Dylan Aïssi authored 1 year ago and Walter Lozano committed 1 year ago

Sometimes, we need to backport a package version to a previous
Apertis release. But, a fast-forward backport is not always
possible as some changes may have occurred that are not suitable
for previous Apertis releases (like a change in debian/apertis/component
or debian/apertis/copyright needs to be refreshed with a different
version or the toolchain). In order to keep the git history linear,
while doing the backport, unsuitable changes are reverted and a
suffix like "~v2023" is used for versioning.

Because this unsual workflow is a bit tricky, it's not possible
to teach the dashboard about this suffix which makes it complaning
for "Mismatch between OBS XXX~v2023 and Git XXX".

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

This mapping was included from the beginning in the dashboard, but it seems
that is not really needed right now. Having this mapping causes dashboard
to complain with

gnome-settings-daemon apt error Not published on APT
gnome-settings-daemon-data git error GitLab project missing but found on OBS:

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Whitelists may not be useful for downstreams, so it should not
be mandatory.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Both scripts packaging-updates and packaging-check-invariants
contain quite long lists of packages to ignore. This makes both
scripts and lists a bit difficult to read. Instead move the lists
into a specific file at data/whitelists.yaml. This should make
maintenance easier.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Dylan Aïssi authored 1 year ago and Walter Lozano committed 1 year ago

Due to the new anchor added in 3c55dc09 for v2024,
the indentation was wrong. Because of this wrong
indentation, the source field for v2025dev*
is filled with bullseye values instead of bookworm.
This completely confuses the dashboard which compares
Apertis bullseye-based releases to the bookworm-bases
releases.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

For most of these packages Apertis is upstream,
so no need to run uscan.
For some others, the version used in Apertis is frozen
due to the upstream license change (to GPL-3), so again
no need to run uscan since we know this is outdated version
on purpose.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Make the past an exception instead of the future.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Dashboard goes through all the packages in GitLab and reports
an error if they are not published in apt repositories. Unfortunately,
this behavior does not take into account the fact that packages can
be dropped in a release, so they are no longer published.
To fix that, don't report missing packages on apt for packages no
longer in an active Apertis release.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Make the past an exception instead of the future.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

These definitions are inherited through anchors.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

No need to keep a separate definition without non-free.
"-security", "-updates" and "-backports" may contain
non-free packages that have been imported in Apertis.

We need to make sure we stay in sync with all sources
even for the non-free component.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Since Bookworm, some Debian packages moves to non-free-firmware instead
of non-free:
- https://lists.debian.org/debian-devel/2022/12/msg00176.html
- https://lists.debian.org/debian-project/2023/01/msg00018.html



We need to track this component to ensure some of our packages
are up to date (e.g. raspi-firmware or firmware-nonfree).

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Dylan Aïssi authored 1 year ago and Walter Lozano committed 1 year ago

For some packages, Apertis is upstream that means the main and
only git repository is https://gitlab.apertis.org/pkg/*

, so
there is no need to run uscan to scan another repository and for
them a missing debian/watch is expected.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Otherwise, it can generate too much noise in the gitlab job log.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Otherwise uscan fails with a message saying required files
are badly formatted.

This issue is not observed in Apertis because all our repositories
are public, but it affects downstream since their repositories are
not publicly available.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

In Apertis is very common to disable the build of some binaries to avoid
license issues, reduce the number of dependencies or disable unsupported
architectures. To do it, usually d/rules is tweak to skip some binary
packages. However, this approach makes dashboard to complain about missing
binaries.

A possible approach could be to remove the binary packages from d/control
but this approach adds more delta.

To avoid getting a wrong idea of status of the packages, ignore this issue
for the packages we know we tweaked.

The approach of using wildcards to reduce the number of entries was discussed
however, to improve readability and to avoid extra complexity while checking
the idea was discarded.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

The Apertis development channel is still based on Bookworm, but
some specific packages are backported from Trixie. Not tracking
Trixie will leave us with outdated packages.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Walter Lozano authored 1 year ago and Andrej Shadura committed 1 year ago

Since v2024dev3 has already been mirrored by downstreams drop it from
channels to save time and space.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

After switching to JSON in commit 254b71ed, the output of the dashboard is
not longer sorted, which is a bit annoying. To fix that, force the sort
of the output before rendering it.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

In commit 254b71ed dashboard switched to JSON to improve efficiency. While
doing that several functions and strings were renamed to point to the new
format. Unfortunately there are still references to YAML in several places
which are misleading.

Fix remaining references to YAML to avoid confusions.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

During the switch from YAML to JSON (commit 254b71ed),
the saved object was changed by mistake. Instead of saving the
results of updates analysis, the input data were saved in
the JSON file. Because of wrong data written, the dashboard
no longer complains about missing updates from Debian.
This commit reverts restore the right object to be saved in
the JSON file.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

In commit 254b71ed the format of the intermediate files was changed to JSON
to improve performance and save resources. With this change the logic to
dump version information was adapted, but unfortunately not all the cases
were covered, causing version to be dumped as null. As a consequence, the
checks for updates also fail.

Fix the dumping of version information by using string.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

In commit 09e15175 the default value for license report was changed
to True to match the most common case. However, this change was not
propagated to whole code.

Fix the issue by using the correct default value.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

After switching to use JSON as format for the outcome of the different
steps in the dashboard the resources needed drop dramatically. In this
context the use of lightweight runners should be sufficient and it is
preferable to reduce costs.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

YAML is a nice format, very easy to read, unfortunately the Python YAML
library is very inefficient both CPU and memory wise. Loading the same
content using JSON takes 10 times less memory and time.

Since dashboard is always struggling with OOM, let's use JSON for the data
it produces.

As reference, below results of importing a 10 MB file with YAML and JSON
are presented

yaml-json $ ./test.py yaml
Time 15.507138013839722 seg
Memory (70914086, 394044198) bytes (current, peak)

yaml-json $ ./test.py json
Time 0.6210496425628662 seg
Memory (58913059, 67501787) bytes (current, peak)

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

In commit 797862 the logic continues even if the cache file cannot be
downloaded. Unfortunately this approach is buggy since wget can create
empty files if the download fails.

To avoid passing an invalid cache file, check if file is not empty.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The new tenacity-based code limits the overall time interval during
which it retries failed calls and no longer limits the actual number of
retries on its own.

Since in some cases the called function may not be retry-safe, just
expose a boolean to make retrying opt-in and avoid some potential foot
shooting in the future.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The concept used for licensing checks is to have as default the most
common value, which allows us to ommit it and save space in our yaml
files.

Following this approach, change the default value for license report
to True.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Report for licensing issues tend to be rare, so most of the related entries
are False, even though we populate the yaml files with lot of entries of
these entries. In order to reduce the size of the yaml file, only add
entries for not default values.

This is specially true for components like development, where checks are not
performed but data is reported.

At the same time, fix a bug in the cache support for licensing, caused by the
fact that upstream branches do not provide license reports which was
interpreted as missing data forcing the logic to retrieve information from
Gitlab.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 1 year ago and Emanuele Aina committed 1 year ago

In some circumstances there is no valid artifact to download to be
used as cache causing the job to fail. In such a case, instead of
aborting the job just continue without cache.

As reference:

$ CACHE_ARGS=""
$ ARTIFACT_URL=${ARTIFACT_URL:-$CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/$CI_DEFAULT_BRANCH/raw/packaging-cache.yaml?job=pages}
$ if [ "$ARTIFACT_URL" != none ] && [ "$DISABLE_CACHE" == "no" ] && [ "$FILTER_ON_CACHE" == "yes" ] # collapsed multi-line command
--2023-10-24 18:16:17--  https://gitlab.apertis.org/api/v4/projects/6587/jobs/artifacts/master/raw/packaging-cache.yaml?job=pages


Resolving gitlab.apertis.org (gitlab.apertis.org)... 116.203.10.182, 2a01:4f8:1c0c:80ad::1
Connecting to gitlab.apertis.org (gitlab.apertis.org)|116.203.10.182|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-10-24 18:16:18 ERROR 404: Not Found.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The current naive approach at retrying is not exactly gentle: if the
server is overloaded we retry right ahead making the problem just worse.

See https://encore.dev/blog/retries

 for a fun explanation of how the
naive approach can be catastrophic.

Fortunately the `tenacity` library allow us to add a bounded exponential
backoff while also making the code easier to understand.

This should make the dashboard behave much better toward GitLab and OBS
when they hit some issue.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>