These definitions are inherited through anchors.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

No need to keep a separate definition without non-free.
"-security", "-updates" and "-backports" may contain
non-free packages that have been imported in Apertis.

We need to make sure we stay in sync with all sources
even for the non-free component.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Since Bookworm, some Debian packages moves to non-free-firmware instead
of non-free:
- https://lists.debian.org/debian-devel/2022/12/msg00176.html
- https://lists.debian.org/debian-project/2023/01/msg00018.html



We need to track this component to ensure some of our packages
are up to date (e.g. raspi-firmware or firmware-nonfree).

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Dylan Aïssi authored 1 year ago and Walter Lozano committed 1 year ago

For some packages, Apertis is upstream that means the main and
only git repository is https://gitlab.apertis.org/pkg/*

, so
there is no need to run uscan to scan another repository and for
them a missing debian/watch is expected.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Otherwise, it can generate too much noise in the gitlab job log.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Otherwise uscan fails with a message saying required files
are badly formatted.

This issue is not observed in Apertis because all our repositories
are public, but it affects downstream since their repositories are
not publicly available.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

In Apertis is very common to disable the build of some binaries to avoid
license issues, reduce the number of dependencies or disable unsupported
architectures. To do it, usually d/rules is tweak to skip some binary
packages. However, this approach makes dashboard to complain about missing
binaries.

A possible approach could be to remove the binary packages from d/control
but this approach adds more delta.

To avoid getting a wrong idea of status of the packages, ignore this issue
for the packages we know we tweaked.

The approach of using wildcards to reduce the number of entries was discussed
however, to improve readability and to avoid extra complexity while checking
the idea was discarded.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

The Apertis development channel is still based on Bookworm, but
some specific packages are backported from Trixie. Not tracking
Trixie will leave us with outdated packages.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Walter Lozano authored 1 year ago and Andrej Shadura committed 1 year ago

Since v2024dev3 has already been mirrored by downstreams drop it from
channels to save time and space.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

After switching to JSON in commit 254b71ed, the output of the dashboard is
not longer sorted, which is a bit annoying. To fix that, force the sort
of the output before rendering it.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

In commit 254b71ed dashboard switched to JSON to improve efficiency. While
doing that several functions and strings were renamed to point to the new
format. Unfortunately there are still references to YAML in several places
which are misleading.

Fix remaining references to YAML to avoid confusions.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

During the switch from YAML to JSON (commit 254b71ed),
the saved object was changed by mistake. Instead of saving the
results of updates analysis, the input data were saved in
the JSON file. Because of wrong data written, the dashboard
no longer complains about missing updates from Debian.
This commit reverts restore the right object to be saved in
the JSON file.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

In commit 254b71ed the format of the intermediate files was changed to JSON
to improve performance and save resources. With this change the logic to
dump version information was adapted, but unfortunately not all the cases
were covered, causing version to be dumped as null. As a consequence, the
checks for updates also fail.

Fix the dumping of version information by using string.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

In commit 09e15175 the default value for license report was changed
to True to match the most common case. However, this change was not
propagated to whole code.

Fix the issue by using the correct default value.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

After switching to use JSON as format for the outcome of the different
steps in the dashboard the resources needed drop dramatically. In this
context the use of lightweight runners should be sufficient and it is
preferable to reduce costs.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

YAML is a nice format, very easy to read, unfortunately the Python YAML
library is very inefficient both CPU and memory wise. Loading the same
content using JSON takes 10 times less memory and time.

Since dashboard is always struggling with OOM, let's use JSON for the data
it produces.

As reference, below results of importing a 10 MB file with YAML and JSON
are presented

yaml-json $ ./test.py yaml
Time 15.507138013839722 seg
Memory (70914086, 394044198) bytes (current, peak)

yaml-json $ ./test.py json
Time 0.6210496425628662 seg
Memory (58913059, 67501787) bytes (current, peak)

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

In commit 797862 the logic continues even if the cache file cannot be
downloaded. Unfortunately this approach is buggy since wget can create
empty files if the download fails.

To avoid passing an invalid cache file, check if file is not empty.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The new tenacity-based code limits the overall time interval during
which it retries failed calls and no longer limits the actual number of
retries on its own.

Since in some cases the called function may not be retry-safe, just
expose a boolean to make retrying opt-in and avoid some potential foot
shooting in the future.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The concept used for licensing checks is to have as default the most
common value, which allows us to ommit it and save space in our yaml
files.

Following this approach, change the default value for license report
to True.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Report for licensing issues tend to be rare, so most of the related entries
are False, even though we populate the yaml files with lot of entries of
these entries. In order to reduce the size of the yaml file, only add
entries for not default values.

This is specially true for components like development, where checks are not
performed but data is reported.

At the same time, fix a bug in the cache support for licensing, caused by the
fact that upstream branches do not provide license reports which was
interpreted as missing data forcing the logic to retrieve information from
Gitlab.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 1 year ago and Emanuele Aina committed 1 year ago

In some circumstances there is no valid artifact to download to be
used as cache causing the job to fail. In such a case, instead of
aborting the job just continue without cache.

As reference:

$ CACHE_ARGS=""
$ ARTIFACT_URL=${ARTIFACT_URL:-$CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/$CI_DEFAULT_BRANCH/raw/packaging-cache.yaml?job=pages}
$ if [ "$ARTIFACT_URL" != none ] && [ "$DISABLE_CACHE" == "no" ] && [ "$FILTER_ON_CACHE" == "yes" ] # collapsed multi-line command
--2023-10-24 18:16:17--  https://gitlab.apertis.org/api/v4/projects/6587/jobs/artifacts/master/raw/packaging-cache.yaml?job=pages


Resolving gitlab.apertis.org (gitlab.apertis.org)... 116.203.10.182, 2a01:4f8:1c0c:80ad::1
Connecting to gitlab.apertis.org (gitlab.apertis.org)|116.203.10.182|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-10-24 18:16:18 ERROR 404: Not Found.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The current naive approach at retrying is not exactly gentle: if the
server is overloaded we retry right ahead making the problem just worse.

See https://encore.dev/blog/retries

 for a fun explanation of how the
naive approach can be catastrophic.

Fortunately the `tenacity` library allow us to add a bounded exponential
backoff while also making the code easier to understand.

This should make the dashboard behave much better toward GitLab and OBS
when they hit some issue.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 1 year ago and Walter Lozano committed 1 year ago

With commit fd2d8220 "Check license issues only in current OS" we
started fetching licensing data only where appropriate.

However, the YAML properties still got set for each of the not-relevant
branches to their default values, polluting the YAML output.

Make sure we output the licensing properties only for the branches where
it is appropriate.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>
Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Emanuele Aina authored 1 year ago and Walter Lozano committed 1 year ago

Skip irrelevant tags like `upstream/*` and only capture the `debian/*`
and `apertis/*` ones to cut down the captured data.

This is even more beneficial for downstreams as they do not need to
track the `debian/*` tags since they source from the `apertis/*` ones.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For consistency with other releases

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

v2024dev3 was released and v2024pre is used for development

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

v2024dev3 was released, now development happens on
v2024pre.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Skip OBS projects that are not matching the configured components.

For instance, the dashboard should not immediately become confused if
somebody were to create an `apertis:v2023:foobar` project, which happens
to be a relatively common case for our downstreams.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Gathering the YAML files from different jobs and merging them locally
when trying to reproduce an issue when rendering the dashboard is really
tedious, so let's capture the merged `packaging.yaml` even on failures.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

GitLab does not currently grant access to the raw files API endpoints to
CI_JOB_TOKEN in any way. However, it allows for CI_JOB_TOKEN to be used
to clone via `git` so let's use that until the situation improves:
https://gitlab.com/gitlab-org/gitlab/-/issues/424161



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Walter Lozano authored 1 year ago and Emanuele Aina committed 1 year ago

Dashboard summarizes license issues by checking the copyright report for
all the available branches. This works fine in Apertis since the filter
for releases newer than v2021 also filters Debian branches.

On downstreams from Apertis, this filter allows the same issue to be reported
both in apertis and downstream branch, creating lot of noise.

Improve the filtering to only take into account the branches for the current
OS.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

This will avoid failures due to unexpected removal of gitlab-rulez
from trixie

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>