After switching to use JSON as format for the outcome of the different
steps in the dashboard the resources needed drop dramatically. In this
context the use of lightweight runners should be sufficient and it is
preferable to reduce costs.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

YAML is a nice format, very easy to read, unfortunately the Python YAML
library is very inefficient both CPU and memory wise. Loading the same
content using JSON takes 10 times less memory and time.

Since dashboard is always struggling with OOM, let's use JSON for the data
it produces.

As reference, below results of importing a 10 MB file with YAML and JSON
are presented

yaml-json $ ./test.py yaml
Time 15.507138013839722 seg
Memory (70914086, 394044198) bytes (current, peak)

yaml-json $ ./test.py json
Time 0.6210496425628662 seg
Memory (58913059, 67501787) bytes (current, peak)

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

In commit 797862 the logic continues even if the cache file cannot be
downloaded. Unfortunately this approach is buggy since wget can create
empty files if the download fails.

To avoid passing an invalid cache file, check if file is not empty.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The new tenacity-based code limits the overall time interval during
which it retries failed calls and no longer limits the actual number of
retries on its own.

Since in some cases the called function may not be retry-safe, just
expose a boolean to make retrying opt-in and avoid some potential foot
shooting in the future.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The concept used for licensing checks is to have as default the most
common value, which allows us to ommit it and save space in our yaml
files.

Following this approach, change the default value for license report
to True.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Report for licensing issues tend to be rare, so most of the related entries
are False, even though we populate the yaml files with lot of entries of
these entries. In order to reduce the size of the yaml file, only add
entries for not default values.

This is specially true for components like development, where checks are not
performed but data is reported.

At the same time, fix a bug in the cache support for licensing, caused by the
fact that upstream branches do not provide license reports which was
interpreted as missing data forcing the logic to retrieve information from
Gitlab.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 1 year ago and Emanuele Aina committed 1 year ago

In some circumstances there is no valid artifact to download to be
used as cache causing the job to fail. In such a case, instead of
aborting the job just continue without cache.

As reference:

$ CACHE_ARGS=""
$ ARTIFACT_URL=${ARTIFACT_URL:-$CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/$CI_DEFAULT_BRANCH/raw/packaging-cache.yaml?job=pages}
$ if [ "$ARTIFACT_URL" != none ] && [ "$DISABLE_CACHE" == "no" ] && [ "$FILTER_ON_CACHE" == "yes" ] # collapsed multi-line command
--2023-10-24 18:16:17--  https://gitlab.apertis.org/api/v4/projects/6587/jobs/artifacts/master/raw/packaging-cache.yaml?job=pages


Resolving gitlab.apertis.org (gitlab.apertis.org)... 116.203.10.182, 2a01:4f8:1c0c:80ad::1
Connecting to gitlab.apertis.org (gitlab.apertis.org)|116.203.10.182|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-10-24 18:16:18 ERROR 404: Not Found.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The current naive approach at retrying is not exactly gentle: if the
server is overloaded we retry right ahead making the problem just worse.

See https://encore.dev/blog/retries

 for a fun explanation of how the
naive approach can be catastrophic.

Fortunately the `tenacity` library allow us to add a bounded exponential
backoff while also making the code easier to understand.

This should make the dashboard behave much better toward GitLab and OBS
when they hit some issue.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 1 year ago and Walter Lozano committed 1 year ago

With commit fd2d8220 "Check license issues only in current OS" we
started fetching licensing data only where appropriate.

However, the YAML properties still got set for each of the not-relevant
branches to their default values, polluting the YAML output.

Make sure we output the licensing properties only for the branches where
it is appropriate.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>
Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Emanuele Aina authored 1 year ago and Walter Lozano committed 1 year ago

Skip irrelevant tags like `upstream/*` and only capture the `debian/*`
and `apertis/*` ones to cut down the captured data.

This is even more beneficial for downstreams as they do not need to
track the `debian/*` tags since they source from the `apertis/*` ones.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For consistency with other releases

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

v2024dev3 was released and v2024pre is used for development

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

v2024dev3 was released, now development happens on
v2024pre.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Skip OBS projects that are not matching the configured components.

For instance, the dashboard should not immediately become confused if
somebody were to create an `apertis:v2023:foobar` project, which happens
to be a relatively common case for our downstreams.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Gathering the YAML files from different jobs and merging them locally
when trying to reproduce an issue when rendering the dashboard is really
tedious, so let's capture the merged `packaging.yaml` even on failures.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

GitLab does not currently grant access to the raw files API endpoints to
CI_JOB_TOKEN in any way. However, it allows for CI_JOB_TOKEN to be used
to clone via `git` so let's use that until the situation improves:
https://gitlab.com/gitlab-org/gitlab/-/issues/424161



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Walter Lozano authored 1 year ago and Emanuele Aina committed 1 year ago

Dashboard summarizes license issues by checking the copyright report for
all the available branches. This works fine in Apertis since the filter
for releases newer than v2021 also filters Debian branches.

On downstreams from Apertis, this filter allows the same issue to be reported
both in apertis and downstream branch, creating lot of noise.

Improve the filtering to only take into account the branches for the current
OS.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

This will avoid failures due to unexpected removal of gitlab-rulez
from trixie

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Make downstream customization slightly easier by moving the `apertis`
bit to a separate and cleaner to customize variable, which could be
easily set at the group level.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Move the call to `gitlab-rulez` to a separate job to shorten a bit the
already extra-long `packaging-data-fetch-downstream` job, and shift it
to the `check` stage as it seems more appropriate.

This also causes the pipeline to properly honor the `$GITLAB_RULES_URL`
variable, as it stops using the hardcoded
gitlab.apertis.org/infrastructure/apertis-infrastructure URL in
`bin/packaging-data-fetch-downstream`.

Reporting is also improved by showing details about the incorrect
settings by using the recently introduced `--output json` option in
`gitlab-rulez`, currently only available starting from Debian Trixie.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Provide a way for external users to map project names and projects.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Address the below deprecation warning:

    DeprecationWarning: `as_list=False` is deprecated and will be
    removed in a future version. Use `iterator=True` instead.
    (python-gitlab: /usr/lib/python3/dist-packages/gitlab/client.py:917)

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 1 year ago and Dylan Aïssi committed 1 year ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For every package updated in v2024dev3 we get an error about the version
in v2024dev2 being outdated.

Mark v2024dev2 as closed so we avoid them.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For downstreams the `apertis/` branches are the actual upstreams, so
let's not hardcode `debian/` when limiting the branches we track.

This fixes a regression for downstreams introduced by commit
025d7c56 "Ignore not active anymore descendant branches".

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Downstreams usually require authentication to access their GitLab
instances, let's make `localtest` work for them as well

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

No need to add `--filter-packages` now that it is already used
everywhere appropriate in the actual pipeline.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit a27543ba "gitlab-ci: Extend filtering by package to all
retrieval jobs" broke `localtest` since `PROJECTS_NAMESPACE` was not set
and no GitLab project could be properly retrieved.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Our scans are quite heavyweight, so let's not run more than one at
a time.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 1 year ago and Andrej Shadura committed 1 year ago

Downstreams usually require authentication to access their GitLab
instances, let's fix the retrieval of the ruleset in that case.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 1 year ago and Andrej Shadura committed 1 year ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Set options for variable with limited choices to give GitLab the chance
to render dropdowns when triggering pipelines manually, reducing the
chance of user errors.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Currently `FILTER` only works for GitLab projects, the OBS and APT jobs
always retrieve everything, which is particularly costly for OBS as it
scans the content of each package.

Drop the `FILTER` variable and replace it with a `PROJECTS_NAMESPACE`
variable for GitLab and a `FILTER_PACKAGES` variable to be used by
all jobs to narrow the amount of data retrieved, which is often very
useful for testing.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit 85802ab7 "HACK: Skip OBS scanning by default" worked around a
performance issue with the initial deploymento of OBS 2.10.

With ce731f22 "Do not skip OBS scan" it got re-enabled again, albeit
needing more time than before.

Time to drop the hack.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Dylan Aïssi authored 1 year ago and Emanuele Aina committed 1 year ago

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

With 30 concurrent threads we were ovehelming both our client-side HTTP
connection pool and the GitLab server itself.

In the best case, we only got many warnings on the client side like:

    INFO:root:Fetching component and license_report for 5745 projects
    WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: gitlab-apertispro.boschdevcloud.com

In other cases we got GitLab failing with many exceptions:

    gitlab.exceptions.GitlabHttpError: 500

Some of those errors mapped to server side logs indicating exhaustion of
the database connection slots:

    connection to server at "10.0.69.74", port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>