GitLab does not currently grant access to the raw files API endpoints to
CI_JOB_TOKEN in any way. However, it allows for CI_JOB_TOKEN to be used
to clone via `git` so let's use that until the situation improves:
https://gitlab.com/gitlab-org/gitlab/-/issues/424161



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Walter Lozano authored 1 year ago and Emanuele Aina committed 1 year ago

Dashboard summarizes license issues by checking the copyright report for
all the available branches. This works fine in Apertis since the filter
for releases newer than v2021 also filters Debian branches.

On downstreams from Apertis, this filter allows the same issue to be reported
both in apertis and downstream branch, creating lot of noise.

Improve the filtering to only take into account the branches for the current
OS.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

This will avoid failures due to unexpected removal of gitlab-rulez
from trixie

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Make downstream customization slightly easier by moving the `apertis`
bit to a separate and cleaner to customize variable, which could be
easily set at the group level.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Move the call to `gitlab-rulez` to a separate job to shorten a bit the
already extra-long `packaging-data-fetch-downstream` job, and shift it
to the `check` stage as it seems more appropriate.

This also causes the pipeline to properly honor the `$GITLAB_RULES_URL`
variable, as it stops using the hardcoded
gitlab.apertis.org/infrastructure/apertis-infrastructure URL in
`bin/packaging-data-fetch-downstream`.

Reporting is also improved by showing details about the incorrect
settings by using the recently introduced `--output json` option in
`gitlab-rulez`, currently only available starting from Debian Trixie.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Provide a way for external users to map project names and projects.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Address the below deprecation warning:

    DeprecationWarning: `as_list=False` is deprecated and will be
    removed in a future version. Use `iterator=True` instead.
    (python-gitlab: /usr/lib/python3/dist-packages/gitlab/client.py:917)

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 1 year ago and Dylan Aïssi committed 1 year ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For every package updated in v2024dev3 we get an error about the version
in v2024dev2 being outdated.

Mark v2024dev2 as closed so we avoid them.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For downstreams the `apertis/` branches are the actual upstreams, so
let's not hardcode `debian/` when limiting the branches we track.

This fixes a regression for downstreams introduced by commit
025d7c56 "Ignore not active anymore descendant branches".

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Downstreams usually require authentication to access their GitLab
instances, let's make `localtest` work for them as well

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

No need to add `--filter-packages` now that it is already used
everywhere appropriate in the actual pipeline.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit a27543ba "gitlab-ci: Extend filtering by package to all
retrieval jobs" broke `localtest` since `PROJECTS_NAMESPACE` was not set
and no GitLab project could be properly retrieved.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Our scans are quite heavyweight, so let's not run more than one at
a time.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 1 year ago and Andrej Shadura committed 1 year ago

Downstreams usually require authentication to access their GitLab
instances, let's fix the retrieval of the ruleset in that case.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 1 year ago and Andrej Shadura committed 1 year ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Set options for variable with limited choices to give GitLab the chance
to render dropdowns when triggering pipelines manually, reducing the
chance of user errors.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Currently `FILTER` only works for GitLab projects, the OBS and APT jobs
always retrieve everything, which is particularly costly for OBS as it
scans the content of each package.

Drop the `FILTER` variable and replace it with a `PROJECTS_NAMESPACE`
variable for GitLab and a `FILTER_PACKAGES` variable to be used by
all jobs to narrow the amount of data retrieved, which is often very
useful for testing.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit 85802ab7 "HACK: Skip OBS scanning by default" worked around a
performance issue with the initial deploymento of OBS 2.10.

With ce731f22 "Do not skip OBS scan" it got re-enabled again, albeit
needing more time than before.

Time to drop the hack.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Dylan Aïssi authored 1 year ago and Emanuele Aina committed 1 year ago

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

With 30 concurrent threads we were ovehelming both our client-side HTTP
connection pool and the GitLab server itself.

In the best case, we only got many warnings on the client side like:

    INFO:root:Fetching component and license_report for 5745 projects
    WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: gitlab-apertispro.boschdevcloud.com

In other cases we got GitLab failing with many exceptions:

    gitlab.exceptions.GitlabHttpError: 500

Some of those errors mapped to server side logs indicating exhaustion of
the database connection slots:

    connection to server at "10.0.69.74", port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The final rendering stage is now running out of memory even on
larger runners.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Dylan Aïssi authored 1 year ago and Emanuele Aina committed 1 year ago

v2024dev2 has been released and all the development happens on
v2024dev3, no need to track v2024dev1

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

For packages not imported from Debian, run uscan to check if new upstream
releases are available. If a new version is found report it on the dashboard.
Also report missing and broken debian/watch files to ensure we continue to track
the correct upstream.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Expand the list of packages that had to be renamed on GitLab because
their name contains characters that are not valid for project names.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Vignesh Raman authored 2 years ago and Ritesh Raj Sarraf committed 1 year ago

v2023 is stable, we should ensure v2023-updates and
v2023-security are consistent too.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

For the v2023 release

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Ritesh Raj Sarraf authored 2 years ago and Dylan Aïssi committed 1 year ago

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Since v2021 is EOL and no other Apertis release is Buster-based,
both v2021 and buster can be dropped.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Fetching upstream sources data for all Debian packages (~ 38100 pkgs)
and not only those in Apertis (~ 5600 pkgs) generates big YAML files
containing useless data. Processing these files lead to frequent
out-of-memory issues. To reduce the memory consumption, we can filter out
all packages not available in Apertis based on the cache file.

This job produces a 42 MB YAML file, merging this file with others YAML
files leads to out-of-memory issues in downstream jobs. Filtering the
generated YAML file at this step reduces the size to only 8 MB.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Source packages contain a "Package-list" field listing all binary packages
generated. All these generated binary packages should be published and
listed in the package index file to avoid possible inconsistencies
between the published Source/Binary packages in the APT repository.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

In the case of releases which use reprepro publisher, there was only
one published version of a package.  This assumption is no longer holds
with the aptly publisher and it can have multiple versions of the
package in the repository and it is as per design. aptly doesn't
remove older versions of a package when a new version is uploaded.
Due to this we see errors like Mismatch between APT source and binary.

If there are multiple versions of a binary, check if there is a
correspondent source for them, and report error if not.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Vignesh Raman authored 2 years ago and Dylan Aïssi committed 2 years ago

In case of releases which use reprepro publisher, duplicates sources
were reported as errors since reprepro publisher never produced
duplicates. This assumption no longer holds with the aptly publisher.
So drop the check for checking duplicates on the releases that use
aptly if the component is the same. Dashboard should report errors if
there's any mismatch between the components.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>