Currently `FILTER` only works for GitLab projects, the OBS and APT jobs
always retrieve everything, which is particularly costly for OBS as it
scans the content of each package.

Drop the `FILTER` variable and replace it with a `PROJECTS_NAMESPACE`
variable for GitLab and a `FILTER_PACKAGES` variable to be used by
all jobs to narrow the amount of data retrieved, which is often very
useful for testing.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit 85802ab7 "HACK: Skip OBS scanning by default" worked around a
performance issue with the initial deploymento of OBS 2.10.

With ce731f22 "Do not skip OBS scan" it got re-enabled again, albeit
needing more time than before.

Time to drop the hack.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Dylan Aïssi authored 1 year ago and Emanuele Aina committed 1 year ago

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

With 30 concurrent threads we were ovehelming both our client-side HTTP
connection pool and the GitLab server itself.

In the best case, we only got many warnings on the client side like:

    INFO:root:Fetching component and license_report for 5745 projects
    WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: gitlab-apertispro.boschdevcloud.com

In other cases we got GitLab failing with many exceptions:

    gitlab.exceptions.GitlabHttpError: 500

Some of those errors mapped to server side logs indicating exhaustion of
the database connection slots:

    connection to server at "10.0.69.74", port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The final rendering stage is now running out of memory even on
larger runners.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Dylan Aïssi authored 1 year ago and Emanuele Aina committed 1 year ago

v2024dev2 has been released and all the development happens on
v2024dev3, no need to track v2024dev1

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

For packages not imported from Debian, run uscan to check if new upstream
releases are available. If a new version is found report it on the dashboard.
Also report missing and broken debian/watch files to ensure we continue to track
the correct upstream.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Expand the list of packages that had to be renamed on GitLab because
their name contains characters that are not valid for project names.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Vignesh Raman authored 2 years ago and Ritesh Raj Sarraf committed 1 year ago

v2023 is stable, we should ensure v2023-updates and
v2023-security are consistent too.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

For the v2023 release

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Ritesh Raj Sarraf authored 2 years ago and Dylan Aïssi committed 1 year ago

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Since v2021 is EOL and no other Apertis release is Buster-based,
both v2021 and buster can be dropped.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Fetching upstream sources data for all Debian packages (~ 38100 pkgs)
and not only those in Apertis (~ 5600 pkgs) generates big YAML files
containing useless data. Processing these files lead to frequent
out-of-memory issues. To reduce the memory consumption, we can filter out
all packages not available in Apertis based on the cache file.

This job produces a 42 MB YAML file, merging this file with others YAML
files leads to out-of-memory issues in downstream jobs. Filtering the
generated YAML file at this step reduces the size to only 8 MB.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Source packages contain a "Package-list" field listing all binary packages
generated. All these generated binary packages should be published and
listed in the package index file to avoid possible inconsistencies
between the published Source/Binary packages in the APT repository.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

In the case of releases which use reprepro publisher, there was only
one published version of a package.  This assumption is no longer holds
with the aptly publisher and it can have multiple versions of the
package in the repository and it is as per design. aptly doesn't
remove older versions of a package when a new version is uploaded.
Due to this we see errors like Mismatch between APT source and binary.

If there are multiple versions of a binary, check if there is a
correspondent source for them, and report error if not.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Vignesh Raman authored 2 years ago and Dylan Aïssi committed 2 years ago

In case of releases which use reprepro publisher, duplicates sources
were reported as errors since reprepro publisher never produced
duplicates. This assumption no longer holds with the aptly publisher.
So drop the check for checking duplicates on the releases that use
aptly if the component is the same. Dashboard should report errors if
there's any mismatch between the components.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Currently the dashboard shows only counters for total
errors. In addition to the total errors, show other
errors for different domains.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Branch name is already displayed in the error
and the reduntant one can be removed from
the error message.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Report error if lintian job is not enabled in development branch
for packages not from debian.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

No need to retrieve a file which is not used.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Currently, the cache file is quite big ~ 70MB and parsing it during
the packaging-data-fetch-downstream job gives frequent out-of-memory issues.

This file contains a lot of unused data (information for all packages available
in Debian). In order to keep only used data (i.e. for packages available in Apertis),
A new file packaging-cache.yaml is generated containing only data for cache. It
is a subset of the packaging.yaml and it is used only during the
packaging-data-fetch-downstream job to load cache without triggering the oom issue.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Some packages are fetched from untracked Debian releases, we need a way to
list them in order to check if they require an update or to be sync against
a tracked Debian release.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Some Apertis packages come from a different Debian release. In order keep
them in sync, active descendant branches are analyzed to determine in which
apertis versions they landed. Then, this information is used to report on the
dashboard which apertis releases need to be updated.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Some packages are imported from bullseye-backports, bookworm (testing)
or sid (unstable). In order to keep Apertis packages in sync with
the source channel, we need to fetch data from these different sources.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Thus settings don't have to be stored in several locations and
the rulez.yaml is the only right location to define correct settings
for gitlab repositories

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Drop v2023pre and v2024dev0 since they were already released.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Mirroring on the Apertispro side is complete for
v2023dev3. So we can stop monitoring v2023dev3.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

No need to run the gitlab-rulez diff because the dashboard already
reports issues.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

These branches are not used anymore, so no need to retrieve data
about them. This will reduce the size of the cache file.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Parsing the big cache file can sometimes lead to an OOM issue with
packaging-data-fetch-downstream. This option allows to disable the
cache at this step to unblock dashboard updates.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

In case of wrong settings detected, gitlab-rulez returns a non
0 value and it makes the job marked as failed. So, this case can
be ignored. But gitlab-rulez running in `apply` mode should not
fail.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>