By default reqwest does not return an actual error for non-success HTTP
status codes like HTTP 404.

This lead to parser failures when trying to read the HTML error message
from a 404 reply as a deb822 document.

Use `Response::error_for_status()` to actually check for successful
status codes.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The OBS 2.10 update made things sensibly slower, causing timeouts in the
packaging-data-fetch-obs job. Disable it for now.

Set `SKIP_OBS` to `no` to run the job again.

See https://phabricator.apertis.org/T8880



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 2 years ago and Andrej Shadura committed 2 years ago

OBS 2.10 has just been deployed and we started hitting timeouts. Be more
patient, maybe the new OBS version some time to get accustomed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 2 years ago and Andrej Shadura committed 2 years ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

GitLab makes retrying failed jobs easy, but sometimes it's preferable to
retrigger the whole pipeline from scratch for consistency, so let's make
it more convenient by adding a dedicated button to each failure report.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 2 years ago and Vignesh Raman committed 2 years ago

Ensure a log message is printed when loading the cache to give a clearer
context when debugging issues.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 2 years ago and Vignesh Raman committed 2 years ago

Retrieving the published data can be complicated on downstream
deployment where the data is auth protected.

Since GitLab stores it as an artifact accessible using the job token,
use it to side-step the issue.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add packages that dynamically generate the version of the
binaries they produce to BINARY_VERSION_IGNORELIST.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

commits moving d/a/component from development to sdk
and disabling java support were not backported to
v2022 from v2023dev2. So the dashboard reports
error for these lagging packages.

Hardcode a list of release, package, version to be
ignored when reporting these lagging packages.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Currently, all the downstream channels that are based on the source
branch are checked to be merged. This doesn't match the linux package
configuration, as it doesn't pull updates from the default release
upstream sources.

For instance, apertis/v2021 release is based on debian/buster, but the
apertis/v2021 linux package pulls from debian/bullseye.

Let's ignore the linux package upstream branches checking invariant as
this is reporting wrong entries and there's another script which takes
care of handling all linux package updates:

  ./bin/packaging-updates-upstream-linux

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Emanuele Aina authored 2 years ago and Vignesh Raman committed 2 years ago

Let downstream override a variable to source the cached data from the
right place, or disable it completely.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

v2022 is stable, we should ensure v2022-updates and v2022-security
are consistent too.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When iterating with a thread pool, always try to compute a useful
identifier rather than doing so only when debug is enabled, since the
itemid is also used when reporting about exceptions and retrying.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The rubygems 3.2.5-2apertis0 package currently produces:
* bundler_2.2.5-2apertis0bv2022dev2b1_all.deb
* ruby-bundler_2.2.5-2apertis0bv2022dev2b1_all.deb
* ruby-rubygems_3.2.5-2apertis0bv2022dev2b1_all.deb

Since `bundler` and `ruby-bundler` have a version that does not match
the version of the sources let's skip the check.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Arnaud Ferraris authored 3 years ago and Detlev Casanova committed 3 years ago

In order to be able to easily identify licensing issues, those should be
reported in a different way than other issues. This is achieved by
changing the severity of those issues to "warning" and using "license"
as the text for the severity badge. This also allows us to easily filter
those issues based on the selected severity.

This commit also changes the filtering checkboxes labels to better
reflect which issues are being taken into account.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Arnaud Ferraris authored 3 years ago and Detlev Casanova committed 3 years ago

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Arnaud Ferraris authored 3 years ago and Detlev Casanova committed 3 years ago

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Arnaud Ferraris authored 3 years ago and Detlev Casanova committed 3 years ago

In order to facilitate license compliance, the dashboard should report
the most severe license issues found in packages from the "target"
repository: this includes cases where we override the existing license
information (especially when specifying a global default license) or
packages for which whitelisting is enabled for the whole source tree.

As part of this effort, this commit fetches the copyright override and
whitelisting files, and process those in order to detect major issues.
The corresponding new per-branch flags are then set accordingly and
added to the output file.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Arnaud Ferraris authored 3 years ago and Detlev Casanova committed 3 years ago

The `filter_cache()` method used an optional `check_tags` argument to
decide between several code paths. This does not properly reflect the
purpose of this argument, which is to differentiate between 2 use cases:
- check and populate information about the descendants of a given ref
- check and populate component and license information

As additional fields will have to be taken into account in the future,
this rework aims at making the aforementioned method more flexible,
while making it clear to which use-case belongs each code path. To that
effect, the `check_flags` argument is renamed to `purpose` and now uses
an enum to distinguish between use-cases. It also only processes the
relevant data for the current use-case:
- when checking for descendants, only descendants data is looked up and
  updated
- when checking for license info, only this information is checked and
  updated

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Now that v2023dev1 has been released do not complain about updates that
need to be landed to it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

All the development happens on v2022 and the mirroring on the
Apertispro side is complete, so we can stop monitoring v2022pre.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

This avoids making unnecessary requests when branches and tags didn't
change since the last run.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Reduce the severity of the logs about retrieved licensing logs since
they are currently drowning everything else.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Retrieve all descendants to correctly detect when a branch got merged in
another and which tags apply.

Unfortunately `commit.refs()` does not currently handle pagination, so
only the first 20 items were downloaded.

This meant that in some cases no tags were returned, so no version could
be computed. In other cases this may have cause incorrect reports of
branches not being merged into their downstream, or other issues due to
computing the wrong version for the branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Fix the typing class path, it's `collections.abc.Sequence`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Vignesh Raman committed 3 years ago

When retrieving the files under `debian/apertis` only expect the
404 Not Found error and re-raise everything else: we want to cleanly
handle missing files, but timeouts (for instance) are actual errors.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We should report an error on the package if it missing the
`debian/apertis/component` file rather than crashing badly.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Vignesh Raman committed 3 years ago

Applying filters one-at-a-time can lead do broken behavior:

* deselect both the `info` and `error` severity checkboxes
* all reports are correctly hidden
* deselect the `apertis/v2021` channels checkbox
* all the other reports incorrectly show up again, regardless of
  their severity

This is because when applying the updated channels filter the code does
not look at the other active filters.

Tweak the code to always check *all* filters regardless of the one that
has been updated.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Nowe that we have a single list of components, drop the
`_NONFREE` suffix.

No functional changes.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

With v2020 reaching its end-of-support there's no need to keep around
the list of component without `non-free`, which was introduced in v2021.

No functional changes.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When getting an exception, ensure the current package name is printed to
debug which one has faulty data.

For instance this is now the output, previously only the first exception
was printed:

    Traceback (most recent call last):
      File "./bin/packaging-check-invariants", line 586, in <module>
        checker.check_missing_license_report(package)
      File "./bin/packaging-check-invariants", line 533, in check_missing_license_report
        if branch.component == "target" and not branch.license_report:
      File "./bin/packaging-check-invariants", line 90, in __getattr__
        item = super(DictWrapper, self).__getitem__(attr)
    KeyError: 'component'

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "./bin/packaging-check-invariants", line 588, in <module>
        raise Error(package.name) from e
    __main__.Error: aalib

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Get rid of an indentation level by moving the iteration over the
packages out of the check functions.

This also will make improving error reporting easier since the package
context of each check is now clearly defined when calling it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Point to the GitLab compare tool when a branch is lagging behind the
others, to let people find easily the missing commits and potentially
create a merge request for them using the handy button in the comparison
page itself.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When we release a channel like `apertis/v2023dev0` we still want to keep
it monitored for errors to catch issues that may impact downstreams when
they mirror it, but we do not want to hear complaints about packages
missing updates and lagging behind as that's on purpose.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Rather than iterating over a list and check for membership in another
list, just iterate on the other list since we already know it is a
strict subset.

No behavioral changes.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>