Emanuele Aina authored 3 years ago and Vignesh Raman committed 3 years ago

When retrieving the files under `debian/apertis` only expect the
404 Not Found error and re-raise everything else: we want to cleanly
handle missing files, but timeouts (for instance) are actual errors.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We should report an error on the package if it missing the
`debian/apertis/component` file rather than crashing badly.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Vignesh Raman committed 3 years ago

Applying filters one-at-a-time can lead do broken behavior:

* deselect both the `info` and `error` severity checkboxes
* all reports are correctly hidden
* deselect the `apertis/v2021` channels checkbox
* all the other reports incorrectly show up again, regardless of
  their severity

This is because when applying the updated channels filter the code does
not look at the other active filters.

Tweak the code to always check *all* filters regardless of the one that
has been updated.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Nowe that we have a single list of components, drop the
`_NONFREE` suffix.

No functional changes.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

With v2020 reaching its end-of-support there's no need to keep around
the list of component without `non-free`, which was introduced in v2021.

No functional changes.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When getting an exception, ensure the current package name is printed to
debug which one has faulty data.

For instance this is now the output, previously only the first exception
was printed:

    Traceback (most recent call last):
      File "./bin/packaging-check-invariants", line 586, in <module>
        checker.check_missing_license_report(package)
      File "./bin/packaging-check-invariants", line 533, in check_missing_license_report
        if branch.component == "target" and not branch.license_report:
      File "./bin/packaging-check-invariants", line 90, in __getattr__
        item = super(DictWrapper, self).__getitem__(attr)
    KeyError: 'component'

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "./bin/packaging-check-invariants", line 588, in <module>
        raise Error(package.name) from e
    __main__.Error: aalib

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Get rid of an indentation level by moving the iteration over the
packages out of the check functions.

This also will make improving error reporting easier since the package
context of each check is now clearly defined when calling it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Point to the GitLab compare tool when a branch is lagging behind the
others, to let people find easily the missing commits and potentially
create a merge request for them using the handy button in the comparison
page itself.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When we release a channel like `apertis/v2023dev0` we still want to keep
it monitored for errors to catch issues that may impact downstreams when
they mirror it, but we do not want to hear complaints about packages
missing updates and lagging behind as that's on purpose.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Rather than iterating over a list and check for membership in another
list, just iterate on the other list since we already know it is a
strict subset.

No behavioral changes.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Avoid assigning two different things to the same variable by giving a
more descriptive name to the second one and add a comment.

No behavioral changes.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Create a new method to reduce the nesting of the code.

No behavioral changes.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Reduce nesting using an early `continue`.

No behavioral changes.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The job retrieving data from OBS often reaches the default 1h timeout,
so let's be more lenient.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For some of the longer running jobs it is sufficient for the service
they interface with to be marginally slower than usual to trigger a
timeout, either in the overall job execution time on on specific
requests from the tools, causing the whole pipeline to fail.

In those case we can afford to retry the job to see if things improve.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Listing which e.g. `debian/*` tag is contained in a `apertis/*` branch
is rather pointless and is a significant waste of resources.

Limit the reported tags to the one with a matching prefix, so only
`debian/*` tags are listed on `debian/*` branches and only `apertis/*`
tags are listed on `apertis/*` branches.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since we already list all the tag contained in each branch, including
the symmetrical list of descendant branches for each tag in the YAML is
wasting bytes with duplicated information.

We still use the list of descendant branches internally to compute which
tags are contained in each branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Let's save a few bytes by getting rid of some pointless information.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This saves a few bytes for each `debian/*` and `upstream/*` branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Ryan Gonzalez committed 3 years ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Walter Lozano committed 3 years ago

The `./localtest` script was skipping the `lint` job, not the `lint`
stage, so when I added the job to exercise `./localtest` itself during
the `lint` stage I have caused `./localtest` to appear in its own
output, which means that if you pipe it into a shell it will call
itself. Luckily recursion will stop there and this is about not doing
something pointless rather than actually fixing something broken.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Highlight the `Filter` button when any filter is active and reports may
be hidden.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Rather than using plaintext error messages, use readable codes and
structured metadata for errors and updates to make them easier
to process.

This will be particularly useful for filtering: for instance we
preserve the branch information rather than muddling it in the
error message.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When multiple sources are published, check the version of the
most recent one rather than whatever comes first.

This avoids reporting a mismatched version when things are just
ambiguous.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add more parallel workers whe retrieving the component and licensing
report from git to speed up the process.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since we moved all git repositories directly under a single `pkg/` group
instead of the `pkg/$component/` nested categories, checking for
duplicates is no longer relevant.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

No functional changes, just get rid of a level of indentiation.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add a script to run the pipeline locally during development:

    ./localtest --skip-job=storage-usage --filter-packages=apache-log4j2 | sh

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Let users focus on a subset of the available packages to speed up the
test cycle during development.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This does not introduce any functional change, it only makes things
slightly more consistent.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

GitLab can show a textual description for variables that can be set when
manually triggering a pipeline, see
https://docs.gitlab.com/ee/ci/yaml/index.html#variablesdescription



Use that to document how `TRIGGER_UPDATES` should be used.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>