Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Dylan Aïssi authored 1 year ago and Walter Lozano committed 11 months ago

Packages versions coming from Debian are correctly parsed, so we don't have to
whitelist them, however in Apertis we use the ~v202X suffix which is to
misinterpreted by dashboard and lead to confusion when comparing versions.
For example, the dashboard the of downstream complains about:

Version in apertis/v2023/5.2.5~ds0-1+apertis0 is higher than version 5.2.5~ds0-1+apertis0~v2023 in upstream apertis/v2023

Which is true, but due to our workflow, the version apertis/v2023/5.2.5~ds0-1+apertis0
should not land in v2023 this is the reason of why we add the problematic
suffix allowing us to backport targeted changes to a previous release.
Adding the logic in the dashboard to correctly interpret the ~v202X is
not trivial, so for now we can teach it to ignore some versions.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Only packaging data are required in the cache file,
that was the previous behavior, but it was mistakenly
reverted in 254b71ed.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Just after adding apertis/v202*-security and -updates branches to be
tracked by the dashboard, the returned version is "None" because the
cache file is not yet updated. This "None" value makes the pipeline
failing with:

Traceback (most recent call last):
  File "/builds/infrastructure/dashboard/./bin/packaging-updates-upstream-linux", line 209, in <module>
    results = compute_linux_updates(data)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/infrastructure/dashboard/./bin/packaging-updates-upstream-linux", line 134, in compute_linux_updates
    process = subprocess.run(
              ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', '-c', 'versionsort.suffix=-', 'ls-remote', '--exit-code', '--tags', '--refs', '--sort=version:refname', 'https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

', 'vNone*']' returned non-zero exit status 2.

To avoid failures, just skip this branch for this iteration.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

If a package only appears on OBS the dashboard fails with an exception:

    Traceback (most recent call last):
      File "/builds/-RogsxNE/0/infrastructure/dashboard/./bin/packaging-check-invariants", line 854, in <module>
        checker.check_git_and_obs_versions(package)
      File "/builds/-RogsxNE/0/infrastructure/dashboard/./bin/packaging-check-invariants", line 238, in check_git_and_obs_versions
        if package.git.default_branch:
           ^^^^^^^^^^^
      File "/builds/-RogsxNE/0/infrastructure/dashboard/./bin/packaging-check-invariants", line 79, in __getattr__
        item = super(DictWrapper, self).__getitem__(attr)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    KeyError: 'git'

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Sometimes, we need to import and modify packages from Debian.
To make the initial review easier, an apertis/v202* branch is created
from the debian one, then a merge request is created against it adding
debian/apertis/* files. During the review, tha apertis branch
doesn't have any debian/apertis/* files and the dashboard fails with:

  Traceback (most recent call last):
    File "/builds/infrastructure/dashboard/./bin/packaging-check-invariants", line 857, in <module>
      checker.check_git_and_obs_versions(package)
    File "/builds/infrastructure/dashboard/./bin/packaging-check-invariants", line 271, in check_git_and_obs_versions
      component = branch.component
                ^^^^^^^^^^^^^^^^
    File "/builds/infrastructure/dashboard/./bin/packaging-check-invariants", line 79, in __getattr__
      item = super(DictWrapper, self).__getitem__(attr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  KeyError: 'component'
  The above exception was the direct cause of the following exception:
  Traceback (most recent call last):
    File "/builds/infrastructure/dashboard/./bin/packaging-check-invariants", line 867, in <module>
      raise Error(package.name) from e

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Apertis v2022 is no longer supported, so drop its configuration to save
resources and avoid misunderstandings.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

The version of this tool used in Apertis is frozen
due to the upstream license change (to GPL-3), so
no need to run uscan since we know this is outdated version
on purpose.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

For some packages, the watch file use the "git" mode to check git
repository, in that case git is required otherwise it fails with:

uscan issue: Command 'git' not found in /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin at /usr/share/perl5/Devscripts/Utils.pm line 24.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

When uscan reports a package is up to date, its return code is 1,
which is interpreted as an issue with uscan/watch by the dashboard.
So, before returning an issue for this package, check if the
output of uscan contains "Package is up to date".

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Dylan Aïssi authored 1 year ago and Walter Lozano committed 1 year ago

Sometimes, we need to backport a package version to a previous
Apertis release. But, a fast-forward backport is not always
possible as some changes may have occurred that are not suitable
for previous Apertis releases (like a change in debian/apertis/component
or debian/apertis/copyright needs to be refreshed with a different
version or the toolchain). In order to keep the git history linear,
while doing the backport, unsuitable changes are reverted and a
suffix like "~v2023" is used for versioning.

Because this unsual workflow is a bit tricky, it's not possible
to teach the dashboard about this suffix which makes it complaning
for "Mismatch between OBS XXX~v2023 and Git XXX".

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

This mapping was included from the beginning in the dashboard, but it seems
that is not really needed right now. Having this mapping causes dashboard
to complain with

gnome-settings-daemon apt error Not published on APT
gnome-settings-daemon-data git error GitLab project missing but found on OBS:

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Whitelists may not be useful for downstreams, so it should not
be mandatory.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Both scripts packaging-updates and packaging-check-invariants
contain quite long lists of packages to ignore. This makes both
scripts and lists a bit difficult to read. Instead move the lists
into a specific file at data/whitelists.yaml. This should make
maintenance easier.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Dylan Aïssi authored 1 year ago and Walter Lozano committed 1 year ago

Due to the new anchor added in 3c55dc09 for v2024,
the indentation was wrong. Because of this wrong
indentation, the source field for v2025dev*
is filled with bullseye values instead of bookworm.
This completely confuses the dashboard which compares
Apertis bullseye-based releases to the bookworm-bases
releases.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

For most of these packages Apertis is upstream,
so no need to run uscan.
For some others, the version used in Apertis is frozen
due to the upstream license change (to GPL-3), so again
no need to run uscan since we know this is outdated version
on purpose.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Make the past an exception instead of the future.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Dashboard goes through all the packages in GitLab and reports
an error if they are not published in apt repositories. Unfortunately,
this behavior does not take into account the fact that packages can
be dropped in a release, so they are no longer published.
To fix that, don't report missing packages on apt for packages no
longer in an active Apertis release.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Make the past an exception instead of the future.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

These definitions are inherited through anchors.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

No need to keep a separate definition without non-free.
"-security", "-updates" and "-backports" may contain
non-free packages that have been imported in Apertis.

We need to make sure we stay in sync with all sources
even for the non-free component.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Since Bookworm, some Debian packages moves to non-free-firmware instead
of non-free:
- https://lists.debian.org/debian-devel/2022/12/msg00176.html
- https://lists.debian.org/debian-project/2023/01/msg00018.html



We need to track this component to ensure some of our packages
are up to date (e.g. raspi-firmware or firmware-nonfree).

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Dylan Aïssi authored 1 year ago and Walter Lozano committed 1 year ago

For some packages, Apertis is upstream that means the main and
only git repository is https://gitlab.apertis.org/pkg/*

, so
there is no need to run uscan to scan another repository and for
them a missing debian/watch is expected.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Otherwise, it can generate too much noise in the gitlab job log.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Otherwise uscan fails with a message saying required files
are badly formatted.

This issue is not observed in Apertis because all our repositories
are public, but it affects downstream since their repositories are
not publicly available.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

In Apertis is very common to disable the build of some binaries to avoid
license issues, reduce the number of dependencies or disable unsupported
architectures. To do it, usually d/rules is tweak to skip some binary
packages. However, this approach makes dashboard to complain about missing
binaries.

A possible approach could be to remove the binary packages from d/control
but this approach adds more delta.

To avoid getting a wrong idea of status of the packages, ignore this issue
for the packages we know we tweaked.

The approach of using wildcards to reduce the number of entries was discussed
however, to improve readability and to avoid extra complexity while checking
the idea was discarded.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>