The `public/data.yaml` generated by the dashboard contains all the
combined data to let triggers do their job, using it gives more
flexibility later on.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When retrieving an artifacts from a past job with a job token, the
`JOB-TOKEN``header should be used rather than the `PRIVATE-HEADER` and
the API URL should be used rather than the enduser route.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Repeat with me: `test -e` stands for "file exists", *not* for "string is
empty", as that's `test -z`.

Update the test in `trigger-updates` accordingly to properly print the
help text when the variable is not set.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

If we already dropped a package from the development channel there is
no point in complaining if its upstream got dropped from Debian, even
if we still ship in stable releases.

If instead it's still in a development branch, show the branch affected.

This silences errors on bootchart2 which is still in v2020 while it got
dropped from Debian Buster and we no longer ship it in v2021 and later.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In some cases we emit some strings with characters like `<` and since
these didn't get escaped we were generating malformed HTML.

Fix that by enabling autoescaping.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `gcc-*-cross` binaries get their versions from the matching `gcc`
source package, so ignore source/binaries mismatches.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Expand the list of packages that had to be renamed on GitLab because
their name contains characters that are not valid for project names.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

List a bunch of packages that dynamically generate the version of the
binaries they produce.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When an update is available on e.g. `buster-security` try to trigger the
update pipeline on the `debian/buster` branch rather than the
`debian/buster-security` one, for consistency.

The latter is currently buggy, but that's a bug in the rules of the
infrastructure/ci-package-builder> pipeline that needs to be fixed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We never backported the commits adding `debian/apertis/component` files
to v2020 to avoid unnecessary churn in a stable release but this had two
drawbacks:

1. pipelines that got re-triggered for any reason would fail
1. almost all v2020 package lag compared to v2021

Since we're not going to address these issues, explicitly ignore them to
reduce the amount of noise on the dashboard.

Ignoring lagging packages may cause us to miss some backports, but at
the moment they'd get drowned in the noise anyway.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Do not re-download the list of published packages but re-use the data
retrieved by the `packaging-data-fetch-sources-published` job, avoiding
a large chunk of duplicated code.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For some imperscrutable reason the YAML file has the executable bit set,
unset it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Kill old pipelines when triggering a new one on the same branch, there
are no side effects in doing that, except for the trigger-updates job.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Now that v20219.7 has been released we can consider v2019 in read-only
mode, no need to monitor it any further.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

v2021 is stable, we should ensure v2021-updates and v2021-security are
consistent too.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The current code was not reporting updates for destinations like
`debian/buster-updates` unless the branch already existed.

Tweak the code to check all the sources even if only a subset of the
matching branches exists, using a new `base` key in the source
definition to group them.

For instance, if a repository has only the `debian/buster-updates`
branch the `base` key points to `debian/buster` which in turn matches
the `debian/buster`, `debian/buster-security`, and
`debian/buster-updates` sources. The availability of updates will be
checked for each of them.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `packaging-updates` tool operates exclusively on local data and it
does not need a connection to the GitLab API, so drop the relevant code.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Frederic Danis authored 3 years ago and Emanuele Aina committed 3 years ago

Some packages like cross-toolchain-base and cross-toolchain-base-ports do
special things for which the source version does not match at all the
binaries because it reflects the one from the gcc version it uses.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Frederic Danis authored 3 years ago and Emanuele Aina committed 3 years ago

With the new published binary data, these stages are failing on
`lightweight` runners.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Frederic Danis authored 3 years ago and Emanuele Aina committed 3 years ago

Check that each published package entry:
- is an array with exactly one 'source' key, otherwise add an error about
  a package published with missing or multiple sources
- is an array with exactly one 'binaries' key, otherwise add an error
  about a package being not published or published from multiple components
  (e.g. we have moved it and didn't clean up)
- that the version in the 'source' key matches all the versions for the
  'binaries' key, after stripping the build counter somehow (e.g. assume
  it's bv\d\d\d\d.*b\d\+$), otherwise add an error about mismatching
  versions

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Frederic Danis authored 3 years ago and Emanuele Aina committed 3 years ago

This is based on packaging-data-fetch-sources

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Frederic Danis authored 3 years ago and Emanuele Aina committed 3 years ago

Add 'source' key entry for source data. This will allow to add 'binaries'
key entry later.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This script extracts the packages list from the respective Apertis
releases Sources metadata file. It extracts the following fields:

* Source Package Names
* Package Version
* Repository Component

It then writes these values in a TSV format to a file per release.

The tsv data is stored in a folder 'tsv/', treated as an artifact by the
CI.

The same artifact is pushed to the rendered dashboard for serving over
HTTP

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

By handling two different base distributions (Bullseye for v2022
and Buster for everything else) the packages have started to diverge so
much that the increase of the amount of data about different versions
causes rendering to hit MemoryError on the current lightweight runner
nodes, so switch to the default one which are much bigger.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We currentl load all the YAML in memory at the same time, which leads to
a MemoryError on runners with 2GB or RAM.

Try to load the YAML more lazily to lower the peak memory consumption.

This still does not allow the step to run on the ligtweight runners when
we start tracking the Bullseye-based v2022dev2.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

If a `NETRC` file variable is set in the CI/CD settings copy it to
`~/.netrc` so python-requests can use it out of the box.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This skips an indirection step when triggering updates on a single
package. Keep the indirection for the "update all" button.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Git refnames are relatively free-form and can contain all sort for
special characters, not just `/ and `#`, see
http://git-scm.com/docs/git-check-ref-format

In particular, Debian's DEP-14 standard for storing packaging in git
repositories mandates the use of the `%` character in tags in some
cases like `debian/2%2.6-21`.

Unfortunately python-gitlab currently only escapes `/` to `%2F` and in
some cases `#` to `%23`. This means that when using the commit API to
retrieve information about the `debian/2%2.6-21` tag only the slash is
escaped before being inserted in the URL path and the `%` is left
untouched, resulting in something like
`/api/v4/projects/123/repository/commits/debian%2F2%2.6-21`. When
urllib3 seees that it detects the invalid `%` escape and then urlencodes
the whole string, resulting in
`/api/v4/projects/123/repository/commits/debian%252F2%252.6-21`, where
the original `/` got escaped twice and produced `%252F`.

This works around the issue while waiting for the upstream fix,
see https://github.com/python-gitlab/python-gitlab/pull/1336



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When the latest commit on a branch is not tagged (that is, there are
unreleased commits) picking the first descendant tag is not correct
since it can end up picking tags from descendant branches.

For instance, if `apertis/v2022dev0` has unreleased commits and they get
released in `apertis/v2022dev1` the current code gets confused and
consider them to have the same version.

To avoid that, check which tags are actually contained in each branch
and pick the latest.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 4 years ago and Martyn Welch committed 4 years ago

Help maintainers spot failed packaging pipelines where manual
intervention is needed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Let users start a pipeline that triggers all the upstream pulls at the
same time. This should hopefully reduce the amount of clicks maintainers
have to do on a daily basis.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When clicking the new update links generated by the dashboard eveything
gets recomputed from scratch: this means that we waste resources but
more importantly we slow down the developers who triggered the updates.

To avoid that, point the pipeline triggering the updates to the
previously computed data and skip the jobs that are now superfluous.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add a job that, if `TRIGGER_UPDATES` is set, triggers all the matching
update pipelines.

Set TRIGGER_UPDATES by manually triggering the pipeline to actually initiate the updates

* use "*" to match everything
* use "dash" to only process the dash package

If TRIGGER_UPDATES is left empty, do a dry run (this is the default).

For instance: https://gitlab.apertis.org/infrastructure/dashboard/-/pipelines/new?var[TRIGGER_UPDATES]=*



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Rename `--pipeline-url` to `--current-pipeline-url` so we can later add
more parameter about other pipelines, for instance the URL to trigger
enw pipelines.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

With the switch to the external gitlab-ci pipeline definition in the
packaging repositories the `debian/buster-gitlab-update-job` branches
are no longer needed since they were only used to host the upstream pull
pipeline definition which it could not reside on the `debian/` branches.

Now we can kick the pipeline on the `debian/` branches directly using
the external config, so let's do that.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>