Odd that it managed to pass multiple runs of CI but is now suddenly
timing out, but it appears to still be working just fine based on the
log output, so just try to make it a tad faster & make sure it can take
longer.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

This explains the meaning of the different statistics taken.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Now, "unique" is used for unique storage on both repositories *and*
snapshots, and "direct" is used for the storage directly used, rather
than "total".

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Ritesh Raj Sarraf authored 3 years ago and Emanuele Aina committed 3 years ago

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

This adds a new tool, storage_stats, to gather statistics on the used
storage, as well as a new dashboard page to display the statistics. Due
to exceedingly poor performance from attempting to use pure Python to
parse all the packaging files in over 2000 snapshots (a total of several
thousand files to parse & process), Rust was used for the storage_stats
implementation; on my system, it determines the full extent of storage
usage in under 20 minutes.

On the dashboard's side, the common HTML and styling was moved into a
single shared template that both the current index/packages page *and*
the new storage page can use, and navigational links were added to the
pages.

https://phabricator.apertis.org/T8197



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Instead of specifying a per-component URL, now a single root URL is
given, and the component paths are derived from that. In addition, the
code that parses the files was refactored a bit to remove duplicated
code in the `packaging-data-fetch-*` scripts.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Now that all the development happens on v2022pre and the mirroring on
the Apertispro side is complete we can stop monitoring v2022dev3.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Inline the link to the per-release package indices with the summary
instead of just having it misaligned somewhere in the header.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Make it easier to share a pointer to a specific report by making the
anchors more discoverable with self-links that appear on hover.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Still continue to use v2022dev3 because our derivatives will use it for
some time

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

The `golang-k8s-sigs-yaml` update in `apertis/v2021-updates` broke the
rendering for downstream instances since it does not have a base branch:

    golang-k8s-sigs-yaml:
      git:
        path_with_namespace: pkg/golang-k8s-sigs-yaml
      updates:
      - branch:
          name: apertis/v2021-updates
          version: 1.2.0-2+apertis1
        upstream:
          component: development
          name: golang-k8s-sigs-yaml
          source: apertis/v2021-updates
          version: 1.2.0-3+apertis1

This is due to the fact that `golang-k8s-sigs-yaml` is a new dependency
introduced by a package that got updated in `apertis/v2021-updates` and
has not been folded into `apertis/v2021` yet: no `golang-k8s-sigs-yaml`
exists in `apertis/v2021`, and for this reason there's no `base` object.

In this case it is enough to trigger the pipeline directly on the
`apertis/v2021-updates` branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Apertis should follow a strict license compliance to avoid shipping software
under not suitable license. Improve dashboard to warn about packages in
target repository which does not provide license report.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Put a value that is extracted a few times in a variable to make the code
a bit more readable.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

No logic changes, exit from a loop early to make the code slightly
more readable.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Move the logic about converting back and forth between OBS project names
and git branches to dedicated functions to better convey their purpose.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Isolate the binary/source version matching by moving it to a dedicated
function, given that it needs a big comment to explain what is going on.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `public/data.yaml` generated by the dashboard contains all the
combined data to let triggers do their job, using it gives more
flexibility later on.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When retrieving an artifacts from a past job with a job token, the
`JOB-TOKEN``header should be used rather than the `PRIVATE-HEADER` and
the API URL should be used rather than the enduser route.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Repeat with me: `test -e` stands for "file exists", *not* for "string is
empty", as that's `test -z`.

Update the test in `trigger-updates` accordingly to properly print the
help text when the variable is not set.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

If we already dropped a package from the development channel there is
no point in complaining if its upstream got dropped from Debian, even
if we still ship in stable releases.

If instead it's still in a development branch, show the branch affected.

This silences errors on bootchart2 which is still in v2020 while it got
dropped from Debian Buster and we no longer ship it in v2021 and later.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In some cases we emit some strings with characters like `<` and since
these didn't get escaped we were generating malformed HTML.

Fix that by enabling autoescaping.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `gcc-*-cross` binaries get their versions from the matching `gcc`
source package, so ignore source/binaries mismatches.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Expand the list of packages that had to be renamed on GitLab because
their name contains characters that are not valid for project names.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

List a bunch of packages that dynamically generate the version of the
binaries they produce.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When an update is available on e.g. `buster-security` try to trigger the
update pipeline on the `debian/buster` branch rather than the
`debian/buster-security` one, for consistency.

The latter is currently buggy, but that's a bug in the rules of the
infrastructure/ci-package-builder> pipeline that needs to be fixed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We never backported the commits adding `debian/apertis/component` files
to v2020 to avoid unnecessary churn in a stable release but this had two
drawbacks:

1. pipelines that got re-triggered for any reason would fail
1. almost all v2020 package lag compared to v2021

Since we're not going to address these issues, explicitly ignore them to
reduce the amount of noise on the dashboard.

Ignoring lagging packages may cause us to miss some backports, but at
the moment they'd get drowned in the noise anyway.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Do not re-download the list of published packages but re-use the data
retrieved by the `packaging-data-fetch-sources-published` job, avoiding
a large chunk of duplicated code.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For some imperscrutable reason the YAML file has the executable bit set,
unset it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Kill old pipelines when triggering a new one on the same branch, there
are no side effects in doing that, except for the trigger-updates job.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Now that v20219.7 has been released we can consider v2019 in read-only
mode, no need to monitor it any further.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

v2021 is stable, we should ensure v2021-updates and v2021-security are
consistent too.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The current code was not reporting updates for destinations like
`debian/buster-updates` unless the branch already existed.

Tweak the code to check all the sources even if only a subset of the
matching branches exists, using a new `base` key in the source
definition to group them.

For instance, if a repository has only the `debian/buster-updates`
branch the `base` key points to `debian/buster` which in turn matches
the `debian/buster`, `debian/buster-security`, and
`debian/buster-updates` sources. The availability of updates will be
checked for each of them.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `packaging-updates` tool operates exclusively on local data and it
does not need a connection to the GitLab API, so drop the relevant code.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Frederic Danis authored 3 years ago and Emanuele Aina committed 3 years ago

Some packages like cross-toolchain-base and cross-toolchain-base-ports do
special things for which the source version does not match at all the
binaries because it reflects the one from the gcc version it uses.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Frederic Danis authored 3 years ago and Emanuele Aina committed 3 years ago

With the new published binary data, these stages are failing on
`lightweight` runners.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>