The job retrieving data from OBS often reaches the default 1h timeout,
so let's be more lenient.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For some of the longer running jobs it is sufficient for the service
they interface with to be marginally slower than usual to trigger a
timeout, either in the overall job execution time on on specific
requests from the tools, causing the whole pipeline to fail.

In those case we can afford to retry the job to see if things improve.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Listing which e.g. `debian/*` tag is contained in a `apertis/*` branch
is rather pointless and is a significant waste of resources.

Limit the reported tags to the one with a matching prefix, so only
`debian/*` tags are listed on `debian/*` branches and only `apertis/*`
tags are listed on `apertis/*` branches.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since we already list all the tag contained in each branch, including
the symmetrical list of descendant branches for each tag in the YAML is
wasting bytes with duplicated information.

We still use the list of descendant branches internally to compute which
tags are contained in each branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Let's save a few bytes by getting rid of some pointless information.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This saves a few bytes for each `debian/*` and `upstream/*` branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Ryan Gonzalez committed 3 years ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Walter Lozano committed 3 years ago

The `./localtest` script was skipping the `lint` job, not the `lint`
stage, so when I added the job to exercise `./localtest` itself during
the `lint` stage I have caused `./localtest` to appear in its own
output, which means that if you pipe it into a shell it will call
itself. Luckily recursion will stop there and this is about not doing
something pointless rather than actually fixing something broken.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Highlight the `Filter` button when any filter is active and reports may
be hidden.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Rather than using plaintext error messages, use readable codes and
structured metadata for errors and updates to make them easier
to process.

This will be particularly useful for filtering: for instance we
preserve the branch information rather than muddling it in the
error message.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When multiple sources are published, check the version of the
most recent one rather than whatever comes first.

This avoids reporting a mismatched version when things are just
ambiguous.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add more parallel workers whe retrieving the component and licensing
report from git to speed up the process.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since we moved all git repositories directly under a single `pkg/` group
instead of the `pkg/$component/` nested categories, checking for
duplicates is no longer relevant.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

No functional changes, just get rid of a level of indentiation.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add a script to run the pipeline locally during development:

    ./localtest --skip-job=storage-usage --filter-packages=apache-log4j2 | sh

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Let users focus on a subset of the available packages to speed up the
test cycle during development.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This does not introduce any functional change, it only makes things
slightly more consistent.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

GitLab can show a textual description for variables that can be set when
manually triggering a pipeline, see
https://docs.gitlab.com/ee/ci/yaml/index.html#variablesdescription



Use that to document how `TRIGGER_UPDATES` should be used.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We pass a list of `OBSEntry` to `thread_pool()`, so add a `id` property
that better identifies the current entry to give enough context should
things go wrong.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

There are more chances that an `id` property is actually unique than a
`name` property, so let's attempt to use it before `name` when
reporting the element `thread_pool()` is working on.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When something fails when called by `thread_pool()` log it as an error,
not just as a debug message.

This usually give more context than just the exception traceback since
we try to find an identifier for the element being processed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Vignesh Raman authored 3 years ago and Emanuele Aina committed 3 years ago

Add a new packaging-updates-upstream-linux script
which checks the version of linux kernel in
apertis and upstream. If the apertis linux requires
an update, it updates the dashboard.

packaging-updates-upstream-linux script matches
major.minor release version to find the appropriate
upstream.

If v2021 has 5.10.x, point the dashboard to the
5.10.x kernel in Bullseye rather than referring
to the upstream git.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>
Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>

Odd that it managed to pass multiple runs of CI but is now suddenly
timing out, but it appears to still be working just fine based on the
log output, so just try to make it a tad faster & make sure it can take
longer.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

This explains the meaning of the different statistics taken.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Now, "unique" is used for unique storage on both repositories *and*
snapshots, and "direct" is used for the storage directly used, rather
than "total".

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Ritesh Raj Sarraf authored 3 years ago and Emanuele Aina committed 3 years ago

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

This adds a new tool, storage_stats, to gather statistics on the used
storage, as well as a new dashboard page to display the statistics. Due
to exceedingly poor performance from attempting to use pure Python to
parse all the packaging files in over 2000 snapshots (a total of several
thousand files to parse & process), Rust was used for the storage_stats
implementation; on my system, it determines the full extent of storage
usage in under 20 minutes.

On the dashboard's side, the common HTML and styling was moved into a
single shared template that both the current index/packages page *and*
the new storage page can use, and navigational links were added to the
pages.

https://phabricator.apertis.org/T8197



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Instead of specifying a per-component URL, now a single root URL is
given, and the component paths are derived from that. In addition, the
code that parses the files was refactored a bit to remove duplicated
code in the `packaging-data-fetch-*` scripts.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Now that all the development happens on v2022pre and the mirroring on
the Apertispro side is complete we can stop monitoring v2022dev3.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Inline the link to the per-release package indices with the summary
instead of just having it misaligned somewhere in the header.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Make it easier to share a pointer to a specific report by making the
anchors more discoverable with self-links that appear on hover.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>