Skip to content

packaging: Gather data and trigger actions on packaging repositories

Emanuele Aina requested to merge wip/em/packaging-upstream-poll into master

Introduce a pipeline to fetch data from multiple sources, cross-check the retrieved information and trigger actions.

Each step emits YAML data that can be consumed by later steps and later merged to render a dashboard, with the goal of easing the addition of more data sources and checks as much as possible.

The current steps are:

  • packaging-data-fetch-upstream: grab package listings from the configured upstream sources
  • packaging-data-fetch-downstream: scan GitLab to collect data about the packaging repositories and branches
  • yaml-merge: dedicated tool to merge data from multiple sources
  • packaging-sanity-check: verify some invariants and report mismatches
  • packaging-updates: compute which packages have a newer upstream and trigger the pipeline to pull them in
  • dashboard: render a basic dashboard listing the identified errors

By triggering only the pipelines where there's a known update pending we avoid the issues with the previous approach that involved running the pipeline on each of the 4000+ repositories every week, which ended up overwhelming GitLab.

The dashboard is published at https://infrastructure.pages.apertis.org/dashboard/ with the styling copied from what @araujo used for https://qa.apertis.org/

Somewhat amusingly, the original purpose of this repository is disabled for the moment: new pipelines are not triggered automatically and for each updatable repository the dashboard only links to the pipeline creation page pointing to the right branch. Once we manually ran some pipelines and ensured the tool is triggering the right ones with no undesired effects the automatic trigger can be trivially enabled by uncommenting the --trigger flag in .gitlab-ci.yml.

Edited by Emanuele Aina

Merge request reports