Currently the code pulling updates from upstream assumes that downstream
branches ending with `dev[0-9]` or `pre` are not stable so updates can
be directly landed there, while other downstream branches (for instance,
`apertis/v2021`) are stable and updates should go through the updates or
security repositories.

However, there's a time right before the first stable release is
published that the "stable" branch is not stable yet. For instance,
before the release of `v2021.0` the `apertis/v2021` should not have been
considered stable and updates should have been landed directly there.

This is even more important for downstreams so that they can pull
updates to their not-released-yet v2021 branch.

To do so, add a `--stable` parameter that accepts the list of branches
to be considered "stable".

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit 788aa27a "Debian commits" was introduced to base images
directly on Debian while the Bullseye import in v2022dev2 is not
complete enought to real Apertis build images.

Unfortunately it broke the ability to tag images as `latest`.

Rework the hack so tagging works again.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Ariel D'Alessandro authored 4 years ago and Emanuele Aina committed 4 years ago

Currently `ci-license-scan` fails if it can't detect the license of some
files, even if are no licensing restrictions in the component that hosts
the package.

This is annoying because it causes the scan to block pipelines where
there's little value in adding the missing licensing information: for
instance, while it is important on target, it is not particularly
relevant everywhere else.

To avoid that, add a flag to define where failing to detect a license is
problematic, otherwise make the command succeed and simply pass through
the `UNKNOWN` entries in the licensing report.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use of `git restore` has been introduced in commit e954d2be
"ci-buildpackage: use wrap-and-sort as a workaround for odd packages".

It works on the Bullseye-based v2022dev2, but on older Buster-based
release `git restore` is not available.

Replace it with the old-style `git checkout`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Make interpolation work in a printed message by turning a plain string
to a proper f-string.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This reverts commit b26ad3fa "Explicitly set TAG to latest"
since it is causing every image that gets built, even the ones on
secondary wip branches, to be tagged as `latest` and thus affecting
production everywhere.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Andrej Shadura authored 4 years ago and Martyn Welch committed 4 years ago

This can happen if the source package has been renamed while still
generating a binary package with the same name. For example, Ruby
bundler is no longer a separate source package, but is shipped in
the rubygems package. Madison returns this for bundler:

---
bundler:
  2.2.5-2:
    bullseye:
      - all

The "source" pseudoarchitecture is missing from the list since it’s not
a source package in bullseye anymore.

Apertis: T7811

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Under early testing of ci-license-scan with docker image apertis/v2022dev2
the copyright.new file generated was missformated, putting the header Format
in the last line instead of the first one.

In order to fix the issue, force line buffering when the file is generated.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

If `--auto-merge` is not set it results in:

    Traceback (most recent call last):
      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 146, in <module>
        main()
      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 92, in main
        auto_merge = args.auto_merge.split(":")
    AttributeError: 'NoneType' object has no attribute 'split'

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `--auto-merge` flag was not meant to be mandatory.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Avoid the underscore in `--auto_merge` and use `--auto-merge` instead.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In order to avoid manual work, enable merge when pipeline succeeds for
trivial changes, such those on d/apertis/* or d/changelog if this feature
is enabled via command line.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Arnaud Ferraris authored 4 years ago and Martyn Welch committed 4 years ago

Building packages into `_build` causes issues with ocaml packages: those
too build into this directory by default, which is therefore removed
during the `clean` stage.

This has been observed for ounit, pcre-ocaml and ocaml-result, and
probably affects all ocaml packages.

Using a less common name for the build directory solves the issue
without having to patch individual packages.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

A lot of `node-*` packages have their Build-Depends formatted in a way
apt can't parse, causing CI failures.

Using wrap-and-sort beforehand in order to make sure the file format is
correct will help working around these issues.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

This reverts commit cf5f4c2d.

Building packages into `_build` causes issues with ocaml packages: those
too build into this directory by default, which is therefore removed
during the `clean` stage.

This has been observed for ounit, pcre-ocaml and ocaml-result, and
probably affects all ocaml packages.

Using a less common name for the build directory solves the issue
without having to patch individual packages.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

`gbp import-dsc` puts all the new changelog entries in the commit
message, generating big walls of text when, for instance, importing the
version from Bullseye on top of the Buster one.

GitLab then puts the whole log message in the `CI_COMMIT_MESSAGE` env
var, which is passed on the docker command line, resulting in a error:

    standard_init_linux.go:219: exec user process caused: argument list too long

See https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26624#note_529234097



To avoid that, trim the message to only keep the first line.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Create a MR even when the local attempt to merge failed. In that case
let the proposed updates branch point to the upstream branch and create
a MR to submit it to the downstream branch.

This makes the failed updates more visbile and easier to track than the
failed pipelines.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Refactor the code to reduce the amount of state carried across loop
iterations and to reduce the nesting of conditionals, so that it's
hopefully easier to understand and modify.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In some cases the current code ensures that a local downstream branch,
for instance `apertis/v2021` or `apertis/v2021-updates`, is available if
a corresponding ref exists in the `origin` remote.

However, in some other cases the local ref is not created, creating an
annoying inconsistency. For instance it is never created for development
branches like `apertis/v2022dev1`. Even for stable branches, when
merging from a security branch like `debian/buster-security` and a
`origin/apertis/v2021-security` remote ref is available, the local
`apertis/v2021-security` ref is not created.

Tweak the code to always ensure there's a local ref for the selected
downstream branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Without it, the generated changelog was wrong like below:

```
node-accepts (1.3.7-1{local_suffix}0) apertis; urgency=medium

  * Sync from Debian debian/bullseye.

 -- Apertis CI <devel@lists.apertis.org>  Wed, 10 Mar 2021 14:56:51 +0000
```

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

The RELEASE string is used commonly by multiple parties
We set a custom CUSTERL variable, for this special case, to push in the
special string "bullseyebootstrap"

For the use case of building pure Debian packages, we only need the
package-source-builder. So, lets just disable the rest of the images.

And also add the Debian specific bits to get the debootstrap work

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Ritesh Raj Sarraf authored 4 years ago and Emanuele Aina committed 4 years ago

Switch from coX to apertisX as the default Debian changelog suffix.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Downstreams need a different local suffix for their packages and we may
change ours at some point.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 4 years ago and Martyn Welch committed 4 years ago

Avoid failing silently in case of fakemachine boot issues.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

License scan for `pkg/dash` and `tests\dash` should not re-use the scan
results from a previous scan of the other project.

Separate license scan using different group id, which is done using
different users.

Strip off the suffix since all branches related to the upstream
Debian release need to be taken into account when pulling updates.

Without this, when the script runs on e.g. buster-security branch,
it would completely ignore the existence of any other buster branches,
leading to incorrectly applied updates.

Apertis: T7681

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Try to download an update for dash from Buster, given a repository
with a version from Lenny. Don’t try to check when exactly is in the
update, as versions may change, but verify a tarball has been imported.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

This will try to merge an "update" of dash on a completely made-up
package history. There are two fixtures, sharing common parts. They
both:

* Start with an ancient dash version from Lenny, 0.5.4-12.
* This version is imported as 0.5.4-12co0 into an imaginary
  Apertis version.
* A "new" version of dash, 0.5.10.2-1 is imported from a 2018
  cut of Buster.
* This new version is merged into Apertis as 0.5.10.2-1co0

From this point, the "modified" fixture differs:
* An imaginary Apertis developer changes the compat level in
  the imported version from 10 to 11, thus introducing a delta
  which however is not going to conflict with any further version
  of dash in Buster.

apertis-pkg-merge-updates is expected to merge the update into the
"unmodified" repository as 0.5.10.2-5co0 and finalise the changelog.

For the "modified" repository, it should import the update as
0.5.10.2-5co1, add "PLEASE SUMMARIZE remaining Apertis changes" note
to the top entry and leave it UNRELEASED.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

We always add some extra files, at least the Apertis component,
plus copyright scanning results. Since those are intended to end up
in the source package, we should append a version suffix regardless
of any other actual changes we add on top.

Since these changes are an addition of metadata only, co0 seems a
good enough starting version.

Instead of dch, we use a custom changelog modification implementation.
This is because dch insists on launching an interactive editor when
a changelog is finalised.

Apertis: T7556

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

checkout-tarballs is no longer needed since pristine-lfs can do
this directly.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Since pristine-lfs can import tarballs automatically, we should
not use a separate but different implementation of the same
functionality.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Include some extra compression tools from the upstream Debos
dockerfile which may be useful.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>