Use of `git restore` has been introduced in commit e954d2be
"ci-buildpackage: use wrap-and-sort as a workaround for odd packages".

It works on the Bullseye-based v2022dev2, but on older Buster-based
release `git restore` is not available.

Replace it with the old-style `git checkout`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Make interpolation work in a printed message by turning a plain string
to a proper f-string.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Andrej Shadura authored 4 years ago and Emanuele Aina committed 3 years ago

This can happen if the source package has been renamed while still
generating a binary package with the same name. For example, Ruby
bundler is no longer a separate source package, but is shipped in
the rubygems package. Madison returns this for bundler:

---
bundler:
  2.2.5-2:
    bullseye:
      - all

The "source" pseudoarchitecture is missing from the list since it’s not
a source package in bullseye anymore.

Apertis: T7811

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Walter Lozano authored 4 years ago and Emanuele Aina committed 3 years ago

Under early testing of ci-license-scan with docker image apertis/v2022dev2
the copyright.new file generated was missformated, putting the header Format
in the last line instead of the first one.

In order to fix the issue, force line buffering when the file is generated.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

If `--auto-merge` is not set it results in:

    Traceback (most recent call last):
      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 146, in <module>
        main()
      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 92, in main
        auto_merge = args.auto_merge.split(":")
    AttributeError: 'NoneType' object has no attribute 'split'

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `--auto-merge` flag was not meant to be mandatory.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Avoid the underscore in `--auto_merge` and use `--auto-merge` instead.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 3 years ago

In order to avoid manual work, enable merge when pipeline succeeds for
trivial changes, such those on d/apertis/* or d/changelog if this feature
is enabled via command line.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Arnaud Ferraris authored 4 years ago and Emanuele Aina committed 3 years ago

Building packages into `_build` causes issues with ocaml packages: those
too build into this directory by default, which is therefore removed
during the `clean` stage.

This has been observed for ounit, pcre-ocaml and ocaml-result, and
probably affects all ocaml packages.

Using a less common name for the build directory solves the issue
without having to patch individual packages.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Arnaud Ferraris authored 4 years ago and Emanuele Aina committed 3 years ago

A lot of `node-*` packages have their Build-Depends formatted in a way
apt can't parse, causing CI failures.

Using wrap-and-sort beforehand in order to make sure the file format is
correct will help working around these issues.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

`gbp import-dsc` puts all the new changelog entries in the commit
message, generating big walls of text when, for instance, importing the
version from Bullseye on top of the Buster one.

GitLab then puts the whole log message in the `CI_COMMIT_MESSAGE` env
var, which is passed on the docker command line, resulting in a error:

    standard_init_linux.go:219: exec user process caused: argument list too long

See https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26624#note_529234097



To avoid that, trim the message to only keep the first line.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Create a MR even when the local attempt to merge failed. In that case
let the proposed updates branch point to the upstream branch and create
a MR to submit it to the downstream branch.

This makes the failed updates more visbile and easier to track than the
failed pipelines.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Refactor the code to reduce the amount of state carried across loop
iterations and to reduce the nesting of conditionals, so that it's
hopefully easier to understand and modify.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In some cases the current code ensures that a local downstream branch,
for instance `apertis/v2021` or `apertis/v2021-updates`, is available if
a corresponding ref exists in the `origin` remote.

However, in some other cases the local ref is not created, creating an
annoying inconsistency. For instance it is never created for development
branches like `apertis/v2022dev1`. Even for stable branches, when
merging from a security branch like `debian/buster-security` and a
`origin/apertis/v2021-security` remote ref is available, the local
`apertis/v2021-security` ref is not created.

Tweak the code to always ensure there's a local ref for the selected
downstream branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Ritesh Raj Sarraf authored 4 years ago and Emanuele Aina committed 3 years ago

Without it, the generated changelog was wrong like below:

```
node-accepts (1.3.7-1{local_suffix}0) apertis; urgency=medium

  * Sync from Debian debian/bullseye.

 -- Apertis CI <devel@lists.apertis.org>  Wed, 10 Mar 2021 14:56:51 +0000
```

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Ritesh Raj Sarraf authored 4 years ago and Emanuele Aina committed 4 years ago

Switch from coX to apertisX as the default Debian changelog suffix.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Downstreams need a different local suffix for their packages and we may
change ours at some point.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Avoid failing silently in case of fakemachine boot issues.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Frederic Danis authored 4 years ago and Emanuele Aina committed 4 years ago

License scan for `pkg/dash` and `tests\dash` should not re-use the scan
results from a previous scan of the other project.

Separate license scan using different group id, which is done using
different users.

Andrej Shadura authored 4 years ago and Emanuele Aina committed 4 years ago

Strip off the suffix since all branches related to the upstream
Debian release need to be taken into account when pulling updates.

Without this, when the script runs on e.g. buster-security branch,
it would completely ignore the existence of any other buster branches,
leading to incorrectly applied updates.

Apertis: T7681

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Andrej Shadura authored 4 years ago and Emanuele Aina committed 4 years ago

Try to download an update for dash from Buster, given a repository
with a version from Lenny. Don’t try to check when exactly is in the
update, as versions may change, but verify a tarball has been imported.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Andrej Shadura authored 4 years ago and Emanuele Aina committed 4 years ago

This will try to merge an "update" of dash on a completely made-up
package history. There are two fixtures, sharing common parts. They
both:

* Start with an ancient dash version from Lenny, 0.5.4-12.
* This version is imported as 0.5.4-12co0 into an imaginary
  Apertis version.
* A "new" version of dash, 0.5.10.2-1 is imported from a 2018
  cut of Buster.
* This new version is merged into Apertis as 0.5.10.2-1co0

From this point, the "modified" fixture differs:
* An imaginary Apertis developer changes the compat level in
  the imported version from 10 to 11, thus introducing a delta
  which however is not going to conflict with any further version
  of dash in Buster.

apertis-pkg-merge-updates is expected to merge the update into the
"unmodified" repository as 0.5.10.2-5co0 and finalise the changelog.

For the "modified" repository, it should import the update as
0.5.10.2-5co1, add "PLEASE SUMMARIZE remaining Apertis changes" note
to the top entry and leave it UNRELEASED.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Andrej Shadura authored 4 years ago and Emanuele Aina committed 4 years ago

We always add some extra files, at least the Apertis component,
plus copyright scanning results. Since those are intended to end up
in the source package, we should append a version suffix regardless
of any other actual changes we add on top.

Since these changes are an addition of metadata only, co0 seems a
good enough starting version.

Instead of dch, we use a custom changelog modification implementation.
This is because dch insists on launching an interactive editor when
a changelog is finalised.

Apertis: T7556

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Andrej Shadura authored 4 years ago and Emanuele Aina committed 4 years ago

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Andrej Shadura authored 4 years ago and Emanuele Aina committed 4 years ago

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Andrej Shadura authored 4 years ago and Emanuele Aina committed 4 years ago

checkout-tarballs is no longer needed since pristine-lfs can do
this directly.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Andrej Shadura authored 4 years ago and Emanuele Aina committed 4 years ago

Since pristine-lfs can import tarballs automatically, we should
not use a separate but different implementation of the same
functionality.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

By applying `latest` tags as soon as the relevant image has been built
we end up with a quite large time window where they are all quite out
of sync: in particular, the tag is applied to the `base` image as soon
as it is ready, before all the depending images get built.

On top of that, if depending images fail, the `latest` tags will stay
out of sync.

This can be quite confusing for users, so let's apply the `latest` tag
only after all the previous stages have completed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We want to ensure that the jobs updating the `latest` tags always reflect
the actual latest pipeline.

When two pipelines are started at the same time on the same branch
sometimes the earlier one can complete last due to scheduling or any
other reason, thus breaking the previous assumption.

To avoid that, make the whole pipeline interruptible up to the tagging jobs
so that earlier pipelines get killed when a new one is started.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When querying madison, the package name needs to be quoted or special
characters may cause issues. For instance, the `+` in the `gtk+3.0`
package needs to be escaped, or madison returns no matches:

    $ curl 'https://qa.debian.org/madison.php?package=gtk+3.0&yaml=on&s=jessie'
    --- {}
    $ curl 'https://qa.debian.org/madison.php?package=gtk%2B3.0&yaml=on&s=jessie

'
    ---
    gtk+3.0:
      3.14.5-1+deb8u1:
        jessie:
          - source

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Frederic Danis authored 4 years ago and Emanuele Aina committed 4 years ago

To improve license scan process we add license scan using a `FOSSology`
server.

The `fossology.py` adds an `ApertisFossolgy` class which is able to:
- request upload of source code from Git server to FOSSology server,
- retrieve references of previous scan for this source code,
- request scan of the source code,
- download analysis report.
- remove FOSSology header, project's name in file path and sort files

This class is used in `ci-licence-scan` to generate the
`debian/apertis/copyright.fossology` file.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

In order to be able to generate the BOM file of the image add python3-debian
package.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Christopher Obbard authored 4 years ago and Sjoerd Simons committed 4 years ago

Include some extra compression tools from the upstream Debos
dockerfile which may be useful.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

The equivs package is used to create dummy Debian packages which can
be useful in development to satisfy dependencies without installing the
real package. Note that this package is not the recommended way of
dealing with broken dependencies: a bug report should be filed instead.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

Run the image-builder smoke-test on a kvm runner and make sure that by
debos runs the recipe using the fakemachine kvm backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

Run the image-builder smoke-test on a shared runner and make sure that
debos runs the recipe using the fakemachine uml backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

The smoke test should retrieve the backend it's running and compare it
against the requested backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

The user-mode-linux support for debos has now been merged upstream, the
packages have also been updated. Switch the builder images to use these
new packaged versions rather than installing from source.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>