Refactor the code to reduce the amount of state carried across loop
iterations and to reduce the nesting of conditionals, so that it's
hopefully easier to understand and modify.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In some cases the current code ensures that a local downstream branch,
for instance `apertis/v2021` or `apertis/v2021-updates`, is available if
a corresponding ref exists in the `origin` remote.

However, in some other cases the local ref is not created, creating an
annoying inconsistency. For instance it is never created for development
branches like `apertis/v2022dev1`. Even for stable branches, when
merging from a security branch like `debian/buster-security` and a
`origin/apertis/v2021-security` remote ref is available, the local
`apertis/v2021-security` ref is not created.

Tweak the code to always ensure there's a local ref for the selected
downstream branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Without it, the generated changelog was wrong like below:

```
node-accepts (1.3.7-1{local_suffix}0) apertis; urgency=medium

  * Sync from Debian debian/bullseye.

 -- Apertis CI <devel@lists.apertis.org>  Wed, 10 Mar 2021 14:56:51 +0000
```

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

The RELEASE string is used commonly by multiple parties
We set a custom CUSTERL variable, for this special case, to push in the
special string "bullseyebootstrap"

For the use case of building pure Debian packages, we only need the
package-source-builder. So, lets just disable the rest of the images.

And also add the Debian specific bits to get the debootstrap work

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Ritesh Raj Sarraf authored 4 years ago and Emanuele Aina committed 4 years ago

Switch from coX to apertisX as the default Debian changelog suffix.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Downstreams need a different local suffix for their packages and we may
change ours at some point.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 4 years ago and Martyn Welch committed 4 years ago

Avoid failing silently in case of fakemachine boot issues.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

License scan for `pkg/dash` and `tests\dash` should not re-use the scan
results from a previous scan of the other project.

Separate license scan using different group id, which is done using
different users.

Strip off the suffix since all branches related to the upstream
Debian release need to be taken into account when pulling updates.

Without this, when the script runs on e.g. buster-security branch,
it would completely ignore the existence of any other buster branches,
leading to incorrectly applied updates.

Apertis: T7681

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Try to download an update for dash from Buster, given a repository
with a version from Lenny. Don’t try to check when exactly is in the
update, as versions may change, but verify a tarball has been imported.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

This will try to merge an "update" of dash on a completely made-up
package history. There are two fixtures, sharing common parts. They
both:

* Start with an ancient dash version from Lenny, 0.5.4-12.
* This version is imported as 0.5.4-12co0 into an imaginary
  Apertis version.
* A "new" version of dash, 0.5.10.2-1 is imported from a 2018
  cut of Buster.
* This new version is merged into Apertis as 0.5.10.2-1co0

From this point, the "modified" fixture differs:
* An imaginary Apertis developer changes the compat level in
  the imported version from 10 to 11, thus introducing a delta
  which however is not going to conflict with any further version
  of dash in Buster.

apertis-pkg-merge-updates is expected to merge the update into the
"unmodified" repository as 0.5.10.2-5co0 and finalise the changelog.

For the "modified" repository, it should import the update as
0.5.10.2-5co1, add "PLEASE SUMMARIZE remaining Apertis changes" note
to the top entry and leave it UNRELEASED.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

We always add some extra files, at least the Apertis component,
plus copyright scanning results. Since those are intended to end up
in the source package, we should append a version suffix regardless
of any other actual changes we add on top.

Since these changes are an addition of metadata only, co0 seems a
good enough starting version.

Instead of dch, we use a custom changelog modification implementation.
This is because dch insists on launching an interactive editor when
a changelog is finalised.

Apertis: T7556

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

checkout-tarballs is no longer needed since pristine-lfs can do
this directly.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Since pristine-lfs can import tarballs automatically, we should
not use a separate but different implementation of the same
functionality.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Include some extra compression tools from the upstream Debos
dockerfile which may be useful.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

The equivs package is used to create dummy Debian packages which can
be useful in development to satisfy dependencies without installing the
real package. Note that this package is not the recommended way of
dealing with broken dependencies: a bug report should be filed instead.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

By applying `latest` tags as soon as the relevant image has been built
we end up with a quite large time window where they are all quite out
of sync: in particular, the tag is applied to the `base` image as soon
as it is ready, before all the depending images get built.

On top of that, if depending images fail, the `latest` tags will stay
out of sync.

This can be quite confusing for users, so let's apply the `latest` tag
only after all the previous stages have completed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We want to ensure that the jobs updating the `latest` tags always reflect
the actual latest pipeline.

When two pipelines are started at the same time on the same branch
sometimes the earlier one can complete last due to scheduling or any
other reason, thus breaking the previous assumption.

To avoid that, make the whole pipeline interruptible up to the tagging jobs
so that earlier pipelines get killed when a new one is started.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When querying madison, the package name needs to be quoted or special
characters may cause issues. For instance, the `+` in the `gtk+3.0`
package needs to be escaped, or madison returns no matches:

    $ curl 'https://qa.debian.org/madison.php?package=gtk+3.0&yaml=on&s=jessie'
    --- {}
    $ curl 'https://qa.debian.org/madison.php?package=gtk%2B3.0&yaml=on&s=jessie

'
    ---
    gtk+3.0:
      3.14.5-1+deb8u1:
        jessie:
          - source

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

To improve license scan process we add license scan using a `FOSSology`
server.

The `fossology.py` adds an `ApertisFossolgy` class which is able to:
- request upload of source code from Git server to FOSSology server,
- retrieve references of previous scan for this source code,
- request scan of the source code,
- download analysis report.
- remove FOSSology header, project's name in file path and sort files

This class is used in `ci-licence-scan` to generate the
`debian/apertis/copyright.fossology` file.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

In order to be able to generate the BOM file of the image add python3-debian
package.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

Run the image-builder smoke-test on a kvm runner and make sure that by
debos runs the recipe using the fakemachine kvm backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

Run the image-builder smoke-test on a shared runner and make sure that
debos runs the recipe using the fakemachine uml backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

The smoke test should retrieve the backend it's running and compare it
against the requested backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

The user-mode-linux support for debos has now been merged upstream, the
packages have also been updated. Switch the builder images to use these
new packaged versions rather than installing from source.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

This reverts commit 7d4bce20. This
change introduces using a keyring from the system path, which requires
features which are not (yet) in our packaged version of debos.

This feature is introduced in debos commit d2ab9d3b08056c416bfbcd63c01199e14b2fc3b9

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Sjoerd Simons authored 4 years ago and Emanuele Aina committed 4 years ago

New debos needs newer fakemachine for the dynamic backends; However that
will also mean a newer slirp-helper etc. So do a quick-fix now to pin it
to the debos versions that has been in use for the past few months while
ensure the new versions don't cause regressions

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Rather than hardcoding the release name in the filename of the YAML
definition of the APT suite, generate it at runtime so that we can
also inherit the osname and APT mirror from the main settings, which
is useful for downstreams.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The base container already ships the GPG key for the Apertis APT
repositories, no need to have multiple copies of it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Downstreams need to figure out what is specific to Apertis and needs to
be customized, so let's not mislead them and drop "apertis" from the
rootfs intermediate artifact as it does not need any customization since
it's purely internal to the pipeline.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>