We want to ensure that the jobs updating the `latest` tags always reflect
the actual latest pipeline.

When two pipelines are started at the same time on the same branch
sometimes the earlier one can complete last due to scheduling or any
other reason, thus breaking the previous assumption.

To avoid that, make the whole pipeline interruptible up to the tagging jobs
so that earlier pipelines get killed when a new one is started.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When querying madison, the package name needs to be quoted or special
characters may cause issues. For instance, the `+` in the `gtk+3.0`
package needs to be escaped, or madison returns no matches:

    $ curl 'https://qa.debian.org/madison.php?package=gtk+3.0&yaml=on&s=jessie'
    --- {}
    $ curl 'https://qa.debian.org/madison.php?package=gtk%2B3.0&yaml=on&s=jessie

'
    ---
    gtk+3.0:
      3.14.5-1+deb8u1:
        jessie:
          - source

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

To improve license scan process we add license scan using a `FOSSology`
server.

The `fossology.py` adds an `ApertisFossolgy` class which is able to:
- request upload of source code from Git server to FOSSology server,
- retrieve references of previous scan for this source code,
- request scan of the source code,
- download analysis report.
- remove FOSSology header, project's name in file path and sort files

This class is used in `ci-licence-scan` to generate the
`debian/apertis/copyright.fossology` file.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

In order to be able to generate the BOM file of the image add python3-debian
package.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

Run the image-builder smoke-test on a kvm runner and make sure that by
debos runs the recipe using the fakemachine kvm backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

Run the image-builder smoke-test on a shared runner and make sure that
debos runs the recipe using the fakemachine uml backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

The smoke test should retrieve the backend it's running and compare it
against the requested backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

The user-mode-linux support for debos has now been merged upstream, the
packages have also been updated. Switch the builder images to use these
new packaged versions rather than installing from source.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

This reverts commit 7d4bce20. This
change introduces using a keyring from the system path, which requires
features which are not (yet) in our packaged version of debos.

This feature is introduced in debos commit d2ab9d3b08056c416bfbcd63c01199e14b2fc3b9

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Sjoerd Simons authored 4 years ago and Emanuele Aina committed 4 years ago

New debos needs newer fakemachine for the dynamic backends; However that
will also mean a newer slirp-helper etc. So do a quick-fix now to pin it
to the debos versions that has been in use for the past few months while
ensure the new versions don't cause regressions

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Rather than hardcoding the release name in the filename of the YAML
definition of the APT suite, generate it at runtime so that we can
also inherit the osname and APT mirror from the main settings, which
is useful for downstreams.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The base container already ships the GPG key for the Apertis APT
repositories, no need to have multiple copies of it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Downstreams need to figure out what is specific to Apertis and needs to
be customized, so let's not mislead them and drop "apertis" from the
rootfs intermediate artifact as it does not need any customization since
it's purely internal to the pipeline.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Calling the recipe `apertis.yaml` is annoying for downstreams which have
to either keep the misleading name or rename it and deal with issues
when pulling updates.

Let's use a slightly more descritive name that does not encode the
name of the OS.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Repeating "apertis" everywhere is not particularly useful but is
annoying for downstreams which have to either keep the misleading
names or rename everything and deal with the resulting mess when
pulling updates.

This also makes our life less painful since not having a common prefix
makes shell completion much more useful. :)

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

It's been a while since we stopped used Phabricator for code review and
moved to GitLab, let's drop the vestigial `.arcconfig` file.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `.build-docker-image` template had an explicit `needs:` pointing to
`build-base-rootfs` which was only correct for `build-base-docker-image`
as other build jobs actually depend on `build-base-docker-image`.

Most of the build jobs did indeed overrid `needs:` except
`build-image-builder-docker-image` and
`build-flatdeb-builder-docker-image` which ended up having a race
condition which caused them to fail if they got executed after
`build-base-rootfs` was done but before `build-base-docker-image` pushed
the base image.

Fix that by getting rid of the `needs:` and simply relying on stages
for sequencing the build jobs.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 4 years ago and Martyn Welch committed 4 years ago

When triggering a pipeline, by setting the `BUILD_ID` CI variable it is
now possible to tell the pipeline to create `$IMAGE:$BUILD_ID` tags
like `v2022dev0-image-builder:20201225.000`.

With this it is possible to have an external orchestrator trigger all
the nightly pipelines with a shared build id, rather than have each
pipeline compute one at different times.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

With the current tagging scheme dynamic child pipelines are not needed
for the test jobs, let's inline the job in the main pipeline.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When targeting development/preview releases the script to merge changes
from upstream works fine, but when aiming at stable releases the part
that tries to detect which branch to target (`-security` vs. `-updates`)
fails to find the refs it looks for because it searches the local refs
instead of the remote ones.

Point it to the refs under the `origin` remote to make it happy.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This is used to upload flatpack package to repository.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Untagged images are cleaned immediately from the GitLab container
registry, so to preserve them for reproducibility purposes we tag every
build with an unique name and have set a cleanup policy to drop these
tags (currently once a week).

Unfortunately the unique ID I choose is not much unique: I reused code
from another project that only builds images when new commits are
pushed, so I used the `$CI_COMMIT_SHA` value which is far from unique
when multiple images are built for the same commit due to the regularly
scheduled daily builds.

Using the `$CI_PIPELINE_ID` should provide an actual unique ID and let
people retrieve the originating pipeline more easily.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The job adding the `latest` tag for the `testcases-builder` image was
missing. Things still worked because the existing tags was still
available and it's not an image which has seen much activity, but now
that we branched v2021pre and v2022dev0 the pipeline rendering the test
cases was failing to find the images.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Sjoerd Simons authored 4 years ago and Ritesh Raj Sarraf committed 4 years ago

Using the docker container for the *current* apertis release to build
the base rootfs for all docker images leads to a bootstrapping issue. So
that should be avoided.

The options for that are:
* Use a docker image from a previous Apertis release
* Use a non-apertis image

Using a docker image from Apertis (e.g. previous stable) is workable but
just pushes the bootstrapping issue down the line. So not really that
helpful (at some point you'll reach a stable release that's not longer
supported and might be dropped).

So avoiding Apertis in the bootstrapping chain seems recommendable. For
that simply pick the upstream debos images so we can keep using debos;
That still requires a kvm based worker, but so would previous stable
apertis releases at this point.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Use `apertis-flatdeb` instead of `debos` to test the new
`apertis-flatdeb-builder` docker image.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This is requested by Gitlab CI to upload flatdeb build result to the
repository.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Frederic Danis authored 4 years ago and Martyn Welch committed 4 years ago

This allows to use flatdeb to build Mildenhall runtime and SDK, using kvm
or UML runners.

`ARG RELEASE` should be declared a second time after `FROM` to be
accessible by `RUN` commands,
cf. https://ryandaniels.ca/blog/docker-dockerfile-arg-from-arg-trouble/



Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Lightweight CI runners have been introduced to run jobs which have low
resource consumption and do not need privileged containers to reduce costs
and latency, so let's use them.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Always tag built images as `test-$COMMIT` so that we keep a bunch of
those around in case a rollback is needed, using a GitLab Container
Registry cleanup policy to keep storage under control.

https://docs.gitlab.com/ee/user/packages/container_registry/#cleanup-policy



We can then apply the `latest` tag to officially publish the built
images only after the tests have passed successfully.

This allows projects to set up a cleanup policy tailored to their needs
and to easily retain images that are needed for long term support, for
instance the ones used to on release builds.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Rather than manually escaping `$CI_COMMIT_BRANCH` as the tag name for
secondary branches, use `$CI_COMMIT_REF_SLUG`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Assume that any string containing a control character is gibberish.

Rely on the `unicodedata` module to know which character is a control
character and what is not.

This addresses some bad interactions with control characters ending up
in the diff output, like u+0098 aka Start-Of-String (SOS) preventing the
output to be displayed on terminal emulators like xterm.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

On the `policykit-1` repository `ci-license-scan` was hitting an
exception:

    Traceback (most recent call last):
      File "/usr/bin/ci-license-scan", line 425, in <module>
        main()
      File "/usr/bin/ci-license-scan", line 370, in main
        fixups[copyright][license].append(p)
    KeyError: License(synopsis='Apache-2.0', text='')

Avoid the error and simplify the code using `setdefault()`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Prevent `BSD-4-Clause` from being misdetected only because the packager
tagged some files as `BSD-4-clause`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>