For the use case of building pure Debian packages, we only need the
package-source-builder. So, lets just disable the rest of the images.

And also add the Debian specific bits to get the debootstrap work

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Ritesh Raj Sarraf authored 4 years ago and Emanuele Aina committed 4 years ago

Switch from coX to apertisX as the default Debian changelog suffix.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Downstreams need a different local suffix for their packages and we may
change ours at some point.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 4 years ago and Martyn Welch committed 4 years ago

Avoid failing silently in case of fakemachine boot issues.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Try to download an update for dash from Buster, given a repository
with a version from Lenny. Don’t try to check when exactly is in the
update, as versions may change, but verify a tarball has been imported.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

This will try to merge an "update" of dash on a completely made-up
package history. There are two fixtures, sharing common parts. They
both:

* Start with an ancient dash version from Lenny, 0.5.4-12.
* This version is imported as 0.5.4-12co0 into an imaginary
  Apertis version.
* A "new" version of dash, 0.5.10.2-1 is imported from a 2018
  cut of Buster.
* This new version is merged into Apertis as 0.5.10.2-1co0

From this point, the "modified" fixture differs:
* An imaginary Apertis developer changes the compat level in
  the imported version from 10 to 11, thus introducing a delta
  which however is not going to conflict with any further version
  of dash in Buster.

apertis-pkg-merge-updates is expected to merge the update into the
"unmodified" repository as 0.5.10.2-5co0 and finalise the changelog.

For the "modified" repository, it should import the update as
0.5.10.2-5co1, add "PLEASE SUMMARIZE remaining Apertis changes" note
to the top entry and leave it UNRELEASED.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

By applying `latest` tags as soon as the relevant image has been built
we end up with a quite large time window where they are all quite out
of sync: in particular, the tag is applied to the `base` image as soon
as it is ready, before all the depending images get built.

On top of that, if depending images fail, the `latest` tags will stay
out of sync.

This can be quite confusing for users, so let's apply the `latest` tag
only after all the previous stages have completed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We want to ensure that the jobs updating the `latest` tags always reflect
the actual latest pipeline.

When two pipelines are started at the same time on the same branch
sometimes the earlier one can complete last due to scheduling or any
other reason, thus breaking the previous assumption.

To avoid that, make the whole pipeline interruptible up to the tagging jobs
so that earlier pipelines get killed when a new one is started.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

Run the image-builder smoke-test on a kvm runner and make sure that by
debos runs the recipe using the fakemachine kvm backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Christopher Obbard authored 4 years ago and Emanuele Aina committed 4 years ago

Run the image-builder smoke-test on a shared runner and make sure that
debos runs the recipe using the fakemachine uml backend.

Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>

Rather than hardcoding the release name in the filename of the YAML
definition of the APT suite, generate it at runtime so that we can
also inherit the osname and APT mirror from the main settings, which
is useful for downstreams.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Calling the recipe `apertis.yaml` is annoying for downstreams which have
to either keep the misleading name or rename it and deal with issues
when pulling updates.

Let's use a slightly more descritive name that does not encode the
name of the OS.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Repeating "apertis" everywhere is not particularly useful but is
annoying for downstreams which have to either keep the misleading
names or rename everything and deal with the resulting mess when
pulling updates.

This also makes our life less painful since not having a common prefix
makes shell completion much more useful. :)

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `.build-docker-image` template had an explicit `needs:` pointing to
`build-base-rootfs` which was only correct for `build-base-docker-image`
as other build jobs actually depend on `build-base-docker-image`.

Most of the build jobs did indeed overrid `needs:` except
`build-image-builder-docker-image` and
`build-flatdeb-builder-docker-image` which ended up having a race
condition which caused them to fail if they got executed after
`build-base-rootfs` was done but before `build-base-docker-image` pushed
the base image.

Fix that by getting rid of the `needs:` and simply relying on stages
for sequencing the build jobs.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 4 years ago and Martyn Welch committed 4 years ago

When triggering a pipeline, by setting the `BUILD_ID` CI variable it is
now possible to tell the pipeline to create `$IMAGE:$BUILD_ID` tags
like `v2022dev0-image-builder:20201225.000`.

With this it is possible to have an external orchestrator trigger all
the nightly pipelines with a shared build id, rather than have each
pipeline compute one at different times.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

With the current tagging scheme dynamic child pipelines are not needed
for the test jobs, let's inline the job in the main pipeline.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Untagged images are cleaned immediately from the GitLab container
registry, so to preserve them for reproducibility purposes we tag every
build with an unique name and have set a cleanup policy to drop these
tags (currently once a week).

Unfortunately the unique ID I choose is not much unique: I reused code
from another project that only builds images when new commits are
pushed, so I used the `$CI_COMMIT_SHA` value which is far from unique
when multiple images are built for the same commit due to the regularly
scheduled daily builds.

Using the `$CI_PIPELINE_ID` should provide an actual unique ID and let
people retrieve the originating pipeline more easily.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The job adding the `latest` tag for the `testcases-builder` image was
missing. Things still worked because the existing tags was still
available and it's not an image which has seen much activity, but now
that we branched v2021pre and v2022dev0 the pipeline rendering the test
cases was failing to find the images.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Sjoerd Simons authored 4 years ago and Ritesh Raj Sarraf committed 4 years ago

Using the docker container for the *current* apertis release to build
the base rootfs for all docker images leads to a bootstrapping issue. So
that should be avoided.

The options for that are:
* Use a docker image from a previous Apertis release
* Use a non-apertis image

Using a docker image from Apertis (e.g. previous stable) is workable but
just pushes the bootstrapping issue down the line. So not really that
helpful (at some point you'll reach a stable release that's not longer
supported and might be dropped).

So avoiding Apertis in the bootstrapping chain seems recommendable. For
that simply pick the upstream debos images so we can keep using debos;
That still requires a kvm based worker, but so would previous stable
apertis releases at this point.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Frederic Danis authored 4 years ago and Martyn Welch committed 4 years ago

This allows to use flatdeb to build Mildenhall runtime and SDK, using kvm
or UML runners.

`ARG RELEASE` should be declared a second time after `FROM` to be
accessible by `RUN` commands,
cf. https://ryandaniels.ca/blog/docker-dockerfile-arg-from-arg-trouble/



Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Lightweight CI runners have been introduced to run jobs which have low
resource consumption and do not need privileged containers to reduce costs
and latency, so let's use them.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Always tag built images as `test-$COMMIT` so that we keep a bunch of
those around in case a rollback is needed, using a GitLab Container
Registry cleanup policy to keep storage under control.

https://docs.gitlab.com/ee/user/packages/container_registry/#cleanup-policy



We can then apply the `latest` tag to officially publish the built
images only after the tests have passed successfully.

This allows projects to set up a cleanup policy tailored to their needs
and to easily retain images that are needed for long term support, for
instance the ones used to on release builds.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Rather than manually escaping `$CI_COMMIT_BRANCH` as the tag name for
secondary branches, use `$CI_COMMIT_REF_SLUG`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The v0.20.0 image ships the fix for the missing platform metadata that
cause images generated with it to not work with `podman`:
https://github.com/GoogleContainerTools/kaniko/issues/997



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 4 years ago

Building the Docker images but not testing them can lead to issues, so
add a test job with a trivial debos recipe to at least smoke test the
image-builder Docker image.

As the Docker image tag which gets pushed is dynamically determined it
can't be defined in the main gitlab-ci.yml. To work around that generate
a child pipeline to trigger afterwards with the dynamic variables
substituted (dotenv artifacts are unfortunately not yet useful here,
as for 12.9 they only work for `environment:url:` keys). Note that the
trigger job uses the depend strategy to ensure it blocks and inherits
the state of the child pipeline, otherwise the main pipeline will
succeed even if the child fails.

In a perfect world the image builder docker image would only be tagged
after a successful test. But that's for later, for now this is useful in
merge requests.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 4 years ago

The build-base-rootfs job uses the apertis v2020 image builder docker
image which needs kvm to run, so tag the job as such.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 4 years ago

Turn the `calculate-release` snippet to a `before_script:` entry;
it's ran on every job anyway and avoids using a YAML anchor inside an
`extends:` entry. This will be helpful later as GitLab CI only supports
one level of expanding script arrays.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Directly use Apertis to build the reference images rather than Buster
and checking out the latest `debos` version from GitHub.

This gives us better control on what we ship, in particular once
we introduce snapshotting in the APT repositories and want to
rebuild old artifacts.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Instead of building all images individually, try and build Apertis-based
ones from the same common ground. First create a fairly basic Apertis
ospack with a few development tools in, then create a Docker image out of
it and layer things on top of it.

This unifies the previously separate overlays for the package builder and
the package source builder, only keeping the real differences in their
respective overlays.

We opt to not use Docker proper since it requires non-trivial
Docker-in-Docker setup. Makisu, which could be used as an alternative,
unfortunately, has buggy implementation of the ADD command, leaving us
the only option to use Kaniko.

The docker building step is templated in .gitlab-ci.yml to avoid
repetition. The Debian-based images don’t normally have to depend on the
rootfs building step, but to be able to run them asynchronously in the
same pipeline stage we need to declare the dependency anyway, as GitLab CI
does not currently support empty "needs" lists.

The branches named apertis/* build their respective Apertis releases,
everything else builds the default defined in the CI YAML file.

The jobs for apertis/* branches tag their build images as "latest", every
other job uses the branch name.

For robustness, the debos rootfs step uses a stable v2020 image builder
step instead of the currently built one.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>