If for any reason `git-buildpackage` fails to detect the orig tarball
extracted from the `pristine-lfs` branch it tries to generate it using
`pristine-tar` or `git archive`.

This is not desirable, we definitely prefer erroring out in case
something goes wrong as we always want to rely on `pristine-lfs` to
ensure that the orig tarballs are bit-identical to the upstream ones.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 5 years ago

The binfmt_misc module loading depends on kmod-static-nodes.service
running which depends on kmod, so install that.

On top of that, to mount the uml kernel modules on /lib/modules that
directory actually needs to exist so create it.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 5 years ago

To build debos images for non-x86 architectures qemu-user-static should
be installed, so install it.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 5 years ago

Building the Docker images but not testing them can lead to issues, so
add a test job with a trivial debos recipe to at least smoke test the
image-builder Docker image.

As the Docker image tag which gets pushed is dynamically determined it
can't be defined in the main gitlab-ci.yml. To work around that generate
a child pipeline to trigger afterwards with the dynamic variables
substituted (dotenv artifacts are unfortunately not yet useful here,
as for 12.9 they only work for `environment:url:` keys). Note that the
trigger job uses the depend strategy to ensure it blocks and inherits
the state of the child pipeline, otherwise the main pipeline will
succeed even if the child fails.

In a perfect world the image builder docker image would only be tagged
after a successful test. But that's for later, for now this is useful in
merge requests.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 5 years ago

The build-base-rootfs job uses the apertis v2020 image builder docker
image which needs kvm to run, so tag the job as such.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 5 years ago

Turn the `calculate-release` snippet to a `before_script:` entry;
it's ran on every job anyway and avoids using a YAML anchor inside an
`extends:` entry. This will be helpful later as GitLab CI only supports
one level of expanding script arrays.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 5 years ago

Setting HOME to /recipes can cause issues as /recipes is not guaranteed
to exist. In particular UML gets upset when HOME doesn't exist.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 5 years ago

When using UML ther is no need to install linux-image-amd64. However
udev is still needed inside the fakemachine for e.g. systemd-networkd to
work properly.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 5 years ago

Make things more clear by referring to the build containers by name not
by their sequence number.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Sjoerd Simons authored 5 years ago and Emanuele Aina committed 5 years ago

slirp-seq is the seq socket tool which should be used together with uml

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

Until everything is landed and packages, build from sources the SLIRP
helper from `libslirp-rs` and the UML fakemachine backend for Debos.

With these, the Docker container can be used to do unprivileged builds
in places where KVM is not available, like cloud CI workers or like
the Apertis SDK under VirtualBox.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Kaniko does not seem to mind, but real Docker complains if variables
used in the `FROM` clause are not explicitly marked with `ARG` clauses
in the `Dockerfile`:

    $ docker build --build-arg RELEASE=v2021dev2 -t tmp apertis-package-source-builder
    Sending build context to Docker daemon  73.22kB
    Step 1/9 : FROM $REGISTRY/$RELEASE-base:$TAG
    invalid reference format

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In our automated Gitlab CI workflow, when an MR is created, the
corresponding source package is uploaded to the :snapshots repository.

Later, when the MR is accepted, the corresponding package is directly
uploaded to the main repostiory.

This change ensures to clean-up the corersponding package from the
:snapshots repository upon acceptance of the MR.

https://phabricator.apertis.org/T6850



Apply suggestion to apertis-package-source-builder/overlay/usr/bin/ci-obs-upload

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Pass `--no-remove` to `apt` so it explicitly fails in case some resolver
weirdness cause it to remove packages we actually want.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For some reason `apt` is now uninstalling the `git-buildpackage` and
`devscripts` packages when trying to install the
`libconfig-model-dpkg-perl` package:

  $ apt install libconfig-model-dpkg-perl
  ...
  The following packages will be REMOVED:
    devscripts git-buildpackage
  The following NEW packages will be installed:
    fuse gnupg2 ...

To work around that, explicitly tell it that `gnupg2` can be installed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Avoid calling `apt update && apt install` too many times to make the
build process faster and avoid excessive layering in the generated
Docker images (when not using Kaniko's `--single-snapshot`, that is).

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Luis Araujo authored 5 years ago and Emanuele Aina committed 5 years ago

Instead of raising an exception, just skip a branch if it doesn't exist.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Directly use Apertis to build the reference images rather than Buster
and checking out the latest `debos` version from GitHub.

This gives us better control on what we ship, in particular once
we introduce snapshotting in the APT repositories and want to
rebuild old artifacts.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

With our workflow, where the CI has 2 steps:
1. deploy-snapshot
2. deploy-release

It becomes cumbersome and unnecessary to create SRs on OBS for each
package. During deploy-snapshot, we already test the fact if our package
has built successfully. So, during deploy-release, we want to upload the
package to the release repository.

We don't do cleanups of the package in the :snapshot repository, because
they get autocleaned there anyways.

Check for the $RELEASE variable to ascertain if it is a release build or
otherwise

Also, print the repository to which the package is uploaded to.

APERTIS: https://phabricator.apertis.org/T6658



Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Apply comments from Emanuele

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

First, ascertain it through debian/apertis/component
If not, ascertain through repository path
Finally, fallback to "development" if neither is available

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

With the previous logic, ensure_suite_branches would not create local
branches for security/pu even when they existed in the upstream repo.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Denis Pynkin authored 5 years ago and Andrej Shadura committed 5 years ago

Added package `imx-code-signing-tool` which is used for signing
U-Boot bootloader and Linux kernel for i.MX-based boards to enable
secure boot based on HAB (High Assurance Boot).
See how-to for details:
https://boundarydevices.com/high-assurance-boot-hab-dummies/



Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

E.g. security updates are available under debian-security, not debian.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Denis Pynkin authored 5 years ago and Emanuele Aina committed 5 years ago

Adding the DTC allows to prepare bootable kernel images for
ARM devices in FIT (Flattened uImage Tree) format.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Since commit 55f061b3 `get_local_version()` is no longer used.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Update `deb-git-version-gen` to properly handle non-native packages being
switched to native packaging by including
https://gitlab.collabora.com/smcv/deb-build-snapshot/commit/131f7f7c2568



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When a merge fails, it's nice to let developers use the same tools as used by
the CI to reproduce and fix the issue.

Unfortunately, `apertis-pkg-merge-updates` cleans up after itself a bit too
much, aborting the merge and thus preventing developer from fixing it.
Stop doing that so that developers can handle the conflicts, GitLab CI will
clean up after itself in any case.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

It seems http pipeline sometimes causes broken downloads; while this really
needs fixing in apt, work it around for now so CI doesn’t hit it.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Instead of building all images individually, try and build Apertis-based
ones from the same common ground. First create a fairly basic Apertis
ospack with a few development tools in, then create a Docker image out of
it and layer things on top of it.

This unifies the previously separate overlays for the package builder and
the package source builder, only keeping the real differences in their
respective overlays.

We opt to not use Docker proper since it requires non-trivial
Docker-in-Docker setup. Makisu, which could be used as an alternative,
unfortunately, has buggy implementation of the ADD command, leaving us
the only option to use Kaniko.

The docker building step is templated in .gitlab-ci.yml to avoid
repetition. The Debian-based images don’t normally have to depend on the
rootfs building step, but to be able to run them asynchronously in the
same pipeline stage we need to declare the dependency anyway, as GitLab CI
does not currently support empty "needs" lists.

The branches named apertis/* build their respective Apertis releases,
everything else builds the default defined in the CI YAML file.

The jobs for apertis/* branches tag their build images as "latest", every
other job uses the branch name.

For robustness, the debos rootfs step uses a stable v2020 image builder
step instead of the currently built one.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

The `tally-revision` script used to skip revision for which a package has
been built already does not really work on new repositories that do not
contain any tag.

In that scenario it fails like this:

  $ tally-revision
  + git rev-parse -q --verify HEAD
  + HEAD=2a2235e0f7182fcc619e755eb14de0983753c6cf
  + git describe --tags --match apertis/* --abbrev=0
  fatal: No names found, cannot describe anything.
  +TAGS=

That's because `set -e` causes the failing `git describe` invocation to exit
the script with an error, which is then interpreted as "skip this revision",
preventing the pipeline from running on new repositories with no tags.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Newer version of HotDoc like 0.10 break the current packages, so let's pull
here the 0.8 hotdoc branch packages before pointing all the affected packages
to them.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Also, only extract the latest tag from the repository

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Like the rest of the scripts, so that we have a better flow of the
commands being executed

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

It checks the revision from HEAD against all aperts/tags revisions
For details, see: https://phabricator.apertis.org/T6647



Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>