Emanuele Aina authored 3 years ago and Martyn Welch committed 3 years ago

We can now drop the `debian/apertis/` cleanup hook since:

1. we no longer use `debian/apertis/gitlab-ci.yml` since we have
   switched to an external reference
2. `debian/apertis/` won't become empty since we always have at least a
   `component` file

This also makes `ci-buildpackage` work with non-Apertis packages which
were previously failing on the `rmdir` invocation no handling well the
lack of the `debian/apertis/` folder.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

By mistake, commit a51aa6de moved a print to stdout after flushing it.
This might produce the message `Using whitelists: ...` to be printed
after the rest of the script output.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Now that license information for all the files is kept, even for those
being whitelisted, filename should be printed in the case of blacklisted
or unknown license entries. Otherwise, there's no way to detect which
copyright entry is the offending one.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Files whitelisted in debian/apertis/copyright.whitelist are used to
instruct the scanner process not to raise an error which allows the
pipeline to succeed and merge the change. However, as a side effect, the
entries in debian/apertis/copyright with offending licenses are being
removed, dropping important information.

BOM generator for Apertis binary packages needs the whole information.
scan-copyright scanner must be called without a whitelist, so the
pipeline keeps license information for all the present files in the
source package.

This MR adapts ci-license-scan script to generate license information
for all the files, without failing on whitelisted offending ones.

Link: https://phabricator.apertis.org/T7878



Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Currently, gitpattern2re is used to convert gitignore patterns to the
Dpkg::Copyright::Scanner-compatible format. As noted in the code, that
requires to prepend ^\./, not just ^.

In order to manually match filename regex to those git patterns, that
prefix must be optional, as the file path from the copyright entry might
not have it.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

v2022dev2 should be self-consistent enough to no longer need the
bullseye repositories that were temporarily added.

This reverts commit c5a817a9.

When pulling from Apertis to downstreams we need to also check branches
like `apertis/v2021-updates` when looking for updates to import.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We moved from `coX` to `apertisX` as the packaging suffix but
unfortunately it turned out to interact badly with the build suffix on
OBS in some scenarios:

1. `bluez` in Debian has the `5.55-3` version
2. it gets uploaded to OBS with no changes, forgetting to add the
   `apertisX` suffix
3. OBS appends the build suffix, generating binaries with version
   `5.55-3bv2022dev2b1`
4. we update `bluez` appending the `apertisX` suffix, yielding
   `5.55-3apertis1`
5. OBS appends the build suffix, generating binaries with version
   `5.55-3apertis1bv2022dev2b1`
6. reprepro gets the new binaries from OBS but the ordering is now wrong
   since `5.55-3apertis1bv2022dev2b1` < `5.55-3bv2022dev2b1` and will
   refuse to publish the newly built binaries

Switching to `+apertisX` has the following advantages:

* it can be rolled out progressively using the standard workflow
* no need to change the build suffix
  * no need to rebuild everything on OBS
  * no need to manually delete entries from reprepro
* backports work correctly, `+apertisX` sorts newer than `coX` and
  the OBS suffixes
* mixing releases like v2021 and v2022 should not produce surprising
  effects like v2021 builds sorting newer than v2022

The drawbacks are:
* every non-Apertis-specific package needs to be tweaked
* if Debian package maintainer releases a new version by appending a
  suffix there are higher chances that the `+apertisX` suffix will sort
  newer (we already have this issue, but this increases the chances to
  hit it)

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Do not hardcode `apertis` for the distribution of the changelog entries
generated when pulling updates, let downstreams customize it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit 4d11675a "pkg-pull-updates: Avoid oversized commit messages"
resolved to amending the commit generated by `gbp import-dsc` to trim
the overly long commit messages since they where causing issues with
GitLab CI/CD.

However, `gbp import-dsc` also creates a git tag for the generated
commit and after the amend the tag was left dangling, pointing to the
original commit rather than the amended one.

Invoke `gbp tag --retag` to update the tag as well, so it get pushed
again with the branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add an overlay for package-source-builder

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

The package-builder image dates back to when we only built a small
subset of packages on Jenkins using a pre-built container with all the
required build dependencies.

With the move to the pkg/ci-package-builder> pipeline on GitLab this has
been no longer needed for quite some time, so it's now a good time to
drop the remnants of the already disabled package-builder image.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This reverts commit d285422a
"Enable only the package-source-builder docker image", with the
exception of the obsolete package-builder image.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When pushing new commits to a MR set to be merged automatically with the
`merge_request.merge_when_pipeline_succeeds` push option GitLab aborts
the automated merge.

This meant that our automerging never worked because we first pushed the
unmerged commit to create the MR and when the local merge attempt
succeeded we pushed the result to the same MR, causing GitLab to abort
the automerging.

To make it work, ensure we only push once to create the MR, either with
the result of the local merge or the unmerged commit if the local
attempt failed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Now that the bulk of the packages have been made available in the
Bullseye based v2022dev2 we can revert
commit 788aa27a.

Avoid fast forward merges, which may happen in specific corner cases,
like a downstream mirroring a repository and making their
`downstream/v2021` branch initially point to `apertis/v2021`: in that
case a pull from `apertis/v2021` would produce a fast-forward and we end
up amending the HEAD of `apertis/v2021` to append the downstream suffix,
rather than amending the merge commit.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We may be merging from Apertis when being run on a dowstream
infrastructure.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add a command line flag to avoid pushing commits, to make local testing easier.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Compare the upstream branch to what we actually propose to be merged
when checking if it can be merged automatically.

The current code compares it to the downstream branch which is not
updated yet, so merge_when_pipeline_succeeds is never actually set as
the diff will always include more files than expected.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In case if file `debian/copyright` is not existing, the `ci-license-scan`
script throws an exception. Do not stop the check if the file is absent.
Example package without the `debian/copyright` file is
`firmware-nonfree`.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Currently the code pulling updates from upstream assumes that downstream
branches ending with `dev[0-9]` or `pre` are not stable so updates can
be directly landed there, while other downstream branches (for instance,
`apertis/v2021`) are stable and updates should go through the updates or
security repositories.

However, there's a time right before the first stable release is
published that the "stable" branch is not stable yet. For instance,
before the release of `v2021.0` the `apertis/v2021` should not have been
considered stable and updates should have been landed directly there.

This is even more important for downstreams so that they can pull
updates to their not-released-yet v2021 branch.

To do so, add a `--stable` parameter that accepts the list of branches
to be considered "stable".

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit 788aa27a "Debian commits" was introduced to base images
directly on Debian while the Bullseye import in v2022dev2 is not
complete enought to real Apertis build images.

Unfortunately it broke the ability to tag images as `latest`.

Rework the hack so tagging works again.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Ariel D'Alessandro authored 3 years ago and Emanuele Aina committed 3 years ago

Currently `ci-license-scan` fails if it can't detect the license of some
files, even if are no licensing restrictions in the component that hosts
the package.

This is annoying because it causes the scan to block pipelines where
there's little value in adding the missing licensing information: for
instance, while it is important on target, it is not particularly
relevant everywhere else.

To avoid that, add a flag to define where failing to detect a license is
problematic, otherwise make the command succeed and simply pass through
the `UNKNOWN` entries in the licensing report.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use of `git restore` has been introduced in commit e954d2be
"ci-buildpackage: use wrap-and-sort as a workaround for odd packages".

It works on the Bullseye-based v2022dev2, but on older Buster-based
release `git restore` is not available.

Replace it with the old-style `git checkout`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Make interpolation work in a printed message by turning a plain string
to a proper f-string.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This reverts commit b26ad3fa "Explicitly set TAG to latest"
since it is causing every image that gets built, even the ones on
secondary wip branches, to be tagged as `latest` and thus affecting
production everywhere.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Andrej Shadura authored 4 years ago and Martyn Welch committed 4 years ago

This can happen if the source package has been renamed while still
generating a binary package with the same name. For example, Ruby
bundler is no longer a separate source package, but is shipped in
the rubygems package. Madison returns this for bundler:

---
bundler:
  2.2.5-2:
    bullseye:
      - all

The "source" pseudoarchitecture is missing from the list since it’s not
a source package in bullseye anymore.

Apertis: T7811

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Under early testing of ci-license-scan with docker image apertis/v2022dev2
the copyright.new file generated was missformated, putting the header Format
in the last line instead of the first one.

In order to fix the issue, force line buffering when the file is generated.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

If `--auto-merge` is not set it results in:

    Traceback (most recent call last):
      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 146, in <module>
        main()
      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 92, in main
        auto_merge = args.auto_merge.split(":")
    AttributeError: 'NoneType' object has no attribute 'split'

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `--auto-merge` flag was not meant to be mandatory.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Avoid the underscore in `--auto_merge` and use `--auto-merge` instead.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In order to avoid manual work, enable merge when pipeline succeeds for
trivial changes, such those on d/apertis/* or d/changelog if this feature
is enabled via command line.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Arnaud Ferraris authored 4 years ago and Martyn Welch committed 4 years ago

Building packages into `_build` causes issues with ocaml packages: those
too build into this directory by default, which is therefore removed
during the `clean` stage.

This has been observed for ounit, pcre-ocaml and ocaml-result, and
probably affects all ocaml packages.

Using a less common name for the build directory solves the issue
without having to patch individual packages.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

A lot of `node-*` packages have their Build-Depends formatted in a way
apt can't parse, causing CI failures.

Using wrap-and-sort beforehand in order to make sure the file format is
correct will help working around these issues.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

This reverts commit cf5f4c2d.

Building packages into `_build` causes issues with ocaml packages: those
too build into this directory by default, which is therefore removed
during the `clean` stage.

This has been observed for ounit, pcre-ocaml and ocaml-result, and
probably affects all ocaml packages.

Using a less common name for the build directory solves the issue
without having to patch individual packages.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

`gbp import-dsc` puts all the new changelog entries in the commit
message, generating big walls of text when, for instance, importing the
version from Bullseye on top of the Buster one.

GitLab then puts the whole log message in the `CI_COMMIT_MESSAGE` env
var, which is passed on the docker command line, resulting in a error:

    standard_init_linux.go:219: exec user process caused: argument list too long

See https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26624#note_529234097



To avoid that, trim the message to only keep the first line.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Create a MR even when the local attempt to merge failed. In that case
let the proposed updates branch point to the upstream branch and create
a MR to submit it to the downstream branch.

This makes the failed updates more visbile and easier to track than the
failed pipelines.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>