Emanuele Aina authored 2 years ago and Ritesh Raj Sarraf committed 2 years ago

The VM type used by the default GitLab runners has been changed to one
that provides nested vistualization support and the runner configuration
has been tweaked to expose it to containers.

The VM type used by the default GitLab runners has been changed to one
that provides nested vistualization support and the runner configuration
has been tweaked to expose it to containers.

Now both the KVM and UML Debos backends can be used on the runners, and
Debos will prefer the KVM as it yields a nice performance speedup.

This means that the test checking the availability of the UML backend
fails since the KVM is picked up automatically. Explicitly setting the
desired backend avoids the reliance on autodetection.

By not forcing the backend we can check that the Debos autodetection now
picks up KVM since it is faster.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This also allows us to easily get the changes in v2021-updates.

https://phabricator.apertis.org/T8507



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

As in some cases there are lot of license issues in a package, it is useful
to have the output sorted.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Try to be compatible tiwh Python 3.7 as shipped in Buster so the tool
can work on projects where the default branch points to v2021, either
because it's been dropped from v2022 or because it's a downstream that
has not started mirroring v2022 yet.

      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 65, in ensure_downstream_branch
        project_id = urllib.parse.quote(url.path.strip("/").removesuffix(".git"), safe="")
    AttributeError: 'str' object has no attribute 'removesuffix'

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Do not enable `merge_when_pipeline_succeeds` when the update pipeline
failed to merge and a MR with the unmerged branch is created.

Otherwise the MR will unexpectedly be merged immediately as soon as the
pipeline succeeds when someone pushes the manually resolved merge.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Tell users what to do when automatic merges fail and they need to fix
conflicts manually by printing a summary like this at the end of the
merge-updates pipeline:

    

  Failed to automatically merge the updates below.

    Check out each proposed branch, manually merge it to the downstream_branch and push it to update the conflicting merge requests:

      Failed to merge debian/bullseye into apertis/v2022dev3 via proposed-updates/debian/bullseye/56f5c7b1:
      git checkout proposed-updates/debian/bullseye/56f5c7b1 && git merge apertis/v2022dev3 && git push origin proposed-updates/debian/bullseye/56f5c7b1

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Ritesh Raj Sarraf authored 3 years ago and Emanuele Aina committed 3 years ago

Gitlab now support merge_request.unassign=
This fits in well with our workflow to have all MRs unassigned by
default.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

When the merge fails, really create a MR where the proposed-updates
branch still points to the from the unmerged upstream commit.

This was the intention all the time, to make failed merge visible in the
list of open MRs and provide a place where the conflicts should be
merged to developers.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When pulling an update for a stable branch it should target a `-updates`
or `-security` branch. However there's no guarantee that they exist, so
the creaton of the merge may fail:

    remote:              WARNINGS: Error encountered with push options
    remote:       'merge_request.create' 'merge_request.remove_source_branch'
    remote:              'merge_request.target=apertis/v2021-security'
    remote:       'merge_request.title=Update from debian/buster-security for
    remote:     apertis/v2021-security': Branch apertis/v2021-security does not
    remote:                                  exist

To avoid that, ensure to push the target branch before creating the MR.

Unfortunately we can't simply use `git push` as the creation of
protected branches is restricted to the Web UI and the API, so let's use
the latter, even if it requires a bit of juggling.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

To ensure the correct ordering of the output, flush stdout before
launching suprocesses.

This prevents the output of `apertis-pkg-merge-upstream-to-downstreams`
from being printed much after the output of the suprocess when not
running on a tty, like when piped to `less` or when being run in the CI.
While it would be normally expected that the output of the merge follows
the `Attempt merging` message, in practice it would be displayed
immediately as it came from a suprocess, while the output of the program
would be buffered until much later.

Add some flushes to correctly handover the output buffering.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Ariel D'Alessandro authored 3 years ago and Emanuele Aina committed 3 years ago

By mistake, commit a51aa6de moved a print to stdout after flushing it.
This might produce the message `Using whitelists: ...` to be printed
after the rest of the script output.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Ariel D'Alessandro authored 3 years ago and Emanuele Aina committed 3 years ago

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Ariel D'Alessandro authored 3 years ago and Emanuele Aina committed 3 years ago

Now that license information for all the files is kept, even for those
being whitelisted, filename should be printed in the case of blacklisted
or unknown license entries. Otherwise, there's no way to detect which
copyright entry is the offending one.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Ariel D'Alessandro authored 3 years ago and Emanuele Aina committed 3 years ago

Files whitelisted in debian/apertis/copyright.whitelist are used to
instruct the scanner process not to raise an error which allows the
pipeline to succeed and merge the change. However, as a side effect, the
entries in debian/apertis/copyright with offending licenses are being
removed, dropping important information.

BOM generator for Apertis binary packages needs the whole information.
scan-copyright scanner must be called without a whitelist, so the
pipeline keeps license information for all the present files in the
source package.

This MR adapts ci-license-scan script to generate license information
for all the files, without failing on whitelisted offending ones.

Link: https://phabricator.apertis.org/T7878



Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Ariel D'Alessandro authored 3 years ago and Emanuele Aina committed 3 years ago

Currently, gitpattern2re is used to convert gitignore patterns to the
Dpkg::Copyright::Scanner-compatible format. As noted in the code, that
requires to prepend ^\./, not just ^.

In order to manually match filename regex to those git patterns, that
prefix must be optional, as the file path from the copyright entry might
not have it.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Since we aren't importing the sh module wholly, we need to hand-import
all things we use from it.

Apply 1 suggestion(s) to 1 file(s)

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

When pulling from Apertis to downstreams we need to also check branches
like `apertis/v2021-updates` when looking for updates to import.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We moved from `coX` to `apertisX` as the packaging suffix but
unfortunately it turned out to interact badly with the build suffix on
OBS in some scenarios:

1. `bluez` in Debian has the `5.55-3` version
2. it gets uploaded to OBS with no changes, forgetting to add the
   `apertisX` suffix
3. OBS appends the build suffix, generating binaries with version
   `5.55-3bv2022dev2b1`
4. we update `bluez` appending the `apertisX` suffix, yielding
   `5.55-3apertis1`
5. OBS appends the build suffix, generating binaries with version
   `5.55-3apertis1bv2022dev2b1`
6. reprepro gets the new binaries from OBS but the ordering is now wrong
   since `5.55-3apertis1bv2022dev2b1` < `5.55-3bv2022dev2b1` and will
   refuse to publish the newly built binaries

Switching to `+apertisX` has the following advantages:

* it can be rolled out progressively using the standard workflow
* no need to change the build suffix
  * no need to rebuild everything on OBS
  * no need to manually delete entries from reprepro
* backports work correctly, `+apertisX` sorts newer than `coX` and
  the OBS suffixes
* mixing releases like v2021 and v2022 should not produce surprising
  effects like v2021 builds sorting newer than v2022

The drawbacks are:
* every non-Apertis-specific package needs to be tweaked
* if Debian package maintainer releases a new version by appending a
  suffix there are higher chances that the `+apertisX` suffix will sort
  newer (we already have this issue, but this increases the chances to
  hit it)

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Do not hardcode `apertis` for the distribution of the changelog entries
generated when pulling updates, let downstreams customize it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit 4d11675a "pkg-pull-updates: Avoid oversized commit messages"
resolved to amending the commit generated by `gbp import-dsc` to trim
the overly long commit messages since they where causing issues with
GitLab CI/CD.

However, `gbp import-dsc` also creates a git tag for the generated
commit and after the amend the tag was left dangling, pointing to the
original commit rather than the amended one.

Invoke `gbp tag --retag` to update the tag as well, so it get pushed
again with the branch.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When pushing new commits to a MR set to be merged automatically with the
`merge_request.merge_when_pipeline_succeeds` push option GitLab aborts
the automated merge.

This meant that our automerging never worked because we first pushed the
unmerged commit to create the MR and when the local merge attempt
succeeded we pushed the result to the same MR, causing GitLab to abort
the automerging.

To make it work, ensure we only push once to create the MR, either with
the result of the local merge or the unmerged commit if the local
attempt failed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Avoid fast forward merges, which may happen in specific corner cases,
like a downstream mirroring a repository and making their
`downstream/v2021` branch initially point to `apertis/v2021`: in that
case a pull from `apertis/v2021` would produce a fast-forward and we end
up amending the HEAD of `apertis/v2021` to append the downstream suffix,
rather than amending the merge commit.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

We may be merging from Apertis when being run on a dowstream
infrastructure.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add a command line flag to avoid pushing commits, to make local testing easier.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Compare the upstream branch to what we actually propose to be merged
when checking if it can be merged automatically.

The current code compares it to the downstream branch which is not
updated yet, so merge_when_pipeline_succeeds is never actually set as
the diff will always include more files than expected.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Denis Pynkin authored 3 years ago and Emanuele Aina committed 3 years ago

In case if file `debian/copyright` is not existing, the `ci-license-scan`
script throws an exception. Do not stop the check if the file is absent.
Example package without the `debian/copyright` file is
`firmware-nonfree`.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Currently the code pulling updates from upstream assumes that downstream
branches ending with `dev[0-9]` or `pre` are not stable so updates can
be directly landed there, while other downstream branches (for instance,
`apertis/v2021`) are stable and updates should go through the updates or
security repositories.

However, there's a time right before the first stable release is
published that the "stable" branch is not stable yet. For instance,
before the release of `v2021.0` the `apertis/v2021` should not have been
considered stable and updates should have been landed directly there.

This is even more important for downstreams so that they can pull
updates to their not-released-yet v2021 branch.

To do so, add a `--stable` parameter that accepts the list of branches
to be considered "stable".

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Ariel D'Alessandro authored 3 years ago and Emanuele Aina committed 3 years ago

Currently `ci-license-scan` fails if it can't detect the license of some
files, even if are no licensing restrictions in the component that hosts
the package.

This is annoying because it causes the scan to block pipelines where
there's little value in adding the missing licensing information: for
instance, while it is important on target, it is not particularly
relevant everywhere else.

To avoid that, add a flag to define where failing to detect a license is
problematic, otherwise make the command succeed and simply pass through
the `UNKNOWN` entries in the licensing report.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use of `git restore` has been introduced in commit e954d2be
"ci-buildpackage: use wrap-and-sort as a workaround for odd packages".

It works on the Bullseye-based v2022dev2, but on older Buster-based
release `git restore` is not available.

Replace it with the old-style `git checkout`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Make interpolation work in a printed message by turning a plain string
to a proper f-string.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Andrej Shadura authored 4 years ago and Emanuele Aina committed 4 years ago

This can happen if the source package has been renamed while still
generating a binary package with the same name. For example, Ruby
bundler is no longer a separate source package, but is shipped in
the rubygems package. Madison returns this for bundler:

---
bundler:
  2.2.5-2:
    bullseye:
      - all

The "source" pseudoarchitecture is missing from the list since it’s not
a source package in bullseye anymore.

Apertis: T7811

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

Under early testing of ci-license-scan with docker image apertis/v2022dev2
the copyright.new file generated was missformated, putting the header Format
in the last line instead of the first one.

In order to fix the issue, force line buffering when the file is generated.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

If `--auto-merge` is not set it results in:

    Traceback (most recent call last):
      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 146, in <module>
        main()
      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 92, in main
        auto_merge = args.auto_merge.split(":")
    AttributeError: 'NoneType' object has no attribute 'split'

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `--auto-merge` flag was not meant to be mandatory.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Avoid the underscore in `--auto_merge` and use `--auto-merge` instead.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

In order to avoid manual work, enable merge when pipeline succeeds for
trivial changes, such those on d/apertis/* or d/changelog if this feature
is enabled via command line.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Arnaud Ferraris authored 4 years ago and Emanuele Aina committed 4 years ago

Building packages into `_build` causes issues with ocaml packages: those
too build into this directory by default, which is therefore removed
during the `clean` stage.

This has been observed for ounit, pcre-ocaml and ocaml-result, and
probably affects all ocaml packages.

Using a less common name for the build directory solves the issue
without having to patch individual packages.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Arnaud Ferraris authored 4 years ago and Emanuele Aina committed 4 years ago

A lot of `node-*` packages have their Build-Depends formatted in a way
apt can't parse, causing CI failures.

Using wrap-and-sort beforehand in order to make sure the file format is
correct will help working around these issues.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

`gbp import-dsc` puts all the new changelog entries in the commit
message, generating big walls of text when, for instance, importing the
version from Bullseye on top of the Buster one.

GitLab then puts the whole log message in the `CI_COMMIT_MESSAGE` env
var, which is passed on the docker command line, resulting in a error:

    standard_init_linux.go:219: exec user process caused: argument list too long

See https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26624#note_529234097



To avoid that, trim the message to only keep the first line.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>