This is a special argument only useful for programs that need user
interaction.
The functionality side-steps a lot of sh's functionality which can cause
unstability in the output of the program.

See
https://amoffat.github.io/sh/sections/special_arguments.html#fg



Signed-off-by: Detlev Casanova <detlev.casanova@collabora.com>

Even though is is reopened for writing after it was for reading, let's
make things safer by closing it before reopening it

Signed-off-by: Detlev Casanova <detlev.casanova@collabora.com>

We may want to run the CI pipeline to pull updates from a debian release
backports upstream. Extend this script to handle those branches.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Documentation files are excluded from installations made to the Docker
images, with a few exceptions to this rule, including `copyright` files.
This extends it to also include `copyright_report` files, as they are
useful to have installed for the purpose of being able to examine
dependencies' reports while building packages.

As the files are all plain text and shorter than the corresponding
`copyright` file (as they omit the full license text), the effect on the
built image sizes should be negligible.

https://phabricator.apertis.org/T8554



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Signed-off-by: Apertis CI <devel@lists.apertis.org>

Any error during the merge attept currently would fail with:

    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 253, in <module>
        main()
      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 221, in main
        except ErrorReturnCode_2:
    NameError: name 'ErrorReturnCode_2' is not defined

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

If a subcommand exits due to an error when parsing its arguments, which
is traditionally reported with exit code 2, it is not appropriate to
consider it like a normal merge conflict, but we should arguably exit
with an error as well.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Report the failure in the exit code if we were unable to merge some
branches.

This also deals with other random errors that may occur and that at the
moment are silently ignored.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

These tools has been moved to `apertis-dev-tools` package, which is now
requested in `package-source-builder` container.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

The archive GPG key is used is when creating the base rootfs, derived
images flatdeb-builder already inherit it automatically, so drop the
useless copies.

There's a copy in image-builder as well, but it is used by the
test recipe.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Pick the appropriate archive keyring package using the specified
`osname` variable, otherwise the test recipe will fail for downstream
since the keyring and the mirror will not match.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Andre Moreira Magalhaes authored 3 years ago and Andre Moreira Magalhaes committed 3 years ago

Add a basic image for toolbox [1] containers.

This image uses the package-source-builder image as base and includes
extra tools required by toolbox.

[1] https://github.com/containers/toolbox



Signed-off-by: Andre Moreira Magalhaes <andre.magalhaes@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

When downstreams need to mirror our packaging repositories they need to
pull the latest but also preserve the downstream changes they
introduced.

For instance assume that a downstream has a `downstream/v2021` branch
that is based on `apertis/v2021-updates` (based on Buster) plus some
customization and needs to mirror `apertis/v2022dev3` (based on
Bullseye) while preserving the customization.

Naively doing a merge would hit conflicts since the
`apertis/v2021-updates` has diverged from the point where
`apertis/v2022dev3` has been branched, and the Buster updates would not
apply on the Bullseye-based branch.

The most sensible option is to rebase the changes: if there are no
downstream changes (the most common case) this becomes a no-op and the
process can proceed easily, otherwise it still has a good chance to be
able to apply simple customizations.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Frederic Danis committed 3 years ago

Avoid copying values and then deleting them by using two distinct lists.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Frederic Danis committed 3 years ago

Commit 86604333 "Ensure the target branch exist" broke the dry-run
mode as it unconditionally tried to create the downstream branches on
the remote repository using the GitLab API.

Tweak the code to honor the dry-run flag.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Frederic Danis committed 3 years ago

Rather than amending the merge commit to add the changelog entry, keep
it in a separate commit to make it easier to insert manual changes
using `git rebase`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since ci-package-builder!199


the CI pipeline is not using `osc-setup` anymore, so let's drop it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Some packages use SVG icons, without the corresponding plugin in the
flatdeb container this will cause build errors.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Walter Lozano authored 3 years ago and Emanuele Aina committed 3 years ago

In order to avoid issues with scan-copyrights not dealing with !!bool in
yaml files, switch to a simpler format.

This kind of issue was seen in packages that ship their own configuration
in d/fill.copyright.blanks.yml, and uses the skip (boolean) option,
which triggers an error an prevents the pipeline to succeed.

As the issue seems to be an upstream bug, as a workaround change the
default_style used by yaml dumper from | to the default one, which
does not add !!bool.

Previously the default_style | was needed as scan-copyright used
YAML::Tiny and it was not able to parse the files properly, however,
after switching to YAML::XS this doesn't seem to be an issue any more,
so default_style can be used with its default.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 3 years ago and Ariel D'Alessandro committed 3 years ago

To avoid messing the output between the stage that prints the changes in
the scan report with the error report flush both stdout and stderr.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Due to an upstream bug scan-copyrights reports license as empty in some
cases. Workaround this issue by treating empty license in the same way
unknown license are treated.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

As in some cases there are lot of license issues in a package, it is useful
to have the output sorted.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Try to be compatible tiwh Python 3.7 as shipped in Buster so the tool
can work on projects where the default branch points to v2021, either
because it's been dropped from v2022 or because it's a downstream that
has not started mirroring v2022 yet.

      File "/usr/bin/apertis-pkg-merge-upstream-to-downstreams", line 65, in ensure_downstream_branch
        project_id = urllib.parse.quote(url.path.strip("/").removesuffix(".git"), safe="")
    AttributeError: 'str' object has no attribute 'removesuffix'

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Do not enable `merge_when_pipeline_succeeds` when the update pipeline
failed to merge and a MR with the unmerged branch is created.

Otherwise the MR will unexpectedly be merged immediately as soon as the
pipeline succeeds when someone pushes the manually resolved merge.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Tell users what to do when automatic merges fail and they need to fix
conflicts manually by printing a summary like this at the end of the
merge-updates pipeline:

    

  Failed to automatically merge the updates below.

    Check out each proposed branch, manually merge it to the downstream_branch and push it to update the conflicting merge requests:

      Failed to merge debian/bullseye into apertis/v2022dev3 via proposed-updates/debian/bullseye/56f5c7b1:
      git checkout proposed-updates/debian/bullseye/56f5c7b1 && git merge apertis/v2022dev3 && git push origin proposed-updates/debian/bullseye/56f5c7b1

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Gitlab now support merge_request.unassign=
This fits in well with our workflow to have all MRs unassigned by
default.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

The {flatdeb,image,package-source}-builder docker images request gnupg
application and libraries to be installed. This requests to force
installation of `gpgv` to replace the `sequoia-gpgv` wrapper.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

When the merge fails, really create a MR where the proposed-updates
branch still points to the from the unmerged upstream commit.

This was the intention all the time, to make failed merge visible in the
list of open MRs and provide a place where the conflicts should be
merged to developers.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When pulling an update for a stable branch it should target a `-updates`
or `-security` branch. However there's no guarantee that they exist, so
the creaton of the merge may fail:

    remote:              WARNINGS: Error encountered with push options
    remote:       'merge_request.create' 'merge_request.remove_source_branch'
    remote:              'merge_request.target=apertis/v2021-security'
    remote:       'merge_request.title=Update from debian/buster-security for
    remote:     apertis/v2021-security': Branch apertis/v2021-security does not
    remote:                                  exist

To avoid that, ensure to push the target branch before creating the MR.

Unfortunately we can't simply use `git push` as the creation of
protected branches is restricted to the Web UI and the API, so let's use
the latter, even if it requires a bit of juggling.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

To ensure the correct ordering of the output, flush stdout before
launching suprocesses.

This prevents the output of `apertis-pkg-merge-upstream-to-downstreams`
from being printed much after the output of the suprocess when not
running on a tty, like when piped to `less` or when being run in the CI.
While it would be normally expected that the output of the merge follows
the `Attempt merging` message, in practice it would be displayed
immediately as it came from a suprocess, while the output of the program
would be buffered until much later.

Add some flushes to correctly handover the output buffering.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since we aren't importing the sh module wholly, we need to hand-import
all things we use from it.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Emanuele Aina authored 3 years ago and Martyn Welch committed 3 years ago

We can now drop the `debian/apertis/` cleanup hook since:

1. we no longer use `debian/apertis/gitlab-ci.yml` since we have
   switched to an external reference
2. `debian/apertis/` won't become empty since we always have at least a
   `component` file

This also makes `ci-buildpackage` work with non-Apertis packages which
were previously failing on the `rmdir` invocation no handling well the
lack of the `debian/apertis/` folder.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

By mistake, commit a51aa6de moved a print to stdout after flushing it.
This might produce the message `Using whitelists: ...` to be printed
after the rest of the script output.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>