"build" is a rather common name which has the potential for file name
conflicts, as is an example below.

Processing triggers for libc-bin (2.28-7co1ba1) ...
Processing triggers for man-db (2.8.5-2bv2020dev0b1) ...
+ dpkg-checkbuilddeps
+ dpkg-parsechangelog -S Version
+ dversion=3.2.26+dfsg-3
+ checkout-tarball build
Traceback (most recent call last):
  File "/usr/bin/checkout-tarball", line 49, in <module>
    main()
  File "/usr/bin/checkout-tarball", line 25, in main
    os.makedirs(dest, exist_ok=True)
  File "/usr/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: 'build'
ERROR: Job failed: exit code 1

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

We keep getting hard-to-debug hash mismatches when downloading packages with
APT, like this one in the https://gitlab.apertis.org/pkg/target/ostree/-/jobs/189090
GitLab pipeline:

    Err:94 https://repositories.apertis.org/apertis v2020dev0/target amd64 libsystemd-dev amd64 241-5co3bv2020dev0b1
      Hash Sum mismatch
      Hashes of expected file:
       - SHA256:41d1ccfd064d8e7aafd19d031f0f35e350279dec3e46c2d6323b9a70159a5bd4
       - SHA1:161fff18c277d0b8bda8a80be7a433e8807aca13 [weak]
       - MD5Sum:2ae73e29e95daa715d07ac2b1ca78412 [weak]
       - Filesize:320672 [weak]
      Hashes of received file:
       - SHA256:bba2cd8458b872069ec6697f2ec23ee47ab5e285d6627f96b6d44914d61c0f14
       - SHA1:384222001e9c4a63552e0017da7aadee5894344b [weak]
       - MD5Sum:63b6130dcff030abf9c75a91074b6731 [weak]
       - Filesize:320672 [weak]
      Last modification reported: Thu, 15 Aug 2019 18:46:42 +0000
    Get:95 https://repositories.apertis.org/apertis v2020dev0/target amd64 libyaml-0-2 amd64 0.2.1-1ba1 [49.2 kB]
    Get:96 https://repositories.apertis.org/apertis v2020dev0/development amd64 python3-yaml amd64 3.13-2bv2020dev0b1 [121 kB]
    E: Failed to fetch https://repositories.apertis.org/apertis/pool/target/s/systemd/libsystemd-dev_241-5co3bv2020dev0b1_amd64.deb  Hash Sum mismatch
       Hashes of expected file:
        - SHA256:41d1ccfd064d8e7aafd19d031f0f35e350279dec3e46c2d6323b9a70159a5bd4
        - SHA1:161fff18c277d0b8bda8a80be7a433e8807aca13 [weak]
        - MD5Sum:2ae73e29e95daa715d07ac2b1ca78412 [weak]
        - Filesize:320672 [weak]
       Hashes of received file:
        - SHA256:bba2cd8458b872069ec6697f2ec23ee47ab5e285d6627f96b6d44914d61c0f14
        - SHA1:384222001e9c4a63552e0017da7aadee5894344b [weak]
        - MD5Sum:63b6130dcff030abf9c75a91074b6731 [weak]
        - Filesize:320672 [weak]
       Last modification reported: Thu, 15 Aug 2019 18:46:42 +0000

It also happens on Jenkins and has been intermittently reproduced on developer machines.

Only APT seems to be affected, so enable as much debugging we can to try to catch it.

See https://phabricator.collabora.com/T15071

 for more information.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

walbottle-tools and libbtrfs-dev
These are needed as build dependency packages.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

The eclipse related build dependency packages are no more available in
the new releases after the rebase to Debian Buster. We did so to avoid
the large set of Java related dependencies that was pulled in by those
packages.

Thus reverting this change which added those non-existent packages

This reverts commit 8a7e0b61.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Extend the script replacing the GPLv2 forks of `tar` and `coreutils` to
also replace the newly introduced `grep-gplv2`, `diffutils-gplv2` and
`findutils-gplv2`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

For testing it was quite useful to silently ignore the case where the
obs project doesn't exist; but now things are going more into production
that should no longer be acceptable.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

for pristine-lfs saved sources we only saves the dsc not the changes
file, so switch to dputting that; That will cause it to not have such a
nice osc revision  log, but that's not so important for now.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

python sh eats up stdout by default (and returns it). For various
complex commands, where we don't use the stdout in the code it would be
better to have their output on the console for diagnostics if things go
wrong.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

To check if the merge job did any changes compare the potential proposed
branch with the merge target (aka downstream) not upstream. Also be
quiet as the diff output might be huge (and we don't need it)

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

The current CI pipeline has one job to merge from one downstream to one
upstream (e.g. debian/buster to apertis/v2020dev0).  While that's
reasonable it's not unlikely quite a few package will be on the same
versions between our different distribution release based on the same
upstream which causes a lot of redundant work as well as merge request
for the same content but different git history.

To solve that add a new helper tool which takes an upstream branch and a
list of downstream branches, keeps track of what git reference each is
at and only generates new actual merges for that are different. If two
or more downstreams have the same state then only one merge is done and
multiple merge requests are sent.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Sjoerd Simons authored 5 years ago and Andrej Shadura committed 5 years ago

If a merge fails bail with an error code such that the CI stops.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Sjoerd Simons authored 5 years ago and Andrej Shadura committed 5 years ago

-m for dpkg-mergechangelogs is quite useful when merge pre-release
branches for Debian but it's not when merging upstream into downstream
repositories. Actually it can cause merge failures if it can't work out
the prereleases to merge.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

The logic got turned up-side down, when ${RELEASE} is so a *non-zero*
string the upload should be done... Make the code reflect that

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Compare the version of the downstream and upstream branches before
trying to merge, such that a merge can be skipped if upstream is newer.

While there also change the argument names to branch names rather then
suites to make the tool a bit more flexible (in case upstream isn't
debian and downstream isn't apertis.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

When populating the build dir with a tarball gbp will create a symlink
(if both are different). While this works fine for the build it does
mean things get confused when just the build dir is stored as an
artifact.

To fix simply make the tarball and build dirs the same as there is no real
benefit in keeping them seperate.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

When the Apertis version and Debian version don't match then on the
apertis side there should be a changelog entry summarizing the
(remaining) changes to the package.

Pre-add a changelog entry to the merge commit to remind the reviewer
that this should be added.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Don't fail if the .git/info directory already exists but the attributes
haven't been set. Also print the exception if failing to setup
attributes for easier debugging of issues.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

ci-buildpackage has two main jobs; it builds the source package and
uploads it to OBS, while also submitting a request iff the build was for
a tag.

This ends up complicating the script as well as taking all policy away
from the gitlab-ci job. Preventing e.g. just running the build for
testing purposes without uploading.

This split ci-buildpackage in a simpler wrapper script around
git-buildpackage which sets up the build environment etc and a new
ci-obs-upload script to dput into obs; Both scripts look at the RELEASE
environment variable for release vs snapshot mode. In snapshot mode
ci-buildpackage adds a deb-git-version-gen based version, while
ci-obs-upload in release mode will also run submitrequest

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

As we have both debian/ tags and apertis/ tags make sure
deb-git-version-gen only considers local hooks.

Patch from:
  https://gitlab.collabora.com/smcv/deb-build-snapshot/merge_requests/2



Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

This will create meaningful (and increasing) version numbers for
building snapshots.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Include deb-git-version-gen from
https://gitlab.collabora.com/smcv/deb-build-snapshot.git

 version
v0.20190704.0. This allows the creationg of meaningful version for git
snapshots of packages.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Make sure we don’t import the same tarball over and over

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Those functions work with packaging versions, not upstream versions

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>