Newer
Older
lastmod = "2021-04-29"
title = "Adding and Updating Components"
Apertis stores the source of all the shipped packages in GitLab and uses a
[GitLab CI pipeline](https://gitlab.apertis.org/infrastructure/ci-package-builder/-/blob/master/ci-package-builder.yml)
to manage the workflows and automate many of the tasks that are required to
release a package:
* Pull updates from upstream distributions like Debian 10 Buster.
* Test any changes made to development and release branches.
* Push updated sources for release branches that pass testing to OBS. From here,
fresh binary packages will either be built and added to the relevant Debian
repository or snapshot repository, depending on the release state of the
component.
## License scans
As contributors submit merge requests to packaged software, the CI pipeline performs
license scans on those packages, scanning all files in the package,
not just the new submission. The pipeline fails or emits a warning (depending on
the configuration) when it finds files with unknown or unclear licensing terms,
or files under licenses not allowed in the package. When such situation arises,
it is the responsibility of the submitter to perform the review of the license
scan results and make updates to the package if necessary.
When the license scan mistakenly identifies a file as being under an incorrect
license, or fails to process it correctly, there are three ways to fix this:
1. Specify the correct copyright and the license in `debian/apertis/copyright.yml`.
The format of the file is specified in the
[Dpkg::Copyright::Scanner](https://manpages.debian.org/buster/libconfig-model-dpkg-perl/Dpkg::Copyright::Scanner.3pm.en.html)
manpage.
In short, it’s a YAML file mapping paths to their licensing information:
debian:
copyright: 2015, Marcel
license: Expat
src/:
copyright: 2016, Joe
license: Expat
.*/NOTICE:
skip: 1
src/garbled/:
'override-copyright': 2016 Marcel MeXzigue
File patterns follow the Perl regular expression rules, and are matched
from the beginning of a path. Patterns are used in the order from the most
specific to the least specific ones.
Please also verify `debian/copyright` specifies the correct license, and if it
doesn’t, submit a patch to Debian.
2. Add the file to the list of ignored files.
`debian/apertis/copyright.whitelist` is formatted the same way as `gitignore`,
please refer to the [gitignore](https://manpages.debian.org/buster/git-man/gitignore.5.en.html)
manpage for more information.
3. If the file is under a license not suitable for Apertis, it can be removed from
the package by either repackaging the tarball or patching it out, in which case
the scanner will not take it into account.
The license scanner will store the automatically generated copyright report file
under `debian/apertis/copyright`, updating the merge request when necessary.
[Dpkg::Copyright::Scanner]: https://manpages.debian.org/testing/libconfig-model-dpkg-perl/Dpkg::Copyright::Scanner.3pm.en.html
[gitignore]: https://manpages.debian.org/testing/git-man/gitignore.5.en.html
## Custom pipelines
When using the packaging pipeline, developers cannot put their CI/CD automation
in `.gitlab-ci.yml` anymore, as the CI config path points to the
ci-package-builder definition.
However, developers can put their jobs in the
`debian/apertis/local-gitlab-ci.yml` file and have them executed in a child
pipeline whenever the main packaging pipeline is executed. This is especially
handy to run tests before the actual packaging process begins.
{{% notice tip %}}
The instructions below assume an Apertis development enviroment. Either boot
the [Apertis SDK]( {{< ref "virtualbox.md" >}} ) or run the
Apertis `package-source-builder` Docker container:
APERTIS_RELEASE=v2021
docker pull registry.gitlab.apertis.org/infrastructure/apertis-docker-images/${APERTIS_RELEASE}-package-source-builder
docker run -it --rm --env HOME -w "$(pwd)" -v "$HOME:$HOME" -v "$HOME:/root" --security-opt label=disable registry.gitlab.apertis.org/infrastructure/apertis-docker-images/${APERTIS_RELEASE}-package-source-builder bash
{{% /notice %}}
# Adding new components
The software components used in Apertis images are packaged in the
[packaging format](https://wiki.debian.org/Packaging) used by Debian.
## Adding new packages from Scratch
To package a component from scratch, Debian provides [a short guide to get
started](https://wiki.debian.org/Packaging/Intro).
{{% notice tip %}}
The VirtualBox-based
[Apertis SDK virtual machine images]( {{< ref "virtualbox.md" >}} ) ship with
all the needed tools installed, providing a reliable, self-contained
environment ready to be used.
{{% /notice %}}
Once the component has been packaged, its sources can be uploaded to GitLab in
a personal project, such that the developer is free to experiment and iterate
until the component is ready to be submitted to the appropriate GitLab project.
## Adding new packages from Debian
This is the process to import a new package from Debian to Apertis:
* create a folder, in the name of the package to import.
* Eg. for package `hello`, run: `mkdir hello`
* chdir to created folder: `cd hello`
* invoke `import-debian-package` from the [packaging-tools
repository:](https://gitlab.apertis.org/infrastructure/packaging-tools/)
* fetch a specific version: `import-debian-package --upstream buster --downstream apertis/v2020dev0 --component target --package hello --version 2.10-2`
* fetch the latest version: `import-debian-package --upstream buster --downstream apertis/v2020dev0 --component target --package hello`
* the argument to `--component` reflects the repository component it is part of (for instance, `target`); it will be stored in `debian/apertis/component`
* multiple downstream branches can be specified, in which case all of them
will be updated to point to the newly imported package version
* the Apertis version of the package will have a local suffix (`apertis0`) appended
* don't use `import-debian-package` on existing repositories, it does not
attempt to merge `apertis/*` branches and instead it re-sets them to new
branches based on the freshly imported Debian package
* create an empty project on GitLab under the `pkg/*` namespaces (for instance, `pkg/hello`)
* configure the origin remote on your local git: `git remote add origin git@gitlab.apertis.org:pkg/hello`
* push your local git contents to the newly created GitLab project: `git push --all --follow-tags origin`
* set it up with `gitlab-rulez apply rulez.yaml --filter pkg/hello` from
the [gitlab-rulez repository](https://gitlab.apertis.org/infrastructure/gitlab-rulez)
* sets the CI config path to `ci-package-builder.yml@infrastructure/ci-package-builder`
* changes the merge request settings:
* only allow fast-forward merges
* ensure merges are only allowed if pipelines succeed
* marks the `apertis/*` and `debian/*` branches as protected
* follow the process described in the [section about landing downstream changes
to the main archive](#landing-downstream-changes-to-the-main-archive) above to
publish the package on OBS.
## Adding updates from a non-default upstream repository of a distribution
There are circumstances, when we deviate from the default upstream. This usually happens
when:
* Packages are not available in the default distribution repository
* Packages in the default distribution repository are outdated
* Newer version of package, available in the non-default repository, is needed
For example, Apertis v2020 ships with a newer version of the Linux kernel
(5.4.x) than Debian Buster (4.9.x) on which it is based. In such cases, special
care needs to be taken to update packages from their respective upstreams.
Below are a set of steps which can be adapted to such exception packages. Let us assume that for
such repository, the package was picked from Debian Unstable instead
* Clone your repository
$ git clone git@gitlab.apertis.org:ritesh/libgpiod.git
Cloning into 'libgpiod'...
remote: Enumerating objects: 114, done.
remote: Counting objects: 100% (114/114), done.
remote: Compressing objects: 100% (85/85), done.
remote: Total 114 (delta 18), reused 110 (delta 18), pack-reused 0
Receiving objects: 100% (114/114), 110.99 KiB | 360.00 KiB/s, done.
Resolving deltas: 100% (18/18), done.
$ cd libgpiod/
* Ensure you have a branch against your deviated upstream. If you are tracking changes from a deviated upstream like **Debian Unstable**, it needs to be ensured that the package's packaging repository has the corresponding branches available. This is needed because the automated machinery tools expect respective branches to be available.
* For example, if you picked a package from Debian Unstable, ensure to have a branch named `debian/unstable` in your git repository.
$ git checkout -b debian/unstable origin/debian/buster
Branch 'debian/unstable' set up to track remote branch 'debian/buster' from 'origin'.
Switched to a new branch 'debian/unstable'
$ git checkout debian/buster
Branch 'debian/buster' set up to track remote branch 'debian/buster' from 'origin'.
Switched to a new branch 'debian/buster'
* Similarly, ensure to have a branch named `upstream/unstable` in your git repository.
$ git checkout upstream/buster
Branch 'upstream/buster' set up to track remote branch 'upstream/buster' from 'origin'.
Switched to a new branch 'upstream/buster'
$ git checkout -b upstream/unstable upstream/buster
Switched to a new branch 'upstream/unstable'
* Ensure that these branches are pushed to your remote `origin`. It is important that these branches are pushed and in sync with the default remote.
$ git push -u origin --all
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for debian/unstable, visit:
remote: https://gitlab.apertis.org/ritesh/libgpiod/-/merge_requests/new?merge_request%5Bsource_branch%5D=debian%2Funstable
remote:
To gitlab.apertis.org:ritesh/libgpiod.git
* [new branch] debian/unstable -> debian/unstable
Branch 'apertis/v2019' set up to track remote branch 'apertis/v2019' from 'origin'.
Branch 'debian/buster' set up to track remote branch 'debian/buster' from 'origin'.
Branch 'upstream/buster' set up to track remote branch 'upstream/buster' from 'origin'.
Branch 'debian/unstable' set up to track remote branch 'debian/unstable' from 'origin'.
* Pull in new updates using the `apertis-pkg-pull-updates` script, instructing it with the deviated upstream
* Eg. `apertis-pkg-pull-updates --package PKGNAME --upstream unstable --mirror http://deb.debian.org/debian`
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
$ git checkout apertis/v2021dev2
Switched to a new branch 'apertis/v2021dev2'
$ ../apertis-pkg-pull-updates --package libgpiod --upstream unstable --mirror http://deb.debian.org/debian
source package libgpiod
running git branch --track -f debian/unstable origin/debian/unstable
Branch 'debian/unstable' set up to track remote branch 'debian/unstable' from 'origin'.
running git branch --track -f upstream/unstable origin/upstream/unstable
running git branch --track -f upstream/unstable origin/upstream/unstable
running git branch --track -f upstream/unstable origin/upstream/unstable
local version: 1.2-3
fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable-security
local version: 1.2-3
fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable-proposed-updates
local version: 1.2-3
fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable
remote version: 1.4.1-4
update to 1.4.1-4
fetch https://snapshot.debian.org/mr/package/libgpiod/1.4.1-4/srcfiles?fileinfo=1
download http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
running dget --download-only --allow-unauthenticated http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 332 100 332 0 0 449 0 --:--:-- --:--:-- --:--:-- 448
100 2294 100 2294 0 0 1381 0 0:00:01 0:00:01 --:--:-- 5474
dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1.orig.tar.xz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 338 100 338 0 0 637 0 --:--:-- --:--:-- --:--:-- 636
100 307k 100 307k 0 0 358k 0 --:--:-- --:--:-- --:--:-- 358k
dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.debian.tar.xz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 342 100 342 0 0 677 0 --:--:-- --:--:-- --:--:-- 677
100 6132 100 6132 0 0 9922 0 --:--:-- --:--:-- --:--:-- 9922
Moving branch debian/unstable to debian/, was: 713dadc
running gbp import-dsc /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1-4.dsc --author-is-committer --author-date-is-committer-date --upstream-branch=upstream/unstable --debian-branch=debian/unstable '--debian-tag=debian/%(version)s' --no-sign-tags --no-pristine-tar
gbp:info: Version '1.4.1-4' imported under '/home/rrs/NoBackup/Gitlab_Packages/packaging-tools/libgpiod'
running ./import-tarballs /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1-4.dsc
Importing /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1.orig.tar.xz
## Adding updates from distribution development repositories
This is another scenario, wherein the user may need updates which are not yet released
into the Upstream Distributions' repositories.
For example, for Apertis, we may need a very newer version of libgpiod, which may not yet
have been released into any of Debian development releases (Unstable, Testing). In
such cases, where the changes may only be available in the packaging repositories, we
need to take extra care when pulling in such updates.
Let us assume libgpiod 1.4.2 has been available in Debian's libgpiod Packaging repository but
is not released into any of the Debian releases. In such case, we can try:
* Clone the remote libgpiod git packaging repository from Debian.
* Generate a source package out of the packaging repository using `gbp buildpackage -S`
* If successful, this will give us a proper *libgpiod source package*.
* Clone the Apertis libgpiod git packaging repostiory
* Use the `pristine-lfs` tool to import the source package generated from the Debian repository into Apertis packaging repository. Eg. `pristine-lfs import-dsc libgpiod-1.4.2-1.dsc`
* Note: The `import-dsc` subcommand imports the new tarball into the git repository and commits it to the `pristine-lfs` branch. While a user can commit to the branch manually by-hand, we recommend the use of `import-dsc` to import new tarballs and committing them to the packaging repository
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
# Updating existing components
## Pulling updates or security fixes from upstream distributions
Updates coming from upstream can be pulled it by triggering a CI pipeline on a
branches like `debian/buster` or `debian/bullseye`.
The pipeline will check the Debian archive for updates, pull them in the
`debian/$RELEASE`-like branch (for instance, `debian/bullseye` or
`debian/bullseye-security`), try to merge the new contents with the matching
`apertis/*` branches and, if successful, push a
proposed updates branch while creating a Merge Request for each `apertis/*`
branches it should be landed on.
The upstream update pipeline is usually triggered from
[the infrastructure dashboard](https://infrastructure.pages.apertis.org/dashboard/)
but can be manually triggered from the GitLab web UI by selecting the
`Run Pipeline` button in the `Pipelines` page of each repository under `pkg/*`
and selecting the `debian/bullseye` branch as the reference.
If the needed `debian/$RELEASE` branch doesn't exist (for example, there is a
`debian/buster` branch but no `debian/bullseye`), the Gitlab Web UI can be used
to create it. In the `Create from` field of the branch creation page, users
should select `debian/${RELEASE-1}`. In this case, the new branch named
`debian/bullseye` will be created from `debian/buster`.
Reviewers can then force-push to the proposed update branch in the Merge
Request to fixup any issue caused by the automated merge, and ultimately land
the MRs to the `apertis/*` branches.
In some situations the automated merge machinery may ask to
`PLEASE SUMMARIZE remaining Apertis Changes`, and in that case
reviewers should:
* check out the proposed update branch
* edit the changelog to list **all** the downstream changes the package still
ships compared to the newly merged upstream package and their reason,
describing the purpose of each downstream patch and of any other change
shown by `git diff` against the `debian/*` branch
* amend the merge commit
* force-push to the proposed update branch
* land the associated Merge Requests as described above
Remember to check that the updated package gets included in the next daily
reference image build and wait for its [QA test
results](https://lavaphabbridge.apertis.org/) to catch regressions timely
and act accordingly.
## Maintaining package from upstream sources
There are likely to be instances where it is desirable to import the latest version of a piece of software into Apertis directly from it's original authors or maintainers.
This may be as a result of the software in question not being packaged by Apertis' default upstream distribution, Debian, or their being a mismatch between the desired version in the upstream distribution and what is required for a specific goal.
The [Apertis release flow]( {{< ref "release-flow.md#apertis-release-flow" >}} ) stipulates that each Apertis release should include the latest mainline kernel LTS release.
Due to Debian's release cadence being approximately half that of Apertis', there are 2 Apertis' releases for each Debian stable release and no guarantees that the kernel's packaged for Debian's current stable or in progress releases will align with Apertis' requirements.
As a result Apertis will need to pull from the mainline kernel LTS tree to satisfy it's requirements.
Such packages will require special attention to assure that they remain up-to-date with any security releases made by the upstream and updated as and when apropriate.
Unless the imported package can be used as-is without any modification, there will be a [patch series]( {{< ref "buildingpatchingandmaintainingtheapertiskernel.md#packaging" >}} ) that potentially needs tweaking to apply after the update.
The workflow set out below takes this into consideration.
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
Below we use the process of importing the Linux kernel for `v2020` as an example, where we track the `linux-5.4.y` stable branch:
- Using the Gitlab Web UI, create an empty project under `pkg/`, for example, `pkg/linux`.
- Download the latest release tarball from upstream. For the `linux-5.4.y` stable branch, it would be `linux-5.4.115.tar.xz` from [kernel.org](https://www.kernel.org/) at the time of this writing.
- Extract the tarball and enter the extracted folder
$ tar xf linux-5.4.114.tar.xz
$ cd linux-5.4.114
- Create a new git repository and commit the whole source tree; this will create the `upstream/apertis` branch containing the untouched upstream source code:
$ git init
$ git checkout -b upstream/apertis
$ git add .
$ git commit
- Commit the original tarball to the `pristine-lfs` branch:
$ pristine-lfs commit ../linux-5.4.114.tar.xz
- Create the packaging branch for the Apertis version you're targetting and add packaging files
{{% notice info %}}
Debian packaging is not covered by this document, users interested in that matter
should refer to the [Debian Policy Manual](https://www.debian.org/doc/debian-policy/)
and [Debian Developer's Reference](https://www.debian.org/doc/manuals/developers-reference/).
{{% /notice %}}
- Add the Gitlab project you created earlier as the `origin` git remote:
$ git remote add origin git@gitlab.apertis.org:pkg/linux.git
- Push your branches to the repository:
$ git push --all --follow-tags origin
- Setup the repository as instructed in the [Adding new packages from Debian](#adding-new-packages-from-debian) section:
$ gitlab-rulez apply rulez.yaml --filter pkg/linux
You can then subsequently update this package by following these steps:
- Using the GitLab web UI, check to ensure that the relevant update branch exists, in the case of the Apertis `v2020` release, kernel updates should be made on the `apertis/v2020-security` branch. Create the required branch if it doesn't exist.
- Clone the existing package from Apertis GitLab and checkout the relevant branch:
$ git clone -b apertis/v2020-security git@gitlab.apertis.org:pkg/linux.git linux-apertis
- Separately clone the Linux kernel LTS repository and checkout the relvant stable branch:
$ git clone -b stable-linux-5.4.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux-stable
- Create a `gbp pq` patch branch and switch back to normal branch. The patch branch will be needed when rebasing the Debian patches:
$ cd linux-apertis
$ gbp pq import
$ gbp pq switch
- Determine the latest stable release and derive the Apertis version to use. To ensure collisions don't occur with any future Debian releases, we will mark the release as the zeroth "Debian" release (`-0`) and first Apertis pre-release (`~apertis1`):
$ RELEASE=`git -C ../linux-stable describe`
$ RELEASE="${RELEASE#v}-0~apertis1"
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
- Add a changelog entry for newest `linux-5.4.y` kernel release (this is needed to get `genorig.py` to use the right release):
$ DEBEMAIL="User Name <user@example.com>" dch -v "$RELEASE" ""
$ git add -f debian/changelog
$ git commit -s -m "Add changelog entry for update to $RELEASE"
- Run `debian/bin/genorig.py` to generate debianised kernel source tarball:
$ debian/bin/genorig.py ../linux-stable
- Import the updated source tarball:
$ gbp import-orig --upstream-branch=upstream/linux-5.4.y --debian-branch=apertis/v2020-security <path to orig tarball>
- Rebase the Debian patches on the new kernel version:
$ gbp pq rebase
This may require some manual intervention following the normal git rebasing process.
- Export updated patch series:
$ gbp pq export
- Tweak `debian/patches/series` file so that it retains the comments that `gbp pq` wants to drop. If no patches have been dropped during rebase then:
$ git checkout debian/patches/series
$ git add -f debian/patches
$ git commit -s -m "Update the debian patches for $RELEASE update"
- Update changelog to released state (set distribution to apertis), document any patches that have been dropped as a result of the update and create new commit:
$ dch -D apertis
$ git add -f debian/changelog
$ git commit -s -m "Release linux version $RELEASE"
- Run `dpkg-buildpackage` to generate the debian packaging, this is needed to use the tools to update the [pristine-lfs branch]( {{< ref "component_structure.md" >}} ):
$ mkdir ../build
$ gbp buildpackage --git-ignore-new \
--git-debian-branch=apertis/v2020-security \
--git-prebuild='debian/rules debian/control || true' \
--git-export-dir='../build' \
-us -S
- Import the generated kernel tarball into pristine-lfs branch with `import-tarballs`:
$ git clone git@gitlab.apertis.org:infrastructure/packaging-tools.git ../packaging-tools
$ ../packaging-tools/import-tarballs ../build/linux_5.4.51-0~apertis1.dsc
$ git push origin pristine-lfs
- Push kernel update to a branch to be reviewed:
$ git push origin apertis/v2020-security:wip/user/kernel-update
## How to manually sync an Apertis package with a new version
Upstream updates are usually handled automatically by the
[`ci-package-builder.yml`](https://gitlab.apertis.org/infrastructure/ci-package-builder/)
Continous Integration pipeline, which
[fetches upstream packages, merges them]({{< ref "component_guide.md#pulling-updates-or-security-fixes-from-upstream-distributions" >}})
with the Apertis contents and directly creates Merge Requests to be reviewed by
[maintainers]({{< ref "contributions.md#the-role-of-maintainers" >}}).
However, in some cases it is necessary to manually pull upstream contents:
1. the CI failed to merge the upstream update with the downstream changes, so a
developer must reproduce the merge and fix the conflicts
1. for some reason it in necessary to manually pull updates from a new upstream
distribution/suite
The steps below can guide you on manually pulling upstream updates:
{{% notice note %}}
This guide assumes the developer is a member of the Apertis development team,
contributors who are not a member of this team will need to fork the package to
their private workspace (as documented in the
[development process]({{< ref "development_process.md" >}}) before manually
updating the package and submit a merge request from there.
{{% /notice %}}
1. On GitLab,
[fork the project](https://docs.gitlab.com/ce/user/project/repository/forking_workflow.html)
to your personal namespace.
1. Check out the source repository
GITLAB_USER=${USER}
UPSTREAM_DISTRIBUTION=buster
MIRROR=https://deb.debian.org/debian
PACKAGE=glib2.0
git clone git@gitlab.apertis.org:${GITLAB_USER}/${PACKAGE}
cd ${PACKAGE}
1. Only if moving to a new upstream distribution, prepare the upstream base
branches using the current upstream branches as a base:
UPSTREAM_DISTRIBUTION=bullseye
UPSTREAM_BASE=buster
git push origin origin/upstream/${UPSTREAM_BASE}:refs/heads/upstream/${UPSTREAM_DISTRIBUTION}
git push origin origin/debian/${UPSTREAM_BASE}:refs/heads/debian/${UPSTREAM_DISTRIBUTION}
1. Sync the upstream contents
apertis-pkg-pull-updates --package=$PACKAGE --upstream=$UPSTREAM_DISTRIBUTION --mirror=$MIRROR
git push origin --follow-tags pristine-lfs upstream/${UPSTREAM_DISTRIBUTION} debian/${UPSTREAM_DISTRIBUTION}
1. Merge the upstream changes
PROPOSED_BRANCH=wip/$GITLAB_USER/update-from-${UPSTREAM_DISTRIBUTION}
TARGET_BRANCH=apertis/v2021dev1
git checkout -b ${PROPOSED_BRANCH} ${TARGET_BRANCH}
apertis-pkg-merge-updates --upstream=debian/${UPSTREAM_DISTRIBUTION} --downstream=${PROPOSED_BRANCH}
1. Only if the merge fails, fix the conflicts and continue
git status
# fix conflicts
git merge --continue
dch --local co --no-auto-nmu
1. Check `debian/changelog` and update it if needed by listing all the remaining Apertis changes
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
$EDITOR debian/changelog
git commit --signoff --amend debian/changelog
1. Push your changes for review
git push origin --follow-tags pristine-lfs upstream/${UPSTREAM_DISTRIBUTION} debian/${UPSTREAM_DISTRIBUTION}
git push -o merge_request.create
# Creating new modifications
The standard [Contribution Process]({{< ref "contributions.md" >}})
applies unchanged to packaging repositories, the Apertis development team and
trusted contributors push changes to `wip/` branches and get them landed to the
`apertis/*` branches via merge requests. Other developers can fork packaging
repositories into their personal space and send merge requests from there.
The only additional requirement imposed by the Debian packaging format is that
changes outside of the `debian/` folder are not allowed and would cause the
source-building pipeline to fail. Check the [Debian Packaging](https://wiki.debian.org/Packaging)
documentation to find how patches affecting code outside of the `debian/`
folder should be handled.
Update `debian/changelog` separately, as the very last step when you issue
a release, by generating the changelog entries from the Git commit log.
Writing good commit messages ensures you don’t have to edit the generated
changelog entries.
Submit a separate merge request on GitLab for each bug or task. To ease
the review process, in particular to avoid churn in the case of rebases, it is
recommended to leave the editing of `debian/changelog` to a dedicated merge
request once all the other MRs have been landed, see the
[section about landing downstream changes to the main
archive](#landing-downstream-changes-to-the-main-archive) below.
{{% notice tip %}}
In order to follow a
[release early, release often philosophy]({{< ref "contributions.md#upstream-early-upstream-often" >}})
it is
also recommended to avoid delaying release commits to include additional
features. This gives the possibility to receive early feedback on the
downstream changes.
{{% /notice %}}
If you still wish to edit `debian/changelog` for any reason, just make sure
that the changelog entry you're writing has the `distribution` field set
to `UNRELEASED`, using `gbp dch --auto --ignore-branch` to ensure the
formatting is correct.
The CI pipeline generate a source package locally for each commit pushed
to the packaging repositories. You can download this package by browsing the
pipeline artifacts. The generated sources are versioned to indicate that
they are not yet suitable for release.
With the `distribution` field set to `UNRELEASED`, package sources get uploaded
to the `:snapshots` OBS project matching the branch (that is,
`apertis:v2020dev0:target:snaphots` when landing changes to the
`apertis/v2020dev0` branch of a `:target` package): the following [section
about landing downstream changes to the main
archive](#landing-downstream-changes-to-the-main-archive) below describes
in detail how to set the `distribution` to land the package to the appropriate
main OBS project.
## Landing downstream changes to the main archive
Once downstream changes to a package are deemed ready to be published in the
Apertis main archive, a proper release needs to be issued.
* Push a `wip/` branch updating `debian/changelog`
* use `GBP_CONF_FILES=/dev/null gbp dch --release -D apertis --debian-branch=apertis/v2020dev0`
to generate a release changelog entry summarizing all the changes already
landed on the `apertis/v2020dev0` branch
* ensure that the `distribution` field has been changed from `UNRELEASED`
to `apertis`
* Create a Merge Request based on your `wip/` branch for the most recent
release branch where you want to land your changes:
* for published stable release, the main branch (for instance
`apertis/v2019`) should **never** be targeted directly, but updates and
fixes should go through the `apertis/v2019-security` or
`apertis/v2019-updates` branches, see [Process after a product
release]( {{< ref "release-flow.md#process-after-a-product-release" >}} )
for more details
* for instance, if you want to land changes to both the development and
stable releases, push your `wip/` source branch and create a merge request for the
development one first (for instance, `apertis/v2020dev0`) and then, once
merged, create a merge request for the stable one (for instance, `apertis/v2019-updates`)
* Get the Merge Request reviewed and landed
* The CI pipeline will then build-check the source package as usual and since
the `distribution` field is no longer `UNRELEASED` it will also:
* add a Git tag for the release version to the repository
* rebuild the release source package
* store the release sources in the `pristine-lfs-source` branch
* upload the release source package to the main project (for instance
`apertis:v2020dev0:target`)
If the `apertis/$RELEASE-updates` or `apertis/$RELEASE-security` branches for
published stable releases do not exist yet, they should be created from the
GitLab web UI since their protected status makes pushing forbidden.
For trivial changes it is also possible to combine the release commit in the same
MR as the changes. Again, developers need to be careful to ensure the changelog
entries are kept up-to-date when the commit messages get changed via rebase.
## Backporting updates or security fixes
Often downstream fixes, upstream updates or security fixes need to be applied
to [multiple active releases]( {{< ref "release-flow.md" >}} ).
Changes should be introduced in the most recent development release where they can
be tested and regressions detected with little impact, following the instructions
in the
[Landing downstream changes to the main archive](#landing-downstream-changes-to-the-main-archive)
and [Pulling updates or security fixes from upstream distributions](#pulling-updates-or-security-fixes-from-upstream-distributions)
sections.
Once the changes have been thoroughly tested paying close attention to the QA
test results, they can then be propagated to the more stable releases, where
any mistake can impact the product teams using Apertis in the field.
For instance, once a fix is landed to `apertis/v2021dev0` and no regressions
are found in the subsequent QA test results, a MR should be create to land
the changes to the stable releases.
If there is no divergence between the packages in the different releases and
the backport can be done with a fast-forward, a MR should be created to submit
the changes from for instance `apertis/v2021dev0` to `apertis/v2020-updates` or
`apertis/v2020-security`, following the
[Landing downstream changes to the main archive](#landing-downstream-changes-to-the-main-archive)
steps, choosing the destination depending on the nature and impact of the fix.
If package diverged across releases, a separate branch has to be created where
the fixes are cherry-picked appropriately before creating the MR. See the
[Diverging release branches](#diverging-release-branches) section for further
details about versioning divergent packages.
## Diverging release branches
Sometimes different downstream patches need to be applied in the different
Apertis release branches. A clear case of that is the `base-files` package
which ships the release name in `/etc/os-release`.
In such situation it is crucial to use different version identifiers in each
branch: the version for a given package needs to be globally unique across the whole
archive since uploading different package sources with the same name/version
would lead to difficult to diagnose errors.
When targeting a specific release, `~${RELEASE}.${COUNTER}` needs to be
appended to the version identifier after the local build suffix:
* `0.42` → append `co0~v2020pre.0` → `0.42co0~v2020pre.0`
* `0.42co3` → bump to `co4` and append `~v2020pre.0`→ `0.42co4~v2020pre.0`
* `0.42co4~v2020pre.0` → increase the release-specific counter → `0.42co4~v2020pre.1`
This uses the fact that `~` in Debian package numbers sorts before anything,
see the [Debian Policy §5.6.12](https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version) for more details.
Adding `~` is necessary so that if a new upstream version `0.42.1` or a new
non-release-specific downstream version `0.42co4` is introduced, they will
replace the release-specific package.
Note that `dpkg` considers `2020.0` to be newer than `2020pre.0`, so the
Apertis release identifiers can be used with no modification (if in doubt,
check with `dpkg --compare-versions 2020pre.0 '<<' 2020.0 && echo ok`).