apertis_packaging_guide.md

---
title: "Apertis Development Guide"
tags: []
---

This guide is a living document that stands as a single point of information on
Packaging, Processes, Guidelines, CI related to the Apertis project.

# Apertis Packaging CI

Apertis stores the source of all the shipped packages in GitLab and uses a set
of GitLab CI pipelines to manage the workflows to:

* land updated sources to OBS which will then build the binary outputs
* pull updates from upstream distributions like Debian 10 Buster

## Adding downstream changes

The standard [Contribution Process](https://wiki.apertis.org/Guidelines/Contribution_process)
applies unchanged to packaging repositories, pushing changes to `wip/` branches
and getting them landed to the `apertis/*` branches via Merge Requests.

The only additional requirement imposed by the Debian packaging format is that
changes outside of the `debian/` folder are not allowed and would cause the
source-building pipeline to fail. Check the [Debian Packaging](https://wiki.debian.org/Packaging)
documentation to find how patches affecting code outside of the `debian/`
folder should be handled.

Updating `debian/changelog` should be done separately as the last step when
issuing a release, generating the changelog entries from the Git commit log,
which makes writing good commit log messages even more important.

A merge request should be submitted on GitLab for each bug or tasks. To ease
the review process, in particular to avoid churn in case or rebases, it is
recommended to leave the editing of `debian/changelog` to a dedicated merge
request once all the other MRs have been landed, see the
[section about landing downstream changes to the main
archive](#landing-downstream-changes-to-the-main-archive) below.

If you still wish to edit `debian/changelog` for any reason, just make sure
that the changelog entry you're writing has the `distribution` field set
to `UNRELEASED`, using `gbp dch --auto --ignore-branch` to ensure the
formatting is correct.

The CI pipeline will locally generate a source package for each commit pushed
to the packaging repositories, which can be retrieved by browsing the
pipeline artifacts. The generated sources will be versioned to indicate that
they are not yet suitable for release.

With the `distribution` field set to `UNRELEASED` package sources get uploaded
to the `:snapshots` OBS project matching the branch (that is,
`apertis:v2020dev0:target:snaphots` when landing changes to the
`apertis/v2020dev0` branch of a `:target` package): the following [section
about landing downstream changes to the main
archive](#landing-downstream-changes-to-the-main-archive) below describes
in detail how to set the `distribution` to land the package to the appropriate
main OBS project.

## Landing downstream changes to the main archive

Once downstream changes to a package are deemed ready to be published in the
Apertis main archive, a proper release needs to be issued.

* Push a `wip/` branch updating `debian/changelog`
  * use `GBP_CONF_FILES=/dev/null gbp dch --release -D apertis --debian-branch=apertis/v2020dev0`
    to generate a release changelog entry summarizing all the changes already
    landed on the `apertis/v2020dev0` branch
  * ensure that the `distribution` field has been changed from `UNRELEASED`
    to `apertis`
* Create a Merge Request based on your `wip/` branch for the most recent
  release branch where you want to land your changes:
  * for published stable release, the main branch (for instance
    `apertis/v2019`) should **never** be targeted directly but updates and
    fixes should go through the `apertis/v2019-security` or
    `apertis/v2019-updates` branches, see [Process after a product
    release](https://designs.apertis.org/latest/release-flow.html#process-after-a-product-release)
    for more details
  * for instance, if you want to land changes to both the development and
    stable releases, push your `wip/` source branch and create a MR for the
    development one first (for instance, `apertis/v2020dev0`) and then, once
    merged, create a MR for the stable one ((for instance, `apertis/v2019-updates`)
* Get the Merge Request reviewed and landed
* The CI pipeline will then build-check the source package as usual and since
  the `distribution` field is no longer `UNRELEASED` it will also:
  * add a Git tag for the release version to the repository
  * rebuild the release source package
  * store the release sources in the `pristine-lfs-source` branch
  * upload the release source package to the main project (for instance
    `apertis:v2020dev0:target`)

If the `apertis/$RELEASE-updates` or `apertis/$RELEASE-security` branches for
published stable releases do not exist yet, they should be created from the
GitLab web UI since their protected status makes pushing forbidden.

For trivial changes it is also possible to combine the release commit in the same
MR as the changes. Again, developers need to be careful to ensure the changelog
entries are kept up-to-date when the commit messages get changed via rebase.

## Pulling updates or security fixes from upstream distributions

A separate set of pipeline steps are configured on the `debian/$RELEASE-gitlab-update-job`
branches (for instance, `debian/buster-gitlab-update-job`) of each package.

The pipeline will check the Debian archive for updates, pull them in the
`debian/$RELEASE` branch (for instance, `debian/buster`), try to merge the new
contents with the matching `apertis/*` branches and, if successful, push a
proposed updates branch while creating a Merge Request for each `apertis/*`
branches it should be landed on.

The upstream update pipeline is scheduled to run automatically each weekend,
but can be manually triggered from the GitLab web UI by selecting the
`Run Pipeline` button in the `Pipelines` page of each repository under `pkg/*`
and selecting the `debian/buster-gitlab-update-job` branch as the reference.

![Run Pipeline button](/images/run-pipeline-button.png)

Reviewers can then force-push to the proposed update branch in the Merge
Request to fixup any issue caused by the automated merge, and ultimately land
the MRs to the `apertis/*` branches.

In some situations the automated merge machinery may ask to
`PLEASE SUMMARIZE remaining Apertis Changes`, and in that case
reviewers should:
* check out the proposed update branch
* edit the changelog to list **all** the downstream changes the package still
  ships compared to the newly merged upstream package and their reason,
  describing the purpose of each downstream patch and of any other change
  shown by `git diff` against the `debian/*` branch
* amend the merge commit
* force-push to the proposed update branch
* land the associated Merge Requests as described above

Remember to check that the updated package gets included in the next daily
reference image build and wait for its [QA test
results](https://lavaphabbridge.apertis.org/) to catch regressions timely
and act accordingly.

## Backporting updates or security fixes

Often downstream fixes, upstream updates or security fixes need to be applied
to [multiple active releases](https://designs.apertis.org/latest/release-flow.html).

Changes should be introduced in the most recent development release where they can
be tested and regression detected with little impact, following the instructions
in the
[Landing downstream changes to the main archive](#landing-downstream-changes-to-the-main-archive)
and [Pulling updates or security fixes from upstream distributions](#pulling-updates-or-security-fixes-from-upstream-distributions)
sections.

Once the changes have been thoroughly tested paying close attention to the QA
test results, they can then be propagated to the more stable releases, where
any mistake can impact the product teams using Apertis in the field.

For instance, once a fix is landed to `apertis/v2021dev0` and no regressions
are found in the subsequent QA test results, a MR should be create to land
the changes to the stable releases.

If there is no divergence between the packages in the different releases and
the backport can be done with a fast-forward, a MR should be created to submit
the changes from for instance `apertis/v2021dev0` to `apertis/v2020-updates` or
`apertis/v2020-security`, following the
[Landing downstream changes to the main archive](#landing-downstream-changes-to-the-main-archive)
steps, choosing the destination depending on the nature and impact of the fix.

If package diverged across releases, a separate branch has to be created where
the fixes are cherry-picked appropriately before creating the MR. See the
[Diverging release branches](#diverging-release-branches) section for further
details about versioning divergent packages.

## Diverging release branches

Sometimes different downstream patches need to be applied in the different
Apertis release branches. A clear case of that is the `base-files` package
which ships the release name in `/etc/os-release`.

In such situation it is crucial to use different version identifiers in each
branch: the version for a given package needs to be globally unique across the whole
archive since uploading different package sources with the same name/version
would lead to errors difficult to diagnose.

When targeting a specific release, `~${RELEASE}.${COUNTER}` needs to be
appended to the version identifier after the local build suffix:

* `0.42` → append `co1~v2020pre.0` → `0.42co1~v2020pre.0`
* `0.42co3` → bump to `co4` and append `~v2020pre.0`→ `0.42co4~v2020pre.0`
* `0.42co4~v2020pre.0` → increase the release-specific counter → `0.42co4~v2020pre.1`

This uses the fact that `~` in Debian package numbers sorts before anything,
see the [Debian Policy §5.6.12](https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version) for more details.
Adding `~` is necessary so that if a new upstream version `0.42.1` or a new
non-release-specific downstream version `0.42co4` is introduced, they will
replace the release-specific package.

Note that `dpkg` considers `2020.0` to be newer than `2020pre.0`, so the
Apertis release identifiers can be used with no modification (if in doubt,
check with `dpkg --compare-versions 2020pre.0 '<<' 2020.0 && echo ok`).

## Adding new packages from Debian

This is the process to import a new package from Debian to Apertis:

* locally create a new git repository
* invoke `import-debian-package` from the [packaging-tools
  repository](https://gitlab.apertis.org/infrastructure/packaging-tools/)
  to populate the local git repository:
  * fetch a specific version: `import-debian-package --upstream buster --downstream apertis/v2020dev0 --create-ci-branches --package hello --version 2.10-2`
  * fetch the latest version: `import-debian-package --upstream buster --downstream apertis/v2020dev0 --create-ci-branches --package hello`
  * multiple downstream branches can be specified, in which case all of them
    will be updated to point to the newly imported package version
  * don't use `import-debian-package` on existing repositories, it does not
    attempt to merge `apertis/*` branches and instead it re-sets them to new
    branches based on the freshly imported Debian package
* create an empty project on GitLab under the `pkg/*` namespaces (for instance, `pkg/target/hello`)
* configure the origin remote on your local git: `git remote add origin git@gitlab.apertis.org:pkg/target/hello`
* push your local git contents to the newly created GitLab project: `git push --all --follow-tags origin`
* set it up with `gitlab-rulez apply rulez.yaml --filter $REPOSITORYPATH` from
  the [gitlab-rulez repository](https://gitlab.apertis.org/infrastructure/gitlab-rulez)
  * sets the CI config path to `debian/apertis/gitlab-ci.yml`
  * changes the merge request settings:
      * only allow fast-forward merges
      * ensure merges are only allowed if pipelines succeed
  * adds a schedule on the `debian/buster-gitlab-update-job` branch to run weekly
  * marks the `apertis/*` and `debian/*` branches as protected
* follow the process described in the [section about landing downstream changes
  to the main archive](#landing-downstream-changes-to-the-main-archive) above to
  publish the package on OBS.

## Adding updates from a non-default upstream repository of a distribution

There are circumstances, when we deviate from the default upstream. This usually happens
when:
* Packages are not available in the default distribution repository
* Packages in the default distribution repository are outdated
* Newer version of package, available in the non-default repository, is needed

For example, Debian Buster ships an older version of the Linux kernel (4.9.x) than what we
have in Apertis (> 5.4). In such cases, special care needs to be taken to update packages from
their respective upstreams.


Below are a set of steps which can be adapted to such exception packages. Let us assume that for
such repository, the package was picked from Debian Unstable instead

* Clone your repository
* Ensure you have a branch against your deviated upstream. If you are tracking changes from a deviated upstream like **Debian Unstable**, it needs to be ensured that the package's packaging repository has the corresponding branches available. This is needed because the automated machinery tools expect respective branches to be available.
  * For example, if you picked a package from Debian Unstable, ensure to have a branch named `debian/unstable` in your git repository.
  * Similarly, ensure to have a branch named `upstream/unstable` in your git repository.
* Ensure that these branches are pushed to your remote `origin`. It is important that these branches are pushed and in sync with the default remote.
* Pull in new updates using the `apertis-pkg-pull-updates` script, instructing it with the deviated upstream
  * Eg. `apertis-pkg-pull-updates --package PKGNAME --upstream unstable --mirror http://deb.debian.org/debian`

Based on the steps mentioned, below is a screen capture with an example package from Apertis, wherein we instruct the tool to pull updates from **Debian Unstable** instead.

    rrs@priyasi:packaging-tools (ritesh-local-branch)$ git clone git@gitlab.apertis.org:ritesh/libgpiod.git
    Cloning into 'libgpiod'...
    remote: Enumerating objects: 114, done.
    remote: Counting objects: 100% (114/114), done.
    remote: Compressing objects: 100% (85/85), done.
    remote: Total 114 (delta 18), reused 110 (delta 18), pack-reused 0
    Receiving objects: 100% (114/114), 110.99 KiB | 360.00 KiB/s, done.
    Resolving deltas: 100% (18/18), done.

    rrs@priyasi:packaging-tools (ritesh-local-branch)$ cd libgpiod/

    rrs@priyasi:libgpiod (apertis/v2019)$ git checkout -b debian/unstable origin/debian/buster 
    Branch 'debian/unstable' set up to track remote branch 'debian/buster' from 'origin'.
    Switched to a new branch 'debian/unstable'

    rrs@priyasi:libgpiod (debian/unstable)$ git checkout debian/buster 
    Branch 'debian/buster' set up to track remote branch 'debian/buster' from 'origin'.
    Switched to a new branch 'debian/buster'

    rrs@priyasi:libgpiod (debian/buster)$ git checkout upstream/buster 
    Branch 'upstream/buster' set up to track remote branch 'upstream/buster' from 'origin'.
    Switched to a new branch 'upstream/buster'


    rrs@priyasi:libgpiod (upstream/buster)$ git push -u origin --all
    Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
    remote: 
    remote: To create a merge request for debian/unstable, visit:
    remote:   https://gitlab.apertis.org/ritesh/libgpiod/-/merge_requests/new?merge_request%5Bsource_branch%5D=debian%2Funstable
    remote: 
    To gitlab.apertis.org:ritesh/libgpiod.git
     * [new branch]      debian/unstable -> debian/unstable
    Branch 'apertis/v2019' set up to track remote branch 'apertis/v2019' from 'origin'.
    Branch 'debian/buster' set up to track remote branch 'debian/buster' from 'origin'.
    Branch 'upstream/buster' set up to track remote branch 'upstream/buster' from 'origin'.
    Branch 'debian/unstable' set up to track remote branch 'debian/unstable' from 'origin'.


    rrs@priyasi:libgpiod (debian/unstable)$ git checkout -b upstream/unstable upstream/buster 
    Switched to a new branch 'upstream/unstable'

    rrs@priyasi:libgpiod (debian/unstable)$ git checkout apertis/v2021dev2
    Switched to a new branch 'apertis/v2021dev2'

    rrs@priyasi:libgpiod (upstream/unstable)$ ../apertis-pkg-pull-updates --package libgpiod --upstream unstable --mirror http://deb.debian.org/debian
    source package libgpiod
    running git branch --track -f debian/unstable origin/debian/unstable
    Branch 'debian/unstable' set up to track remote branch 'debian/unstable' from 'origin'.
    running git branch --track -f upstream/unstable origin/upstream/unstable
    running git branch --track -f upstream/unstable origin/upstream/unstable
    running git branch --track -f upstream/unstable origin/upstream/unstable
    local version: 1.2-3
    fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable-security
    local version: 1.2-3
    fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable-proposed-updates
    local version: 1.2-3
    fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable
    remote version: 1.4.1-4
    update to 1.4.1-4
    fetch https://snapshot.debian.org/mr/package/libgpiod/1.4.1-4/srcfiles?fileinfo=1
    download http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
    running dget --download-only --allow-unauthenticated http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
    dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   332  100   332    0     0    449      0 --:--:-- --:--:-- --:--:--   448
    100  2294  100  2294    0     0   1381      0  0:00:01  0:00:01 --:--:--  5474
    dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1.orig.tar.xz
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   338  100   338    0     0    637      0 --:--:-- --:--:-- --:--:--   636
    100  307k  100  307k    0     0   358k      0 --:--:-- --:--:-- --:--:--  358k
    dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.debian.tar.xz
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   342  100   342    0     0    677      0 --:--:-- --:--:-- --:--:--   677
    100  6132  100  6132    0     0   9922      0 --:--:-- --:--:-- --:--:--  9922
    Moving branch debian/unstable to debian/, was: 713dadc
    running gbp import-dsc /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1-4.dsc --author-is-committer --author-date-is-committer-date --upstream-branch=upstream/unstable --debian-branch=debian/unstable '--debian-tag=debian/%(version)s' --no-sign-tags --no-pristine-tar
    gbp:info: Version '1.4.1-4' imported under '/home/rrs/NoBackup/Gitlab_Packages/packaging-tools/libgpiod'
    running ./import-tarballs /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1-4.dsc
    Importing /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1.orig.tar.xz

## Adding updates from distribution development repositories

This is another scenario, wherein the user may need updates which are not yet released
into the Upstream Distributions's repositories.

For example, for Apertis, we may need a very newer version of libgpiod, which may not yet
have been released into any of Debian development releases (Unstable, Testing). Under
such cases, where the changes may only be available in the packaging repositories, we
need to take extra care when pulling in such updates.

Let us assume libgpiod 1.4.2 has been available in Debian's libgpiod Packaging repository but
is not released into any of the Debian releases. In such case, we can try:

* Clone the remote libgpiod git packaging repository from Debian.
* Generate a source package out of the packaging repository using `gbp buildpackage -S`
  * If successful, this will give us a proper *libgpiod source package*.
* Clone the Apertis libgpiod git packaging repostiory
  * Use the [import-tarballs](https://gitlab.apertis.org/infrastructure/packaging-tools/-/blob/master/import-tarballs) tool to import the source package generated from the Debian repository into Apertis packaging repository. Eg. `import-tarballs libgpiod-1.4.2-1.dsc`
  * Note: The `import-tarballs` script imports the new tarball into the git repository and commits it to the `pristine-lfs` branch. While, a user can commit to the branch manually by-hand, we recommend the use of the `import-tarballs` tool to import new tarballs and commiting them to the packaging repository

## License scans

As merge requests to packaged software are submitted, the CI pipeline performs
license scans on the package. The scans are performed on all files in the package,
not just the new submission. The pipeline fails or emits a warning (depending on
the configuration) when if finds files with unknown or unclear licensing terms,
or files under licenses not allowed in the package. When such situation arises,
it is the responsibility of the submitter to perform the review of the license
scan results and make updates to the package if necessary.

When the license scan mistakenly identifies a file as being under an incorrect
license, or fails to process it correctly, there are three ways to fix this:

 1. Specify the correct copyright and the license in `debian/apertis/copyright.yml`.
    The format of the file is specified in the
    [Dpkg::Copyright::Scanner](https://manpages.debian.org/buster/libconfig-model-dpkg-perl/Dpkg::Copyright::Scanner.3pm.en.html)
    manpage.
    In short, it’s a YAML file mapping paths to their licensing information:

        debian:
          copyright: 2015, Marcel
          license: Expat
        src/:
          copyright: 2016, Joe
          license: Expat
        .*/NOTICE:
          skip: 1
        src/garbled/:
          'override-copyright': 2016 Marcel MeXzigue

    Patterns follow the Perl regular expression rules.

    Please also verify `debian/copyright` specifies the correct license, and if it
    doesn’t, submit a patch to Debian.

 2. Add the file to the list of ignored files.
    `debian/apertis/copyright.whitelist` is formatted the same way as `gitignore`,
    please refer to the [gitignore](https://manpages.debian.org/buster/git-man/gitignore.5.en.html)
    manpage for more information.

 3. If the file is under a license not suitable for Apertis, it can be removed from
    the package by either repackaging the tarball or patching it out, in which case
    the scanner will not take it into account.

The license scanner will store the automatically generated copyright report file
under `debian/apertis/copyright`, updating the merge request when necessary.

[Dpkg::Copyright::Scanner]: https://manpages.debian.org/testing/libconfig-model-dpkg-perl/Dpkg::Copyright::Scanner.3pm.en.html
[gitignore]: https://manpages.debian.org/testing/git-man/gitignore.5.en.html

# Internals

Main components:
* [`ci-package-builder`](https://gitlab.apertis.org/infrastructure/ci-package-builder):
  centralized location of the GitLab-to-OBS and Debian-to-GitLab pipeline definitions
* [`debian/apertis/gitlab-ci.yaml`](https://gitlab.apertis.org/pkg/target/base-files/blob/apertis/v2019/debian/apertis/gitlab-ci.yml):
  imports the `ci-package-builder` pipelines from each packaging repository
* [`apertis-package-source-builder`](https://gitlab.apertis.org/infrastructure/apertis-docker-images/tree/apertis/v2019/apertis-package-source-builder):
  Docker environment for the GitLab pipelines
* [`pristine-lfs`](https://salsa.debian.org/andrewsh/pristine-lfs): stores
  upstream original tarballs and packaging source tarballs using Git-LFS, as a
  more robust replacement for `pristine-tar`

Branches:
* `pristine-lfs`: stores references to the Git-LFS-hosted original tarballs
* `debian/$DEBIAN_RELEASE` (for instance, `debian/buster`): contains the extracted
  upstream sources and packaging information from Debian
* `pristine-lfs-source`: stores references to the Git-LFS-hosted packaging
  tarballs, mainly to ensure that each (package, version) tuple is built only
  once and no conflicts can arise
* `apertis/$APERTIS_RELEASE` (for intance, `apertis/v2020dev0`): contains the
  extracted upstream sources and possibly patched packaging information for
  Apertis, including the `debian/apertis/gitlab-ci.yaml` to set up the
  GitLab-to-OBS pipeline
* `apertis/$APERTIS_RELEASE-security` and `apertis/$APERTIS_RELEASE-updates`
  (for intance, `apertis/v2019-updates`): similar to ``apertis/$APERTIS_RELEASE`
  but respectively target the Security and Updates repositories for published
  stable releases as described in [Process after a product
  release](https://designs.apertis.org/latest/release-flow.html#process-after-a-product-release)
* `debian/$DEBIAN_RELEASE-gitlab-update-job` (for instance,
  `debian/buster-gitlab-update-job`): hosts the `debian/apertis/gitlab-ci.yaml`
  file to configure the Debian-to-GitLab pipeline

Tags:
* `debian/*`: tags for Debian releases in the `debian/*` branches
* `apertis/*`:  tags for the Apertis releases in the `apertis/*` branches