component_guide.md

+++
date = "2020-07-24"
lastmod = "2021-04-29"
weight = 100
toc = true

title = "Adding and Updating Components"
+++

Apertis stores the source of all the shipped packages in GitLab and uses a
[GitLab CI pipeline](https://gitlab.apertis.org/infrastructure/ci-package-builder/-/blob/master/ci-package-builder.yml)
to manage the workflows and automate many of the tasks that are required to
release a package, thus for many tasks such as adding or updating packages,
interaction is only required with GitLab, with automation taking care of the
rest of the process.

{{% notice info %}}
Please see the [Apertis workflow]({{< ref "workflow-guide.md" >}}) for more
information regarding the automation provided by Apertis.
{{% /notice %}}

{{% notice tip %}}
The VirtualBox-based
[Apertis SDK virtual machine images]( {{< ref "virtualbox.md" >}} ) ship with
all the needed tools installed, providing a reliable, self-contained
environment ready to be used. Alternatively, you can run the Apertis
`package-source-builder` Docker container.

For v2021 and newer:

    APERTIS_RELEASE=v2021
    docker pull registry.gitlab.apertis.org/infrastructure/apertis-docker-images/${APERTIS_RELEASE}-package-source-builder
    docker run -it --rm --env HOME -w "$(pwd)" -v "$HOME:$HOME" -v "$HOME:/root" --security-opt label=disable registry.gitlab.apertis.org/infrastructure/apertis-docker-images/${APERTIS_RELEASE}-package-source-builder bash

For v2020 and older:

    APERTIS_RELEASE=v2020
    docker pull docker-registry.apertis.org/apertis/apertis-${APERTIS_RELEASE}-package-source-builder
    docker run -it --rm --env HOME -w "$(pwd)" -v "$HOME:$HOME" -v "$HOME:/root" --security-opt label=disable docker-registry.apertis.org/apertis/apertis-${APERTIS_RELEASE}-package-source-builder bash

{{% /notice %}}

# Adding New Components

The software components used in Apertis images are packaged in the
[`deb` packaging format](https://wiki.debian.org/Packaging), as used by Debian.
It is preferred, where possible, to add additional packages from Debian as
covered by the
[contribution policy]({{< ref "contributions.md#adding-components-to-apertis" >}}).

## Adding New Packages from Debian

{{% notice info %}}
As the adding of existing Debian packages is mostly scripted/automated it is
expected that such actions will be carried out by maintainers after a request
made in line with the
[contribution policy]({{< ref "contributions.md#adding-components-to-apertis" >}}).
{{% /notice %}}

This is the process to import a new package from Debian to Apertis:

* Create a folder, in the name of the package to import.
  * Eg. for package `hello`, run:

        mkdir hello

  * chdir to created folder:

        cd hello

* Invoke `import-debian-package` from the
  [packaging-tools repository](https://gitlab.apertis.org/infrastructure/packaging-tools/).
  To fetch the latest version:

      import-debian-package --upstream buster --downstream apertis/v2021 --component target --package hello

  To fetch a specific version, add the `--version` option:

      import-debian-package --upstream buster --downstream apertis/v2021 --component target --package hello --version 2.10-2

  * The argument to `--component` reflects the repository component it is part
    of (for instance, `target`); it will be stored in
    `debian/apertis/component`.
  * Multiple downstream branches can be specified, in which case all of them
    will be updated to point to the newly imported package version.
  * The Apertis version of the package will have a local suffix (`apertis0`)
    appended.

  {{% notice warning %}}
  Don't use `import-debian-package` on existing repositories, it does not
  attempt to merge `apertis/*` branches and instead it re-sets them to new
  branches based on the freshly imported Debian package.
  {{% /notice %}}

* Create an empty project on GitLab under the `pkg` namespace (for instance,
  `pkg/hello`).
* Configure the origin remote on your local git:

      git remote add origin git@gitlab.apertis.org:pkg/hello

* Push your local git contents to the newly created GitLab project:

      git push --all --follow-tags origin

* Set it up with `gitlab-rulez` from the
  [gitlab-rulez repository](https://gitlab.apertis.org/infrastructure/gitlab-rulez):

      gitlab-rulez apply rulez.yaml --filter pkg/hello

  {{% notice info %}}
  This:
  - sets the CI config path to
    `ci-package-builder.yml@infrastructure/ci-package-builder`
  - changes the merge request settings to:
    - only allow fast-forward merges
    - ensure merges are only allowed if pipelines succeed
  - marks the `apertis/*` and `debian/*` branches as protected
  {{% /notice %}}

* As the CI configuration was set by `gitlab-rulez` after the initial commit,
  it will be necessary to run the CI/CD pipeline on each `apertis/*` branch to
  push the package through to OBS.

  {{% notice warning %}}
  Ensure that one branch successfully completes it's upload to OBS before
  starting other branches to ensure that the different branches don't conflict
  over uploading the tarballs.
  {{% /notice %}}

## Adding New Packages from Upstream Sources

There are likely to be instances where it is desirable to import the latest
version of a piece of software into Apertis directly from it's original authors
or maintainers.  This may be as a result of the software in question not being
packaged by Apertis' default upstream distribution, Debian, or their being a
mismatch between the desired version in the upstream distribution and what is
required for a specific goal.

For example, the
[Apertis release flow]( {{< ref "release-flow.md#apertis-release-flow" >}})
stipulates that each Apertis release should include the latest mainline kernel
LTS release.  Due to Debian's release cadence being approximately half that of
Apertis', there are 2 Apertis releases for each Debian stable release and no
guarantees that the kernel's packaged for Debian's current stable or in
progress releases will align with Apertis' requirements.  As a result it is
expected that Apertis will occasionally need to take its kernel from the
mainline kernel LTS tree to satisfy it's requirements.

{{% notice warning %}}
Such packages will require special attention to assure that they remain
up-to-date with any security releases made by the upstream and updated as and
when apropriate.
{{% /notice %}}

Unless the imported package can be used as-is without any modification, there
will be a
[patch series]( {{< ref "buildingpatchingandmaintainingtheapertiskernel.md#packaging" >}} )
that potentially needs tweaking to apply after the update.  The workflow set
out below takes this into consideration.

Below we use the process of importing
[GNU Hello](https://www.gnu.org/software/hello/) application as an example.

- Using the Gitlab Web UI, create an empty project in your personal area, for
  example called `hello`.

- Download the latest release tarball from upstream. At the time of writing
  this is [`hello-2.10.tar.gz`](https://ftp.gnu.org/gnu/hello/) for GNU Hello.

- Extract the tarball and enter the extracted folder

      $ tar xf hello-2.10.tar.gz
      $ cd hello-2.10

- Create a new git repository and commit the whole source tree; this will
  create the `upstream/apertis` branch containing the untouched upstream source
  code:

      $ git init
      $ git checkout -b upstream/apertis
      $ git add .
      $ git commit -s -m "Initial commit of GNU Hello 2.10"

- Commit the original tarball to the `pristine-lfs` branch:

      $ pristine-lfs commit ../hello-2.10.tar.gz

- Create the packaging branch for the Apertis version you're targetting, for
  example if you are targetting Apertis v2021, create the `apertis/v2021`
  branch:

      $ APERTIS_RELEASE=v2021
      $ git checkout -b apertis/${APERTIS_RELEASE}

- Add packaging files and commit them to the packaging branch.

  {{% notice info %}}
  Debian packaging is not covered by this document, users interested in that
  matter should refer to the
  [Debian Packaging Intro](https://wiki.debian.org/Packaging/Intro),
  [Debian Policy Manual](https://www.debian.org/doc/debian-policy/) and
  [Debian Developer's Reference](https://www.debian.org/doc/manuals/developers-reference/).
  {{% /notice %}}

  {{% notice tip %}}
  If you are packaging a newer version of a component which is already
  available in Debian, starting from the existing Debian packaging may save you
  considerable time.
  {{% /notice %}}

- Ensure that the package builds for you locally:

      $ gbp buildpackage -uc -us --git-debian-branch=apertis/${APERTIS_RELEASE} --git-upstream-tree=upstream/apertis

- Add the Gitlab project you created earlier as the `origin` git remote
  (replacing `<username>` with your GitLab username):

      $ GITLAB_USER=<username>
      $ git remote add origin git@gitlab.apertis.org:${GITLAB_USER}/hello.git

- Push your branches to the repository:

      $ git push --all --follow-tags origin

{{% notice info %}}
If you wish to submit this package for inclusion in the Apertis packages, please make yourself familiar with the [contribution policy]({{< ref "contributions.md#adding-components-to-apertis" >}}).
{{% /notice %}}

# Updating Existing Components

Once we have components added to Apertis, it is important that we keep them up
to date with any security updates that are made available. 

## Updating Components from Debian

Updates coming from Debian have been automated using a CI/CD pipeline. Such
updates are usually triggered from the
[infrastructure dashboard](https://infrastructure.pages.apertis.org/dashboard/)
but can be manually triggered from the GitLab web UI.

{{% notice note %}}
This functionality will only be available to users with sufficient GitLab
privileges to enable them to run CI/CD pipelines.
{{% /notice %}}

To trigger the updates pipeline:
- Navigate to the "Pipelines" page of the component in GitLab.
- Click on the `Run pipeline` button in the top right-hand corner.
![Run Pipeline button](/images/run-pipeline-button.png)
- Select the relevant `debian/*` branch that you would like to update (such as
  `debian/bullseye`) under `Run for branch name or tag` and click the `Run
  pipeline` button shown on that page.

The pipeline will check the Debian archive for updates, pull them in to the
`debian/$RELEASE`-like branch (for instance, `debian/bullseye` or
`debian/bullseye-security`), try to merge the new contents with the matching
`apertis/*` branches and, if successful, will create a merge request for
each `apertis/*` branch that requires updating.

## Manually Updating Components from Debian

Upstream updates are usually handled automatically by the
[`ci-package-builder.yml`](https://gitlab.apertis.org/infrastructure/ci-package-builder/)
Continous Integration pipeline, which
[fetches upstream packages, merges them]({{< ref "component_guide.md#updating-components-from-debian" >}})
with the Apertis contents and directly creates Merge Requests to be reviewed by
[maintainers]({{< ref "contributions.md#the-role-of-maintainers" >}}).

{{% notice warning %}}
The
[automated pipeline]({{< ref "component_guide.md#updating-components-from-debian" >}})
should be used to perform this task where possible.

This guide uses tools that are only provided in the package-source-builder
image. The easiest way to perform this task is to utilise the docker container.
{{% /notice %}}

However, in some cases it may be necessary to manually perform these
operations.

- Check out the source repository

      $ GITLAB_GROUP=pkg
      $ PACKAGE=hello
      $ git clone git@gitlab.apertis.org:${GITLAB_GROUP}/${PACKAGE}
      $ cd ${PACKAGE}

  {{% notice note %}}
  When handling packages in your personal GitLab namespace, `GITLAB_GROUP`
  should point to that, which is your GitLab username, rather than the main
  Apertis application group `pkg`.
  {{% /notice %}}

- Ensure we have the required `debian/` and `upstream/` branches locally

      $ CURRENT_UPSTREAM=buster
      $ git branch debian/${CURRENT_UPSTREAM} origin/debian/${CURRENT_UPSTREAM}
      $ git branch upstream/${CURRENT_UPSTREAM} origin/upstream/${CURRENT_UPSTREAM}

- Ensure we have a `pristine-lfs` branch locally:

      $ git branch pristine-lfs origin/pristine-lfs

- Pull in the latest updates Debian from using the `apertis-pkg-pull-updates` script

      $ APERTIS_RELEASE=v2021
      $ MIRROR=https://deb.debian.org/debian
      $ git checkout -b apertis/${APERTIS_RELEASE} origin/apertis/${APERTIS_RELEASE}
      $ apertis-pkg-pull-updates --package=${PACKAGE} --upstream=${CURRENT_UPSTREAM} --mirror=${MIRROR}

  This will update the `debian/...`, `upstream/...` and `pristine-lfs` branches
  as prepared previously.

- Ensure that the changes to these branches are pushed to your remote `origin`.
  It is important that these branches are pushed and in sync with the default
  remote.

      $ git push origin --follow-tags pristine-lfs debian/${CURRENT_UPSTREAM} upstream/${CURRENT_UPSTREAM}

- The upstream changes now need to be merged into the apertis branch. We will
  do this via a development branch that can be pushed for review.

      $ PROPOSED_BRANCH=wip/${GITLAB_GROUP}/${APERTIS_RELEASE}-update
      $ git checkout -b ${PROPOSED_BRANCH} apertis/${APERTIS_RELEASE}
      $ apertis-pkg-merge-updates --upstream=debian/${CURRENT_UPSTREAM} --downstream=${PROPOSED_BRANCH}

- If the merge fails, fix the conflicts and continue

      $ git add ...
      $ git merge --continue

- Create a new changelog entry for the new version

      $ DEBEMAIL="User Name <user@example.com>" dch -D apertis --local +apertis --no-auto-nmu ""

- Check `debian/changelog` and update it if needed by listing all the remaining
  Apertis changes then commit the changelog.

      $ DEBIAN_VERSION=$(dpkg-parsechangelog -S Version)
      $ git commit -s -m "Release ${PACKAGE} version ${DEBIAN_VERSION}" debian/changelog

- Push your changes for review

      $ git push origin ${PROPOSED_BRANCH}:${PROPOSED_BRANCH}

  Follow the instructions provided by the above command to create a merge request

## Updating Components to a New Debian Release

The CI/CD pipeline can also be used to pull in a new release of Debian. 
If the needed `debian/$RELEASE` branch doesn't exist (for example, there is a
`debian/buster` branch but no `debian/bullseye`), the Gitlab Web UI can be used
to create it:
- Navigate to the `Branches` page and click on the `New branch` button.
- Enter the new branch name as `debian/$RELEASE`, where `$RELEASE` is the new
  release name, for example `debian/bullseye`.
- In the `Create from` field select the previous Debian release. For example,
  the release that preceeds Debian Bullseye is Buster, so pick `debian/buster`.
- Click the `Create branch` button.

In some situations manual intervention may be required to tweak the packaging,
patches or the automated merge machinery may ask to `PLEASE SUMMARIZE remaining
Apertis Changes`. When this happens:

- Check out the proposed update branch.
- If asked to summarize the changes, edit the changelog to list **all** the
  downstream changes the package still ships compared to the newly merged
  upstream package and their reason, describing the purpose of each downstream
  patch and of any other change shown by `git diff` against the `debian/*`
  branch.
- Where other issues are found, either modify the packaging data or add/modify
  the Apertis patch series to address the issue.
- Amend the merge commit.
- Force-push to the proposed update branch
- If changes beyond summarizing the changelog are made, the update branch
  should be reviewed before it is merged.

## Manually Updating Components from a Newer Debian Release

There are circumstances, when we deviate from the default upstream. This
usually happens when:

- Packages are not available in the default distribution repository
- Packages in the default distribution repository are outdated
- Newer version of package, available in the non-default repository, is needed

For example, Apertis v2021 ships with a newer version of the Linux kernel
(5.10.x) than Debian Buster (4.9.x) on which it is based, which is the kernel
found in the next Debian release called Bullseye. In such cases, special care
needs to be taken to update packages from their respective upstreams.

{{% notice warning %}}
The
[automated pipeline]({{< ref "component_guide.md#updating-components-from-debian" >}})
should be used to perform this task where possible.

This guide uses tools that are only provided in the package-source-builder
image. The easiest way to perform this task is to utilise the docker container.
{{% /notice %}}

Below are a set of steps which can be adapted to such exception packages. Let
us assume that for such repository, the package was picked from Debian Bullseye
instead of Buster

- Clone the repository:

      $ GITLAB_GROUP=pkg
      $ PACKAGE=hello
      $ git clone git@gitlab.apertis.org:${GITLAB_GROUP}/${PACKAGE}
      $ cd ${PACKAGE}

- If the required `debian/` and `upstream/` branches don't yet exist in your
  repository, create them based on the version that you have for instance. For
  instance, if you currently have a version of Apertis based on Debian Buster
  and wish to use the updated package in Bullseye:

      $ CURRENT_UPSTREAM=buster
      $ DESIRED_UPSTREAM=bullseye
      $ git push origin origin/upstream/${CURRENT_UPSTREAM}:refs/heads/upstream/${DESIRED_UPSTREAM}
      $ git push origin origin/debian/${CURRENT_UPSTREAM}:refs/heads/debian/${DESIRED_UPSTREAM}

- Ensure you have the required branches locally:

      $ DESIRED_UPSTREAM=bullseye
      $ git branch debian/${DESIRED_UPSTREAM} origin/debian/${DESIRED_UPSTREAM}
      $ git branch upstream/${DESIRED_UPSTREAM} origin/upstream/${DESIRED_UPSTREAM}

- Ensure we have a `pristine-lfs` branch locally:

      $ git branch pristine-lfs origin/pristine-lfs

- Pull in new updates using the `apertis-pkg-pull-updates` script, passing in
  the deviated upstream:
  
      $ APERTIS_RELEASE=v2021
      $ MIRROR=https://deb.debian.org/debian
      $ git checkout -b apertis/${APERTIS_RELEASE} origin/apertis/${APERTIS_RELEASE}
      $ apertis-pkg-pull-updates --package ${PACKAGE} --upstream ${DESIRED_UPSTREAM} --mirror ${MIRROR}

  This will update the `debian/...` and `upstream/...` branch as prepared
  previously.

- Ensure that the changes to these branches are pushed to your remote `origin`.
  It is important that these branches are pushed and in sync with the default
  remote.

      $ git push origin --follow-tags pristine-lfs debian/${DESIRED_UPSTREAM} upstream/${DESIRED_UPSTREAM}

- The upstream changes now need to be merged into the apertis branch. We will
  do this via a development branch that can be pushed for review.

      $ PROPOSED_BRANCH=wip/${GITLAB_GROUP}/${APERTIS_RELEASE}-update-from-${DESIRED_UPSTREAM}
      $ git checkout -b ${PROPOSED_BRANCH} apertis/${APERTIS_RELEASE}
      $ apertis-pkg-merge-updates --upstream=debian/${DESIRED_UPSTREAM} --downstream=${PROPOSED_BRANCH}

- If the merge fails, fix the conflicts and continue

      $ git add ...
      $ git merge --continue

- Create a new changelog entry for the new version

      $ DEBEMAIL="User Name <user@example.com>" dch -D apertis --local +apertis --no-auto-nmu ""

- Check `debian/changelog` and update it if needed by listing all the remaining
  Apertis changes then commit the changelog.

      $ DEBIAN_VERSION=$(dpkg-parsechangelog -S Version)
      $ git commit -s -m "Release ${PACKAGE} version ${DEBIAN_VERSION}" debian/changelog

- Push your changes for review

      $ git push origin ${PROPOSED_BRANCH}:${PROPOSED_BRANCH}

  Follow the instructions provided by the above command to create a merge request

## Updating from Debian Development Repositories

This is another scenario, wherein the user may need updates which are not yet released
into the Upstream Distributions' package repositories.

For example, for Apertis, we may need a very newer version of a package, which
may not yet have been released into any of Debian development releases
(Unstable, Testing). In such cases, where the changes may only be available in
the packaging repositories. We need to take extra care when pulling in such
updates as they will not have been through the same levels of testing as
packages taken from stable or even testing releases.

- Clone the remote git packaging repository from Debian.

      $ PACKAGE=hello
      $ DEBIAN_REPO=https://salsa.debian.org/debian/${PACKAGE}.git
      $ DEBIAN_BRANCH=debian/master
      $ git clone -b ${DEBIAN_BRANCH} ${DEBIAN_REPO} ${PACKAGE}-debian
      $ cd ${PACKAGE}-debian

- Generate a source package out of the packaging repository

      $ gbp buildpackage -S --no-sign --git-debian-branch=${DEBIAN_BRANCH}

  If successful, this will give us a proper source packages in the parent
  directory

- Clone the Apertis git packaging repository for the same component

      $ cd ..
      $ GITLAB_GROUP=pkg
      $ git clone git@gitlab.apertis.org:${GITLAB_GROUP}/${PACKAGE}
      $ cd ${PACKAGE}

- If the required `debian/` and `upstream/` branches don't yet exist in your
  repository, create them based on the version that you have for instance. For
  instance, if you currently have a version of Apertis based on Debian Buster
  and wish to use the updated package in Bullseye:

      $ CURRENT_UPSTREAM=buster
      $ DESIRED_UPSTREAM=experimental
      $ git push origin origin/upstream/${CURRENT_UPSTREAM}:refs/heads/upstream/${DESIRED_UPSTREAM}
      $ git push origin origin/debian/${CURRENT_UPSTREAM}:refs/heads/debian/${DESIRED_UPSTREAM}

- Ensure you have the required branches locally:

      $ git branch debian/${DESIRED_UPSTREAM} origin/debian/${DESIRED_UPSTREAM}
      $ git branch upstream/${DESIRED_UPSTREAM} origin/upstream/${DESIRED_UPSTREAM}

- Run `gbp import-dsc` to update the `debian/` and `upstream/` branches.

      $ NEW_DSC=$(ls -t ../*.dsc | head -n 1)
      $ GBP_CONF_FILES=/dev/null gbp import-dsc --author-is-committer --author-date-is-committer-date \
        --upstream-branch=upstream/${DESIRED_UPSTREAM} --debian-branch=debian/${DESIRED_UPSTREAM} \
        --debian-tag='debian/%(version)s' --no-sign-tags --no-pristine-tar ${NEW_DSC}

- Use the `pristine-lfs` tool to import the source package generated from the Debian repository into Apertis packaging repository.

      $ pristine-lfs import-dsc ${NEW_DSC}

- Ensure that the changes to these branches are pushed to your remote `origin`.
  It is important that these branches are pushed and in sync with the default
  remote.

      $ git push origin --follow-tags pristine-lfs debian/${DESIRED_UPSTREAM} upstream/${DESIRED_UPSTREAM}

- The upstream changes now need to be merged into the apertis branch. We will
  do this via a development branch that can be pushed for review.

      $ APERTIS_RELEASE=v2021
      $ PROPOSED_BRANCH=wip/${GITLAB_GROUP}/${APERTIS_RELEASE}-update-from-${DESIRED_UPSTREAM}
      $ git checkout -b ${PROPOSED_BRANCH} apertis/${APERTIS_RELEASE}
      $ apertis-pkg-merge-updates --upstream=debian/${DESIRED_UPSTREAM} --downstream=${PROPOSED_BRANCH}

- If the merge fails, fix the conflicts and continue

      $ git add ...
      $ git merge --continue

- Create a new changelog entry for the new version

      $ DEBEMAIL="User Name <user@example.com>" dch -D apertis --local +apertis --no-auto-nmu ""

- Check `debian/changelog` and update it if needed by listing all the remaining
  Apertis changes then commit the changelog.

      $ DEBIAN_VERSION=$(dpkg-parsechangelog -S Version)
      $ git commit -s -m "Release ${PACKAGE} version ${DEBIAN_VERSION}" debian/changelog

- Push your changes for review

      $ git push origin ${PROPOSED_BRANCH}:${PROPOSED_BRANCH}

  Follow the instructions provided by the above command to create a merge request

# Creating New Modifications

The standard [Contribution Process]({{< ref "contributions.md" >}})
applies unchanged to packaging repositories, the Apertis development team and
trusted contributors push changes to `wip/` branches and get them landed to the
`apertis/*` branches via merge requests. Other developers can fork packaging
repositories into their personal space and send merge requests from there.

The only additional requirement imposed by the Debian packaging format is that
changes outside of the `debian/` folder are not allowed and would cause the
source-building pipeline to fail. Check the [Debian Packaging](https://wiki.debian.org/Packaging)
documentation to find how patches affecting code outside of the `debian/`
folder should be handled.

## The Debian Changelog and Releases

Beyond informing developers and interested users of what has changed between
sucessive releases of a component, the changelog controls the release process
in a number of ways:

- **Package versioning**: The changelog is the canonical location used by the
  Debian packaging process to determine the version number used for the binary
  packages.
- **Release control**: The "distribution" field of a changelog entry controls
  whether the component is ready to be cut as a new release.

{{% notice info %}}
The changelog format is documented in the
[Debian Policy Manual](https://www.debian.org/doc/debian-policy/ch-source.html#s-dpkgchangelog.)
{{% /notice %}}

It is therefore important to correctly handle the changelog file when creating
modifications to components.

The CI pipeline generates a source package locally for each commit pushed
to the packaging repositories. You can download this package by browsing the
pipeline artifacts. The generated sources are versioned to indicate that
they are not yet suitable for release.

With the "distribution" field set to `UNRELEASED`, package sources get uploaded
to the `:snapshots` OBS project with the matching branch (that is, when targetting the `apertis/v2021` branch of a component from the `target` suite, the source package will be uploaded to the `apertis:v2021:target:snaphots` project).

It is advisable to update the `debian/changelog` file in a separate commit, as
the very last step when you issue a release. Such changelog entries should be
generated from the Git commit log. This can be achieved by using `gbp dch`.
Writing good commit messages ensures that you don’t have to spend time editing
the generated changelog entries.

It is best to submit a separate merge request on GitLab for each bug or task.
As performing a separate release between each merge request may not be wanted
and to avoid churn in the case of rebases (in particular to ease the review
process), it is recommended to leave the editing of `debian/changelog` to a
dedicated merge request once all the other MRs have been landed.

{{% notice tip %}}
In order to follow a 
[release early, release often philosophy]({{< ref "contributions.md#upstream-early-upstream-often" >}})
it is
also recommended to avoid delaying release commits to include additional
features. This gives the possibility to receive early feedback on the
downstream changes.
{{% /notice %}}

If you still wish to edit `debian/changelog` as you go along for some reason,
make sure that the changelog entry you're writing has the `distribution` field
set to `UNRELEASED` and use `gbp dch --auto --ignore-branch` to ensure the
formatting of the changelog is correct.

## Landing Downstream Changes to the Main Archive

Once downstream changes to a package are deemed ready to be published in the
Apertis main archive, a proper release needs to be issued. To achieve this create a merge request containing an update to the changelog. The description should contain details of all the changes since the last release. This can be easily achieved with `gbp dch` if the commits have been well written:

    $ APERTIS_RELEASE=v2021
    $GBP_CONF_FILES=/dev/null gbp dch --release -D apertis --debian-branch=apertis/${APERTIS_RELEASE}

Ensure that the "distribution" field has been changed from `UNRELEASED` to
`apertis`.

If making changes to more than one release, start with the most recent release
branch where you want to land your changes. For instance, if you want to land
changes to both the development and stable releases, submit a merge request for
the development one first (for instance, `apertis/v2020dev0`) and then, once
merged, create a merge request for the stable one (for instance,
`apertis/v2019-updates`)

Once a merge request has been reviewed and landed, the CI pipeline will
build-check the source package as usual. Since the `distribution` field is no
longer `UNRELEASED` it will also:
* add a Git tag for the release version to the repository
* rebuild the release source package
* store the release sources in the `pristine-lfs-source` branch
* upload the release source package to the main project on OBS (for instance
  `apertis:v2020dev0:target`)

{{% notice warning %}}
Published stable release branches (for instance `apertis/v2019`) should
**never** be targeted directly. Updates to these should
[go through the `-security` or `-updates` branches]({{< ref "release-flow.md#process-after-a-product-release" >}})
(for example `apertis/v2019-security` or `apertis/v2019-updates).
{{% /notice %}}

If the `apertis/$RELEASE-updates` or `apertis/$RELEASE-security` branches for
published stable releases do not exist yet, they should be created from the
GitLab web UI since their protected status makes pushing forbidden.

For trivial changes it is also possible to combine the release commit in the
same MR as the changes. Again, developers need to be careful to ensure the
changelog entries are kept up-to-date when the commit messages get changed via
rebase.

## Backporting Updates or Security Fixes

Often downstream fixes, upstream updates or security fixes need to be applied
to [multiple active releases]({{< ref "release-flow.md" >}}). Changes should be
introduced in the most recent development release where they can be tested and
regressions detected with little impact. Once the changes have been thoroughly
tested, paying close attention to the QA test results, they can then be
propagated to the more stable releases, where any mistake can impact the
product teams using Apertis in the field.

For instance, once a fix is landed to `apertis/v2021dev0` and no regressions
are found in the subsequent QA test results, an MR should be create to land
the changes to the relevant stable releases.

If there is no divergence between the packages in the different releases and
the backport can be done with a fast-forward, an MR should be created to submit
the changes from for instance `apertis/v2021dev0` to `apertis/v2020-updates` or
`apertis/v2020-security`. Choose the required destination depending on the nature and impact of the fix.

If package diverged across releases, a separate branch has to be created where
the fixes are cherry-picked appropriately before creating the MR.

{{% notice warning %}}
Care needs to be taken to give these diverging release branches suitable
version numbers.
{{% /notice %}}

## Diverging Release Branches

Sometimes different downstream patches need to be applied in the different
Apertis release branches. A clear case of that is the `base-files` package
which ships the release name in `/etc/os-release`.

In such situation it is crucial to use different version identifiers in each
branch: the version for a given package needs to be globally unique across the whole
archive since uploading different package sources with the same name/version
would lead to difficult to diagnose errors.

When targeting a specific release, `~${RELEASE}.${COUNTER}` needs to be
appended to the version identifier after the local build suffix:

| Previous Version | New release specific version (for `v2020`) | Description |
| --- | --- | --- |
| `0.42+apertis0` | `0.42+apertis0~v2020.0` | Increment Apertis release, append release specific identifier |
| `0.42+apertis0~v2020.0` | `0.42+apertis4~v2020.1` | Increment the release specific counter |

This uses the fact that `~` in Debian package numbers
[sorts before anything](https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version).
Adding `~` is necessary so that if a new upstream version `0.42.1` or a new
non-release-specific downstream version `0.42+apertis4` is introduced, they
will replace the release-specific package.

Note that `dpkg` considers `2020.0` to be newer than `2020pre.0`, so the
Apertis release identifiers can be used with no modification. If in doubt,
check with `dpkg --compare-versions`:

    $ dpkg --compare-versions 0.42+apertis0~2020pre.0 '<<' 0.42+apertis0~2020.0 && echo true || echo false
    true
    $ dpkg --compare-versions 0.42+apertis1 '<<' 0.42+apertis0~2020.0 && echo true || echo false
    false

# Advanced Configuration

## License Scans

As merge requests to components are submitted, the CI pipeline performs license
scans on the package. The scans are performed on all files in the package, not
just the new submission. The pipeline fails or emits a warning (depending on
the configuration) if it finds files with unknown or unclear licensing terms,
or files under licenses not allowed in the package. When such a situation
arises, it is the responsibility of the submitter to perform the review of the
license scan results and make updates to the package if necessary.

When the license scan mistakenly identifies a file as being under an incorrect
license, or fails to process it correctly, there are three ways to fix this:

1. Specify the correct copyright and the license in
   `debian/apertis/copyright.yml`.  The format of the file is specified in the
   [Dpkg::Copyright::Scanner](https://manpages.debian.org/buster/libconfig-model-dpkg-perl/Dpkg::Copyright::Scanner.3pm.en.html)
   manpage.  In short, it’s a YAML file mapping paths to their licensing
   information:

   ``` yaml
   debian:
     copyright: 2015, Marcel
     license: Expat
   src/:
     copyright: 2016, Joe
     license: Expat
   .*/NOTICE:
     skip: 1
   src/garbled/:
     'override-copyright': 2016 Marcel MeXzigue
    ```

   Patterns follow the Perl regular expression rules.

   Please also verify `debian/copyright` specifies the correct license, and if
   it doesn’t, submit a patch to Debian.

2. Add the file to the list of ignored files,
   `debian/apertis/copyright.whitelist`. This file is formatted the same way as
   `gitignore`, please refer to the
   [gitignore](https://manpages.debian.org/buster/git-man/gitignore.5.en.html)
   manpage for more information.

3. If the file is under a license not suitable for Apertis, it can be removed
   from the package by either repackaging the tarball or patching it out, in
   which case the scanner will not take it into account.

The license scanner will store the automatically generated copyright report
file under `debian/apertis/copyright`, updating the merge request when
necessary.

## Custom Pipelines

When using the packaging pipeline, developers cannot put their CI/CD automation
in `.gitlab-ci.yml` anymore, as the CI config path points to the
ci-package-builder definition.

However, developers can put their jobs in the
`debian/apertis/local-gitlab-ci.yml` file and have them executed in a child
pipeline whenever the main packaging pipeline is executed. This is specially
handy to run tests before the actual packaging process begins.