apertis_packaging_guide.md

+++
date = "2020-07-24"
lastmod = "2021-04-22"
weight = 100

title = "Apertis Development Guide"
+++

# Apertis Packaging CI

Apertis stores the source of all the shipped packages in GitLab and uses a
[GitLab CI pipeline](https://gitlab.apertis.org/infrastructure/ci-package-builder/-/blob/master/ci-package-builder.yml)
to manage the workflows to:

* land updated sources to OBS which will then build the binary outputs
* pull updates from upstream distributions like Debian 10 Buster

## Adding downstream changes

The standard [Contribution Process]({{< ref "contributions.md" >}})
applies unchanged to packaging repositories, the Apertis development team and
trusted contributors push changes to `wip/` branches and get them landed to the
`apertis/*` branches via Merge Requests. Other developers can fork packaging
repositories into their personal space and send Merge Requests from there.

The only additional requirement imposed by the Debian packaging format is that
changes outside of the `debian/` folder are not allowed and would cause the
source-building pipeline to fail. Check the [Debian Packaging](https://wiki.debian.org/Packaging)
documentation to find how patches affecting code outside of the `debian/`
folder should be handled.

Updating `debian/changelog` should be done separately as the last step when
issuing a release, generating the changelog entries from the Git commit log,
which makes writing good commit log messages even more important.

A merge request should be submitted on GitLab for each bug or task. To ease
the review process, in particular to avoid churn in the case of rebases, it is
recommended to leave the editing of `debian/changelog` to a dedicated merge
request once all the other MRs have been landed, see the
[section about landing downstream changes to the main
archive](#landing-downstream-changes-to-the-main-archive) below.

{{% notice tip %}}
In order to follow a 
[release early, release often philosophy]({{< ref "contributions.md#upstream-early-upstream-often" >}})
it is
also recommended to avoid delaying release commits to include additional
features. This gives the possibility to receive early feedback on the
downstream changes.
{{% /notice %}}

If you still wish to edit `debian/changelog` for any reason, just make sure
that the changelog entry you're writing has the `distribution` field set
to `UNRELEASED`, using `gbp dch --auto --ignore-branch` to ensure the
formatting is correct.

The CI pipeline will locally generate a source package for each commit pushed
to the packaging repositories, which can be retrieved by browsing the
pipeline artifacts. The generated sources will be versioned to indicate that
they are not yet suitable for release.

With the `distribution` field set to `UNRELEASED` package sources get uploaded
to the `:snapshots` OBS project matching the branch (that is,
`apertis:v2020dev0:target:snaphots` when landing changes to the
`apertis/v2020dev0` branch of a `:target` package): the following [section
about landing downstream changes to the main
archive](#landing-downstream-changes-to-the-main-archive) below describes
in detail how to set the `distribution` to land the package to the appropriate
main OBS project.

## Landing downstream changes to the main archive

Once downstream changes to a package are deemed ready to be published in the
Apertis main archive, a proper release needs to be issued.

* Push a `wip/` branch updating `debian/changelog`
  * use `GBP_CONF_FILES=/dev/null gbp dch --release -D apertis --debian-branch=apertis/v2020dev0`
    to generate a release changelog entry summarizing all the changes already
    landed on the `apertis/v2020dev0` branch
  * ensure that the `distribution` field has been changed from `UNRELEASED`
    to `apertis`
* Create a Merge Request based on your `wip/` branch for the most recent
  release branch where you want to land your changes:
  * for published stable release, the main branch (for instance
    `apertis/v2019`) should **never** be targeted directly but updates and
    fixes should go through the `apertis/v2019-security` or
    `apertis/v2019-updates` branches, see [Process after a product
    release]( {{< ref "release-flow.md#process-after-a-product-release" >}} )
    for more details
  * for instance, if you want to land changes to both the development and
    stable releases, push your `wip/` source branch and create a MR for the
    development one first (for instance, `apertis/v2020dev0`) and then, once
    merged, create a MR for the stable one ((for instance, `apertis/v2019-updates`)
* Get the Merge Request reviewed and landed
* The CI pipeline will then build-check the source package as usual and since
  the `distribution` field is no longer `UNRELEASED` it will also:
  * add a Git tag for the release version to the repository
  * rebuild the release source package
  * store the release sources in the `pristine-lfs-source` branch
  * upload the release source package to the main project (for instance
    `apertis:v2020dev0:target`)

If the `apertis/$RELEASE-updates` or `apertis/$RELEASE-security` branches for
published stable releases do not exist yet, they should be created from the
GitLab web UI since their protected status makes pushing forbidden.

For trivial changes it is also possible to combine the release commit in the same
MR as the changes. Again, developers need to be careful to ensure the changelog
entries are kept up-to-date when the commit messages get changed via rebase.

## Pulling updates or security fixes from upstream distributions

Updates coming from upstream can be pulled it by triggering a CI pipeline on a
branches like `debian/buster` or `debian/bullseye`.

The pipeline will check the Debian archive for updates, pull them in the
`debian/$RELEASE`-like branch (for instance, `debian/bullseye` or
`debian/bullseye-security`), try to merge the new contents with the matching
`apertis/*` branches and, if successful, push a
proposed updates branch while creating a Merge Request for each `apertis/*`
branches it should be landed on.

The upstream update pipeline is usually triggered from
[the infrastructure dashboard](https://infrastructure.pages.apertis.org/dashboard/)
but can be manually triggered from the GitLab web UI by selecting the
`Run Pipeline` button in the `Pipelines` page of each repository under `pkg/*`
and selecting the `debian/bullseye` branch as the reference.

![Run Pipeline button](/images/run-pipeline-button.png)

Reviewers can then force-push to the proposed update branch in the Merge
Request to fixup any issue caused by the automated merge, and ultimately land
the MRs to the `apertis/*` branches.

In some situations the automated merge machinery may ask to
`PLEASE SUMMARIZE remaining Apertis Changes`, and in that case
reviewers should:
* check out the proposed update branch
* edit the changelog to list **all** the downstream changes the package still
  ships compared to the newly merged upstream package and their reason,
  describing the purpose of each downstream patch and of any other change
  shown by `git diff` against the `debian/*` branch
* amend the merge commit
* force-push to the proposed update branch
* land the associated Merge Requests as described above

Remember to check that the updated package gets included in the next daily
reference image build and wait for its [QA test
results](https://lavaphabbridge.apertis.org/) to catch regressions timely
and act accordingly.

## Backporting updates or security fixes

Often downstream fixes, upstream updates or security fixes need to be applied
to [multiple active releases]( {{< ref "release-flow.md" >}} ).

Changes should be introduced in the most recent development release where they can
be tested and regressions detected with little impact, following the instructions
in the
[Landing downstream changes to the main archive](#landing-downstream-changes-to-the-main-archive)
and [Pulling updates or security fixes from upstream distributions](#pulling-updates-or-security-fixes-from-upstream-distributions)
sections.

Once the changes have been thoroughly tested paying close attention to the QA
test results, they can then be propagated to the more stable releases, where
any mistake can impact the product teams using Apertis in the field.

For instance, once a fix is landed to `apertis/v2021dev0` and no regressions
are found in the subsequent QA test results, a MR should be create to land
the changes to the stable releases.

If there is no divergence between the packages in the different releases and
the backport can be done with a fast-forward, a MR should be created to submit
the changes from for instance `apertis/v2021dev0` to `apertis/v2020-updates` or
`apertis/v2020-security`, following the
[Landing downstream changes to the main archive](#landing-downstream-changes-to-the-main-archive)
steps, choosing the destination depending on the nature and impact of the fix.

If package diverged across releases, a separate branch has to be created where
the fixes are cherry-picked appropriately before creating the MR. See the
[Diverging release branches](#diverging-release-branches) section for further
details about versioning divergent packages.

## Diverging release branches

Sometimes different downstream patches need to be applied in the different
Apertis release branches. A clear case of that is the `base-files` package
which ships the release name in `/etc/os-release`.

In such situation it is crucial to use different version identifiers in each
branch: the version for a given package needs to be globally unique across the whole
archive since uploading different package sources with the same name/version
would lead to difficult to diagnose errors.

When targeting a specific release, `~${RELEASE}.${COUNTER}` needs to be
appended to the version identifier after the local build suffix:

* `0.42` → append `co0~v2020pre.0` → `0.42co0~v2020pre.0`
* `0.42co3` → bump to `co4` and append `~v2020pre.0`→ `0.42co4~v2020pre.0`
* `0.42co4~v2020pre.0` → increase the release-specific counter → `0.42co4~v2020pre.1`

This uses the fact that `~` in Debian package numbers sorts before anything,
see the [Debian Policy §5.6.12](https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version) for more details.
Adding `~` is necessary so that if a new upstream version `0.42.1` or a new
non-release-specific downstream version `0.42co4` is introduced, they will
replace the release-specific package.

Note that `dpkg` considers `2020.0` to be newer than `2020pre.0`, so the
Apertis release identifiers can be used with no modification (if in doubt,
check with `dpkg --compare-versions 2020pre.0 '<<' 2020.0 && echo ok`).

## Adding new packages from Debian

This is the process to import a new package from Debian to Apertis:

* create a folder, in the name of the package to import.
  * Eg. for package `hello`, run: `mkdir hello`
  * chdir to created folder: `cd hello`
* invoke `import-debian-package` from the [packaging-tools
  repository:](https://gitlab.apertis.org/infrastructure/packaging-tools/)
  * fetch a specific version: `import-debian-package --upstream buster --downstream apertis/v2020dev0 --component target --package hello --version 2.10-2`
  * fetch the latest version: `import-debian-package --upstream buster --downstream apertis/v2020dev0 --component target --package hello`
  * the argument to `--component` reflects the repository component it is part of (for instance, `target`); it will be stored in `debian/apertis/component`
  * multiple downstream branches can be specified, in which case all of them
    will be updated to point to the newly imported package version
  * the Apertis version of the package will have a local suffix (`co0`) appended
  * don't use `import-debian-package` on existing repositories, it does not
    attempt to merge `apertis/*` branches and instead it re-sets them to new
    branches based on the freshly imported Debian package
* create an empty project on GitLab under the `pkg/*` namespaces (for instance, `pkg/hello`)
* configure the origin remote on your local git: `git remote add origin git@gitlab.apertis.org:pkg/hello`
* push your local git contents to the newly created GitLab project: `git push --all --follow-tags origin`
* set it up with `gitlab-rulez apply rulez.yaml --filter pkg/hello` from
  the [gitlab-rulez repository](https://gitlab.apertis.org/infrastructure/gitlab-rulez)
  * sets the CI config path to `ci-package-builder.yml@infrastructure/ci-package-builder`
  * changes the merge request settings:
      * only allow fast-forward merges
      * ensure merges are only allowed if pipelines succeed
  * marks the `apertis/*` and `debian/*` branches as protected
* follow the process described in the [section about landing downstream changes
  to the main archive](#landing-downstream-changes-to-the-main-archive) above to
  publish the package on OBS.

## Adding updates from a non-default upstream repository of a distribution

There are circumstances, when we deviate from the default upstream. This usually happens
when:
* Packages are not available in the default distribution repository
* Packages in the default distribution repository are outdated
* Newer version of package, available in the non-default repository, is needed

For example, Apertis v2020 ships with a newer version of the Linux kernel
(5.4.x) than Debian Buster (4.9.x) on which it is based. In such cases, special
care needs to be taken to update packages from their respective upstreams.

Below are a set of steps which can be adapted to such exception packages. Let us assume that for
such repository, the package was picked from Debian Unstable instead

* Clone your repository

        $ git clone git@gitlab.apertis.org:ritesh/libgpiod.git
        Cloning into 'libgpiod'...
        remote: Enumerating objects: 114, done.
        remote: Counting objects: 100% (114/114), done.
        remote: Compressing objects: 100% (85/85), done.
        remote: Total 114 (delta 18), reused 110 (delta 18), pack-reused 0
        Receiving objects: 100% (114/114), 110.99 KiB | 360.00 KiB/s, done.
        Resolving deltas: 100% (18/18), done.

        $ cd libgpiod/

* Ensure you have a branch against your deviated upstream. If you are tracking changes from a deviated upstream like **Debian Unstable**, it needs to be ensured that the package's packaging repository has the corresponding branches available. This is needed because the automated machinery tools expect respective branches to be available.
  * For example, if you picked a package from Debian Unstable, ensure to have a branch named `debian/unstable` in your git repository.

        $ git checkout -b debian/unstable origin/debian/buster
        Branch 'debian/unstable' set up to track remote branch 'debian/buster' from 'origin'.
        Switched to a new branch 'debian/unstable'

        $ git checkout debian/buster
        Branch 'debian/buster' set up to track remote branch 'debian/buster' from 'origin'.
        Switched to a new branch 'debian/buster'

  * Similarly, ensure to have a branch named `upstream/unstable` in your git repository.

        $ git checkout upstream/buster
        Branch 'upstream/buster' set up to track remote branch 'upstream/buster' from 'origin'.
        Switched to a new branch 'upstream/buster'

        $ git checkout -b upstream/unstable upstream/buster
        Switched to a new branch 'upstream/unstable'

* Ensure that these branches are pushed to your remote `origin`. It is important that these branches are pushed and in sync with the default remote.

        $ git push -u origin --all
        Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
        remote:
        remote: To create a merge request for debian/unstable, visit:
        remote:   https://gitlab.apertis.org/ritesh/libgpiod/-/merge_requests/new?merge_request%5Bsource_branch%5D=debian%2Funstable
        remote:
        To gitlab.apertis.org:ritesh/libgpiod.git
         * [new branch]      debian/unstable -> debian/unstable
        Branch 'apertis/v2019' set up to track remote branch 'apertis/v2019' from 'origin'.
        Branch 'debian/buster' set up to track remote branch 'debian/buster' from 'origin'.
        Branch 'upstream/buster' set up to track remote branch 'upstream/buster' from 'origin'.
        Branch 'debian/unstable' set up to track remote branch 'debian/unstable' from 'origin'.

* Pull in new updates using the `apertis-pkg-pull-updates` script, instructing it with the deviated upstream
  * Eg. `apertis-pkg-pull-updates --package PKGNAME --upstream unstable --mirror http://deb.debian.org/debian`

        $ git checkout apertis/v2021dev2
        Switched to a new branch 'apertis/v2021dev2'

        $ ../apertis-pkg-pull-updates --package libgpiod --upstream unstable --mirror http://deb.debian.org/debian
        source package libgpiod
        running git branch --track -f debian/unstable origin/debian/unstable
        Branch 'debian/unstable' set up to track remote branch 'debian/unstable' from 'origin'.
        running git branch --track -f upstream/unstable origin/upstream/unstable
        running git branch --track -f upstream/unstable origin/upstream/unstable
        running git branch --track -f upstream/unstable origin/upstream/unstable
        local version: 1.2-3
        fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable-security
        local version: 1.2-3
        fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable-proposed-updates
        local version: 1.2-3
        fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable
        remote version: 1.4.1-4
        update to 1.4.1-4
        fetch https://snapshot.debian.org/mr/package/libgpiod/1.4.1-4/srcfiles?fileinfo=1
        download http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
        running dget --download-only --allow-unauthenticated http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
        dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
          % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                         Dload  Upload   Total   Spent    Left  Speed
        100   332  100   332    0     0    449      0 --:--:-- --:--:-- --:--:--   448
        100  2294  100  2294    0     0   1381      0  0:00:01  0:00:01 --:--:--  5474
        dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1.orig.tar.xz
          % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                         Dload  Upload   Total   Spent    Left  Speed
        100   338  100   338    0     0    637      0 --:--:-- --:--:-- --:--:--   636
        100  307k  100  307k    0     0   358k      0 --:--:-- --:--:-- --:--:--  358k
        dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.debian.tar.xz
          % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                         Dload  Upload   Total   Spent    Left  Speed
        100   342  100   342    0     0    677      0 --:--:-- --:--:-- --:--:--   677
        100  6132  100  6132    0     0   9922      0 --:--:-- --:--:-- --:--:--  9922
        Moving branch debian/unstable to debian/, was: 713dadc
        running gbp import-dsc /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1-4.dsc --author-is-committer --author-date-is-committer-date --upstream-branch=upstream/unstable --debian-branch=debian/unstable '--debian-tag=debian/%(version)s' --no-sign-tags --no-pristine-tar
        gbp:info: Version '1.4.1-4' imported under '/home/rrs/NoBackup/Gitlab_Packages/packaging-tools/libgpiod'
        running ./import-tarballs /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1-4.dsc
        Importing /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1.orig.tar.xz


## Adding updates from distribution development repositories

This is another scenario, wherein the user may need updates which are not yet released
into the Upstream Distributions' repositories.

For example, for Apertis, we may need a very newer version of libgpiod, which may not yet
have been released into any of Debian development releases (Unstable, Testing). In
such cases, where the changes may only be available in the packaging repositories, we
need to take extra care when pulling in such updates.

Let us assume libgpiod 1.4.2 has been available in Debian's libgpiod Packaging repository but
is not released into any of the Debian releases. In such case, we can try:

* Clone the remote libgpiod git packaging repository from Debian.
* Generate a source package out of the packaging repository using `gbp buildpackage -S`
  * If successful, this will give us a proper *libgpiod source package*.
* Clone the Apertis libgpiod git packaging repostiory
  * Use the `pristine-lfs` tool to import the source package generated from the Debian repository into Apertis packaging repository. Eg. `pristine-lfs import-dsc libgpiod-1.4.2-1.dsc`
  * Note: The `import-dsc` subcommand imports the new tarball into the git repository and commits it to the `pristine-lfs` branch. While a user can commit to the branch manually by-hand, we recommend the use of `import-dsc` to import new tarballs and committing them to the packaging repository

## Maintaining package from upstream sources

There are likely to be instances where it is desirable to import the latest version of a piece of software into Apertis directly from it's original authors or maintainers.
This may be as a result of the software in question not being packaged by Apertis' default upstream distribution, Debian, or their being a mismatch between the desired version in the upstream distribution and what is required for a specific goal.

The [Apertis release flow]( {{< ref "release-flow.md#apertis-release-flow" >}} ) stipulates that each Apertis release should include the latest mainline kernel LTS release.
Due to Debian's release cadence being approximately half that of Apertis', there are 2 Apertis' releases for each Debian stable release and no guarantees that the kernel's packaged for Debian's current stable or in progress releases will align with Apertis' requirements.
As a result Apertis will need to pull from the mainline kernel LTS tree to satisfy it's requirements.

Such packages will require special attention to assure that they remain up-to-date with any security releases made by the upstream and updated as and when apropriate.

Unless the imported package can be used as-is without any modification, there will be a [patch series]( {{< ref "buildingpatchingandmaintainingtheapertiskernel.md#packaging" >}} ) that potentially needs tweaking to apply after the update.
The workflow set out below takes this into consideration.

Below we use the process of updating the Linux kernel for `v2020` as an example, where we track the `linux-5.4.y` stable branch:

- Using the GitLab web UI, check to ensure that the relevant update branch exists, in the case of the Apertis `v2020` release, kernel updates should be made on the `apertis/v2020-security` branch. Create the required branch if it doesn't exist.
- Clone the existing package from Apertis GitLab and checkout the relevant branch:

      $ git clone -b apertis/v2020-security git@gitlab.apertis.org:pkg/target/linux.git linux-apertis

- Separately clone the Linux kernel LTS repository and checkout the relvant stable branch:

      $ git clone -b stable-linux-5.4.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux-stable

- Create a `gbp pq` patch branch and switch back to normal branch. The patch branch will be needed when rebasing the Debian patches:

      $ cd linux-apertis
      $ gbp pq import
      $ gbp pq switch

- Determine the latest stable release and derive the Apertis version to use. To ensure collisions don't occur with any future Debian releases, we will mark the release as the zeroth "Debian" release (`-0`) and first Apertis pre-release, as applied by Collabora (`~co1`):

      $ RELEASE=`git -C ../linux-stable describe`
      $ RELEASE="${RELEASE#v}-0~co1"

- Add a changelog entry for newest `linux-5.4.y` kernel release (this is needed to get `genorig.py` to use the right release):

      $ DEBEMAIL="User Name <user@example.com>" dch -v "$RELEASE" ""
      $ git add -f debian/changelog
      $ git commit -s -m "Add changelog entry for update to $RELEASE"

- Run `debian/bin/genorig.py` to generate debianised kernel source tarball:

      $ debian/bin/genorig.py ../linux-stable

- Import the updated source tarball:

      $ gbp import-orig --upstream-branch=upstream/linux-5.4.y --debian-branch=apertis/v2020-security <path to orig tarball>

- Rebase the Debian patches on the new kernel version:

      $ gbp pq rebase

  This may require some manual intervention following the normal git rebasing process.

- Export updated patch series:

      $ gbp pq export

- Tweak `debian/patches/series` file so that it retains the comments that `gbp pq` wants to drop. If no patches have been dropped during rebase then:

      $ git checkout debian/patches/series
      $ git add -f debian/patches
      $ git commit -s -m "Update the debian patches for $RELEASE update"

- Update changelog to released state (set distribution to apertis), document any patches that have been dropped as a result of the update and create new commit:

      $ dch -D apertis
      $ git add -f debian/changelog
      $ git commit -s -m "Release linux version $RELEASE"

- Run `dpkg-buildpackage` to generate the debian packaging, this is needed to use the tools to update the [pristine-lfs branch]( {{< ref "gitlab-based_packaging_workflow.md" >}} ):

      $ mkdir ../build
      $ gbp buildpackage --git-ignore-new \
          --git-debian-branch=apertis/v2020-security \
          --git-prebuild='debian/rules debian/control || true' \
          --git-export-dir='../build' \
          -us -S

- Import the generated kernel tarball into pristine-lfs branch with `import-tarballs`:

      $ git clone git@gitlab.apertis.org:infrastructure/packaging-tools.git ../packaging-tools
      $ ../packaging-tools/import-tarballs ../build/linux_5.4.51-0~co1.dsc
      $ git push origin pristine-lfs

- Push kernel update to a branch to be reviewed:

      $ git push origin apertis/v2020-security:wip/user/kernel-update

## License scans

As merge requests to packaged software are submitted, the CI pipeline performs
license scans on the package. The scans are performed on all files in the package,
not just the new submission. The pipeline fails or emits a warning (depending on
the configuration) when if finds files with unknown or unclear licensing terms,
or files under licenses not allowed in the package. When such situation arises,
it is the responsibility of the submitter to perform the review of the license
scan results and make updates to the package if necessary.

When the license scan mistakenly identifies a file as being under an incorrect
license, or fails to process it correctly, there are three ways to fix this:

 1. Specify the correct copyright and the license in `debian/apertis/copyright.yml`.
    The format of the file is specified in the
    [Dpkg::Copyright::Scanner](https://manpages.debian.org/buster/libconfig-model-dpkg-perl/Dpkg::Copyright::Scanner.3pm.en.html)
    manpage.
    In short, it’s a YAML file mapping paths to their licensing information:

        debian:
          copyright: 2015, Marcel
          license: Expat
        src/:
          copyright: 2016, Joe
          license: Expat
        .*/NOTICE:
          skip: 1
        src/garbled/:
          'override-copyright': 2016 Marcel MeXzigue

    Patterns follow the Perl regular expression rules.

    Please also verify `debian/copyright` specifies the correct license, and if it
    doesn’t, submit a patch to Debian.

 2. Add the file to the list of ignored files.
    `debian/apertis/copyright.whitelist` is formatted the same way as `gitignore`,
    please refer to the [gitignore](https://manpages.debian.org/buster/git-man/gitignore.5.en.html)
    manpage for more information.

 3. If the file is under a license not suitable for Apertis, it can be removed from
    the package by either repackaging the tarball or patching it out, in which case
    the scanner will not take it into account.

The license scanner will store the automatically generated copyright report file
under `debian/apertis/copyright`, updating the merge request when necessary.

[Dpkg::Copyright::Scanner]: https://manpages.debian.org/testing/libconfig-model-dpkg-perl/Dpkg::Copyright::Scanner.3pm.en.html
[gitignore]: https://manpages.debian.org/testing/git-man/gitignore.5.en.html

## Custom pipelines

When using the packaging pipeline, developers cannot put their CI/CD automation
in `.gitlab-ci.yml` anymore, as the CI config path points to the
ci-package-builder definition.

However, developers can put their jobs in the
`debian/apertis/local-gitlab-ci.yml` file and have them executed in a child
pipeline whenever the main packaging pipeline is executed. This is specially
handy to run tests before the actual packaging process begins.


# Internals

Main components:
* [`ci-package-builder`](https://gitlab.apertis.org/infrastructure/ci-package-builder):
  centralized location of the GitLab-to-OBS and Debian-to-GitLab pipeline definitions
* [`apertis-package-source-builder`](https://gitlab.apertis.org/infrastructure/apertis-docker-images/tree/apertis/v2019/apertis-package-source-builder):
  Docker environment for the GitLab pipelines
* [`pristine-lfs`](https://salsa.debian.org/andrewsh/pristine-lfs): stores
  upstream original tarballs and packaging source tarballs using Git-LFS, as a
  more robust replacement for `pristine-tar`

![DEP-14 in Apertis](/images/apertis-dep-14-gitlab-curves.svg)

Branches:
* `pristine-lfs`: stores references to the Git-LFS-hosted original tarballs
* `debian/$DEBIAN_RELEASE` (for instance, `debian/buster`): contains the extracted
  upstream sources and packaging information from Debian
* `pristine-lfs-source`: stores references to the Git-LFS-hosted packaging
  tarballs, mainly to ensure that each (package, version) tuple is built only
  once and no conflicts can arise
* `apertis/$APERTIS_RELEASE` (for instance, `apertis/v2020dev0`): contains the
  extracted upstream sources and possibly patched packaging information for
  Apertis, including the `debian/apertis/gitlab-ci.yaml` to set up the
  GitLab-to-OBS pipeline
* `apertis/$APERTIS_RELEASE-security` and `apertis/$APERTIS_RELEASE-updates`
  (for intance, `apertis/v2019-updates`): similar to `apertis/$APERTIS_RELEASE`
  but respectively target the Security and Updates repositories for published
  stable releases as described in [Process after a product
  release]( {{< ref "release-flow.md#process-after-a-product-release" >}} )

Tags:
* `debian/*`: tags for Debian releases in the `debian/*` branches
* `apertis/*`:  tags for the Apertis releases in the `apertis/*` branches

Git:

Gitattributes is a feature that enables modifications to be automatically made by git when certain
operations are done (such as `checkout`, `check-in` and `clone`) to enforce certain project standards.
but this has the potential effect of modifying files and that's not what we want when packaging. More
details about [gitattributes](https://git-scm.com/docs/gitattributes).

* Many upstream projects ship a `.gitattributes` file in their source repositories, which are
  tightly tied to their development workflows. At times, certain settings in `.gitattributes`
  can lead to problems in our packaging workflow, as summarized above. Thus, it is advised to execute
  the following command, which will override any such inherited settings from the
  `.gitattributes` file:

  ```
  echo "* -text -eol -crlf -ident -filter -working-tree-encoding -export-subst" > .git/info/attributes
  ```
  * This will override the settings in the `.gitattributes` file from the repository, for all files and for the mentioned features