Newer
Older
lastmod = "2021-04-29"
Apertis stores the source of all the shipped packages in GitLab and uses a
[GitLab CI pipeline](https://gitlab.apertis.org/infrastructure/ci-package-builder/-/blob/master/ci-package-builder.yml)
to manage the workflows to:
* land updated sources to OBS which will then build the binary outputs
* pull updates from upstream distributions like Debian 10 Buster
## Adding downstream changes
The standard [Contribution Process]({{< ref "contributions.md" >}})
applies unchanged to packaging repositories, the Apertis development team and
trusted contributors push changes to `wip/` branches and get them landed to the
`apertis/*` branches via Merge Requests. Other developers can fork packaging
repositories into their personal space and send Merge Requests from there.
The only additional requirement imposed by the Debian packaging format is that
changes outside of the `debian/` folder are not allowed and would cause the
source-building pipeline to fail. Check the [Debian Packaging](https://wiki.debian.org/Packaging)
documentation to find how patches affecting code outside of the `debian/`
folder should be handled.
Updating `debian/changelog` should be done separately as the last step when
issuing a release, generating the changelog entries from the Git commit log,
which makes writing good commit log messages even more important.
A merge request should be submitted on GitLab for each bug or task. To ease
the review process, in particular to avoid churn in the case of rebases, it is
recommended to leave the editing of `debian/changelog` to a dedicated merge
request once all the other MRs have been landed, see the
[section about landing downstream changes to the main
archive](#landing-downstream-changes-to-the-main-archive) below.
{{% notice tip %}}
In order to follow a
[release early, release often philosophy]({{< ref "contributions.md#upstream-early-upstream-often" >}})
it is
also recommended to avoid delaying release commits to include additional
features. This gives the possibility to receive early feedback on the
downstream changes.
{{% /notice %}}
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
If you still wish to edit `debian/changelog` for any reason, just make sure
that the changelog entry you're writing has the `distribution` field set
to `UNRELEASED`, using `gbp dch --auto --ignore-branch` to ensure the
formatting is correct.
The CI pipeline will locally generate a source package for each commit pushed
to the packaging repositories, which can be retrieved by browsing the
pipeline artifacts. The generated sources will be versioned to indicate that
they are not yet suitable for release.
With the `distribution` field set to `UNRELEASED` package sources get uploaded
to the `:snapshots` OBS project matching the branch (that is,
`apertis:v2020dev0:target:snaphots` when landing changes to the
`apertis/v2020dev0` branch of a `:target` package): the following [section
about landing downstream changes to the main
archive](#landing-downstream-changes-to-the-main-archive) below describes
in detail how to set the `distribution` to land the package to the appropriate
main OBS project.
## Landing downstream changes to the main archive
Once downstream changes to a package are deemed ready to be published in the
Apertis main archive, a proper release needs to be issued.
* Push a `wip/` branch updating `debian/changelog`
* use `GBP_CONF_FILES=/dev/null gbp dch --release -D apertis --debian-branch=apertis/v2020dev0`
to generate a release changelog entry summarizing all the changes already
landed on the `apertis/v2020dev0` branch
* ensure that the `distribution` field has been changed from `UNRELEASED`
to `apertis`
* Create a Merge Request based on your `wip/` branch for the most recent
release branch where you want to land your changes:
* for published stable release, the main branch (for instance
`apertis/v2019`) should **never** be targeted directly but updates and
fixes should go through the `apertis/v2019-security` or
`apertis/v2019-updates` branches, see [Process after a product
release]( {{< ref "release-flow.md#process-after-a-product-release" >}} )
for more details
* for instance, if you want to land changes to both the development and
stable releases, push your `wip/` source branch and create a MR for the
development one first (for instance, `apertis/v2020dev0`) and then, once
merged, create a MR for the stable one ((for instance, `apertis/v2019-updates`)
* Get the Merge Request reviewed and landed
* The CI pipeline will then build-check the source package as usual and since
the `distribution` field is no longer `UNRELEASED` it will also:
* add a Git tag for the release version to the repository
* rebuild the release source package
* store the release sources in the `pristine-lfs-source` branch
* upload the release source package to the main project (for instance
`apertis:v2020dev0:target`)
If the `apertis/$RELEASE-updates` or `apertis/$RELEASE-security` branches for
published stable releases do not exist yet, they should be created from the
GitLab web UI since their protected status makes pushing forbidden.
For trivial changes it is also possible to combine the release commit in the same
MR as the changes. Again, developers need to be careful to ensure the changelog
entries are kept up-to-date when the commit messages get changed via rebase.
## Pulling updates or security fixes from upstream distributions
Updates coming from upstream can be pulled it by triggering a CI pipeline on a
branches like `debian/buster` or `debian/bullseye`.
The pipeline will check the Debian archive for updates, pull them in the
`debian/$RELEASE`-like branch (for instance, `debian/bullseye` or
`debian/bullseye-security`), try to merge the new contents with the matching
`apertis/*` branches and, if successful, push a
proposed updates branch while creating a Merge Request for each `apertis/*`
branches it should be landed on.
The upstream update pipeline is usually triggered from
[the infrastructure dashboard](https://infrastructure.pages.apertis.org/dashboard/)
but can be manually triggered from the GitLab web UI by selecting the
`Run Pipeline` button in the `Pipelines` page of each repository under `pkg/*`
and selecting the `debian/bullseye` branch as the reference.
If the needed `debian/$RELEASE` branch doesn't exist (for example, there is a
`debian/buster` branch but no `debian/bullseye`), the Gitlab Web UI can be used
to create it. In the `Create from` field of the branch creation page, users
should select `debian/${RELEASE-1}`. In this case, the new branch named
`debian/bullseye` will be created from `debian/buster`.
Reviewers can then force-push to the proposed update branch in the Merge
Request to fixup any issue caused by the automated merge, and ultimately land
the MRs to the `apertis/*` branches.
In some situations the automated merge machinery may ask to
`PLEASE SUMMARIZE remaining Apertis Changes`, and in that case
reviewers should:
* check out the proposed update branch
* edit the changelog to list **all** the downstream changes the package still
ships compared to the newly merged upstream package and their reason,
describing the purpose of each downstream patch and of any other change
shown by `git diff` against the `debian/*` branch
* amend the merge commit
* force-push to the proposed update branch
* land the associated Merge Requests as described above
Remember to check that the updated package gets included in the next daily
reference image build and wait for its [QA test
results](https://lavaphabbridge.apertis.org/) to catch regressions timely
and act accordingly.
## Backporting updates or security fixes
Often downstream fixes, upstream updates or security fixes need to be applied
to [multiple active releases]( {{< ref "release-flow.md" >}} ).
Changes should be introduced in the most recent development release where they can
be tested and regressions detected with little impact, following the instructions
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
in the
[Landing downstream changes to the main archive](#landing-downstream-changes-to-the-main-archive)
and [Pulling updates or security fixes from upstream distributions](#pulling-updates-or-security-fixes-from-upstream-distributions)
sections.
Once the changes have been thoroughly tested paying close attention to the QA
test results, they can then be propagated to the more stable releases, where
any mistake can impact the product teams using Apertis in the field.
For instance, once a fix is landed to `apertis/v2021dev0` and no regressions
are found in the subsequent QA test results, a MR should be create to land
the changes to the stable releases.
If there is no divergence between the packages in the different releases and
the backport can be done with a fast-forward, a MR should be created to submit
the changes from for instance `apertis/v2021dev0` to `apertis/v2020-updates` or
`apertis/v2020-security`, following the
[Landing downstream changes to the main archive](#landing-downstream-changes-to-the-main-archive)
steps, choosing the destination depending on the nature and impact of the fix.
If package diverged across releases, a separate branch has to be created where
the fixes are cherry-picked appropriately before creating the MR. See the
[Diverging release branches](#diverging-release-branches) section for further
details about versioning divergent packages.
## Diverging release branches
Sometimes different downstream patches need to be applied in the different
Apertis release branches. A clear case of that is the `base-files` package
which ships the release name in `/etc/os-release`.
In such situation it is crucial to use different version identifiers in each
branch: the version for a given package needs to be globally unique across the whole
archive since uploading different package sources with the same name/version
would lead to difficult to diagnose errors.
When targeting a specific release, `~${RELEASE}.${COUNTER}` needs to be
appended to the version identifier after the local build suffix:
* `0.42` → append `co0~v2020pre.0` → `0.42co0~v2020pre.0`
* `0.42co3` → bump to `co4` and append `~v2020pre.0`→ `0.42co4~v2020pre.0`
* `0.42co4~v2020pre.0` → increase the release-specific counter → `0.42co4~v2020pre.1`
This uses the fact that `~` in Debian package numbers sorts before anything,
see the [Debian Policy §5.6.12](https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version) for more details.
Adding `~` is necessary so that if a new upstream version `0.42.1` or a new
non-release-specific downstream version `0.42co4` is introduced, they will
replace the release-specific package.
Note that `dpkg` considers `2020.0` to be newer than `2020pre.0`, so the
Apertis release identifiers can be used with no modification (if in doubt,
check with `dpkg --compare-versions 2020pre.0 '<<' 2020.0 && echo ok`).
## Adding new packages from Debian
This is the process to import a new package from Debian to Apertis:
* create a folder, in the name of the package to import.
* Eg. for package `hello`, run: `mkdir hello`
* chdir to created folder: `cd hello`
* invoke `import-debian-package` from the [packaging-tools
repository:](https://gitlab.apertis.org/infrastructure/packaging-tools/)
* fetch a specific version: `import-debian-package --upstream buster --downstream apertis/v2020dev0 --component target --package hello --version 2.10-2`
* fetch the latest version: `import-debian-package --upstream buster --downstream apertis/v2020dev0 --component target --package hello`
* the argument to `--component` reflects the repository component it is part of (for instance, `target`); it will be stored in `debian/apertis/component`
* multiple downstream branches can be specified, in which case all of them
will be updated to point to the newly imported package version
* the Apertis version of the package will have a local suffix (`co0`) appended
* don't use `import-debian-package` on existing repositories, it does not
attempt to merge `apertis/*` branches and instead it re-sets them to new
branches based on the freshly imported Debian package
* create an empty project on GitLab under the `pkg/*` namespaces (for instance, `pkg/hello`)
* configure the origin remote on your local git: `git remote add origin git@gitlab.apertis.org:pkg/hello`
* push your local git contents to the newly created GitLab project: `git push --all --follow-tags origin`
* set it up with `gitlab-rulez apply rulez.yaml --filter pkg/hello` from
the [gitlab-rulez repository](https://gitlab.apertis.org/infrastructure/gitlab-rulez)
* sets the CI config path to `ci-package-builder.yml@infrastructure/ci-package-builder`
* changes the merge request settings:
* only allow fast-forward merges
* ensure merges are only allowed if pipelines succeed
* marks the `apertis/*` and `debian/*` branches as protected
* follow the process described in the [section about landing downstream changes
to the main archive](#landing-downstream-changes-to-the-main-archive) above to
publish the package on OBS.
## Adding updates from a non-default upstream repository of a distribution
There are circumstances, when we deviate from the default upstream. This usually happens
when:
* Packages are not available in the default distribution repository
* Packages in the default distribution repository are outdated
* Newer version of package, available in the non-default repository, is needed
For example, Apertis v2020 ships with a newer version of the Linux kernel
(5.4.x) than Debian Buster (4.9.x) on which it is based. In such cases, special
care needs to be taken to update packages from their respective upstreams.
Below are a set of steps which can be adapted to such exception packages. Let us assume that for
such repository, the package was picked from Debian Unstable instead
* Clone your repository
$ git clone git@gitlab.apertis.org:ritesh/libgpiod.git
Cloning into 'libgpiod'...
remote: Enumerating objects: 114, done.
remote: Counting objects: 100% (114/114), done.
remote: Compressing objects: 100% (85/85), done.
remote: Total 114 (delta 18), reused 110 (delta 18), pack-reused 0
Receiving objects: 100% (114/114), 110.99 KiB | 360.00 KiB/s, done.
Resolving deltas: 100% (18/18), done.
$ cd libgpiod/
* Ensure you have a branch against your deviated upstream. If you are tracking changes from a deviated upstream like **Debian Unstable**, it needs to be ensured that the package's packaging repository has the corresponding branches available. This is needed because the automated machinery tools expect respective branches to be available.
* For example, if you picked a package from Debian Unstable, ensure to have a branch named `debian/unstable` in your git repository.
$ git checkout -b debian/unstable origin/debian/buster
Branch 'debian/unstable' set up to track remote branch 'debian/buster' from 'origin'.
Switched to a new branch 'debian/unstable'
$ git checkout debian/buster
Branch 'debian/buster' set up to track remote branch 'debian/buster' from 'origin'.
Switched to a new branch 'debian/buster'
* Similarly, ensure to have a branch named `upstream/unstable` in your git repository.
$ git checkout upstream/buster
Branch 'upstream/buster' set up to track remote branch 'upstream/buster' from 'origin'.
Switched to a new branch 'upstream/buster'
$ git checkout -b upstream/unstable upstream/buster
Switched to a new branch 'upstream/unstable'
* Ensure that these branches are pushed to your remote `origin`. It is important that these branches are pushed and in sync with the default remote.
$ git push -u origin --all
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for debian/unstable, visit:
remote: https://gitlab.apertis.org/ritesh/libgpiod/-/merge_requests/new?merge_request%5Bsource_branch%5D=debian%2Funstable
remote:
To gitlab.apertis.org:ritesh/libgpiod.git
* [new branch] debian/unstable -> debian/unstable
Branch 'apertis/v2019' set up to track remote branch 'apertis/v2019' from 'origin'.
Branch 'debian/buster' set up to track remote branch 'debian/buster' from 'origin'.
Branch 'upstream/buster' set up to track remote branch 'upstream/buster' from 'origin'.
Branch 'debian/unstable' set up to track remote branch 'debian/unstable' from 'origin'.
* Pull in new updates using the `apertis-pkg-pull-updates` script, instructing it with the deviated upstream
* Eg. `apertis-pkg-pull-updates --package PKGNAME --upstream unstable --mirror http://deb.debian.org/debian`
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
$ git checkout apertis/v2021dev2
Switched to a new branch 'apertis/v2021dev2'
$ ../apertis-pkg-pull-updates --package libgpiod --upstream unstable --mirror http://deb.debian.org/debian
source package libgpiod
running git branch --track -f debian/unstable origin/debian/unstable
Branch 'debian/unstable' set up to track remote branch 'debian/unstable' from 'origin'.
running git branch --track -f upstream/unstable origin/upstream/unstable
running git branch --track -f upstream/unstable origin/upstream/unstable
running git branch --track -f upstream/unstable origin/upstream/unstable
local version: 1.2-3
fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable-security
local version: 1.2-3
fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable-proposed-updates
local version: 1.2-3
fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable
remote version: 1.4.1-4
update to 1.4.1-4
fetch https://snapshot.debian.org/mr/package/libgpiod/1.4.1-4/srcfiles?fileinfo=1
download http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
running dget --download-only --allow-unauthenticated http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 332 100 332 0 0 449 0 --:--:-- --:--:-- --:--:-- 448
100 2294 100 2294 0 0 1381 0 0:00:01 0:00:01 --:--:-- 5474
dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1.orig.tar.xz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 338 100 338 0 0 637 0 --:--:-- --:--:-- --:--:-- 636
100 307k 100 307k 0 0 358k 0 --:--:-- --:--:-- --:--:-- 358k
dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.debian.tar.xz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 342 100 342 0 0 677 0 --:--:-- --:--:-- --:--:-- 677
100 6132 100 6132 0 0 9922 0 --:--:-- --:--:-- --:--:-- 9922
Moving branch debian/unstable to debian/, was: 713dadc
running gbp import-dsc /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1-4.dsc --author-is-committer --author-date-is-committer-date --upstream-branch=upstream/unstable --debian-branch=debian/unstable '--debian-tag=debian/%(version)s' --no-sign-tags --no-pristine-tar
gbp:info: Version '1.4.1-4' imported under '/home/rrs/NoBackup/Gitlab_Packages/packaging-tools/libgpiod'
running ./import-tarballs /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1-4.dsc
Importing /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1.orig.tar.xz
## Adding updates from distribution development repositories
This is another scenario, wherein the user may need updates which are not yet released
into the Upstream Distributions' repositories.
For example, for Apertis, we may need a very newer version of libgpiod, which may not yet
have been released into any of Debian development releases (Unstable, Testing). In
such cases, where the changes may only be available in the packaging repositories, we
need to take extra care when pulling in such updates.
Let us assume libgpiod 1.4.2 has been available in Debian's libgpiod Packaging repository but
is not released into any of the Debian releases. In such case, we can try:
* Clone the remote libgpiod git packaging repository from Debian.
* Generate a source package out of the packaging repository using `gbp buildpackage -S`
* If successful, this will give us a proper *libgpiod source package*.
* Clone the Apertis libgpiod git packaging repostiory
* Use the `pristine-lfs` tool to import the source package generated from the Debian repository into Apertis packaging repository. Eg. `pristine-lfs import-dsc libgpiod-1.4.2-1.dsc`
* Note: The `import-dsc` subcommand imports the new tarball into the git repository and commits it to the `pristine-lfs` branch. While a user can commit to the branch manually by-hand, we recommend the use of `import-dsc` to import new tarballs and committing them to the packaging repository
## Maintaining package from upstream sources
There are likely to be instances where it is desirable to import the latest version of a piece of software into Apertis directly from it's original authors or maintainers.
This may be as a result of the software in question not being packaged by Apertis' default upstream distribution, Debian, or their being a mismatch between the desired version in the upstream distribution and what is required for a specific goal.
The [Apertis release flow]( {{< ref "release-flow.md#apertis-release-flow" >}} ) stipulates that each Apertis release should include the latest mainline kernel LTS release.
Due to Debian's release cadence being approximately half that of Apertis', there are 2 Apertis' releases for each Debian stable release and no guarantees that the kernel's packaged for Debian's current stable or in progress releases will align with Apertis' requirements.
As a result Apertis will need to pull from the mainline kernel LTS tree to satisfy it's requirements.
Such packages will require special attention to assure that they remain up-to-date with any security releases made by the upstream and updated as and when apropriate.
Unless the imported package can be used as-is without any modification, there will be a [patch series]( {{< ref "buildingpatchingandmaintainingtheapertiskernel.md#packaging" >}} ) that potentially needs tweaking to apply after the update.
The workflow set out below takes this into consideration.
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
Below we use the process of importing the Linux kernel for `v2020` as an example, where we track the `linux-5.4.y` stable branch:
- Using the Gitlab Web UI, create an empty project under `pkg/`, for example, `pkg/linux`.
- Download the latest release tarball from upstream. For the `linux-5.4.y` stable branch, it would be `linux-5.4.115.tar.xz` from [kernel.org](https://www.kernel.org/) at the time of this writing.
- Extract the tarball and enter the extracted folder
$ tar xf linux-5.4.114.tar.xz
$ cd linux-5.4.114
- Create a new git repository and commit the whole source tree; this will create the `upstream/apertis` branch containing the untouched upstream source code:
$ git init
$ git checkout -b upstream/apertis
$ git add .
$ git commit
- Commit the original tarball to the `pristine-lfs` branch:
$ pristine-lfs commit ../linux-5.4.114.tar.xz
- Create the packaging branch for the Apertis version you're targetting and add packaging files
{{% notice info %}}
Debian packaging is not covered by this document, users interested in that matter
should refer to the [Debian Policy Manual](https://www.debian.org/doc/debian-policy/)
and [Debian Developer's Reference](https://www.debian.org/doc/manuals/developers-reference/).
{{% /notice %}}
- Add the Gitlab project you created earlier as the `origin` git remote:
$ git remote add origin git@gitlab.apertis.org:pkg/linux.git
- Push your branches to the repository:
$ git push --all --follow-tags origin
- Setup the repository as instructed in the [Adding new packages from Debian](#adding-new-packages-from-debian) section:
$ gitlab-rulez apply rulez.yaml --filter pkg/linux
You can then subsequently update this package by following these steps:
- Using the GitLab web UI, check to ensure that the relevant update branch exists, in the case of the Apertis `v2020` release, kernel updates should be made on the `apertis/v2020-security` branch. Create the required branch if it doesn't exist.
- Clone the existing package from Apertis GitLab and checkout the relevant branch:
$ git clone -b apertis/v2020-security git@gitlab.apertis.org:pkg/linux.git linux-apertis
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
- Separately clone the Linux kernel LTS repository and checkout the relvant stable branch:
$ git clone -b stable-linux-5.4.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux-stable
- Create a `gbp pq` patch branch and switch back to normal branch. The patch branch will be needed when rebasing the Debian patches:
$ cd linux-apertis
$ gbp pq import
$ gbp pq switch
- Determine the latest stable release and derive the Apertis version to use. To ensure collisions don't occur with any future Debian releases, we will mark the release as the zeroth "Debian" release (`-0`) and first Apertis pre-release, as applied by Collabora (`~co1`):
$ RELEASE=`git -C ../linux-stable describe`
$ RELEASE="${RELEASE#v}-0~co1"
- Add a changelog entry for newest `linux-5.4.y` kernel release (this is needed to get `genorig.py` to use the right release):
$ DEBEMAIL="User Name <user@example.com>" dch -v "$RELEASE" ""
$ git add -f debian/changelog
$ git commit -s -m "Add changelog entry for update to $RELEASE"
- Run `debian/bin/genorig.py` to generate debianised kernel source tarball:
$ debian/bin/genorig.py ../linux-stable
- Import the updated source tarball:
$ gbp import-orig --upstream-branch=upstream/linux-5.4.y --debian-branch=apertis/v2020-security <path to orig tarball>
- Rebase the Debian patches on the new kernel version:
$ gbp pq rebase
This may require some manual intervention following the normal git rebasing process.
- Export updated patch series:
$ gbp pq export
- Tweak `debian/patches/series` file so that it retains the comments that `gbp pq` wants to drop. If no patches have been dropped during rebase then:
$ git checkout debian/patches/series
$ git add -f debian/patches
$ git commit -s -m "Update the debian patches for $RELEASE update"
- Update changelog to released state (set distribution to apertis), document any patches that have been dropped as a result of the update and create new commit:
$ dch -D apertis
$ git add -f debian/changelog
$ git commit -s -m "Release linux version $RELEASE"
- Run `dpkg-buildpackage` to generate the debian packaging, this is needed to use the tools to update the [pristine-lfs branch]( {{< ref "gitlab-based_packaging_workflow.md" >}} ):
$ mkdir ../build
$ gbp buildpackage --git-ignore-new \
--git-debian-branch=apertis/v2020-security \
--git-prebuild='debian/rules debian/control || true' \
--git-export-dir='../build' \
-us -S
- Import the generated kernel tarball into pristine-lfs branch with `import-tarballs`:
$ git clone git@gitlab.apertis.org:infrastructure/packaging-tools.git ../packaging-tools
$ ../packaging-tools/import-tarballs ../build/linux_5.4.51-0~co1.dsc
$ git push origin pristine-lfs
- Push kernel update to a branch to be reviewed:
$ git push origin apertis/v2020-security:wip/user/kernel-update
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
## License scans
As merge requests to packaged software are submitted, the CI pipeline performs
license scans on the package. The scans are performed on all files in the package,
not just the new submission. The pipeline fails or emits a warning (depending on
the configuration) when if finds files with unknown or unclear licensing terms,
or files under licenses not allowed in the package. When such situation arises,
it is the responsibility of the submitter to perform the review of the license
scan results and make updates to the package if necessary.
When the license scan mistakenly identifies a file as being under an incorrect
license, or fails to process it correctly, there are three ways to fix this:
1. Specify the correct copyright and the license in `debian/apertis/copyright.yml`.
The format of the file is specified in the
[Dpkg::Copyright::Scanner](https://manpages.debian.org/buster/libconfig-model-dpkg-perl/Dpkg::Copyright::Scanner.3pm.en.html)
manpage.
In short, it’s a YAML file mapping paths to their licensing information:
debian:
copyright: 2015, Marcel
license: Expat
src/:
copyright: 2016, Joe
license: Expat
.*/NOTICE:
skip: 1
src/garbled/:
'override-copyright': 2016 Marcel MeXzigue
Patterns follow the Perl regular expression rules.
Please also verify `debian/copyright` specifies the correct license, and if it
doesn’t, submit a patch to Debian.
2. Add the file to the list of ignored files.
`debian/apertis/copyright.whitelist` is formatted the same way as `gitignore`,
please refer to the [gitignore](https://manpages.debian.org/buster/git-man/gitignore.5.en.html)
manpage for more information.
3. If the file is under a license not suitable for Apertis, it can be removed from
the package by either repackaging the tarball or patching it out, in which case
the scanner will not take it into account.
The license scanner will store the automatically generated copyright report file
under `debian/apertis/copyright`, updating the merge request when necessary.
[Dpkg::Copyright::Scanner]: https://manpages.debian.org/testing/libconfig-model-dpkg-perl/Dpkg::Copyright::Scanner.3pm.en.html
[gitignore]: https://manpages.debian.org/testing/git-man/gitignore.5.en.html
## Custom pipelines
When using the packaging pipeline, developers cannot put their CI/CD automation
in `.gitlab-ci.yml` anymore, as the CI config path points to the
ci-package-builder definition.
However, developers can put their jobs in the
`debian/apertis/local-gitlab-ci.yml` file and have them executed in a child
pipeline whenever the main packaging pipeline is executed. This is specially
handy to run tests before the actual packaging process begins.
# Internals
Main components:
* [`ci-package-builder`](https://gitlab.apertis.org/infrastructure/ci-package-builder):
centralized location of the GitLab-to-OBS and Debian-to-GitLab pipeline definitions
* [`apertis-package-source-builder`](https://gitlab.apertis.org/infrastructure/apertis-docker-images/tree/apertis/v2019/apertis-package-source-builder):
Docker environment for the GitLab pipelines
* [`pristine-lfs`](https://salsa.debian.org/andrewsh/pristine-lfs): stores
upstream original tarballs and packaging source tarballs using Git-LFS, as a
more robust replacement for `pristine-tar`
Branches:
* `pristine-lfs`: stores references to the Git-LFS-hosted original tarballs
* `debian/$DEBIAN_RELEASE` (for instance, `debian/buster`): contains the extracted
upstream sources and packaging information from Debian
* `pristine-lfs-source`: stores references to the Git-LFS-hosted packaging
tarballs, mainly to ensure that each (package, version) tuple is built only
once and no conflicts can arise
* `apertis/$APERTIS_RELEASE` (for instance, `apertis/v2020dev0`): contains the
extracted upstream sources and possibly patched packaging information for
Apertis, including the `debian/apertis/gitlab-ci.yaml` to set up the
GitLab-to-OBS pipeline
* `apertis/$APERTIS_RELEASE-security` and `apertis/$APERTIS_RELEASE-updates`
(for intance, `apertis/v2019-updates`): similar to `apertis/$APERTIS_RELEASE`
but respectively target the Security and Updates repositories for published
stable releases as described in [Process after a product
release]( {{< ref "release-flow.md#process-after-a-product-release" >}} )
Tags:
* `debian/*`: tags for Debian releases in the `debian/*` branches
* `apertis/*`: tags for the Apertis releases in the `apertis/*` branches
Gitattributes is a feature that enables modifications to be automatically made by git when certain
operations are done (such as `checkout`, `check-in` and `clone`) to enforce certain project standards.
but this has the potential effect of modifying files and that's not what we want when packaging. More
details about [gitattributes](https://git-scm.com/docs/gitattributes).
* Many upstream projects ship a `.gitattributes` file in their source repositories, which are
tightly tied to their development workflows. At times, certain settings in `.gitattributes`
can lead to problems in our packaging workflow, as summarized above. Thus, it is advised to execute
the following command, which will override any such inherited settings from the
`.gitattributes` file:
```
echo "* -text -eol -crlf -ident -filter -working-tree-encoding -export-subst" > .git/info/attributes
```
* This will override the settings in the `.gitattributes` file from the repository, for all files and for the mentioned features