Skip to content
Snippets Groups Projects

Folding documentation

Merged Folding documentation
wip/andrewsh/folding into master
All threads resolved!
Merged Andrej Shadura requested to merge wip/andrewsh/folding into master
All threads resolved!
Compare
and
5 files
+ 56
− 93
Compare changes
  • Side-by-side
  • Inline
Files
5
content/guides/maintenance/apertis_release_process.md
+ 56
− 93
+++
date = "2020-07-14"
lastmod = "2024-06-12"
weight = 100
toc = true
@@ -111,7 +112,7 @@ the pipeline can be re-triggered.
After the completion of the **Release Branching** CI Jobs, a certain set of manual steps need to be performed, which are outlined below.
* Add a directory for the release to the LAVA NFS path at: `/srv/images/public/lava/nfs/` on host images.apertis.org. The LAVA test jobs pick the kernel and initrd image from the release's directory path. Refer to the README for details.
* Instantiate a *new timer to rotate images* for the new release (eg. On host `aurora`, run command `systemctl enable --now rotate-images@v2019.timer`).
* Instantiate a *new timer to rotate images* for the new release (eg. On host `aurora`, run command `systemctl enable --now rotate-images@v2024.timer`).
* Update the [ci-package-builder](https://gitlab.apertis.org/infrastructure/ci-package-builder/) to target the next release.
* Update the [gitlab-rulez recipe](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/-/blob/main/gitlab-scripts/rulez.yaml) to point to the new development release.
* Set the default development branch for all git projects.
@@ -129,7 +130,6 @@ A Stable Point Release includes:
* Security Updates to software packages (funneled through $RELEASE-security). These include security vulnerability fixes.
* Normal updates to software packages (funneled through $RELEASE-updates). These include bug fixes to existing version of software packages.
* Backport of software packages (funneled through $RELEASE-backports). These include newer version of software packages backported to older OS releases
## Stable Point Release Steps
@@ -152,26 +152,50 @@ those steps specific for stable point releases.
Once the **Release Preparation** is complete, the next step in the **Release Process** is to run the **Release Folding** steps.
At a higher level, **Release Folding** is the process of merging the updates from `security`, `updates` and `backports` branches into the main ones, and includes the following steps which needs to be run in close co-ordination.
At a higher level, **Release Folding** is the process of merging the updates from `security` and `updates` branches into the main ones, and includes the following steps which needs to be run in close co-ordination.
* Update [Build Suffix](#build-suffix)
* Fold changes in `security`, `updates` and `backports` branches into the main ones
* Fold changes in `security` and `updates` branches into the main ones
* Confirm that newer versions of the packages are available in the main repositories
* Drop `security`, `updates` and `backports` branches for the release in GitLab
* Drop `security`, `updates` and `backports` repositories for the release in OBS
* Drop `security` and `updates` branches for the release in GitLab
* Drop `security` and `updates` repositories for the release in OBS
To perform these changes launch `fold-security-update-branches` script from the *apertis-infrastructure/release-scripts/gitlab/*, which automates many of the steps in a *Stable Point Release Preparation*
These changes are performed by the `fold-security-update-branches` script from the [apertis-infrastructure/release-scripts/gitlab/](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/-/blob/main/release-scripts/gitlab/fold-security-update-branches), which automates many of the steps in a *Stable Point Release Preparation*:
* Increment the [Build Suffix](#build-suffix) in the Project configuration for each repository relevant to the release
* List the packages in the `:security`, `:updates` and other similar repositories
* Fast Forward Merge `--ff-only` the respective security and updates branches on Git. For example, `apertis/v2020-security` and `apertis/v2020-updates` should be merged back to `apertis/v2020`. Note: Not able to perform a fast forward merge means something got broken and needs to be investigated and fixed.
* Poke the pipeline and ensure that all the new packages (from the merged back changes to `apertis/v2020`) get built on OBS and land into the main repository.
* Drop the *updates*, *security* branches (Eg. `apertis/v2020-updates`, `apertis/v2020-security`)
* Delete the packages from the *updates*, *security* etc repositories on OBS (Eg. `apertis:v2020:updates:target`, `apertis:v2020:security:development`)
* Fast Forward Merge `--ff-only` the respective security and updates branches on Git. For example, `apertis/v2024-security` and `apertis/v2024-updates` should be merged back to `apertis/v2024`. Note: Not able to perform a fast forward merge means something got broken and needs to be investigated and fixed.
* Poke the pipeline and ensure that all the new packages (from the merged back changes to `apertis/v2024`) get built on OBS and land into the main repository.
* Drop the *updates*, *security* branches (Eg. `apertis/v2024-updates`, `apertis/v2024-security`)
* Delete the packages from the *updates*, *security* etc repositories on OBS (Eg. `apertis:v2024:updates:target`, `apertis:v2024:security:development`)
Note: The cleanup of the public apt repositories is to be run separately, after validation of the folded packages in the main repositories
The script can be run in a semi-automatic mode through a GitLab CI pipeline.
Similarly as with release branching, the pipeline runs in the dry-run mode by default.
To actually perform the changes, set `NOACT` variable to `0` when triggering the pipeline.
See [Folding examples](#folding-examples) for more information.
![Triggering folding pipeline](/images/folding.png)
The pipeline is organised in the following stages:
![Folding pipeline stages](/images/folding-stages.png)
1. **Fold**. In this stage, the script submits merge requests to all packages with updates in `updates` and `security`.
Packages where these merge requests can be automatically merged, are set to auto-merge once the pipelines succeed.
2. **Manual review**. Since the script cannot properly handle all updates yet, manual merges are often necessary. See below for more details.
3. **Cleanup**. After performing the reviews and verifying that all packages have been built and published in the main repositories, run this step to clean-up the GitLab branches and public apt repositories.
To make the review process easier, each component is processed in a separate job in the "fold" child pipeline:
![Folding child pipeline with separate jobs](/images/folding-subpipeline.png)
At the end of each job’s logs, there’s a summary with status of each package and links to the merge requests:
![Folding job’s log with links to merge requests](/images/folding-log-review.png)
The meaning of each status:
* `CLEAN`: this package needs no further processing other than possibly a clean-up
* `NEEDS-MERGE`: this package is out of sync and needs a merge, but the dry-run mode is in force
* `MERGE`: a merge request has been submitted for this package
* `SKIP`: a package was found in one of the OBS repositories, but no corresponding branch was found in GitLab
* `ERROR`: an error occurred during processing this package, check the detailed logs above
A known situation when the pipeline submits a merge request that GitLab cannot process is when a new package has been introduced in *updates* or *security*. In such cases, the merge request is created against a non-existent branch, and it cannot be edited or merged using the GitLab UI, and the branch needs to be manually pushed.
# Common tasks during release
@@ -233,7 +257,7 @@ Some notes about the `Archive Rebuild` task
* Run the `copy-release` script on the images server (`aurora`) to prepare the RC.
* `copy-release -v 20201203.0116 v2021pre.0rc1`
* `copy-release -v -i 20201203.0116 v221pre.0rc1`
* `copy-release -v -i 20201203.0116 v2021pre.0rc1`
* First for the public images and then second for Internal images (-i option).
* The source folder is picked from the daily image builds.
* Send release announcement for the release.
@@ -247,8 +271,8 @@ The suffix string is constructed of: `b + Release Name + b<B_CNT>`
* String `Release Name` will refer to the Apertis release's name
* String, `b<B_CNT>` refers to being a binary build along with the build count
For example, for an Apertis Developer Release of `v2020dev0`, we add string `bv2020dev0b<B_CNT>` to the project configuration on OBS.
Similarly, for an Apertis Product Release of `v2019.0`, we add string `bv2019.0b<B_CNT>` to the project configuration on OBS
For example, for an Apertis Developer Release of `v2025dev0`, we add string `bv2025dev0b<B_CNT>` to the project configuration on OBS.
Similarly, for an Apertis Product Release of `v2024.0`, we add string `bv2024.0b<B_CNT>` to the project configuration on OBS
## Stable Build Suffix
@@ -256,13 +280,13 @@ The `Build Suffix` is a crucial metadata for an Apertis Stable release. Setting
In a typical Apertis Release, we have multiple repository components
* Main repository components like: `apertis:v2023:development`, `apertis:v2023:target`
* Security repository components like: `apertis:v2023:security:development`, `apertis:v2023:security:target`
* Updates repository components like: `apertis:v2023:updates:development`, `apertis:v2023:updates:target`
* Backports repository components like: `apertis:v2023:backports:development`, `apertis:v2023:backports:target`
* Main repository components like: `apertis:v2024:development`, `apertis:v2024:target`
* Security repository components like: `apertis:v2024:security:development`, `apertis:v2024:security:target`
* Updates repository components like: `apertis:v2024:updates:development`, `apertis:v2024:updates:target`
* Backports repository components like: `apertis:v2024:backports:development`, `apertis:v2024:backports:target`
For packages landing into an Apertis Stable Release, it would land through either of `security`, `updates`, or `backports` repository components.
For packages landing into an Apertis Stable Release, it would land through either of `security` or `updates` repository components.
Packages accumulate into either of the repositories and eventually get folded into the main respective repository, just like the stable release procedure followed in the Debian project.
The above structure allows Apertis users to have immediate access to updates through either of the repository channels, provided those repositories are enabled in their apt configuration.
@@ -273,33 +297,33 @@ Given the structure above, of the lifetime of a package, across different reposi
To achieve the above goal, consider the below project config metadata:
```
bv2023.0db<B_CNT> in v2023
bv2023.0cb<B_CNT> in v2023-security
bv2023.0bb<B_CNT> in v2023-updates
bv2023.0ab<B_CNT> in v2023-backports
bv2024.0db<B_CNT> in v2024
bv2024.0cb<B_CNT> in v2024-security
bv2024.0bb<B_CNT> in v2024-updates
bv2024.0ab<B_CNT> in v2024-backports
```
In the above example, the revisioning is carefully ordered to ensure that when Stable Update packages finally land into the main repository, they have the proper revision.
Consider the above with an example of package `foo` and version `1.0-1`. Under Apertis, its typical reflection would be something like: `foo-1.0-1+apertis0bv2023.0db1_amd64.deb` which would be the initial version in the main repository. Over the course, we may push a new security update reflecting as `foo-1.0-3+apertis0bv2023.0cb1_amd64.deb` which would live for a defined time until the next Stable Point Release in the security repository, like `apertis:v2023:security:development`. Eventually, as part of the Stable Point Release, the package will be folded into the main repository, at which point, its reflection in the main repository will become: `foo-1.0-3+apertis0bv2023.0db1_amd64.deb`.
Consider the above with an example of package `foo` and version `1.0-1`. Under Apertis, its typical reflection would be something like: `foo-1.0-1+apertis0bv2024.0db1_amd64.deb` which would be the initial version in the main repository. Over the course, we may push a new security update reflecting as `foo-1.0-3+apertis0bv2024.0cb1_amd64.deb` which would live for a defined time until the next Stable Point Release in the security repository, like `apertis:v2024:security:development`. Eventually, as part of the Stable Point Release, the package will be folded into the main repository, at which point, its reflection in the main repository will become: `foo-1.0-3+apertis0bv2024.0db1_amd64.deb`.
Over its lifecycle in the Apertis v2023 release so far, package `foo` has had 3 versions: `1.0-1+apertis0bv2023.0db1`, `1.0-3+apertis0bv2023.0cb1` and `1.0-3+apertis0bv2023.0db1`
Over its lifecycle in the Apertis v2024 release so far, package `foo` has had 3 versions: `1.0-1+apertis0bv2024.0db1`, `1.0-3+apertis0bv2024.0cb1` and `1.0-3+apertis0bv2024.0db1`
From apt/dpkg point of view:
```
$ dpkg --compare-versions 1.0-1+apertis0bv2023.0db1 lt 1.0-3+apertis0bv2023.0cb1 && echo "Latter version is greater"
$ dpkg --compare-versions 1.0-1+apertis0bv2024.0db1 lt 1.0-3+apertis0bv2024.0cb1 && echo "Latter version is greater"
Latter version is greater
```
Similarly, now for the example security update folded into main:
```
$ dpkg --compare-versions 1.0-3+apertis0bv2023.0cb1 lt 1.0-3+apertis0bv2023.0db1 && echo "Latter version is greater"
$ dpkg --compare-versions 1.0-3+apertis0bv2024.0cb1 lt 1.0-3+apertis0bv2024.0db1 && echo "Latter version is greater"
Latter version is greater
$ dpkg --compare-versions 1.0-3+apertis0bv2023.0cb2 lt 1.0-3+apertis0bv2023.0db1 && echo "Latter version is greater"
$ dpkg --compare-versions 1.0-3+apertis0bv2024.0cb2 lt 1.0-3+apertis0bv2024.0db1 && echo "Latter version is greater"
Latter version is greater
```
@@ -427,67 +451,6 @@ Host images.apertis.org
* Restart the OBS schedulers
* Update build suffix on OBS Project configuration (eg. `bv2021.0b<B_CNT>`, `bv2020dev0b<B_CNT>`). Please refer to section **Build Suffix** and **Stable Build Suffix** for more details about it
## Folding examples
Following are the example runs for the automation script.
### Example Run with help sub-command
$ ./fold-security-update-branches help
Usage: BASE_RELEASE='v2021' ./fold-security-update-branches [apply] [gen-pkg-list] [dry-run] [clean-obs] [clean-reprepro] [clean-obs-reprepro] [help]
Example: BASE_RELEASE='v2021' ./fold-security-update-branches apply 2>&1 | tee -a console.log
### Example Run with gen-pkg-list sub-command
The `gen-pkg-list` sub-command generates a bunch of data files, based on the Apertis Release and Apertis Repositories. These generated data files are used by the script to process changes (Security and Regular Updates) in Apertis point releases.
rrs@priyasi:~/rrs-home/devel/CCU/Apertis/apertis-infrastructure/release-scripts/gitlab (v2020-folding-work-data)$ ./fold-security-update-branches gen-pkg-list
Package lists extracted from OBS' repository v2020-security-target.txt
Package lists extracted from OBS' repository v2020-updates-target.txt
Package lists extracted from OBS' repository v2020-security-development.txt
Package lists extracted from OBS' repository v2020-updates-development.txt
Package lists extracted from OBS' repository v2020-security-sdk.txt
Package lists extracted from OBS' repository v2020-updates-sdk.txt
Package lists extracted from OBS' repository v2020-security-hmi.txt
Package lists extracted from OBS' repository v2020-updates-hmi.txt
### Example Run with dry-run sub-command
The `dry-run` sub-command will run the actual commands without any effect on the server-side. This is very handy to check and determine the actual actions that will be run.
$ ./fold-security-update-branches dry-run
Package apt is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package bluez is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package boost1.67 is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package cups is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
### Example Run with apply sub-command
The `apply` sub-command does the actual job.
$ ./fold-security-update-branches apply
Running in apply mode
Package apt is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/apt/-/merge_requests/10
Package bluez is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/bluez/-/merge_requests/12
Package boost1.67 is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/boost1.67/-/merge_requests/5
Package cups is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/cups/-/merge_requests/8
Package curl is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/curl/-/merge_requests/7
Package cyrus-sasl2 is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
## Removing an old distribution - Reprepro
Perform the following steps to remove an old distributions. This would usually be the
Apertis Website Terms of Use Privacy Policy