Add Apertis distribution rebasing guide

Add a guide covering the step by step process of rebasing the Apertis
distribution. This is aimed primarily at the core Apertis developers, so
adding a new "maintenance" session for such documentation.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

content/maintenance/_index.md

++++
+date = "2024-04-24"
+weight = 100
+
+title = "Maintenance"
++++
+
+The documents provided in this section is targeted towards the Apertis maintainers, covering the tasks required to maintain and develop the core Apertis distribution.
+
+For higher level descriptions of the technology employed in Apertis, please see
+the [architecture documents]({{< ref "architecture/_index.md" >}}). For
+documentation targeted at developers and users of Apertis, please see the
+[guides]({{< ref "guides/_index.md" >}}).

content/maintenance/rebasing.md

++++
+date = "2024-04-24"
+weight = 100
+toc = true
+
+title = "Rebasing Apertis"
++++
+
+Each Apertis release is based on the current Debian stable release. This means
+that approximately bi-yearly Apertis makes the jump to be based on a new
+version of Debian. This process is known as "rebasing".
+
+{{% notice info %}}
+Please see
+[Release flow and product lines]({{< ref "policies/release-flow.md" >}}) for
+more information about this policy.
+{{% /notice %}}
+
+The following terms will be used here:
+
+- `OLD_DEV_RELEASE`: The existing Apertis release on which the new release will
+  be based.
+- `NEW_DEV_RELEASE`: The new Apertis development release that we are creating
+  during the rebasing process.
+- `OLD_DEBIAN_RELEASE`: The Debian release on which `OLD_DEV_RELEASE` is based.
+- `NEW_DEBIAN_RELEASE`: The Debian release on which `NEW_DEV_RELEASE` will be
+  based.
+
+# Early Tests
+
+These tests should be carried out in Q4 ready for the rebase in the follwoing
+Q1.
+
+## A. Test Rebase Workflow
+
+1. Follow the rebase plan, but with a limited set of packages from target using
+   `NEW_DEV_RELEASE` set to `rebase-test`.
+1. Check that pipelines run as expected
+1. Check that the scripts used to check status work as expected
+
+{{% notice note %}}
+It is not required to merge all the changes, focus on ensuring that the
+workflow is working as expected.
+{{% /notice %}}
+
+## B. Test Debian Sources
+
+If sufficient time is available early in Q1, it is advantageous to ensure that
+we have a relatively accurate seed list and have already imported new
+dependencies where possible. It is unlikely that the seed list will be perfect
+even after this, however any remaining missing dependencies, etc. will be
+flagged by OBS. The aim here is to reduce the time required when rebasing.
+
+1. Use [germinate](https://gitlab.apertis.org/infrastructure/germinate) to
+   generate seeds for `NEW_DEBIAN_RELEASE`.
+1. Compare "existing-packages" lists generated for `NEW_DEBIAN_RELEASE` with
+   the list of packages in the current Apertis release. Review differences and
+   update the new seed list as required.
+1. Review packages listed in "new-packages" lists for addition to Apertis
+
+# Preparing for Rebase
+
+## C. Prepare Development Workflow
+
+During the rebase `NEW_DEV_RELEASE` will be unstable as many packages will be
+updated and images will not be generated. To allow development work during this
+time, `OLD_DEV_RELEASE` will be used as development branch. For that reason,
+after `OLD_DEV_RELEASE` freeze ends, it will be used for development.
+{{% notice note %}}
+For repositories that diverge during the rebase cycle, after the rebase is
+completed, changes will need forward porting to `NEW_DEV_RELEASE`.
+{{% /notice %}}
+
+
+## D. Prepare Infrastructure
+
+1. Create `NEW_DEV_RELEASE` branch for `apertis-docker-images`
+1. Tweak the `package-source-builder` dockerfile in the `NEW_DEV_RELEASE`
+   release to enable Debian `NEW_DEBIAN_RELEASE`
+1. Enable apertis `OLD_DEV_RELEASE` repositories in dockerfile, to allow
+   the installation of Apertis specific packages, such as `apertis-dev-tools`
+1. Temporarily disable other docker images as they won't be needed
+1. Create new OBS projects for `NEW_DEV_RELEASE`
+1. Disable publish of `NEW_DEV_RELEASE` on OBS projects
+1. Add `NEW_DEBIAN_RELEASE` DoD on `NEW_DEV_RELEASE` OBS repositories
+1. Create Apertis `NEW_DEV_RELEASE` branches for the repositories in the
+   following groups based on the `OLD_DEV_RELEASE` branches in Gitlab. (This
+   should create new branches but not trigger any pipelines): 
+   - [infrastructure](https://gitlab.apertis.org/infrastructure)
+   - [tests](https://gitlab.apertis.org/tests)
+   - [sample-applications](https://gitlab.apertis.org/sample-applications)
+1. Continue to use `OLD_DEV_RELEASE` for development work. 
+
+# Perform Rebase
+
+## E. Upgrade packages from NEW\_DEBIAN\_RELEASE 
+
+Extract per repository component package list generated from germinate. Work
+through the package lists for `target`, `non-free`, `development` and `sdk`
+repositories in this order:
+
+{{% notice note %}}
+These commands act on a single package rather than on lists of them and thus will need to be run for each package.
+
+It is better to process the list of packages in batches, starting with a small number of packages from target, to make sure that no unexpected issues happen, before continuing with subsequent batches, working through target and then onto the other repositories in order.
+{{% /notice %}}
+
+1. Create `debian/NEW_DEBIAN_RELEASE` branches in Gitlab for these packages in
+   their git repositories. This can be done using the
+   [`create-branches`](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/-/blob/main/release-scripts/create-branches)
+   script:
+   ```
+   $ ./create-branches --filter pkg/${pkg} --branch-prefix debian --new-release ${NEW_DEBIAN_RELEASE} --base-release ${OLD_DEBIAN_RELEASE}
+   ```
+1. Create `upstream/NEW_DEBIAN_RELEASE` branches in Gitlab for these packages
+   in their git repositories. This can also be done using the
+   [`create-branches`](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/-/blob/main/release-scripts/create-branches)
+   script:
+   ```
+   $ ./create-branches --filter pkg/${pkg} --branch-prefix upstream --new-release ${NEW_DEBIAN_RELEASE} --base-release ${OLD_DEBIAN_RELEASE}
+   ```
+
+1. Trigger the pipeline on the `debian/NEW_DEBIAN_RELEASE` branches using the
+   [`trigger-pipeline`](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/-/blob/main/release-scripts/gitlab/trigger-pipeline)
+   script:
+   ```
+   $ ./trigger-pipeline ${pkg} debian/${NEW_DEBIAN_RELEASE}
+   ```
+   This will trigger a chain of pipelines that will sync with the branch with
+   the required Debian release and attempt to merge with the `NEW_DEV_RELEASE`
+   branch.  This results in open MRs with failed pipelines where this merge
+   can't happen automatically.
+
+   {{% notice note %}}
+   Since Debian DoD will be available no additional configuration should be
+   required to resolve dependencies
+   {{% /notice %}}
+
+## F. Check status of each package
+
+As there will be a large number of merge requests open, it is important to
+carefully prioritise the order in which packages are targeted. The packages
+should be checked in the following order:
+
+- Packages required for bootstrapping (get a list based on
+  `NEW_DEBIAN_RELEASE`)
+- Packages in the `target` and `non-free` repositories
+- Other packages, such as those in `development` and `sdk`.
+
+The
+[analise-rebases.py](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/-/blob/main/release-scripts/analyse-rebases.py)
+script can be used to aid with this process.
+
+There will be different scenarios depending on the package:
+1. Most of the packages will have an update available in `NEW_DEBIAN_RELEASE`
+   and an MR should be available for it in one of the following states:
+   * MR pipeline failure due to unable to merge: Manually merge the branch.
+   * MR pipeline failure due to license scan failure: Manually fix license scan
+   * MR pipeline failure due to OBS build error: Manually check and/or retry
+   * MR pipeline success with not relevant changes from `NEW_DEBIAN_RELEASE`
+     (only changelog or debian/apertis): Should be automerged
+   * MR pipeline success with relevant changes (delta) to `NEW_DEBIAN_RELEASE`
+     (changes other than in changelog or debian/apertis): Review and determine if delta is still
+     valid
+1. Some packages will not have an update available in `NEW_DEBIAN_RELEASE`, run
+   the pipeline for `NEW_DEV_RELEASE` in these repositories
+1. Some new packages will need to be imported
+
+All successful pipelines will result in a package being uploaded to OBS
+`proposed-updates/debian/NEW_DEBIAN_RELEASE` branches.  Merging all the changes
+is a time consuming task, apply common sense to avoid waiting time in things
+that can be addressed later.
+
+{{% notice warning %}}
+Keep track of delta introduced as this will need to be documented, rectified or forwarded upstream.
+{{% /notice %}}
+
+# Image generation and testing
+
+## G. Target image generation
+
+1. Run local bootstrap tests by downloading packages required for bootsrapping
+   directly from OBS and fix any issue found.
+1. Once packages in `target` and `non-free` have been built it is possible to
+   enable publishing of these repositories. It is important to do this to
+   decouple the rebase work from the image generation/testing.
+1. During this stage we focus in target images (fixed function and hmi)
+1. Run local tests to building target images using `NEW_DEV_RELEASE`
+   and fix any issue found.
+1. Run image pipeline only for fixed function and HMI (disable the rest):
+   1. Fix any issue found by either proposing proper fixes or workarounds
+      (disabling functionality)
+   1. The goal of this stage is to have a minimal image that can be tested on
+      the different reference boards
+   1. In case of temporary workarounds create tasks to address them later
+   1. Test image on LAVA by using `UPLOAD_ARTIFACTS` and `SUBMIT_LAVA_JOBS`
+
+## H. Check status of OBS projects
+
+1. Check that all the packages in germinate are in OBS using
+   [analyse-missing.py](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/-/blob/main/release-scripts/analyse-missing-packages.py)
+1. Check if all build statuses are green on OBS
+1. Check that OBS is NOT using Debian DoD to build packages using the
+   [`check-for-dod`](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/-/blob/main/release-scripts/check-for-dod) script.
+1. Drop Bookworm DoD repositories from OBS
+1. Fix any leftover cracks:
+   * New package imports
+   * Build dependency trimming
+   * Feature/plugin [dis|en]ablement
+1. Enable publishing on `NEW_DEV_RELEASE` OBS `development` and `sdk`
+   repositories
+1. Validate the public apt repositories for `NEW_DEV_RELEASE`
+
+## I. Image generation
+
+1. Run image pipeline for all the images (reenable previously disabled images)
+1. Fix any issue found by either proposing proper fixes of workarounds
+   (disabling functionality)
+1. The goal of this stage is to have a minimal image that can be tested on the
+   different reference boards
+1. In case of temporary workarounds, create tasks to address them later
+1. Test image on LAVA by using `UPLOAD_ARTIFACTS` and `SUBMIT_LAVA_JOBS`
+
+## J. Cleanup infrastructure
+
+1. Tweak docker images for package-source-builder in `NEW_DEV_RELEASE` to only
+   use `NEW_DEV_RELEASE`
+1. Re-enable the other docker images
+1. Build new docker images
+1. Rebuild whole archive 
+1. Generate a list of packages with a `NEW_DEV_RELEASE` branch in GitLab but
+   which are not present in OBS.  Dashboard will report the following error for
+   such packages and the list can be generated from this.
+   ```
+   Missing on OBS apertis:NEW_DEV_RELEASE, found on Git apertis/NEW_DEV_RELEASE
+   ```
+1. Drop these packages from the new release:
+   - Remove the `apertis/NEW_DEV_RELEASE` branch from the package in Gitlab
+   - Set the default branch as `apertis/OLD_DEV_RELEASE` using
+     [set-delete-branches](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/-/blob/main/release-scripts/set-delete-branches).
+   - Update
+     [gitlab-rulez](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/-/blob/main/gitlab-scripts/rulez.yaml)
+     to contain a list for `OLD_DEV_RELEASE` containing the packages for which
+     it is the last release. This causes gitlab-rulez to leave the default
+     branch at this release for these packages.
+
+## K. Forward port any changes to OLD\_DEV\_RELEASE
+
+1. Check which repositories have diverged between `OLD_DEV_RELEASE` and
+`NEW_DEV_RELEASE`. This means for repositories where `OLD_DEV_RELEASE` is not a
+descendant of `NEW_DEV_RELEASE` and thus further changes have been merged to
+`OLD_DEV_RELEASE`.
+1. Evaluate and prepare forward ports for changes where required.
+1. Fix any temporary workarounds that were added during rebasing.
+
+## L. Rejoice! Rebase party!