Release Branching Automation through CI Pipeline

This document describes the new Release Branching Automation through the
Gitlab CI Pipeline

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

content/guides/apertis_release_process.md

@@ -77,55 +77,60 @@ At a higher level, **Release Branching** includes the following steps which need
 * Branching docker image recipes
 * Branching all OBS repositories
 
-Given the number of steps involved in **Release Branching**, the following **Branching Checklists** should be executed in close co-ordination.
-
-# Branching Checklists
-
-There are a couple of checks to be performed on individual host machines. The machine names mentioned are specific and should be adopted to users' environment
-
-  * Host `niobium`: Hosts the OBS service
-  * Host `images.apertis.org`: Hosts the Apertis images repository
-  * Host `gitlab.apertis.org`: Hosts the Apertis Gitlab instance
-
-# Branching Preparation
-
-* set up `~/.ssh/config` such that the `niobium` alias is available with appropriate `root` privilege.
-
-```
-Host niobium
-    User andrewshR
-
-Host images.apertis.org
-    User andrewshR
-```
-
-* Check available disk space on OBS Server
-  * *Release Branching* is a task that will consume a significant amount of disk space. Thus, it is very important to ensure that enough free space is available before commencing a release branching. Depending upon the size of a project, the amount of disk space required can vary.
-  * Running out of disk space, halfway during the *release branching* process, may have severe side-effects. Hence, it is advised to consider over provision of free disk space before initiating a *release branching*.
-* Install the `jq` package. It is already packaged in Apertis
-* Install the [osc-plugin-clone](https://gitlab.collabora.com/andrewsh/osc-plugin-clone) plugin for the OBS client
-* Install the `osc-plugins-dput`. It is already packaged in Apertis
-* Create a [GitLab personal access token](https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html) and configure it in `$HOME/.gitlab-apertis-org-token`, e.g. `GITLAB_TOKEN=XXXXxxxxxXXXXXX > $HOME/.gitlab-apertis-org-token`
-  * Make sure your API token is active. When you create your API token, you have an option to set an expiry date. So make sure to verify the validity of your granted token
-* Ensure that the `osc` tool is configured with the `Admin` user credentials for `OBS`. This is essential for the creation of additional repositories
-* Check out the [apertis-infrastructure repository](https://gitlab.apertis.org/infrastructure/apertis-infrastructure.git/)
-* Install `gitlab-rulez` tool and unprotect the current release branch for all repositories.
-  * Verify that all relevant package groups, like `pkg/*`, have `push_access_level: DEVELOPER_ACCESS` in `apertis-infrastructure/gitlab-scripts/rulez.yaml`
-  * Apply the rule with `gitlab-rulez apply apertis-infrastructure/gitlab-scripts/rulez.yaml`
-* Change the working directory to release-scripts/
-* Launch `export RELEASE=v2019; export NEXT_RELEASE=v2020dev0; ./do-branching.sh` which will:
-  * Set new release to string *v2020dev0* and set current release to string *v2019*, for example.
-  * Create a new debootstrap script entry on the image builder host
-  * Update the debootstrap package on OBS
-  * Update the apertis-customizations, apertis-image-customization, apertist-tests repositories, both public and internal
-  * Create branches in every repository on `gitlab.apertis.org`
-  * Stop the OBS schedulers
-  * Fork the OBS projects, copying packages and metadata
-  * Create a new reprepro repository forcing a re-export of the binaries
-  * Restart OBS to pick up the new reprepro entry (the magic is in `/etc/obs/BSConfig.apertis.pm`)
-  * Clone and republish the OBS binary repositories
-  * Restart the OBS schedulers
-* Update build suffix on OBS Project configuration (eg. `bv2019.0b<B_CNT>`, `bv2020dev0b<B_CNT>`). Please refer to section **Build Suffix** for more details about it
+## Release Branching through Gitlab CI
+
+The release process has been semi-automated with the Gitlab CI framework. The following screenshots/notes will give a run through of the steps, to make a release.
+
+Some terms in regard to *release branching*:
+
+* `RELEASE`: The current release codename
+* `NEXT_RELEASE`: The new release codename that should be branched to
+* `PURPOSE`: Use string `branching` to instruct CI that it is a release procedure job
+* `NOACT`: Set to `0` to trigger the release
+
+By default, the release procedure job will run in `dry-run` mode, i.e. with `NOACT:1`. It is a good practice to first run the release process in `dry-run` mode to uncover any possible issues. When ready, re-run the CI pipeline jobs with `NOACT:0` to run in `real` mode.
+
+![500px|thumb|Release Branching Trigger Summary](/images/Initial_Trigger.png)
+
+This is the inital step to trigger release branching. Here:
+
+* We specify the current `RELEASE` to be branched, i.e. `v2023dev0` 
+* `NEXT_RELEASE` is the name of the new release that is to be branched, i.e. `v2023dev1`
+* `PURPOSE` is set to `branching` to tell the CI Pipeline that this is a release branching job.
+
+Note: `NOACT` has been deliberately left out in this example screenshot. So, in this case, the entire procedure will run in `dry-run` mode. To run in effective mode, pass `NOACT` with value `0`
+
+
+![500px|thumb|Stages Overview](/images/Stages.png)
+
+This is an overview of all the release branching jobs queued in the pipeline
+
+![500px|thumb|Stages Dependency Overview](/images/Job_Dependencies.png)
+
+This is an example of the same jobs, with their dependencies chalked out. There are multiple jobs that depend on one another. This is to ensure that jobs, that depend on certain outputs or tasks to be completed, are run only after.
+ 
+![500px|thumb|Dependent Run Overview](/images/Dependent_Run.png)
+
+This example highlights the dependency in action. A job is only run *after* its parent dependency job is run successfully. Jobs that do not have a connecting line are independent jobs that are run in parallel. In this exmaple, most jobs in the `branch` stage are *independent* and run in *parallel*
+
+![500px|thumb|Manual Job Overview](/images/Manual_Job_Summary.png)
+
+This example highlights *Manual Jobs* and their dependencies. *Manual Jobs* are differentiated with the *Play* Button. A *Manual Job* is to be run manually by a user. In the context of this document, a manual job constitues of certain commands that the user is expected to run by hand on respective servers. Most manual jobs, in this case, are to be run on the **OBS Backend** server. For each manual job, the exact commands will be displayed in the job's console view.
+
+![500px|thumb|Manual Job Run Detail](/images/Manual_Job_Run_Detail.png)
+
+This example highlights the exact set of commands that should be manually run. Most commands to be manually run, are to be run on specific servers, which will be mentioned in the console output for the particular job.
+
+![500px|thumb|Manual Job Run Dependency Chain](/images/Manual_Job_Run_Dependency_Chain.png)
+
+Most manual jobs are made dependent on other job. This is to ensure that steps are performed in a particular order. In the above example, the dependency is defined as: `obs_prjconf => obs_reprepro => obs_clone_binary`, where `obs_prjconf` runs a certain set of tasks, which are needed by `obs_reprepro`. Similarly `obs_reprepro` performs a certain set of tasks which are a pre-requisite for `obs_clone_binary`. The job chain is strictly defined and users should ensure to successfully run the manual jobs on respective servers, before progressing to the next job. 
+
+The above example complete the *semi-automated* aspect of the **Release Branching** steps.
+
+## Post CI Manual Steps
+
+After the completion of the **Release Branching** CI Jobs, a certain set of manual steps need to be performed, which are outlined below.
+
 * Add a directory for the release to the LAVA NFS path at: `/srv/images/public/lava/nfs/` on host images.apertis.org. The LAVA test jobs pick the kernel and initrd image from the release's directory path. Refer to the README for details
 * Instantiate a *new timer to rotate images* for the new release (eg. On host `aura`, run command `systemctl enable --now rotate-images@v2019.timer`)
 * Update the [ci-package-builder](https://gitlab.apertis.org/infrastructure/ci-package-builder/) to target the next release