guides: obs-apt-publisher: Document OBS aptly integration

Task https://phabricator.apertis.org/T8672

Aptly has been integrated to OBS as the publisher backend. It is aimed to replace reprepro completely.

content/guides/obs-aptly-publisher.md

++++
+title = "OBS aptly publisher integration"
+toc = true
+date = "2022-05-27"
++++
+
+This document outlines the OBS aptly publisher backend that has been implemented
+on the Collabora instance of OBS. The upstreaming process of this feature is
+currently in-progress.
+
+# Background
+
+Apertis relies on [OBS]( {{< ref "workflow-guide.md" >}} ) for building and
+publishing binary packages. However, upstream OBS provides an APT publisher
+based on `dpkg-scanpackages`, which is not suitable for a project the scale of
+Apertis, where a single OBS project contains a lot of packages.
+
+For a period of time, our OBS instance used a custom publisher based on
+`reprepro`, but it is still subject to some limitations that became more
+noticeable as the scale of Apertis increased considerably.
+
+## Aptly
+
+[Aptly](https://www.aptly.info/) is a complete solution for Debian repository
+management, including mirroring, snapshots and publication. It has been chosen
+as the target APT publisher and therefore it has been integrated to OBS.
+
+For a quick introduction and in-depth comparison on aptly features check this
+[article]( {{< ref "apt-publisher.md" >}} ).
+
+# OBS aptly integration overview
+
+OBS backend has been integrated to aptly, using a locally running aptly instance
+(no remote API).
+
+The only requirement to start using the aptly publisher backend on a
+project/repository is to have it defined in the OBS configuration files (i.e.
+`BSConfig.pm`), check the OBS aptly
+[documentation](https://gitlab.collabora.com/obs/open-build-service/-/blob/collabora/staging/README.aptly.md).
+
+The following features are integrated to OBS:
+* OBS repositories gets automatically created/removed on aptly.
+* Each arch with the publish flag enabled in a repository gets published on
+  aptly.
+* Package binaries get added to the aptly repository right after they get built.
+  Old versions are automatically removed.
+* Every time a package is published, a snapshot is taken and published by
+  calling the aptly publisher hook (if defined in the OBS configuration).
+* Aptly database cleanup routine is called periodically running a cron job.
+
+# Testing
+
+## Configuration requirements
+
+* Make sure the required environments variables are properly set as follows
+(default values are shown here):
+
+```
+$ export GNUPGHOME=/srv/obs/gnupg
+$ export OBS_FRONTEND_HOST=obs-frontend
+```
+
+* Add the following configuration to `/etc/obs/BSConfig.pm`, setting the
+`gpg-key` configuration to a valid GPG key:
+
+```perl
+my @reprepro_releases = ("v2022");
+my @aptly_releases = ("v2023");
+my @apertis_components = ("target", "development", "sdk");
+my $apertis_prefix = "shared/apertis/public/apertis";
+
+our $publishedhook_use_regex = 1;
+our $publishedhook = {};
+
+our $aptly_defconfig = {
+  "gpg-key" => "D4B4146191456112D814554C1DB3714017240E34",
+};
+
+foreach $release (@reprepro_releases) {
+  $publishedhook->{"apertis:$release:.*"} = "/usr/lib/obs/server/reprepro-snapshot";
+};
+
+foreach $release (@aptly_releases) {
+  $publishedhook->{"apertis:$release:.*"} = "/usr/lib/obs/server/bs_published_hook_aptly_snapshot";
+};
+
+our $reprepository = {};
+
+foreach $release (@reprepro_releases) {
+  foreach $component (@apertis_components) {
+    $reprepository->{"apertis:$release:$component/default"} = {
+      "repository" => $apertis_prefix,
+      "codename" => $release,
+      "component" => $component
+    };
+    $reprepository->{"apertis:$release:updates:$component/default"} = {
+      "repository" => $apertis_prefix,
+      "codename" => "$release-updates",
+      "component" => $component
+    };
+    $reprepository->{"apertis:$release:security:$component/default"} = {
+      "repository" => $apertis_prefix,
+      "codename" => "$release-security",
+      "component" => $component
+    };
+  };
+};
+
+our $aptly_config = {};
+
+foreach $release (@aptly_releases) {
+  foreach $component (@apertis_components) {
+    $aptly_config->{$apertis_prefix}{$release}{"components"}{$component} = {
+      "project" => "apertis:$release:$component",
+      "repository" => "default",
+    };
+    $aptly_config->{$apertis_prefix}{$release."-updates"}{"components"}{$component} = {
+      "project" => "apertis:$release:updates:$component",
+      "repository" => "default",
+    };
+    $aptly_config->{$apertis_prefix}{$release."-security"}{"components"}{$component} = {
+      "project" => "apertis:$release:security:$component",
+      "repository" => "default",
+    };
+  };
+};
+
+1;
+```
+
+## OBS aptly integration test
+
+Simple test setting up aptly repositories to validate OBS aptly backend
+integration.
+
+* Copy the following test files to a temporary location with write permissions
+to the `obsrun` user: `tests/aptly/aptly.sh tests/aptly/common.sh`
+
+* Run the test script, which should show this message on success:
+
+```
+$ ./aptly.sh
+[...]
+PASSED: all tests successfully passed!
+```
+
+## Reprepro to aptly migration test
+
+This test set a few reprepro repositories on OBS, then migrates them to aptly.
+
+* Copy the following test files to a temporary location with write permissions
+to the `obsrun` user: `tests/aptly/common.sh tests/aptly/migration.sh tests/aptly/reprepro.sh`
+
+* Run the tests scripts, which should show these messages on success:
+
+```
+$ ./reprepro.sh
+[...]
+PASSED: all tests successfully passed!
+$ ./migration.sh
+[...]
+PASSED: all tests successfully passed!
+```
+
+# Upgrading downstream instances to aptly publisher
+
+This section describes how to upgrade a downstream OBS instance to use aptly
+publisher backend on project/repositories.
+
+## Branch a new release
+
+A new Apertis release can be branched out of a previous release following the
+same [release process]({{<ref "apertis_release_process">}}) as done previously
+with reprepro. Note that this branching process is supported by the Apertis
+[infrastracture gitlab CI](https://gitlab.apertis.org/infrastructure/apertis-infrastructure/),
+which has been updated to switch from reprepro published backend to aptly.
+
+The only requirement before running the gitlab CI steps is to have the
+`NEXT_RELEASE` projects setup in the OBS configuration file so these use aptly
+as the publisher backend.
+
+## Migrate a reprepro repository to aptly
+
+The script `./src/backend/bs_aptly_migration` can be used to migrate reprepro
+repositories and snapshots to aptly. Note that this script needs to be run as
+`obsrun` user.
+
+Before running this script, OBS configuration must be updated moving the
+repositories from reprepro backend to aptly. As reference, check the OBS aptly
+[documentation](https://gitlab.collabora.com/obs/open-build-service/-/blob/collabora/staging/README.aptly.md).
+
+OBS services (srcserver and publisher) should be stopped before running this
+script and re-enabled afterwards, e.g.:
+
+```
+$ supervisorctl stop srcserver
+$ supervisorctl stop publisher
+[...] # Run migration
+$ supervisorctl start srcserver
+$ supervisorctl start publisher
+```
+
+Note that OBS projects are not modified by this script at all. Once the services
+are restarted it will start using the new aptly publisher instead of reprepro.
+All the previous packages and snapshots should be available and re-published
+transparently.
+
+On the other hand, reprepro repositories and snapshots are not modified by this
+script. These should be cleaned up manually from the filesystem afterwards.
+
+### Cleanup migrated reprepro repository
+
+After a reprepro-to-aptly migration has been performed, it might be useful to
+remove the previous reprepro repository entirely.
+
+The following script `reprepro_remove.sh` is a reference snippet that can be
+used to clean up a reprepro repository, along with all its snapshots.
+
+Before running this script, make sure the configuration entry for the target
+release has been removed from the reprepro conf/distributions file.
+
+```bash
+#!/bin/bash
+
+set -eu
+
+# Before running this script, make sure the configuration entry for the target
+# release has been removed from the reprepro conf/distributions file.
+
+ROOT_DIR="$(dirname -- "${BASH_SOURCE}")"
+ABS_ROOT_DIR="$(realpath $ROOT_DIR)"
+REPOS_DIR=/srv/obs/repos
+
+if [ $(id -nu) != obsrun ]; then
+	echo "ERROR: needs to be run as obsrun" >&2
+	exit 1
+fi
+
+if [ $# -ne 2 ]; then
+	echo "ERROR: wrong arguments" >&2
+	echo "Usage: $0 <osname> <distro>" >&2
+	exit 1
+fi
+
+prj_osname="$1"
+prj_distro="$2"
+
+# Publish path is assumed to have the following format
+prj_public_prefix="shared/$prj_osname/public/$prj_osname"
+
+reprepro="reprepro --gnupghome $GNUPGHOME -Vb $REPOS_DIR/$prj_public_prefix"
+
+$reprepro --delete clearvanished
+for d in $REPOS_DIR/$prj_public_prefix/dists/$prj_distro/snapshots/*; do
+	timestamp=$(basename $d)
+	$reprepro _removereferences s=$prj_distro=$timestamp
+done
+$reprepro deleteunreferenced
+rm -rf $REPOS_DIR/$prj_public_prefix/dists/$prj_distro
+```
+
+The above script receives two arguments `<osname> <distro>` pointing to the
+target reprepro release to remove. For example, to drop `apertis:v2022` release,
+you should run:
+
+```
+./reprepro_remove.sh apertis v2022
+```