From 3ec77fd0b4c1e50e2a3ceb8171383bc6910a70fc Mon Sep 17 00:00:00 2001
From: Emanuele Aina <emanuele.aina@collabora.com>
Date: Wed, 27 Jan 2021 10:47:50 +0100
Subject: [PATCH] immutablerootfstests: Overhaul the instructions
Give some more contetx about the Apertis approach to standalone tests
and update the instructions to be more detailed and reflect the
current status (bye bye Jenkins).
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>
---
content/qa/immutablerootfstests.md | 143 +++++++++++++++++++++++------
1 file changed, 114 insertions(+), 29 deletions(-)
diff --git a/content/qa/immutablerootfstests.md b/content/qa/immutablerootfstests.md
index 4ac24a39d..946b6b64c 100644
--- a/content/qa/immutablerootfstests.md
+++ b/content/qa/immutablerootfstests.md
@@ -31,39 +31,124 @@ should not require dependencies to be installed on the rootfs.
## Writing self-contained tests
- - Each tests is shipped in a separate git repository under the
- **[tests](https://gitlab.apertis.org/tests/)** GitLab group
- - Check existing tests like the
- **[D-Bus reply time](https://gitlab.apertis.org/tests/dbus-dos-reply-time/)**
- ones or the
- **[Canterbury](https://gitlab.apertis.org/tests/canterbury)** ones
- - Add a subtree of the
- [`common`](https://gitlab.apertis.org/tests/common) folder. For
+The approach used by Apertis to create self contained tests is to ship all the
+required test code via dedicated git repositories under the
+**[tests](https://gitlab.apertis.org/tests/)** GitLab group which get deployed
+by LAVA on the device before executing the tests.
+
+Those repositories can either contain interpreted code that does not need to be
+built, for instance by using the `dash` POSIX shell, or they include binaries
+built on OBS and synchronized with a GitLab CI/CD pipeline.
+
+For instance, the [`apparmor-dbus`](https://qa.apertis.org/latest/apparmor-dbus.html)
+testcase involves:
+
+1. The [`apparmor-dbus.yaml`](https://gitlab.apertis.org/tests/apertis-test-cases/-/blob/apertis/v2021/test-cases/apparmor-dbus.yaml)
+ testcase definition
+2. The [`tests/apparmor-dbus`](https://gitlab.apertis.org/tests/apparmor-dbus)
+ git repository
+3. The [`apparmor-dbus-tester`](https://gitlab.apertis.org/pkg/apparmor-dbus-tester)
+ deb package
+
+### The testcase definition
+
+Both manual and automated testcases are defined via YAML files in the
+[`apertis-test-cases`](https://gitlab.apertis.org/tests/apertis-test-cases/).
+
+The YAML files are used to generate the [qa.apertis.org](https://qa.apertis.org/)
+website and the automated ones are executed by LAVA.
+
+Usually testcase need supporting files and scripts and these are shipped via
+git repositories with the below directive:
+
+```yaml
+install:
+ git-repos:
+ - url: https://gitlab.apertis.org/tests/apparmor-dbus.git
+ branch: 'apertis/v2021'
+```
+
+### The git repository
+
+The easiest way to deploy supporting files, scripts and binaries on the device
+under test is by storing them in standalone git repositories that then get
+unpacked on the device before the test is executed.
+
+In the simplest case the support git repository only ships a shell script and
+maybe some static test samples. Keep in mind that the Apertis reference images
+do not ship `bash` on minimal/target images but use `dash`, so stick to POSIX
+features and avoid [bashisms](https://mywiki.wooledge.org/Bashism).
+
+If instead compiled binaries are needed, they need to be built on OBS and
+regularly synchronized in the git repository. Apertis provides a reusable
+GitLab pipeline to retrieve those external binaries and commit them in the
+git repository. To set it up:
+
+1. Create a new GitLab project under the
+ [`test`](https://gitlab.apertis.org/tests) group.
+2. Add a subtree of the
+ [`common`](https://gitlab.apertis.org/tests/common) folder. For
convenience, use the
- [`common-subtree.sh`](https://gitlab.apertis.org/tests/common/blob/master/common-subtree.sh)
+ [`common-subtree.sh`](https://gitlab.apertis.org/tests/common/blob/apertis/v2021/common-subtree.sh)
script to add a *git-subtree* into your repository:
- ../common/common-subtree.sh add
+ ```shell
+ ../common/common-subtree.sh add
+ ```
See the
- [README](https://gitlab.apertis.org/tests/common/blob/master/README.md)
+ [README](https://gitlab.apertis.org/tests/common/blob/apertis/v2021/README.md)
for further details
- - For tests requiring additional binaries not shipped on the base
- image:
- - Add an
- [`external-binaries.cfg`](https://gitlab.apertis.org/tests/dbus-dos-reply-time/blob/master/external-binaries.cfg)
- file
- - Run the
- [`update_test_binaries.sh`](https://gitlab.apertis.org/tests/helper-tools/blob/master/update_test_binaries.sh)
- script:
- ./helper-tools/update_test_binaries.sh --repository <name-of-repo-under-gitlab-tests> --release 18.06 --dry-run
- `     This should add a commit with the binaries required by your test repository`
- - Tests should use compiled executables included with
- `update_test_binaries.sh` and/or POSIX shell scripts (avoid
- [bashisms](https://mywiki.wooledge.org/Bashism) and assume that
- `/bin/sh` points to `dash`)
- - Test the scripts in a `minimal` or `ostree` image
- - Add your repository to the
- [`test-repositories.txt` list](https://gitlab.apertis.org/tests/helper-tools/blob/master/test-repositories.txt)
- and Jenkins will keep the binaries automatically updated
+3. Add an
+ [`external-binaries.cfg`](https://gitlab.apertis.org/tests/apparmor-dbus/blob/apertis/v2021/external-binaries.cfg)
+ file which describes the binaries that need to be retrieved and committed.
+ All the files that would otherwise need to be installed via `apt` should be
+ listed here: check what the dependencies of the test are and inspect their
+ contents, filtering out the files which are not needed to execute the test.
+ The format is a simple list of `$PACKAGE $FILEPATH` on each line.
+ The retrieved binaries will be committed under the appropriate `$ARCH/bin`
+ subdirectory in the repository. An optional `prefix=` attribute is supported
+ to control where the retrieved file should be installed, for instance
+ from the
+ [`canterbury`](https://gitlab.apertis.org/tests/canterbury/-/blob/apertis/v2021/external-binaries.cfg)
+ test repository:
+
+ ```
+ canterbury-tests-additions usr/lib/installed-tests/canterbury-0/store-app-bundle-apparmor prefix=lib/installed-tests/canterbury-0
+ canterbury-core-tests /etc/apparmor.d/canterbury-tests prefix=etc/apparmor.d
+ ```
+
+4. Add a [`.gitlab-ci.yml`](https://gitlab.apertis.org/tests/apparmor-dbus/-/blob/apertis/v2021/.gitlab-ci.yml)
+ which defines the GitLab CI/CD pipeline that is responsible of keeping the
+ binaries updated by pulling them from the built packages and committing them
+ to the repository:
+
+ ```yaml
+ variables:
+ osname: apertis
+ release: v2021
+ include:
+ - project: tests/helper-tools
+ ref: apertis/v2021
+ file: /gitlab-ci/update-test-binaries.yaml
+ update-test-binaries:
+ stage: build
+ extends:
+ - .update-test-binaries
+ ```
+
+5. Configure a [CI/CD schedule](https://gitlab.apertis.org/tests/apparmor-dbus/-/pipeline_schedules)
+ to run the pipeline daily.
+
+### The deb package
+
+In some cases the needed binaries can be extracted from existing packages.
+
+For instance the [`glib-gio-fs`](https://gitlab.apertis.org/tests/glib-gio-fs/-/blob/apertis/v2021/external-binaries.cfg)
+testcases sources them from the upstream `libglib2.0-tests` package.
+
+In other cases a new, dedicated package needs to be created to host the test
+utilities. No specific requirements are needed on this package, but
+conventionally they are named `${TESTCASE}-tester`, like
+[`apparmor-dbus-tester`](https://gitlab.apertis.org/tests/glib-gio-fs/-/blob/apertis/v2022dev1/external-binaries.cfg).
## References
--
GitLab