diff --git a/content/qa/immutablerootfstests.md b/content/qa/immutablerootfstests.md index 4ac24a39dd465fd818a46dfff295d00e9e12a5c0..946b6b64cc5ee1ef4332ae6cc036b365c4db495b 100644 --- a/content/qa/immutablerootfstests.md +++ b/content/qa/immutablerootfstests.md @@ -31,39 +31,124 @@ should not require dependencies to be installed on the rootfs. ## Writing self-contained tests - - Each tests is shipped in a separate git repository under the - **[tests](https://gitlab.apertis.org/tests/)** GitLab group - - Check existing tests like the - **[D-Bus reply time](https://gitlab.apertis.org/tests/dbus-dos-reply-time/)** - ones or the - **[Canterbury](https://gitlab.apertis.org/tests/canterbury)** ones - - Add a subtree of the - [`common`](https://gitlab.apertis.org/tests/common) folder. For +The approach used by Apertis to create self contained tests is to ship all the +required test code via dedicated git repositories under the +**[tests](https://gitlab.apertis.org/tests/)** GitLab group which get deployed +by LAVA on the device before executing the tests. + +Those repositories can either contain interpreted code that does not need to be +built, for instance by using the `dash` POSIX shell, or they include binaries +built on OBS and synchronized with a GitLab CI/CD pipeline. + +For instance, the [`apparmor-dbus`](https://qa.apertis.org/latest/apparmor-dbus.html) +testcase involves: + +1. The [`apparmor-dbus.yaml`](https://gitlab.apertis.org/tests/apertis-test-cases/-/blob/apertis/v2021/test-cases/apparmor-dbus.yaml) + testcase definition +2. The [`tests/apparmor-dbus`](https://gitlab.apertis.org/tests/apparmor-dbus) + git repository +3. The [`apparmor-dbus-tester`](https://gitlab.apertis.org/pkg/apparmor-dbus-tester) + deb package + +### The testcase definition + +Both manual and automated testcases are defined via YAML files in the +[`apertis-test-cases`](https://gitlab.apertis.org/tests/apertis-test-cases/). + +The YAML files are used to generate the [qa.apertis.org](https://qa.apertis.org/) +website and the automated ones are executed by LAVA. + +Usually testcase need supporting files and scripts and these are shipped via +git repositories with the below directive: + +```yaml +install: + git-repos: + - url: https://gitlab.apertis.org/tests/apparmor-dbus.git + branch: 'apertis/v2021' +``` + +### The git repository + +The easiest way to deploy supporting files, scripts and binaries on the device +under test is by storing them in standalone git repositories that then get +unpacked on the device before the test is executed. + +In the simplest case the support git repository only ships a shell script and +maybe some static test samples. Keep in mind that the Apertis reference images +do not ship `bash` on minimal/target images but use `dash`, so stick to POSIX +features and avoid [bashisms](https://mywiki.wooledge.org/Bashism). + +If instead compiled binaries are needed, they need to be built on OBS and +regularly synchronized in the git repository. Apertis provides a reusable +GitLab pipeline to retrieve those external binaries and commit them in the +git repository. To set it up: + +1. Create a new GitLab project under the + [`test`](https://gitlab.apertis.org/tests) group. +2. Add a subtree of the + [`common`](https://gitlab.apertis.org/tests/common) folder. For convenience, use the - [`common-subtree.sh`](https://gitlab.apertis.org/tests/common/blob/master/common-subtree.sh) + [`common-subtree.sh`](https://gitlab.apertis.org/tests/common/blob/apertis/v2021/common-subtree.sh) script to add a *git-subtree* into your repository: - ../common/common-subtree.sh add + ```shell + ../common/common-subtree.sh add + ``` See the - [README](https://gitlab.apertis.org/tests/common/blob/master/README.md) + [README](https://gitlab.apertis.org/tests/common/blob/apertis/v2021/README.md) for further details - - For tests requiring additional binaries not shipped on the base - image: - - Add an - [`external-binaries.cfg`](https://gitlab.apertis.org/tests/dbus-dos-reply-time/blob/master/external-binaries.cfg) - file - - Run the - [`update_test_binaries.sh`](https://gitlab.apertis.org/tests/helper-tools/blob/master/update_test_binaries.sh) - script: - ./helper-tools/update_test_binaries.sh --repository <name-of-repo-under-gitlab-tests> --release 18.06 --dry-run - `     This should add a commit with the binaries required by your test repository` - - Tests should use compiled executables included with - `update_test_binaries.sh` and/or POSIX shell scripts (avoid - [bashisms](https://mywiki.wooledge.org/Bashism) and assume that - `/bin/sh` points to `dash`) - - Test the scripts in a `minimal` or `ostree` image - - Add your repository to the - [`test-repositories.txt` list](https://gitlab.apertis.org/tests/helper-tools/blob/master/test-repositories.txt) - and Jenkins will keep the binaries automatically updated +3. Add an + [`external-binaries.cfg`](https://gitlab.apertis.org/tests/apparmor-dbus/blob/apertis/v2021/external-binaries.cfg) + file which describes the binaries that need to be retrieved and committed. + All the files that would otherwise need to be installed via `apt` should be + listed here: check what the dependencies of the test are and inspect their + contents, filtering out the files which are not needed to execute the test. + The format is a simple list of `$PACKAGE $FILEPATH` on each line. + The retrieved binaries will be committed under the appropriate `$ARCH/bin` + subdirectory in the repository. An optional `prefix=` attribute is supported + to control where the retrieved file should be installed, for instance + from the + [`canterbury`](https://gitlab.apertis.org/tests/canterbury/-/blob/apertis/v2021/external-binaries.cfg) + test repository: + + ``` + canterbury-tests-additions usr/lib/installed-tests/canterbury-0/store-app-bundle-apparmor prefix=lib/installed-tests/canterbury-0 + canterbury-core-tests /etc/apparmor.d/canterbury-tests prefix=etc/apparmor.d + ``` + +4. Add a [`.gitlab-ci.yml`](https://gitlab.apertis.org/tests/apparmor-dbus/-/blob/apertis/v2021/.gitlab-ci.yml) + which defines the GitLab CI/CD pipeline that is responsible of keeping the + binaries updated by pulling them from the built packages and committing them + to the repository: + + ```yaml + variables: + osname: apertis + release: v2021 + include: + - project: tests/helper-tools + ref: apertis/v2021 + file: /gitlab-ci/update-test-binaries.yaml + update-test-binaries: + stage: build + extends: + - .update-test-binaries + ``` + +5. Configure a [CI/CD schedule](https://gitlab.apertis.org/tests/apparmor-dbus/-/pipeline_schedules) + to run the pipeline daily. + +### The deb package + +In some cases the needed binaries can be extracted from existing packages. + +For instance the [`glib-gio-fs`](https://gitlab.apertis.org/tests/glib-gio-fs/-/blob/apertis/v2021/external-binaries.cfg) +testcases sources them from the upstream `libglib2.0-tests` package. + +In other cases a new, dedicated package needs to be created to host the test +utilities. No specific requirements are needed on this package, but +conventionally they are named `${TESTCASE}-tester`, like +[`apparmor-dbus-tester`](https://gitlab.apertis.org/tests/glib-gio-fs/-/blob/apertis/v2022dev1/external-binaries.cfg). ## References