Skip to content
Snippets Groups Projects
Commit 8bf6aa11 authored by Dylan Aïssi's avatar Dylan Aïssi
Browse files

Add new guide for package testing using LAVA

parent c9a9a16f
No related branches found
No related tags found
1 merge request!587Add new guide for package testing using LAVA
Pipeline #666392 passed with warnings
+++
title = "Apertis packages testing with LAVA"
date = "2023-12-20"
weight = 100
toc = true
+++
Automatic testing of packages is essential to ensure high quality software. This
document describe how to define tests and run them as part of the Apertis
`ci-package-builder` pipeline. A job called `package-centric-testing` will
install freshly built packages in an Apertis image and will run a test command
to ensure the package works as expected.
# How to enable testing of packages
The first step is to create a file `debian/apertis/package-centric-testing.yaml`
in the source package. This file is used to define the test environment (image ID,
image type, architecture, board type, installed packages) and the test itself
(it can be a simple command or a test script for a more complex test).
The presence of this file will add a testing job in the `ci-package-builder`
pipeline.
`package-centric-testing.yaml` is a YAML file with a syntax similar to the one
used for [Apertis Test Cases]( {{< ref "test_cases_guidelines.md" >}} ). Here is
a simple example for package `liblc3`:
```yaml
tests:
image-types:
fixedfunction: [ armhf, arm64 ]
hmi: [ armhf, arm64 ]
basesdk: [ amd64 ]
sdk: [ amd64 ]
bin_pkg_list: >
liblc3-0
liblc3-tools
pkg_test_cmd: "echo Hello World"
```
In this example, we will test our package in 6 different environments which are:
- on `armhf` with a `fixedfunction` image;
- on `arm64` with a `fixedfunction` image;
- on `armhf` with a `hmi` image;
- on `arm64` with a `hmi` image;
- on `amd64` with a `basesdk` image;
- on `amd64` with a `sdk` image;
For each environment, freshly built packages listed in `bin_pkg_list` will be
installed then the command defined in `pkg_test_cmd` will be run.
In other words, this test will trigger 6 different LAVA jobs.
It's possible to refine the test environment by defining the board type and the
image ID. By default, the image ID used is always the latest daily successfully
built, but it is possible to override it with the `image_buildid` field.
Regarding the board type, by default `uboot` is used as board type for both
architectures `arm64` and `armhf`, so based on the [devices.yaml](https://gitlab.apertis.org/tests/apertis-test-cases/-/raw/apertis/v2025dev1/lava/devices.yaml)
`arm64` tests will run on an Renesas R-Car board, this can be overridden by selecting
another board type, like in the following test definition:
```yaml
tests:
image-types:
fixedfunction: [ armhf, arm64: [uboot, rpi64] ]
hmi: [ armhf, arm64: [rpi64] ]
basesdk: [ amd64 ]
sdk: [ amd64 ]
image_buildid: "20231211.0015"
bin_pkg_list: >
liblc3-0
liblc3-tools
pkg_test_cmd: "echo Hello World"
```
In this second example, the `arm64` tests will be run on both an Renesas R-Car
board and a Raspberry Pi 4B board with a `fixedfunction` image. And the `arm64`
tests with an `hmi` image will only run on a Raspberry Pi 4B board.
We can also see that the `image_buildid` field is used, that means the image used
won't be the latest daily but the daily built on `20231211.0015`.
The command defined in `pkg_test_cmd` will be run in the `debian/tests/` folder
(if the folder is not available, it will be created). Running the `pkg_test_cmd`
in `debian/tests/` allow us to easily re-use tests written by Debian community
as part of the Debian's [autopkgtest](https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst)
([DEP8](https://dep-team.pages.debian.net/deps/dep8/)).
# How to write complex tests
The first step is to define a test with `debian/apertis/package-centric-testing.yaml`:
```yaml
tests:
...
pkg_test_cmd: "./run_complexe_test.sh"
```
For this test, instead of running a simple command, we will call a script
called `run_complexe_test.sh` which allow us to do a more complex testing.
This script `run_complexe_test.sh` must be located in `debian/tests/` and can
contain several commands or call other scripts like in this example:
```sh
sudo apt update
sudo apt install some-required-packages -y
my-binary --version
my-binary --help
./run-an-autopkgtest
./run-another-autopkgtest
```
This fake example will update package information before installing a package
`some-required-packages`. Then, it will run `my-binary` with the `--version`
argument and then with `--help`. Finally, our script will call other scripts
`run-an-autopkgtest` and `run-another-autopkgtest` which are also located in
`debian/tests/` in our source package.
# Behind the scene
When a `debian/apertis/package-centric-testing.yaml` is available in a source
package, a new job is added to the [ci-package-builder](https://gitlab.apertis.org/infrastructure/ci-package-builder)
pipeline.
This job `package-centric-testing` will clone [apertis-test-cases](https://gitlab.apertis.org/tests/apertis-test-cases)
and will use its logic to generate LAVA jobs. More precisely, it will use
`generate-pkg-centric-jobs.py` which reads the job definition from
`package-centric-testing.yaml` and from there it will call `generate-jobs.py` to
generate LAVA jobs. Then, based on the LAVA jobs created, `generate-test-pipeline.py`
will generate a child pipeline `package-centric-testing-monitor` composed of the
generated LAVA jobs.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment