Skip to content
Snippets Groups Projects
Commit 02c9a0b5 authored by Denis Pynkin's avatar Denis Pynkin Committed by Peter Senna Tschudin
Browse files

guide: add Apertis testing with LAVA


Added the guide how to create Apertis tests for LAVA.

Signed-off-by: default avatarDenis Pynkin <denis.pynkin@collabora.com>
parent 05705347
No related branches found
No related tags found
1 merge request!204Add guide: "Apertis integration testing with LAVA"
+++
title = "Apertis integration testing with LAVA"
date = "2021-03-31"
weight = 100
outputs = [ "html", "pdf-in",]
+++
[LAVA](https://www.lavasoftware.org/) is a testing system allowing to deploy
operation system to physical and virtual devices, sharing access to devices
between developers. As a rule tests are started in non-interactive unattended
mode and LAVA provide logs and results in a human-readable form for analysis.
As a common part of development cycle we need to do some integration testing
of the application and validate it's behavior on different hardware and
software platforms.
## Integration testing example
Let's take the `systemd` service and `systemctl` CLI tool as example to
illustrate how to test the application with D-Bus interface.
The goal could be defined as follows:
> as a developer of `systemctl` CLI tool, I want to ensure what the `systemctl`
> is able to provide correct information about the system state.
## Local testing
To simplify the guide we are testing only the correct status of `systemd` with
command below:
$ systemctl is-system-running
running
To ensure that the `systemctl` tool is really provides the correct information
we may check the system state additionally via `systemd` D-Bus interface:
$ gdbus call --system --dest=org.freedesktop.systemd1 --object-path "/org/freedesktop/systemd1" --method org.freedesktop.DBus.Properties.Get org.freedesktop.systemd1.Manager SystemState
(<'running'>,)
So, for the local testing during development we are able to create a simple script
validating that the `systemctl` works well in our development environment:
```
#!/bin/sh
status=$(systemctl is-system-running)
gdbus call --system --dest=org.freedesktop.systemd1 \
--object-path "/org/freedesktop/systemd1" \
--method org.freedesktop.DBus.Properties.Get org.freedesktop.systemd1.Manager SystemState | \
grep "${status}"
if [ $? -eq 0 ]; then
echo "systemctl is working"
else
echo "systemctl is not working"
fi
```
## Testing in LAVA
As soon as we done with development, we push all changes to GitLab and CI will
prepare a new version of package and OS images. But we do not know if the
updated version of `systemctl` is working well for all supported devices and
OS variants, so we want to have the integration test to be run by LAVA.
Since the LAVA is a part of CI and works in non-interactive unattended mode we
can't use the test script above as is.
To start the test with LAVA automation we need to:
1. Adopt the script for LAVA
2. Integrate the testing script into Apertis LAVA CI
### Changes in testing script
The script above is not suitable for unattended testing in LAVA due some issues:
- LAVA relies to exit code to determine if test passed or not. The example above
always return the `success` code, only human-readable string allows to validate
the status of `systemctl`
- if `systemctl is-system-running` call is failed for some reason (segfault for
instance), the script will proceed further without that error detection and LAVA
will set the test as passed, so we will have a false positive result
- LAVA is able to report separately for any part of the test suite -- just need
to use LAVA-friendly output pattern
So, more sophisticated script suitable both for local and unattended testing in
LAVA could be following:
```
#!/bin/sh
# Test if systemctl is not crashed
testname="test-systemctl-crash"
status=$(systemctl is-system-running)
if [ $? -le 4 ]; then
echo "${testname}: pass"
else
echo "${testname}: fail"
exit 1
fi
# Test if systemctl return non-empty string
testname="test-systemctl-value"
if [ -n "$status" ]; then
echo "${testname}: pass"
else
echo "${testname}: fail"
exit 1
fi
# Test if systemctl is reporting the same status as
# systemd exposing via D-Bus
testname="test-systemctl-dbus-status"
gdbus call --system --dest=org.freedesktop.systemd1 \
--object-path "/org/freedesktop/systemd1" \
--method org.freedesktop.DBus.Properties.Get \
org.freedesktop.systemd1.Manager SystemState | \
grep "${status}"
if [ $? -eq 0 ]; then
echo "${testname}: pass"
else
echo "${testname}: fail"
exit 1
fi
```
Now the script is ready for adding into LAVA testing. Pay attention to
output format which will be used by LAVA to detect separate tests from our
single script. The exit code from the testing script must be non-zero to
indicate the test suite failure.
### Create GIT repository for the test suite
The test script must be accessible by LAVA for downloading. LAVA have support
of several methods for downloading but for Apertis the GIT fetch is preferable
since we are using separate version of test scripts for each release.
It is strongly recommended to create separate repository with test scripts
and tools for each single test suite.
As a first step we need a fresh and empty GIT repository anywhere, for
example in personal space of GitLab instance and it's local copy:
git clone git@gitlab.apertis.org:d4s/test-systemctl.git
cd test-systemctl
By default the branch name is set to `main` but Apertis automation require
to use the branch name aimed to selected release (for instance
`apertis/v2022dev1`), so need to create it:
git checkout HEAD -b apertis/v2022dev1
Copy your script into GIT repository, commit and push it into GitLab:
chmod a+x test-systemctl.sh
git add test-systemctl.sh
git commit -s -m "Add test script" test-systemctl.sh
git push -u origin apertis/v2022dev1
### Add the test into Apertis LAVA CI
Apertis test automation could be found in [GIT repository with Apertis test
cases](https://gitlab.apertis.org/tests/apertis-test-cases),
so we need to fetch the local copy and create a work branch `wip/example`
for our changes:
git clone git@gitlab.apertis.org:tests/apertis-test-cases.git
cd apertis-test-cases
git checkout HEAD -b wip/example
1. Create test case description
First of all we need to create the instruction for LAVA with following
information:
- where to get the test
- how to run the test
Create the test case file `test-cases/test-systemctl.yaml` with your
favorite editor:
```
metadata:
name: test-systemctl
format: "Apertis Test Definition 1.0"
image-types:
minimal: [ armhf, arm64, amd64 ]
image-deployment:
- OSTree
type: functional
exec-type: automated
priority: medium
maintainer: "Apertis Project"
description: "Test the systemctl."
expected:
- "The output should show pass."
install:
git-repos:
- url: https://gitlab.apertis.org/d4s/test-systemctl.git
branch: apertis/v2022dev1
run:
steps:
- "# Enter test directory:"
- cd test-systemctl
- "# Execute the following command:"
- lava-test-case test-systemctl --shell ./test-systemctl.sh
parse:
pattern: "(?P<test_case_id>.*):\\s+(?P<result>(pass|fail))"
```
This test is aimed to be run for ostree-based minimal Apertis image for
all supported architectures. However the metadata is mostly needed for
documentation purposes.
Action "install" point to GIT repository as a source for the test, so LAVA
will fetch and deploy this repository for us.
Action "run" provide the step-by-step instruction how to execute the test.
Please note that it is recommended to use wrapper for the test for
integration with LAVA.
Action "parse" provides own detection for the status of test results printed
by script.
2. Push the test case to the GIT repository.
This step is mandatory since the test case would be checked out by LAVA
internally during the test preparation.
git add test-cases/test-systemctl.yaml
git commit -s -m "add test case for systemctl" test-cases/test-systemctl.yaml
git push --set-upstream origin wip/example
3. Add job template to be run in lava. Job template contains all needed
information for LAVA how to boot the target device and deploy the OS image
onto it.
Create the simple template `lava/test-systemctl-tpl.yaml` with your lovely
editor:
```
job_name: systemctl test on {{release_version}} {{pretty}} {{image_date}}
{% if device_type == 'qemu' %}
{% include 'common-qemu-boot-tpl.yaml' %}
{% else %}
{% include 'common-boot-tpl.yaml' %}
{% endif %}
- test:
timeout:
minutes: 15
namespace: system
name: common-tests
definitions:
- repository: https://gitlab.apertis.org/tests/apertis-test-cases.git
revision: 'wip/example'
from: git
path: test-cases/test-systemctl.yaml
name: test-systemctl
```
Hopefully you don't need to deal with HW-related part, since we already
have those instructions for all supported boards and Apertis OS images.
Please pay attention to `revision` -- it must point to your development
branch while you are working on your test.
Instead of creation of the new template you may want to extend appropriate
existing template with additional test definition. In this case next step
could be omitted.
6. Add the template into profile.
Profile file is mapping test jobs to devices under the test. So you need to
add your job template into proper list. For example we may extend
the templates list named `templates-minimal-ostree` in file
`lava/profiles.yaml`:
```
.templates:
- &templates-minimal-ostree
- test-systemctl-tpl.yaml
```
It is highly recommended to temporary remove or comment out the rest of
templates from the list to avoid unnecessary workload on LAVA while you're
developing the test.
7. Check the profile and template locally first.
As a first step you have to define the proper profile name to use for the
test in LAVA.
**NB**: you need to have `lqa` tool installed and configured as described
in [personal LAVA tests](/qa/personal_lava_tests/) tutorial.
Since the LAVA is a part of Apertis OS CI -- it requires some variables to
be provided for using of Apertis profiles and templates. Let's define the
board we will use for testing, as well as the image release and variant:
release=v2022dev1
version=v2022dev1.0rc2
variant=minimal
arch=armhf
board=uboot
baseurl="https://images.apertis.org"
imgpath="release/$release"
profile_name=apertis_ostree-${variant}-${arch}-${board}
image_name=apertis_ostree_${release}-${variant}-${arch}-${board}_${version}
And now we are able to submit the test in a dry-run mode:
lqa submit -g lava/profiles.yaml -p ${profile_name} \
-t visibility:"{'group': ['Apertis']}" -t priority:"high" \
-t imgpath:${imgpath} -t release:${release} -t image_date:${version} \
-t image_name:${image_name} -n
There should not be any error or warning from `lqa`. You may want to add `-v`
argument to see the generated LAVA job.
**NB**: it is recommended to set `visibility` variable to "Apertis" group
during development to avoid any credentials/passwords leak by occasion.
Set the additional variable `priority` to `high` allows you to bypass the
jobs common queue and do not wait your job results for ages.
8. Submit your first job to LAVA.
Just repeat the `lqa` call above without `-n` option.
After the job submission you will see the job ID:
lqa submit -g lava/profiles.yaml -p "${profile_name}" -t visibility:"{'group': ['Apertis']}" -t priority:"high" -t imgpath:${imgpath} -t release:${release} -t image_date:${version} -t image_name:${image_name}
Submitted job test-systemctl-tpl.yaml with id 3463731
So it is possible to check the job status by URL with that ID:
https://lava.collabora.co.uk/scheduler/job/3463731
9. Push your changes from template and profile.
As soon as your test case works as expected you may restore all commented
templates for profile, change the `revision` key in file
`lava/test-systemctl-tpl.yaml` to target branch and submit your changes:
git add lava/test-systemctl-tpl.yaml lava/profiles.yaml
git commit -a -m "hello world template added"
git push
As a last step you need to create merge request in GitLab and as soon as it
would be accepted your test became a part of Apertis testing CI.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment