guide: add Apertis testing with LAVA

Added the guide how to create Apertis tests for LAVA.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

content/guides/lava-apertis-testing.md

++++
+title = "Apertis integration testing with LAVA"
+date = "2021-03-31"
+weight = 100
+outputs = [ "html", "pdf-in",]
++++
+
+
+[LAVA](https://www.lavasoftware.org/) is a testing system allowing to deploy
+operation system to physical and virtual devices, sharing access to devices
+between developers. As a rule tests are started in non-interactive unattended
+mode and LAVA provide logs and results in a human-readable form for analysis.
+
+As a common part of development cycle we need to do some integration testing
+of the application and validate it's behavior on different hardware and
+software platforms.
+
+## Integration testing example
+
+Let's take the `systemd` service and `systemctl` CLI tool as example to
+illustrate how to test the application with D-Bus interface.
+
+The goal could be defined as follows:
+>  as a developer of `systemctl` CLI tool, I want to ensure what the `systemctl`
+>  is able to provide correct information about the system state.
+
+## Local testing
+
+To simplify the guide we are testing only the correct status of `systemd` with
+command below:
+
+    $ systemctl is-system-running
+    running
+
+To ensure that the `systemctl` tool is really provides the correct information
+we may check the system state additionally via `systemd` D-Bus interface:
+
+    $ gdbus call --system --dest=org.freedesktop.systemd1 --object-path "/org/freedesktop/systemd1" --method org.freedesktop.DBus.Properties.Get org.freedesktop.systemd1.Manager SystemState
+    (<'running'>,)
+
+So, for the local testing during development we are able to create a simple script
+validating that the `systemctl` works well in our development environment:
+```
+#!/bin/sh
+
+status=$(systemctl is-system-running)
+
+gdbus call --system --dest=org.freedesktop.systemd1 \
+  --object-path "/org/freedesktop/systemd1" \
+  --method org.freedesktop.DBus.Properties.Get org.freedesktop.systemd1.Manager SystemState | \
+  grep "${status}"
+
+if [ $? -eq 0 ]; then
+  echo "systemctl is working"
+else
+  echo "systemctl is not working"
+fi
+```
+
+## Testing in LAVA
+
+As soon as we done with development, we push all changes to GitLab and CI will
+prepare a new version of package and OS images. But we do not know if the
+updated version of `systemctl` is working well for all supported devices and
+OS variants, so we want to have the integration test to be run by LAVA.
+
+Since the LAVA is a part of CI and works in non-interactive unattended mode we
+can't use the test script above as is.
+
+To start the test with LAVA automation we need to:
+1. Adopt the script for LAVA
+2. Integrate the testing script into Apertis LAVA CI
+
+### Changes in testing script
+
+The script above is not suitable for unattended testing in LAVA due some issues:
+
+- LAVA relies to exit code to determine if test passed or not. The example above
+  always return the `success` code, only human-readable string allows to validate
+  the status of `systemctl`
+- if `systemctl is-system-running` call is failed for some reason (segfault for
+  instance), the script will proceed further without that error detection and LAVA
+  will set the test as passed, so we will have a false positive result
+- LAVA is able to report separately for any part of the test suite -- just need
+  to use LAVA-friendly output pattern
+
+So, more sophisticated script suitable both for local and unattended testing in
+LAVA could be following:
+
+```
+#!/bin/sh
+
+# Test if systemctl is not crashed
+testname="test-systemctl-crash"
+status=$(systemctl is-system-running)
+if [ $? -le 4 ]; then
+  echo "${testname}: pass"
+else
+  echo "${testname}: fail"
+  exit 1
+fi
+
+# Test if systemctl return non-empty string
+testname="test-systemctl-value"
+if [ -n "$status" ]; then
+  echo "${testname}: pass"
+else
+  echo "${testname}: fail"
+  exit 1
+fi
+
+# Test if systemctl is reporting the same status as
+# systemd exposing via D-Bus
+testname="test-systemctl-dbus-status"
+gdbus call --system --dest=org.freedesktop.systemd1 \
+  --object-path "/org/freedesktop/systemd1" \
+  --method org.freedesktop.DBus.Properties.Get \
+  org.freedesktop.systemd1.Manager SystemState | \
+  grep "${status}"
+if [ $? -eq 0 ]; then
+  echo "${testname}: pass"
+else
+  echo "${testname}: fail"
+  exit 1
+fi
+```
+
+Now the script is ready for adding into LAVA testing. Pay attention to
+output format which will be used by LAVA to detect separate tests from our
+single script. The exit code from the testing script must be non-zero to
+indicate the test suite failure.
+
+### Create GIT repository for the test suite
+
+The test script must be accessible by LAVA for downloading. LAVA have support
+of several methods for downloading but for Apertis the GIT fetch is preferable
+since we are using separate version of test scripts for each release.
+
+It is strongly recommended to create separate repository with test scripts
+and tools for each single test suite.
+
+As a first step we need a fresh and empty GIT repository anywhere, for
+example in personal space of GitLab instance and it's local copy:
+
+    git clone git@gitlab.apertis.org:d4s/test-systemctl.git
+    cd test-systemctl
+
+
+By default the branch name is set to `main` but Apertis automation require
+to use the branch name aimed to selected release (for instance
+`apertis/v2022dev1`), so need to create it:
+   
+    git checkout HEAD -b apertis/v2022dev1
+
+Copy your script into GIT repository, commit and push it into GitLab:
+
+    chmod a+x test-systemctl.sh
+    git add test-systemctl.sh
+    git commit -s -m "Add test script" test-systemctl.sh
+    git push -u origin apertis/v2022dev1
+   
+
+### Add the test into Apertis LAVA CI
+
+Apertis test automation could be found in [GIT repository with Apertis test
+cases](https://gitlab.apertis.org/tests/apertis-test-cases),
+so we need to fetch the local copy and create a work branch `wip/example`
+for our changes:
+
+    git clone git@gitlab.apertis.org:tests/apertis-test-cases.git
+    cd apertis-test-cases
+    git checkout HEAD -b wip/example
+
+1. Create test case description
+
+   First of all we need to create the instruction for LAVA with following
+   information:
+   - where to get the test
+   - how to run the test
+
+   Create the test case file `test-cases/test-systemctl.yaml` with your
+   favorite editor:
+   ```
+   metadata:
+     name: test-systemctl
+     format: "Apertis Test Definition 1.0"
+     image-types:
+       minimal:  [ armhf, arm64, amd64 ]
+     image-deployment:
+       - OSTree
+     type: functional
+     exec-type: automated
+     priority: medium
+     maintainer: "Apertis Project"
+     description: "Test the systemctl."
+
+     expected:
+       - "The output should show pass."
+
+   install:
+     git-repos:
+       - url: https://gitlab.apertis.org/d4s/test-systemctl.git
+         branch: apertis/v2022dev1
+
+   run:
+     steps:
+       - "# Enter test directory:"
+       - cd test-systemctl
+       - "# Execute the following command:"
+       - lava-test-case test-systemctl --shell ./test-systemctl.sh
+
+   parse:
+     pattern: "(?P<test_case_id>.*):\\s+(?P<result>(pass|fail))"
+
+   ```
+   This test is aimed to be run for ostree-based minimal Apertis image for
+   all supported architectures. However the metadata is mostly needed for
+   documentation purposes.
+
+   Action "install" point to GIT repository as a source for the test, so LAVA
+   will fetch and deploy this repository for us.
+
+   Action "run" provide the step-by-step instruction how to execute the test.
+   Please note that it is recommended to use wrapper for the test for
+   integration with LAVA.
+
+   Action "parse" provides own detection for the status of test results printed
+   by script.
+
+2. Push the test case to the GIT repository.
+   This step is mandatory since the test case would be checked out by LAVA
+   internally during the test preparation.
+
+       git add test-cases/test-systemctl.yaml
+       git commit -s -m "add test case for systemctl" test-cases/test-systemctl.yaml
+       git push --set-upstream origin wip/example
+
+3. Add job template to be run in lava. Job template contains all needed
+   information for LAVA how to boot the target device and deploy the OS image
+   onto it.
+
+   Create the simple template `lava/test-systemctl-tpl.yaml` with your lovely
+   editor:
+   ```
+    job_name: systemctl test on {{release_version}} {{pretty}} {{image_date}}
+    {% if device_type == 'qemu' %}
+    {% include 'common-qemu-boot-tpl.yaml' %}
+    {% else %}
+    {% include 'common-boot-tpl.yaml' %}
+    {% endif %}
+
+      - test:
+          timeout:
+            minutes: 15
+          namespace: system
+          name: common-tests
+          definitions:
+            - repository: https://gitlab.apertis.org/tests/apertis-test-cases.git
+              revision: 'wip/example'
+              from: git
+              path: test-cases/test-systemctl.yaml
+              name: test-systemctl
+   ```
+   Hopefully you don't need to deal with HW-related part, since we already
+   have those instructions for all supported boards and Apertis OS images.
+
+   Please pay attention to `revision` -- it must point to your development
+   branch while you are working on your test.
+
+   Instead of creation of the new template you may want to extend appropriate
+   existing template with additional test definition. In this case next step
+   could be omitted.
+
+6. Add the template into profile.
+
+   Profile file is mapping test jobs to devices under the test. So you need to
+   add your job template into proper list. For example we may extend
+   the templates list named `templates-minimal-ostree` in file
+   `lava/profiles.yaml`:
+   ```
+   .templates:
+     - &templates-minimal-ostree
+       - test-systemctl-tpl.yaml
+   ```
+   It is highly recommended to temporary remove or comment out the rest of
+   templates from the list to avoid unnecessary workload on LAVA while you're
+   developing the test.
+
+7. Check the profile and template locally first.
+
+   As a first step you have to define the proper profile name to use for the
+   test in LAVA.
+
+   **NB**: you need to have `lqa` tool installed and configured as described
+   in [personal LAVA tests](/qa/personal_lava_tests/) tutorial.
+
+   Since the LAVA is a part of Apertis OS CI -- it requires some variables to
+   be provided for using of Apertis profiles and templates. Let's define the
+   board we will use for testing, as well as the image release and variant:
+
+       release=v2022dev1
+       version=v2022dev1.0rc2
+       variant=minimal
+       arch=armhf
+       board=uboot
+       baseurl="https://images.apertis.org"
+       imgpath="release/$release"
+       profile_name=apertis_ostree-${variant}-${arch}-${board}
+       image_name=apertis_ostree_${release}-${variant}-${arch}-${board}_${version}
+
+   And now we are able to submit the test in a dry-run mode:
+
+       lqa submit -g lava/profiles.yaml -p ${profile_name} \
+         -t visibility:"{'group': ['Apertis']}" -t priority:"high" \
+         -t imgpath:${imgpath} -t release:${release} -t image_date:${version} \
+         -t image_name:${image_name} -n
+
+   There should not be any error or warning from `lqa`. You may want to add `-v`
+   argument to see the generated LAVA job.
+
+   **NB**: it is recommended to set `visibility` variable to "Apertis" group
+   during development to avoid any credentials/passwords leak by occasion.
+   Set the additional variable `priority` to `high` allows you to bypass the
+   jobs common queue and do not wait your job results for ages.
+
+8. Submit your first job to LAVA.
+
+   Just repeat the `lqa` call above without `-n` option.
+   After the job submission you will see the job ID:
+
+       lqa submit -g lava/profiles.yaml -p "${profile_name}" -t visibility:"{'group': ['Apertis']}" -t priority:"high" -t imgpath:${imgpath} -t release:${release} -t image_date:${version} -t image_name:${image_name} 
+       Submitted job test-systemctl-tpl.yaml with id 3463731
+
+   So it is possible to check the job status by URL with that ID:
+   https://lava.collabora.co.uk/scheduler/job/3463731
+
+9. Push your changes from template and profile.
+
+   As soon as your test case works as expected you may restore all commented
+   templates for profile, change the `revision` key in file
+   `lava/test-systemctl-tpl.yaml` to target branch and submit your changes:
+
+       git add lava/test-systemctl-tpl.yaml lava/profiles.yaml
+       git commit -a -m "hello world template added"
+       git push
+
+As a last step you need to create merge request in GitLab and as soon as it
+would be accepted your test became a part of Apertis testing CI.
+