Added the `dry-run` mode feature testing.
Added the network polling time change feature testing.
Updated output for manual OTA rollback upgrade test.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Benani Sagar Kishore authored 4 years ago and Benani Sagar Kishore committed 3 years ago

Signed-off-by: Sagar Kishore Benani <SagarKishore.Benani@in.bosch.com>

The common tests group currently blows up on LAVA when submitted to
QEMU-based DUTs as we do for the SDK and base SDK images because the
dispatcher runs out of disk space while unpacking the overlay before
uploading it to the DUT:

    Job error: tar_in: tar subcommand failed on directory: /:
    tar: 0/tests/1_disk-rootfs-fsck/apertis-tests/.git/objects/pack/pack-c1d7de14a133c29fdcacfeef5eba98291ada6c91.pack:
    Wrote only 2048 of 10240 bytes tar: 0/tests/1_disk-rootfs-fsck/apertis-tests/.git/packed-refs:
    Cannot write: No space left on device

In particular this is because the git repositoriee installed by the
apparmor-gstreamer1.0 and gstreamer1.0-decode testcases used to ship a
large copy of the Big Buck Bunny movie. Even if the test file has been
dropped from the HEAD of both repositories in favor of the media sample
file for the same movie pre-installed on the images, LAVA fails because
it does not do a shallow checkout but it ends up putting the whole git
hisotry in the overlay.

To side-step the issue, move all the apparmor tests still in the
`common-standalone` group to the `apparmor-standalone` group, but keep
skipping them on minimal images.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Added templates list for ostree-based profiles without AUM tests.
At the moment we do not support AUM tests for Amd64 and for internal
Arm32 images, hence use a base list for them.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

This reverts commit 0dc36df3.

Revert this commit since it is breaking tests submission to LAVA with
`lqa` tool started in Apertis environment:

 Submitting job aum-ota-rollback-blacklist.yaml: <Fault 400: 'Problem
 with submitted job data: Loading job submission failed: while parsing
 a block mapping\n  in "<unicode string>", line 2, column 1\ndid not
 find expected key\n  in "<unicode string>", line 154, column 1.'>

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Emanuele Aina authored 4 years ago and Denis Pynkin committed 4 years ago

Commit 39e6ec4f made `tar` output the extracted filenames while
deployng the overlays and later commit 39e6ec4f made it
customizable since the AUM tests do many reboot cycles each with their
own (redundant) overlay deployment, causing the logs to grow massively
and going over the limits imposed by the LAVA web UI, which refuses to
display them and only offers to download them in JSON format.

Unfortunately the fix on 39e6ec4f worked only partially and the
overlays in the loop were still upacked in verbose mode, resulting in
unreadable logs.

For some reason that currently escapes me related to the way Jinja2
imports work, re-importing the template with the macros from the
looping template fixes the issue.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Submitting tests to LAVA from this repository has been broken for a long
time and has proven to be not useful. Yet, it generates plenty of noise
on the LAVA side, so disable the actual submission.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add missing screenshot which shows the popup functionality of agl-compositor.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The new `psdk-test` requests `paramiko` to run.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

For AUM tests unpack of overlay with tests in verbose mode
produce too lot of almost useless information. This is the reason
why the job log exceeding 5MB limit and should be downloaded as
plain text which is hard to analyze.

Added common macros `tar_unpack_cmd()` which create dynamic unpack
string depending on variable `tar_additional_options` ('-v' by default).

Additional verbose option for `tar` was switched off for all AUM tests
to simplify log analysis in WEb interface.

It is possible to override default options with `lqa` command, for
example: `lqa -t tar_additional_options:""`.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Without quoting they could be assumed as number which will lead to
incorrect usage. For instance build `20210119.0130` was cut to
`20210119.013` leading to incorrect path to delta URL construction.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The new `psdk-test` perform the 3 previous tests (etc, home-user and
sysroot) without user interaction.
It can use a pre-downloaded virtual disk image, or directly the URL.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

python-dialog should be installed by apertis-tests, dialog itself is a
dependency of python-dialog so will be installed as well.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Starting from kernel 5.5 FDT names were split and
renamed to align with config options. This leads to
boot failure of old or new Apertis images depending of
naming schema used in `fdtfile` environment variable.

U-Boot used in Apertis and LAVA have additional feature
allowing to try the legacy FDT naming schema in cases
of boot failure with default naming schema.

On v2021 for the switching branches test we have to boot
images with old naming for DTB file in v2020 (5.4 kernel) and
new naming schema in v2021 (5.9 kernel).
So it is not possible to hardcode proper variable `fdtfile`
for booting both versions in LAVA.

Instead we remove hardcoded DTB file and use additional boot
target `legacy_mmc0` which automatically set the "old" name
for the DTB file in case if boot with a new one was failed.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The most of the tests are covered by automatic tests or by
other manual tests. Hence remove unneeded manual tests.

Lower priority for manual OTA rollback test since we have an
automation counterpair.

We keep both rollback tests for offline and online upgrades to
have a check point of our automation -- due complexity of the
testing process, automatic test _may_ have errors or false-positive
behavior in case if we change the AUM, U-Boot or libostree.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Fixed the name for the test -- it was the same as already existing.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The correct testing of rollback relies to U-Boot behavior.
In Apertis we expect the U-Boot should to use rollback configuration
in case if boot counter is above the limit.

Unfortunately we break the normal boot of SabreLite and Renesas boards
in LAVA by running direct boot with `run bootcmd_mmc0` command.

This completely breaks the correct workflow for rollback since it relies
to command `altbootcmd` which is a part of Apertis distro boot
modification.
The U-Boot bootloader should to drop the warning if boot counter is
truggering the rollback:

  File System is consistent
  file found, deleting
  File System is consistent
  Warning: Bootlimit (3) exceeded. Using altbootcmd.

and proceed with the call of `altbootcmd` instead of `distro_bootcmd`.
But since we stop the boot and do a manual boot from mmc0, alternative
boot is never called.

To proceed with the correct behavior after breaking the boot process we
check the boot limit additionally and choose the proper boot variant.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The path to git repository should be:
 https://gitlab.apertis.org/pkg/apertis-tests

otherwise LAVA throw an error:
 Infrastructure error: Unable to fetch git repository
 'https://gitlab.apertis.org/infrastructure/apertis-tests

'

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Emanuele Aina authored 4 years ago and Ritesh Raj Sarraf committed 4 years ago

This project was never actually uploaded as a package, as it would not
be particularly useful to do so.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 4 years ago and Ritesh Raj Sarraf committed 4 years ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

Add a new test case to confirm that a licensing summary is generated for minimal
and target images.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Additional variable `priority` has been introduced and used in all
relevant templates. This allows to change the tests priority via CLI
during tests development and debug to avoid queues in LAVA:
  lqa submit -g lava/profiles.yaml -t priority:high

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add the new template for testing the offline upgrade from previous
release. For this test the image from previous release should be used.

To deploy the proper image version new variables were added into global
list pointing to requested version of the image:
 - upgradable_url_prefix
 - upgradable_release
 - upgradable_build

The test 'aum-offline-upgrade-branch' introduce the custom path
'image_base_url' to the image for deploy, based on variables above.
This custom path is used in macros `image_url()` allowing to download
and deploy requested image version instead of the image under the test.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Macros for path calculation to the testing image for deploy.
It accepts only one string parameter -- file extension to be added
to the base image URL.

In case if global variable 'image_base_url' is set -- it is returned
with requested extension instead of on-the-fly calculation.

As a side effect -- it is possible to define the image for the test
via CLI without changing other variables, for instance:
  lqa -g lava/profiles.yaml -t image_base_url:'https://URL/name.img

'

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

This allows to use the same macroses across all templates.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

This test ensures that Apertis is capable to upgrade the previos version
to the recent. Check that update manager is able to fix all refs after
successful update.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

The NFS rootfs does not ship connmand-wait-online, so at the moment
this only produces an error message as the error is ignored:

    ash: connmand-wait-online: not found

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In some cases, like the `group-common-standalone-tpl.yaml` test set, the
overlay is quite big and causes out-of-space errors when `wget` tries to
save it in the tmpfs-backed `/tmp` directory.

To avoid that, use a pipe to unpack the tarball while `wget` streams it
on standard out, avoiding any intermediate storage.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Include additional steps to show that agl-compositor is able to show popups
based on the agl-compositor protocol.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

Include steps to test gnome-menus support. These steps consist in install
a new application, check that it is shown in the menu and check that it
can be launched.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Commit 1cd0f488 "lava: enable ostree tests for arm64" enabled the
tests in the profiles but unfortunately that's not enough to make them
show up in the reports on https://lavaphabbridge.apertis.org/ nor to
flag them for arm64 on https://qa.apertis.org/



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

As the final goal is to use agl-compositor on target images change the
test to use it. Since target images already provide a compositor, change
the configuration to use the agl-compositor instead.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>