Add missing screenshot which shows the popup functionality of agl-compositor.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The new `psdk-test` requests `paramiko` to run.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

For AUM tests unpack of overlay with tests in verbose mode
produce too lot of almost useless information. This is the reason
why the job log exceeding 5MB limit and should be downloaded as
plain text which is hard to analyze.

Added common macros `tar_unpack_cmd()` which create dynamic unpack
string depending on variable `tar_additional_options` ('-v' by default).

Additional verbose option for `tar` was switched off for all AUM tests
to simplify log analysis in WEb interface.

It is possible to override default options with `lqa` command, for
example: `lqa -t tar_additional_options:""`.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Without quoting they could be assumed as number which will lead to
incorrect usage. For instance build `20210119.0130` was cut to
`20210119.013` leading to incorrect path to delta URL construction.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The new `psdk-test` perform the 3 previous tests (etc, home-user and
sysroot) without user interaction.
It can use a pre-downloaded virtual disk image, or directly the URL.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

python-dialog should be installed by apertis-tests, dialog itself is a
dependency of python-dialog so will be installed as well.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Starting from kernel 5.5 FDT names were split and
renamed to align with config options. This leads to
boot failure of old or new Apertis images depending of
naming schema used in `fdtfile` environment variable.

U-Boot used in Apertis and LAVA have additional feature
allowing to try the legacy FDT naming schema in cases
of boot failure with default naming schema.

On v2021 for the switching branches test we have to boot
images with old naming for DTB file in v2020 (5.4 kernel) and
new naming schema in v2021 (5.9 kernel).
So it is not possible to hardcode proper variable `fdtfile`
for booting both versions in LAVA.

Instead we remove hardcoded DTB file and use additional boot
target `legacy_mmc0` which automatically set the "old" name
for the DTB file in case if boot with a new one was failed.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The most of the tests are covered by automatic tests or by
other manual tests. Hence remove unneeded manual tests.

Lower priority for manual OTA rollback test since we have an
automation counterpair.

We keep both rollback tests for offline and online upgrades to
have a check point of our automation -- due complexity of the
testing process, automatic test _may_ have errors or false-positive
behavior in case if we change the AUM, U-Boot or libostree.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Fixed the name for the test -- it was the same as already existing.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The correct testing of rollback relies to U-Boot behavior.
In Apertis we expect the U-Boot should to use rollback configuration
in case if boot counter is above the limit.

Unfortunately we break the normal boot of SabreLite and Renesas boards
in LAVA by running direct boot with `run bootcmd_mmc0` command.

This completely breaks the correct workflow for rollback since it relies
to command `altbootcmd` which is a part of Apertis distro boot
modification.
The U-Boot bootloader should to drop the warning if boot counter is
truggering the rollback:

  File System is consistent
  file found, deleting
  File System is consistent
  Warning: Bootlimit (3) exceeded. Using altbootcmd.

and proceed with the call of `altbootcmd` instead of `distro_bootcmd`.
But since we stop the boot and do a manual boot from mmc0, alternative
boot is never called.

To proceed with the correct behavior after breaking the boot process we
check the boot limit additionally and choose the proper boot variant.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The path to git repository should be:
 https://gitlab.apertis.org/pkg/apertis-tests

otherwise LAVA throw an error:
 Infrastructure error: Unable to fetch git repository
 'https://gitlab.apertis.org/infrastructure/apertis-tests

'

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Emanuele Aina authored 4 years ago and Ritesh Raj Sarraf committed 4 years ago

This project was never actually uploaded as a package, as it would not
be particularly useful to do so.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 4 years ago and Ritesh Raj Sarraf committed 4 years ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

Add a new test case to confirm that a licensing summary is generated for minimal
and target images.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Additional variable `priority` has been introduced and used in all
relevant templates. This allows to change the tests priority via CLI
during tests development and debug to avoid queues in LAVA:
  lqa submit -g lava/profiles.yaml -t priority:high

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add the new template for testing the offline upgrade from previous
release. For this test the image from previous release should be used.

To deploy the proper image version new variables were added into global
list pointing to requested version of the image:
 - upgradable_url_prefix
 - upgradable_release
 - upgradable_build

The test 'aum-offline-upgrade-branch' introduce the custom path
'image_base_url' to the image for deploy, based on variables above.
This custom path is used in macros `image_url()` allowing to download
and deploy requested image version instead of the image under the test.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Macros for path calculation to the testing image for deploy.
It accepts only one string parameter -- file extension to be added
to the base image URL.

In case if global variable 'image_base_url' is set -- it is returned
with requested extension instead of on-the-fly calculation.

As a side effect -- it is possible to define the image for the test
via CLI without changing other variables, for instance:
  lqa -g lava/profiles.yaml -t image_base_url:'https://URL/name.img

'

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

This allows to use the same macroses across all templates.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

This test ensures that Apertis is capable to upgrade the previos version
to the recent. Check that update manager is able to fix all refs after
successful update.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

The NFS rootfs does not ship connmand-wait-online, so at the moment
this only produces an error message as the error is ignored:

    ash: connmand-wait-online: not found

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

In some cases, like the `group-common-standalone-tpl.yaml` test set, the
overlay is quite big and causes out-of-space errors when `wget` tries to
save it in the tmpfs-backed `/tmp` directory.

To avoid that, use a pipe to unpack the tarball while `wget` streams it
on standard out, avoiding any intermediate storage.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Include additional steps to show that agl-compositor is able to show popups
based on the agl-compositor protocol.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

Include steps to test gnome-menus support. These steps consist in install
a new application, check that it is shown in the menu and check that it
can be launched.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Commit 1cd0f488 "lava: enable ostree tests for arm64" enabled the
tests in the profiles but unfortunately that's not enough to make them
show up in the reports on https://lavaphabbridge.apertis.org/ nor to
flag them for arm64 on https://qa.apertis.org/



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

As the final goal is to use agl-compositor on target images change the
test to use it. Since target images already provide a compositor, change
the configuration to use the agl-compositor instead.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

After updating of Renesas boards in LAVA Lab with U-Boot
version 2020.04+dfsg-2co5 we are able to test all ostree-based
capabilities on Arm64 platform.

Enable all manual AUM/OSTree tests for Renesas boards.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

After updating of Renesas boards in LAVA Lab with U-Boot
version 2020.04+dfsg-2co5 we are able to test all ostree-based
capabilities on Arm64 platform.
Moved all ostree tests into `templates-minimal-ostree` group since
only amd64 have no full ostree support.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Frederic Danis authored 4 years ago and Emanuele Aina committed 4 years ago

AUM has now more specific error messages when trying to use an incorrect
static update bundle.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

As agl-compositor in declared as a dependency in maynard remove it
from the installation command.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

As now all the dependecies were moved to target repository there is
no need to enable the development respository, so remove the this
step from the test case.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Samir Kumar Sarangi authored 4 years ago and Emanuele Aina committed 4 years ago

Some testcases not longer work due to bitrot. They were initially set up
to test specific enhancements applied to the WebKit-Clutter codebase but
now that we ship a standard WebKitGTK+ they do not att much value.

Since the webkit2gtk-gstreamer1.0, webkit2gtk-mt-touch-events and
webkit2gtk-ac-3d-rendering already cover most of the web stack already
and also exercise the important integration points we decided to drop
the bitrotted testcases:

* webkit2gtk-ac-animations
* webkit2gtk-ac-scrolling
* webkit2gtk-drag-and-drop
* webkit2gtk-event-handling-redesign

Apertis: T7104

Signed-off-by: Samir <external.SamirKumar.Sarangi@in.bosch.com>

Samir Kumar Sarangi authored 4 years ago and Emanuele Aina committed 4 years ago

The manual testcases below have an automated counterpart, so let's drop
them as they are redundant:

* apertis-update-manager-api
* apertis-update-manager-diskfull
* apertis-update-manager-ota-auto
* apertis-update-manager-ota-diskfull
* apertis-update-manager

Apertis: T7104

Signed-off-by: Samir <external.SamirKumar.Sarangi@in.bosch.com>

Our guidelines say that to contribute people should fork repositories,
but at the moment this gives them a broken pipeline.

Tweak things in a way that the main project triggers
tests/apertis-test-cases-web to collect and indirectly publish all the
release branches while forked repositories directly publish their own
branches via Pages for easier testing.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Minnowboard currently get stuck for a few minutes during boot with
this message:

    Clear memory in MRC per MOR request Start, Please wait for some minutes...

This indicates that the firmware is busy overwriting (slowly) the RAM
contents. This is triggered by the `CONFIG_RESET_ATTACK_MITIGATION`
kernel option, enabled by default in Debian kernels, setting the
`MemoryOverwriteRequestControl`/`MemoryOverwriteRequestControlLock`
UEFI variables.

A request to drop the kernel option has been submitted upstream:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970861

Further details about the UEFI variables and their behaviour in the
EDK2 TianoCore firmware here:

https://edk2.groups.io/g/discuss/topic/77180459#381



The memory cleaning process seems to be largely outside of the control
of the free firmware, so the suggested option is to avoid triggering
it from the OS kernel. The best way to do so is still unclear and
should be discussed in the above Debian bug. In the meantime, increase
the timeouts.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Include a basic test for agl-compositor which downloads the required
packages and launches the compositor.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Emanuele Aina authored 4 years ago and Martyn Welch committed 4 years ago

The test uses `/usr/lib/chaiwala-tests/bluez/simple-agent` shipped in
the `chaiwala-tests` package, let's explicitly depend on it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>