The most of the tests are covered by automatic tests or by
other manual tests. Hence remove unneeded manual tests.

Lower priority for manual OTA rollback test since we have an
automation counterpair.

We keep both rollback tests for offline and online upgrades to
have a check point of our automation -- due complexity of the
testing process, automatic test _may_ have errors or false-positive
behavior in case if we change the AUM, U-Boot or libostree.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Fixed the name for the test -- it was the same as already existing.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The correct testing of rollback relies to U-Boot behavior.
In Apertis we expect the U-Boot should to use rollback configuration
in case if boot counter is above the limit.

Unfortunately we break the normal boot of SabreLite and Renesas boards
in LAVA by running direct boot with `run bootcmd_mmc0` command.

This completely breaks the correct workflow for rollback since it relies
to command `altbootcmd` which is a part of Apertis distro boot
modification.
The U-Boot bootloader should to drop the warning if boot counter is
truggering the rollback:

  File System is consistent
  file found, deleting
  File System is consistent
  Warning: Bootlimit (3) exceeded. Using altbootcmd.

and proceed with the call of `altbootcmd` instead of `distro_bootcmd`.
But since we stop the boot and do a manual boot from mmc0, alternative
boot is never called.

To proceed with the correct behavior after breaking the boot process we
check the boot limit additionally and choose the proper boot variant.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The path to git repository should be:
 https://gitlab.apertis.org/pkg/apertis-tests

otherwise LAVA throw an error:
 Infrastructure error: Unable to fetch git repository
 'https://gitlab.apertis.org/infrastructure/apertis-tests

'

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Additional variable `priority` has been introduced and used in all
relevant templates. This allows to change the tests priority via CLI
during tests development and debug to avoid queues in LAVA:
  lqa submit -g lava/profiles.yaml -t priority:high

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add the new template for testing the offline upgrade from previous
release. For this test the image from previous release should be used.

To deploy the proper image version new variables were added into global
list pointing to requested version of the image:
 - upgradable_url_prefix
 - upgradable_release
 - upgradable_build

The test 'aum-offline-upgrade-branch' introduce the custom path
'image_base_url' to the image for deploy, based on variables above.
This custom path is used in macros `image_url()` allowing to download
and deploy requested image version instead of the image under the test.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Macros for path calculation to the testing image for deploy.
It accepts only one string parameter -- file extension to be added
to the base image URL.

In case if global variable 'image_base_url' is set -- it is returned
with requested extension instead of on-the-fly calculation.

As a side effect -- it is possible to define the image for the test
via CLI without changing other variables, for instance:
  lqa -g lava/profiles.yaml -t image_base_url:'https://URL/name.img

'

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

This allows to use the same macroses across all templates.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

This test ensures that Apertis is capable to upgrade the previos version
to the recent. Check that update manager is able to fix all refs after
successful update.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Emanuele Aina authored 4 years ago and Denis Pynkin committed 4 years ago

The NFS rootfs does not ship connmand-wait-online, so at the moment
this only produces an error message as the error is ignored:

    ash: connmand-wait-online: not found

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit 1cd0f488 "lava: enable ostree tests for arm64" enabled the
tests in the profiles but unfortunately that's not enough to make them
show up in the reports on https://lavaphabbridge.apertis.org/ nor to
flag them for arm64 on https://qa.apertis.org/



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Denis Pynkin authored 4 years ago and Emanuele Aina committed 4 years ago

After updating of Renesas boards in LAVA Lab with U-Boot
version 2020.04+dfsg-2co5 we are able to test all ostree-based
capabilities on Arm64 platform.

Enable all manual AUM/OSTree tests for Renesas boards.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 4 years ago and Emanuele Aina committed 4 years ago

After updating of Renesas boards in LAVA Lab with U-Boot
version 2020.04+dfsg-2co5 we are able to test all ostree-based
capabilities on Arm64 platform.
Moved all ostree tests into `templates-minimal-ostree` group since
only amd64 have no full ostree support.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

In some cases, like the `group-common-standalone-tpl.yaml` test set, the
overlay is quite big and causes out-of-space errors when `wget` tries to
save it in the tmpfs-backed `/tmp` directory.

To avoid that, use a pipe to unpack the tarball while `wget` streams it
on standard out, avoiding any intermediate storage.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Include additional steps to show that agl-compositor is able to show popups
based on the agl-compositor protocol.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Include steps to test gnome-menus support. These steps consist in install
a new application, check that it is shown in the menu and check that it
can be launched.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

As the final goal is to use agl-compositor on target images change the
test to use it. Since target images already provide a compositor, change
the configuration to use the agl-compositor instead.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Frederic Danis authored 4 years ago and Emanuele Aina committed 4 years ago

AUM has now more specific error messages when trying to use an incorrect
static update bundle.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

As agl-compositor in declared as a dependency in maynard remove it
from the installation command.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

As now all the dependecies were moved to target repository there is
no need to enable the development respository, so remove the this
step from the test case.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Samir Kumar Sarangi authored 4 years ago and Emanuele Aina committed 4 years ago

Some testcases not longer work due to bitrot. They were initially set up
to test specific enhancements applied to the WebKit-Clutter codebase but
now that we ship a standard WebKitGTK+ they do not att much value.

Since the webkit2gtk-gstreamer1.0, webkit2gtk-mt-touch-events and
webkit2gtk-ac-3d-rendering already cover most of the web stack already
and also exercise the important integration points we decided to drop
the bitrotted testcases:

* webkit2gtk-ac-animations
* webkit2gtk-ac-scrolling
* webkit2gtk-drag-and-drop
* webkit2gtk-event-handling-redesign

Apertis: T7104

Signed-off-by: Samir <external.SamirKumar.Sarangi@in.bosch.com>

Samir Kumar Sarangi authored 4 years ago and Emanuele Aina committed 4 years ago

The manual testcases below have an automated counterpart, so let's drop
them as they are redundant:

* apertis-update-manager-api
* apertis-update-manager-diskfull
* apertis-update-manager-ota-auto
* apertis-update-manager-ota-diskfull
* apertis-update-manager

Apertis: T7104

Signed-off-by: Samir <external.SamirKumar.Sarangi@in.bosch.com>

Our guidelines say that to contribute people should fork repositories,
but at the moment this gives them a broken pipeline.

Tweak things in a way that the main project triggers
tests/apertis-test-cases-web to collect and indirectly publish all the
release branches while forked repositories directly publish their own
branches via Pages for easier testing.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Minnowboard currently get stuck for a few minutes during boot with
this message:

    Clear memory in MRC per MOR request Start, Please wait for some minutes...

This indicates that the firmware is busy overwriting (slowly) the RAM
contents. This is triggered by the `CONFIG_RESET_ATTACK_MITIGATION`
kernel option, enabled by default in Debian kernels, setting the
`MemoryOverwriteRequestControl`/`MemoryOverwriteRequestControlLock`
UEFI variables.

A request to drop the kernel option has been submitted upstream:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970861

Further details about the UEFI variables and their behaviour in the
EDK2 TianoCore firmware here:

https://edk2.groups.io/g/discuss/topic/77180459#381



The memory cleaning process seems to be largely outside of the control
of the free firmware, so the suggested option is to avoid triggering
it from the OS kernel. The best way to do so is still unclear and
should be discussed in the above Debian bug. In the meantime, increase
the timeouts.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Walter Lozano authored 4 years ago and Emanuele Aina committed 4 years ago

Include a basic test for agl-compositor which downloads the required
packages and launches the compositor.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The test uses `/usr/lib/chaiwala-tests/bluez/simple-agent` shipped in
the `chaiwala-tests` package, let's explicitly depend on it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Benani Sagar Kishore authored 4 years ago and Emanuele Aina committed 4 years ago

Signed-off-by: Benani Sagar Kishore (RBEI/ECO2) <SagarKishore.Benani@in.bosch.com>

Use the passed URL to Tiny RFS rather than re-construct the URL
with script hardcoded function. This allows to test the RFS prepared
in users GIT branches.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

After commit 45dae71b fixed the boot on Renesas boards with
linux >= 5.7 due to the change of the dtb name, the tests exercising
the NFS boot of the current release as the first stage broke since
they do not use `boot_commands` and thus didn't get the override.

To fix that, rather than constructing the path to the DTB from the
LAVA device name, which no longer match the board type with
linux >= 5.7, explicitly pass the DTB path for the first stage and the
release under test separately.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Lightweight CI runners have been introduced to run jobs which have low
resource consumption and do not need privileged containers to reduce costs
and latency, so let's use them.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add some additional YAML anchors so we further shrink the
*-internal-mx6qsabrelite profile definitions.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Two of the test cases point to videos to show what a successful run looks
like. These were randomly stored on the old wiki machine. Rather than
having unlinked static content on the website, add these to the
`images` folder and update links.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Commit 822fb8b8 dropped a few testcases, but the profiles still
point to them, resulting in errors like on
https://lava.collabora.co.uk/scheduler/job/2572900

:

    Job error: Unable to open test definition 'test-cases/tracker-indexing-local-storage.yaml':
    [Errno 2] No such file or directory:
    '/var/lib/lava/dispatcher/tmp/2572900/lava-overlay-knf8ynqa/var/lib/lava-2572900/2/tests/21_tracker-indexing-local-storage/test-cases/tracker-indexing-local-storage.yaml'

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

After the discussions following test cases will be dropped.
apparmor-tracker
folks-ofono-backend
telepathy-ring
tracker-indexing-local-storage
tracker-indexing-mass-storage
traffic-control-basic

Apertis: T5734
Signed-off-by: Tarun Baghmar <tarun.baghmar@in.bosch.com>

Denis Pynkin authored 4 years ago and Emanuele Aina committed 4 years ago

Ask to wait a few seconds before to check if undeploy is really
finished. Sometimes the human is too fast to enter commands
so undeploy have no chance to finish before the check.
That is leading to false result of manual test while it works well.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Frederic Danis authored 4 years ago and Denis Pynkin committed 4 years ago

wget needs the complete URL to be able to download key0.key file.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Earlier, we were using an old kerne and rootfs from 18.09.xx.

We've now switched to using a much newer one from v2020, which is where
it requires an update of our assumpt login prompts

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

In commit 83772e1 and later commits DTB names in Linux kernel was
renamed to align names with config options.
Hence the old DTB name `r8a7796-m3ulcb.dtb` is `8a77960-ulcb.dtb` for
Renesas M3 boards used in LAVA.

Old U-Boot versions are using the old naming schema including version
used in v2021dev3 atm.

This commit set proper name only for v2021dev3 branch allowing to boot
images for Renesas boards in LAVA.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Rather than looping over branches (in included template from `pages`),
generate the test cases here, provide these as an artifact and deploy
in a separate job. This avoids the regeneration of pages that have
not changed.

Use the apertis-test-cases-web as part of a multi-project pipeline to
deploy. Remove the deployment tools from this branch.

Whilst we are here, utilise an environment variable for the release to
make it easier to migrate this between releases.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>