The most of the tests are covered by automatic tests or by
other manual tests. Hence remove unneeded manual tests.

Lower priority for manual OTA rollback test since we have an
automation counterpair.

We keep both rollback tests for offline and online upgrades to
have a check point of our automation -- due complexity of the
testing process, automatic test _may_ have errors or false-positive
behavior in case if we change the AUM, U-Boot or libostree.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

After updating of Renesas boards in LAVA Lab with U-Boot
version 2020.04+dfsg-2co5 we are able to test all ostree-based
capabilities on Arm64 platform.

Enable all manual AUM/OSTree tests for Renesas boards.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 4 years ago and Emanuele Aina committed 4 years ago

Ask to wait a few seconds before to check if undeploy is really
finished. Sometimes the human is too fast to enter commands
so undeploy have no chance to finish before the check.
That is leading to false result of manual test while it works well.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add the description of how to add the proxy address for OSTree allowing
online upgrades both for `ostree` and AUM in restricted networks.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Use timedatectl to workaround NTP issues on some networks

With battery backed devices on networks with no properly reachable NTP servers
the boards would fail to automatically get the right time, so testers set it up
manually with `date`.

However, the hardware clock need to be configured as well for the time
configuration to survive a reboot, needed by the Apertis update manager OTA
test.

Without this, the device will fail to establish any TLS connection after the
reboot since every certificate will look like they are out of their validity
period.

This is only a workaround until we ensure NTP servers are reachable on the
testers' network, see https://phabricator.apertis.org/T6115



Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Denis Pynkin committed 5 years ago

Test if rollback and blacklist mechanisms works for network updates.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Frédéric Dalleau authored 5 years ago and Frédéric Dalleau committed 5 years ago

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Frédéric Dalleau authored 5 years ago and Frédéric Dalleau committed 5 years ago

That command is no longer needed because the u-boot environment is now
partially stored directly in the image. Everytime the image is reflashed, the
environment is wiped out.

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Removed the mention of 'update_available' U-Boot variable -- we do not
use environment-based rollback driver anymore.
Updated the output for blacklisted part -- now we have additional
information about Collection ID.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Some issues were reported in comments from T5439.
This patch fixes them.

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

After completion of APERTIS-5594 the offline upgrade test has been changed.
Now it is not possible to use random update bundle hence we must to
prepare the system to be upgraded to the same commit.
Added the description how to do that into all AUM manual tests.

Fixes: APERTIS-5808

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Denis Pynkin committed 5 years ago

Mark some manual tests as 'low' since there are automatic versions
but we need to keep the steps for manual testing.
Added links to automated versions of these tests.
But we still need 2 tests with USB HW for manual testing.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

This field merges the image variant and arch fields in order to clearly
specify in which images a test case should be executed.

This commit adds this new field to each test case file with the specific
images where it should be executed and updates the tests and format
definition files.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

This commit adds support to have multiple image variants in the test
cases files.

It also adds the respective image variants list relevant to each test
case.

This commit also updates the tests files and format definition files.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

This is a madantory field that specifies the deployment type of the
image where this test case can be executed.

The commit adds the image-deployment field for each test case, and it
currently can be a list with the following values: APT, OSTree, Any.

This commit also updates the tests files.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

This commit renames the image-type field to image-variant
in the test cases files.

It also updates the parser, renderer code, test_case template
and unittests files in order to use the new name.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Frédéric Dalleau authored 6 years ago and Denis Pynkin committed 6 years ago

Add a test to exercise the features of the Apertis update manager.
These tests only works on OSTree images and check following:
- API of update manager with `updatectl` utility
- automatic update from mass storage device with static delta
- manual upgrade with static delta file
- automatic rollback from "failed" update
- blacklist feature
- rollback feature of U-Boot bootloader

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>