ade-commands test is failing, and this seems due to long download
times. Increase the timeout of the test.
The sysroot weight 1GB
Refer to APERTIS-6124

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Test cases files should always match their metadata name.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Add a test to check if the persistent disk functionality is working
properly by:
 - Installing a sysroot on the old SDK
 - Configuring the old SDK to use the persistent disk
 - Configuring the new SDK to use the persistent disk
 - Check that the sysroot is available on the new SDK

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

This test used a simple echo pass. The qa report was not able to
understand that the test passed successfully because it eliminated
tests results of the "lava" test group.
Use the specific lava tool for reporting instead

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Add a testcase to ensure that the sample image recipes can be built on the SDK
and that the commands in the documentation work correctly.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Frédéric Dalleau authored 5 years ago and Denis Pynkin committed 5 years ago

Currently paramiko do not support Keys in RFC4716 format that
have landed with the buster update.
So any keys need to be generated in legacy PEM format.

Fixes: APERTIS-5932
Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

This reverts commit c86d18a7.
The problem of low speed bandwidth is kernel crash (APERTIS-6016) which
completely blocking the USB unplug test.

With changes provided in previous commit we provide the binary diff
big enough to catch the proper timing by human and without kernel
crashes.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

With this change we prepare the "outdated" commit without the
content of some directories. This allows to provide the significant
difference between "original" and "outdated" commits, so it is easy
to check it in booted deployment.
The side effect of huge size difference during upgrade is more stable
and convenient testing of "disk full" and "usb unplug" cases.

More detailed description is available here:
https://phabricator.apertis.org/T5439#189061



Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Test if update doesn't break anything if there is not enough free space
on system disk.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Denis Pynkin committed 5 years ago

Test if rollback and blacklist mechanisms works for network updates.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Denis Pynkin committed 5 years ago

Test if AUM is able to correctly read network-related settings
from configuration file and able to do automatic updates.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Denis Pynkin committed 5 years ago

Manual test aimed to check the API between update manager and
`updatectl` tool allowing to start network updates with CLI.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Improve test case description.
Use the new NFS image-deployment value.

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

After booting an nfsroot, an image is flashed and we only want to
ensure the image boots properly to a shell.
This test uses the nfsroot image type.

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Frédéric Dalleau authored 5 years ago and Frédéric Dalleau committed 5 years ago

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Frédéric Dalleau authored 5 years ago and Frédéric Dalleau committed 5 years ago

That command is no longer needed because the u-boot environment is now
partially stored directly in the image. Everytime the image is reflashed, the
environment is wiped out.

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The testcase uses `run-test-in-systemd` which in turn uses `lava-test-case` so
there's no need to set a custom pattern to parse test results.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The expected output doesn't really match what the tests will emit, and
there's little point in documenting the standard LAVA expectations here.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Need to reduce the read speed for the test since recent libostree do not
read the whole update bundle during the test.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Use CGroups blkio to limit the read speed to 1MBps allowing to
remove the USB flash in predictable state even on extremely fast
devices.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Need to generate the recent version of tests description for QA!

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Incorrect name has been used leading to failure of the container startup.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Use branch apertis/v2019pre for all test cases.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Removed the mention of 'update_available' U-Boot variable -- we do not
use environment-based rollback driver anymore.
Updated the output for blacklisted part -- now we have additional
information about Collection ID.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Let `run-test-in-systemd` figure out the appropriate subtest name when
launching `ade-tests-custom.sh` and `ade-tests-defaults.sh`, instead of
confusingly setting the name as `run-test` for both.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use CGroups blkio to limit the read speed to 1MBps allowing to
remove the USB flash in predictable state even on extremely fast
devices.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The current test downloads from a legacy location the same contents already
shipped by the official multimedia demopack under `/home/shared`.

On the target and SDK images it is wasteful and silly, as those exact contents
come pre-installed.

On the minimal images this fails since there's not enough storage.

On the base SDK image the decision to not install the multimedia demopack was
taken explicitly as it is not relevant there.

Change the test to only run on the images where it is appropriate and rely on
the multimedia demopack being available out-of-the-box in `/home/shared`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The "as-root" run of dbus-installed-test failed as it exceeded the time
limit by 0.04s. Increase the time limit from 15 mins to 20 mins to avoid
this happening.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

The check-dbus-services aims to test all services configured as bus
activatable, however such a test is proving to be too naive and requires
significant investment to keep in a passing state.

There are an increasing number of services that have needed to be
blacklisted as they don't pass with reasonable rationale:

* Some require the ability to display a GUI and recieve a response that
  isn't currently viable with the current automated test platform nor is
  a GUI present on many of the image types.
* Some require specific devices to be present for the underlying service
  to start that isn't present in on the test systems.
* Some aren't completely installed and operational in the smaller cut-down
  images.

It has been decided that this test lacks sufficent benefit and significant
overhead and is thus being removed.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Emanuele Aina authored 5 years ago and Luis Araujo committed 5 years ago

The `make-ssl-cert` script specifically targets bash, so let's install it
explicitly.

The `ssl-cert` package is brought in by `postfix`, which is needed by `quilt`,
suggested by `devscript`.

This fixes the error below:
    Setting up ssl-cert (1.0.39ba1) ...
    /var/lib/dpkg/info/ssl-cert.postinst: 48: /var/lib/dpkg/info/ssl-cert.postinst: make-ssl-cert: not found
    dpkg: error processing package ssl-cert (--configure):
     installed ssl-cert package post-installation script subprocess returned error exit status 127

The install error also caused the subsequent tests to be skipped due to the
inconsistent state.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Folks test cases are not so relevant anymore.

The apparmor-folks test case has not been executed for several releases
already (it was not executed for 18.12 and for 18.09 was only executed
for half the platforms with some failures too).

This commit removes the apparmor-folks test case.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

This will allow the didcot test case page to be included in the test
case index page and test report page.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Some issues were reported in comments from T5439.
This patch fixes them.

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Martyn Welch authored 5 years ago and Frédéric Dalleau committed 5 years ago

The regex for parsing the test result is incorrect, leading to the test
reporting a pass rather than a fail as it should be. Change the regex to
make it report the correct result.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

After completion of APERTIS-5594 the offline upgrade test has been changed.
Now it is not possible to use random update bundle hence we must to
prepare the system to be upgraded to the same commit.
Added the description how to do that into all AUM manual tests.

Fixes: APERTIS-5808

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Denis Pynkin committed 5 years ago

Mark some manual tests as 'low' since there are automatic versions
but we need to keep the steps for manual testing.
Added links to automated versions of these tests.
But we still need 2 tests with USB HW for manual testing.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

The test is currently being run on all the image types, update the testcase to
reflect that fact.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

There is no benefit to execute this unit test in LAVA and it is already
being executed in OBS during package build time.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>