Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Use `apertis/v2020pre` branch as default.
Do not test outdated branches.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add the description of how to add the proxy address for OSTree allowing
online upgrades both for `ostree` and AUM in restricted networks.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Use timedatectl to workaround NTP issues on some networks

With battery backed devices on networks with no properly reachable NTP servers
the boards would fail to automatically get the right time, so testers set it up
manually with `date`.

However, the hardware clock need to be configured as well for the time
configuration to survive a reboot, needed by the Apertis update manager OTA
test.

Without this, the device will fail to establish any TLS connection after the
reboot since every certificate will look like they are out of their validity
period.

This is only a workaround until we ensure NTP servers are reachable on the
testers' network, see https://phabricator.apertis.org/T6115



Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Need to test the upgrade behavior on different architectures.
Enabled automatic tests of ostree and upgrade manager behavior for all
supported minimal images.

Not enabled rollback test for Arm64 and Amd64 yet since they have no
enabled bootcounter mechanism in bootloader.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Peter Senna Tschudin committed 5 years ago

Test case allow to check that AUM refuses to use commit signed with
unknown key (or unsigned). Check the well-known place for public keys
in system configuration directory is parsed correctly.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Peter Senna Tschudin committed 5 years ago

New test case allows to check the initial configuration of repository
and ensure that AUM refuses to use unsigned commit and commit signed
with unknown signature.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Commit infrastructure/apertis-test@a699e7d "Check for failed systemd units"
introduced a new call to `systemctl` that gets included even in case the test
is successful.

Update the expected output to avoid any confusion.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 5 years ago and Luis Araujo committed 5 years ago

The test is currently emitting results in this way:

    Testing supported locales:
    ar_QA.utf8: fail
    ar_SA.utf8: fail
    cs_CZ.utf8: fail
    da_DK.utf8: fail
    de_DE.utf8: fail
    el_GR.utf8: fail
    en_GB.utf8: fail
    en_US.utf8: fail
    es_AR.utf8: fail
    es_ES.utf8: fail
    es_MX.utf8: fail
    fi_FI.utf8: fail
    fr_CA.utf8: fail
    fr_FR.utf8: fail
    hu_HU.utf8: fail
    id_ID.utf8: fail
    it_IT.utf8: fail
    ms_MY.utf8: fail
    nb_NO.utf8: fail
    nl_NL.utf8: fail
    pl_PL.utf8: fail
    pt_BR.utf8: fail
    pt_PT.utf8: fail
    ro_RO.utf8: fail
    ru_RU.utf8: fail
    sk_SK.utf8: fail
    sv_SE.utf8: fail
    th_TH.utf8: fail
    tr_TR.utf8: fail
    zh_CN.utf8: fail
    zh_HK.utf8: fail
    31 test(s) failed

See https://lava.collabora.co.uk/scheduler/job/1827695#L5089



However, the current regex only matched uppercase characters,
expecting `PASSED`/`FAILED`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This commit add two test for extending the test coverage of the persistent SDK disk.
The new tests cover changes made to an arbitrary file on /etc and another on
/home/user.

Signed-off-by: Benani Sagar Kishore <SagarKishore.Benani@in.bosch.com>

This test case has been failing for a long a time since the required
telepathy packages are missing from the images.

The test case is also not so relevant anymore and makes current SDK lava
tests jobs to fail, so this commit removes the test case.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Using the same SDK image to setup old and new VM instance results in a
VirtualBox error:

	Locking of attached media failed. A possible reason is that one of the media is attached to a running VM.
	Result Code:
	VBOX_E_INVALID_OBJECT_STATE (0x80BB0007)
	Component:
	SessionMachine
	Interface:
	IMachine {85cd948e-a71f-4289-281e-0ca7ad48cd89}

This commit change the test case to clone the VM instead of creating two
identical VMs.

This patch also fixes other issues that were pointed out by:

	https://phabricator.apertis.org/T5868#195314



Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

Most tests have been converted to use the system journal to check for audit
events, so there's no need to reset the audit.log anymore.

Resetting it is even harmful today since `auditd` is no longer shipped by
default on images, thus the step fails as the file and the directory that
should contain it do not exist.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Except for the SDK image, this test is failing in the remaining images.

This commit removes the test case since it is also not so relevant
anymore and needs to be reviewed later.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

All telepathy folks tests currently fail to execute because the
telepathy-gabble package is not available in the images.

Support for folks and telepathy libs also needs to be revised, so
this commit disables all the folks telepathy tests in the meantime.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Emanuele Aina authored 5 years ago and Peter Senna Tschudin committed 5 years ago

The `auditd` package is being dropped from the images. The audit log can be
retrieved from the system journal, and the `auditd` daemon currently fails to
start on the ostree images.

However, the apparmor-{geoclue,ofono,tracker} testcases use the `run-aa-test`
tool which is not trivial to port away from `audit.log`, so add an explicit
installation step for the `auditd` package to paper over it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The apparmor-tracker testcases uses the `run-aa-test` tool, which is written in
Python 3. This makes it unsuitable to be run on the OSTree images, since none
of them ships Python.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since version 0.4+20150326.3, `apparmor-demo` no longer uses `audit.log` but
relies exclusively on the system journal to check for audit messages.

For this reason, truncating `audit.log` before running the test is no longer
useful, and it is still actually harmful on the images that don't ship auditd
like the minimal ones (other image types may follow).

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use the systemd journal to retrieve audit messages, as the minimal images no
longer ship auditd.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use the systemd journal to retrieve audit messages, as the minimal images no
longer ship auditd.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

With battery backed devices on networks with no properly reachable NTP servers
the boards would fail to automatically get the right time, so testers set it up
manually with `date`.

However, the hardware clock need to be configured as well for the time
configuration to survive a reboot, needed by the Apertis update manager OTA
test.

Without this, the device will fail to establish any TLS connection after the
reboot since every certificate will look like they are out of their validity
period.

This is only a workaround until we ensure NTP servers are reachable on the
testers' network, see https://phabricator.apertis.org/T6115

Fixes: https://phabricator.apertis.org/T6284



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The sdk-persistent-disk.yaml test case suggest using ade to verify that
the persistent disk is working. However ade uses the release version of
the SDK as default value for the sysroot. If two different versions of
the SDK are used, not specifying the release for ade will cause ade
update to fail. This patch adds --release to all calls of ade.

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

Replace aa_log_extract_tokens.pl by aa_log_extract_tokens.sh to remove
dependencies on auditd and libapparmor-perl packages.

Fixes: APERTIS-6233

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Emanuele Aina authored 5 years ago and Luis Araujo committed 5 years ago

We want to ensure that things work out-of-the-box and having the test install
packages would mask issue due to missing packages in the default installation.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 5 years ago and Luis Araujo committed 5 years ago

The lack of links support in the `steps:` list caused the `sdk-persistent-disk`
to lack important instructions, making testers more confused than necessary.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Remove this package dependency from the dbus-installed-tests test case
since this package is already installed by coreutils.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Denis Pynkin authored 5 years ago and Denis Pynkin committed 5 years ago

Use a single step for reboot waiting. Added the example how to track the
upgrade status while waiting for automatic upgrade and reboot.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Use apertis-update-manager-automount for the test case name so it
matches its filename.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This commit splits the sanity-check test case into a specific manual
version of it (sanity-check-manual).

This will solve the issue of overwriting automated tests results for
this test case on the qa-report-app.

The name sanity-check is preserved for the automated version so there is
no need to apply any changes to the LAVA jobs.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

ade-commands test is failing, and this seems due to long download
times. Increase the timeout of the test.
The sysroot weight 1GB
Refer to APERTIS-6124

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Test cases files should always match their metadata name.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Add a test to check if the persistent disk functionality is working
properly by:
 - Installing a sysroot on the old SDK
 - Configuring the old SDK to use the persistent disk
 - Configuring the new SDK to use the persistent disk
 - Check that the sysroot is available on the new SDK

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

This test used a simple echo pass. The qa report was not able to
understand that the test passed successfully because it eliminated
tests results of the "lava" test group.
Use the specific lava tool for reporting instead

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Add a testcase to ensure that the sample image recipes can be built on the SDK
and that the commands in the documentation work correctly.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Frédéric Dalleau authored 5 years ago and Denis Pynkin committed 5 years ago

Currently paramiko do not support Keys in RFC4716 format that
have landed with the buster update.
So any keys need to be generated in legacy PEM format.

Fixes: APERTIS-5932
Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

This reverts commit c86d18a7.
The problem of low speed bandwidth is kernel crash (APERTIS-6016) which
completely blocking the USB unplug test.

With changes provided in previous commit we provide the binary diff
big enough to catch the proper timing by human and without kernel
crashes.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>