Hattihal Basavana Gowda Chethan authored 4 years ago and Emanuele Aina committed 4 years ago

When some test scripts got moved from /usr/share/chaiwala-tests/ to
/usr/lib/apertis-tests/ their path in the test definitions didn't get
updated, causing the tests to fail:

    + common/run-test-in-systemd --basename --user=user -- /usr/share/chaiwala-tests/sdk/automated/test-valgrind.sh
    /var/lib/lava-1826179/1/tests/6_sdk-debug-tools-valgrind/apertis-tests/common/tee-exec: 19: /var/lib/lava-1826179/1/tests/6_sdk-debug-tools-valgrind/apertis-tests/common/tee-exec: Job for generated-test-case-test-valgrind.service failed because the control process exited with error code./usr/share/chaiwala-tests/sdk/automated/test-valgrind.sh: not found

To fix the issue, rather that using the version shipped in
/usr/lib/apertis-tests/ use the one in the git repository since we
already have it checked out.

Fixes: https://phabricator.apertis.org/T6349
Fixes: https://phabricator.apertis.org/T6350
Fixes: https://phabricator.apertis.org/T6351
Fixes: https://phabricator.apertis.org/T6352



Signed-off-by: hattihalbasavanagowda.chethan <hattihalbasavanagowda.chethan@in.bosch.com>

Rather than looping over branches (in included template from `pages`),
generate the test cases here, provide these as an artifact and deploy
in a separate job. This avoids the regeneration of pages that have
not changed.

Rather than using the page branch, use the apertis-test-cases-web as
part of a multi-project pipeline.

Whilst we are here, utilise an environment variable for the release to
make it easier to migrate this between releases.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Commit infrastructure/apertis-test@a699e7d "Check for failed systemd units"
introduced a new call to `systemctl` that gets included even in case the test
is successful.

Update the expected output to avoid any confusion.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The test is currently emitting results in this way:

    Testing supported locales:
    ar_QA.utf8: fail
    ar_SA.utf8: fail
    cs_CZ.utf8: fail
    da_DK.utf8: fail
    de_DE.utf8: fail
    el_GR.utf8: fail
    en_GB.utf8: fail
    en_US.utf8: fail
    es_AR.utf8: fail
    es_ES.utf8: fail
    es_MX.utf8: fail
    fi_FI.utf8: fail
    fr_CA.utf8: fail
    fr_FR.utf8: fail
    hu_HU.utf8: fail
    id_ID.utf8: fail
    it_IT.utf8: fail
    ms_MY.utf8: fail
    nb_NO.utf8: fail
    nl_NL.utf8: fail
    pl_PL.utf8: fail
    pt_BR.utf8: fail
    pt_PT.utf8: fail
    ro_RO.utf8: fail
    ru_RU.utf8: fail
    sk_SK.utf8: fail
    sv_SE.utf8: fail
    th_TH.utf8: fail
    tr_TR.utf8: fail
    zh_CN.utf8: fail
    zh_HK.utf8: fail
    31 test(s) failed

See https://lava.collabora.co.uk/scheduler/job/1827695#L5089



However, the current regex only matched uppercase characters,
expecting `PASSED`/`FAILED`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Benani Sagar Kishore (RBEI/ECO2) authored 5 years ago and Emanuele Aina committed 5 years ago

This commit add two test for extending the test coverage of the persistent SDK disk.
The new tests cover changes made to an arbitrary file on /etc and another on
/home/user.

Signed-off-by: Benani Sagar Kishore <SagarKishore.Benani@in.bosch.com>

Peter Senna Tschudin authored 5 years ago and Emanuele Aina committed 5 years ago

Using the same SDK image to setup old and new VM instance results in a
VirtualBox error:

	Locking of attached media failed. A possible reason is that one of the media is attached to a running VM.
	Result Code:
	VBOX_E_INVALID_OBJECT_STATE (0x80BB0007)
	Component:
	SessionMachine
	Interface:
	IMachine {85cd948e-a71f-4289-281e-0ca7ad48cd89}

This commit change the test case to clone the VM instead of creating two
identical VMs.

This patch also fixes other issues that were pointed out by:

	https://phabricator.apertis.org/T5868#195314



Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

This test case has been failing for a long a time since the required
telepathy packages are missing from the images.

The test case is also not so relevant anymore and makes current SDK lava
tests jobs to fail, so this commit removes the test case.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Most tests have been converted to use the system journal to check for audit
events, so there's no need to reset the audit.log anymore.

Resetting it is even harmful today since `auditd` is no longer shipped by
default on images, thus the step fails as the file and the directory that
should contain it do not exist.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Luis Araujo authored 5 years ago and Emanuele Aina committed 5 years ago

Except for the SDK image, this test is failing in the remaining images.

This commit removes the test case since it is also not so relevant
anymore and needs to be reviewed later.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Luis Araujo authored 5 years ago and Emanuele Aina committed 5 years ago

All telepathy folks tests currently fail to execute because the
telepathy-gabble package is not available in the images.

Support for folks and telepathy libs also needs to be revised, so
this commit disables all the folks telepathy tests in the meantime.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

The `auditd` package is being dropped from the images. The audit log can be
retrieved from the system journal, and the `auditd` daemon currently fails to
start on the ostree images.

However, the apparmor-{geoclue,ofono,tracker} testcases use the `run-aa-test`
tool which is not trivial to port away from `audit.log`, so add an explicit
installation step for the `auditd` package to paper over it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The apparmor-tracker testcases uses the `run-aa-test` tool, which is written in
Python 3. This makes it unsuitable to be run on the OSTree images, since none
of them ships Python.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since version 0.4+20150326.3, `apparmor-demo` no longer uses `audit.log` but
relies exclusively on the system journal to check for audit messages.

For this reason, truncating `audit.log` before running the test is no longer
useful, and it is still actually harmful on the images that don't ship auditd
like the minimal ones (other image types may follow).

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use the systemd journal to retrieve audit messages, as the minimal images no
longer ship auditd.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use the systemd journal to retrieve audit messages, as the minimal images no
longer ship auditd.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Peter Senna Tschudin authored 5 years ago and Emanuele Aina committed 5 years ago

The sdk-persistent-disk.yaml test case suggest using ade to verify that
the persistent disk is working. However ade uses the release version of
the SDK as default value for the sysroot. If two different versions of
the SDK are used, not specifying the release for ade will cause ade
update to fail. This patch adds --release to all calls of ade.

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

Frederic Danis authored 5 years ago and Emanuele Aina committed 5 years ago

Replace aa_log_extract_tokens.pl by aa_log_extract_tokens.sh to remove
dependencies on auditd and libapparmor-perl packages.

Fixes: APERTIS-6233

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

We want to ensure that things work out-of-the-box and having the test install
packages would mask issue due to missing packages in the default installation.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The lack of links support in the `steps:` list caused the `sdk-persistent-disk`
to lack important instructions, making testers more confused than necessary.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Luis Araujo authored 5 years ago and Emanuele Aina committed 5 years ago

Remove this package dependency from the dbus-installed-tests test case
since this package is already installed by coreutils.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Denis Pynkin authored 5 years ago and Emanuele Aina committed 5 years ago

Use a single step for reboot waiting. Added the example how to track the
upgrade status while waiting for automatic upgrade and reboot.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Luis Araujo authored 5 years ago and Emanuele Aina committed 5 years ago

Use apertis-update-manager-automount for the test case name so it
matches its filename.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Frédéric Dalleau authored 5 years ago and Emanuele Aina committed 5 years ago

ade-commands test is failing, and this seems due to long download
times. Increase the timeout of the test.
The sysroot weight 1GB
Refer to APERTIS-6124

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

This commit splits the sanity-check test case into a specific manual
version of it (sanity-check-manual).

This will solve the issue of overwriting automated tests results for
this test case on the qa-report-app.

The name sanity-check is preserved for the automated version so there is
no need to apply any changes to the LAVA jobs.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Emanuele Aina authored 5 years ago and Luis Araujo committed 5 years ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 5 years ago and Luis Araujo committed 5 years ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

With battery backed devices on networks with no properly reachable NTP servers
the boards would fail to automatically get the right time, so testers set it up
manually with `date`.

However, the hardware clock need to be configured as well for the time
configuration to survive a reboot, needed by the Apertis update manager OTA
test.

Without this, the device will fail to establish any TLS connection after the
reboot since every certificate will look like they are out of their validity
period.

This is only a workaround until we ensure NTP servers are reachable on the
testers' network, see https://phabricator.apertis.org/T6115

Fixes: https://phabricator.apertis.org/T6284



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use branch 'apertis/v2019' for all test cases.
gitlab-ci should generate the recent version of tests description for QA!

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Test cases files should always match their metadata name.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Add a test to check if the persistent disk functionality is working
properly by:
 - Installing a sysroot on the old SDK
 - Configuring the old SDK to use the persistent disk
 - Configuring the new SDK to use the persistent disk
 - Check that the sysroot is available on the new SDK

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

This test used a simple echo pass. The qa report was not able to
understand that the test passed successfully because it eliminated
tests results of the "lava" test group.
Use the specific lava tool for reporting instead

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Add a testcase to ensure that the sample image recipes can be built on the SDK
and that the commands in the documentation work correctly.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Frédéric Dalleau authored 5 years ago and Denis Pynkin committed 5 years ago

Currently paramiko do not support Keys in RFC4716 format that
have landed with the buster update.
So any keys need to be generated in legacy PEM format.

Fixes: APERTIS-5932
Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

This reverts commit c86d18a7.
The problem of low speed bandwidth is kernel crash (APERTIS-6016) which
completely blocking the USB unplug test.

With changes provided in previous commit we provide the binary diff
big enough to catch the proper timing by human and without kernel
crashes.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

With this change we prepare the "outdated" commit without the
content of some directories. This allows to provide the significant
difference between "original" and "outdated" commits, so it is easy
to check it in booted deployment.
The side effect of huge size difference during upgrade is more stable
and convenient testing of "disk full" and "usb unplug" cases.

More detailed description is available here:
https://phabricator.apertis.org/T5439#189061



Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Test if update doesn't break anything if there is not enough free space
on system disk.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Denis Pynkin committed 5 years ago

Test if rollback and blacklist mechanisms works for network updates.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Denis Pynkin committed 5 years ago

Test if AUM is able to correctly read network-related settings
from configuration file and able to do automatic updates.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Denis Pynkin committed 5 years ago

Manual test aimed to check the API between update manager and
`updatectl` tool allowing to start network updates with CLI.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Improve test case description.
Use the new NFS image-deployment value.

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>