Rather than looping over branches (in included template from `pages`),
generate the test cases here, provide these as an artifact and deploy
in a separate job. This avoids the regeneration of pages that have
not changed.

Use the apertis-test-cases-web as part of a multi-project pipeline to
deploy.

Whilst we are here, utilise an environment variable for the release to
make it easier to migrate this between releases.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

This reverts commit c8252838.
Disable internal armhf image from supported images types
for OTA update tests since their behavior aren't stable and
there is no value to test AUM on this platform.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Use separate commit for changing the target to avoid
potential merging issues in future.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add internal armhf image into supported images types
for OTA update tests.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add the test for checking non-successful update during OTA update.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Ensure that only updates signed with valid key can be applied.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Test OTA updates behavior with lack of disk space.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add automatic test of unattended network updates.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add automatic test for network updates API.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Xorg is no more expected as process in complain mode.

Fixes: APERTIS-6233

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Fix aa_log_extract_tokens.sh path as it is intalled by
apertis-tests-apparmor-report package.

Fixes: APERTIS-6233

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Emanuele Aina authored 5 years ago and Luis Araujo committed 5 years ago

Target images using the etnaviv gfx stack are now built daily, and test jobs
are already being submitted, reflect this in the testcases.

See https://phabricator.apertis.org/T6395

 for more details.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

It has been observed that the execution time for the test varies.
Icrease the timeout to ensure that the build doesn't get killed halfway
through.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

This reverts commit 96227328.

The AUM tests have been experimentally enabled on arm64 and amd64 but it seems
things are still too broken to officially support them.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since the rebase to Buster we are no longer shipping `clutter/test-text`,
making the test execution impossible to complete.

Replace the test program with the MiniBrowser from WebKitGTK opening
a simple WebGL scene and testing that the output is not stretched.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Use `apertis/v2020pre` branch as default.
Do not test outdated branches.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add the description of how to add the proxy address for OSTree allowing
online upgrades both for `ostree` and AUM in restricted networks.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Use timedatectl to workaround NTP issues on some networks

With battery backed devices on networks with no properly reachable NTP servers
the boards would fail to automatically get the right time, so testers set it up
manually with `date`.

However, the hardware clock need to be configured as well for the time
configuration to survive a reboot, needed by the Apertis update manager OTA
test.

Without this, the device will fail to establish any TLS connection after the
reboot since every certificate will look like they are out of their validity
period.

This is only a workaround until we ensure NTP servers are reachable on the
testers' network, see https://phabricator.apertis.org/T6115



Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Need to test the upgrade behavior on different architectures.
Enabled automatic tests of ostree and upgrade manager behavior for all
supported minimal images.

Not enabled rollback test for Arm64 and Amd64 yet since they have no
enabled bootcounter mechanism in bootloader.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Peter Senna Tschudin committed 5 years ago

Test case allow to check that AUM refuses to use commit signed with
unknown key (or unsigned). Check the well-known place for public keys
in system configuration directory is parsed correctly.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 5 years ago and Peter Senna Tschudin committed 5 years ago

New test case allows to check the initial configuration of repository
and ensure that AUM refuses to use unsigned commit and commit signed
with unknown signature.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Commit infrastructure/apertis-test@a699e7d "Check for failed systemd units"
introduced a new call to `systemctl` that gets included even in case the test
is successful.

Update the expected output to avoid any confusion.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 5 years ago and Luis Araujo committed 5 years ago

The test is currently emitting results in this way:

    Testing supported locales:
    ar_QA.utf8: fail
    ar_SA.utf8: fail
    cs_CZ.utf8: fail
    da_DK.utf8: fail
    de_DE.utf8: fail
    el_GR.utf8: fail
    en_GB.utf8: fail
    en_US.utf8: fail
    es_AR.utf8: fail
    es_ES.utf8: fail
    es_MX.utf8: fail
    fi_FI.utf8: fail
    fr_CA.utf8: fail
    fr_FR.utf8: fail
    hu_HU.utf8: fail
    id_ID.utf8: fail
    it_IT.utf8: fail
    ms_MY.utf8: fail
    nb_NO.utf8: fail
    nl_NL.utf8: fail
    pl_PL.utf8: fail
    pt_BR.utf8: fail
    pt_PT.utf8: fail
    ro_RO.utf8: fail
    ru_RU.utf8: fail
    sk_SK.utf8: fail
    sv_SE.utf8: fail
    th_TH.utf8: fail
    tr_TR.utf8: fail
    zh_CN.utf8: fail
    zh_HK.utf8: fail
    31 test(s) failed

See https://lava.collabora.co.uk/scheduler/job/1827695#L5089



However, the current regex only matched uppercase characters,
expecting `PASSED`/`FAILED`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This commit add two test for extending the test coverage of the persistent SDK disk.
The new tests cover changes made to an arbitrary file on /etc and another on
/home/user.

Signed-off-by: Benani Sagar Kishore <SagarKishore.Benani@in.bosch.com>

This test case has been failing for a long a time since the required
telepathy packages are missing from the images.

The test case is also not so relevant anymore and makes current SDK lava
tests jobs to fail, so this commit removes the test case.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Using the same SDK image to setup old and new VM instance results in a
VirtualBox error:

	Locking of attached media failed. A possible reason is that one of the media is attached to a running VM.
	Result Code:
	VBOX_E_INVALID_OBJECT_STATE (0x80BB0007)
	Component:
	SessionMachine
	Interface:
	IMachine {85cd948e-a71f-4289-281e-0ca7ad48cd89}

This commit change the test case to clone the VM instead of creating two
identical VMs.

This patch also fixes other issues that were pointed out by:

	https://phabricator.apertis.org/T5868#195314



Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

Most tests have been converted to use the system journal to check for audit
events, so there's no need to reset the audit.log anymore.

Resetting it is even harmful today since `auditd` is no longer shipped by
default on images, thus the step fails as the file and the directory that
should contain it do not exist.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Except for the SDK image, this test is failing in the remaining images.

This commit removes the test case since it is also not so relevant
anymore and needs to be reviewed later.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

All telepathy folks tests currently fail to execute because the
telepathy-gabble package is not available in the images.

Support for folks and telepathy libs also needs to be revised, so
this commit disables all the folks telepathy tests in the meantime.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Emanuele Aina authored 5 years ago and Peter Senna Tschudin committed 5 years ago

The `auditd` package is being dropped from the images. The audit log can be
retrieved from the system journal, and the `auditd` daemon currently fails to
start on the ostree images.

However, the apparmor-{geoclue,ofono,tracker} testcases use the `run-aa-test`
tool which is not trivial to port away from `audit.log`, so add an explicit
installation step for the `auditd` package to paper over it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The apparmor-tracker testcases uses the `run-aa-test` tool, which is written in
Python 3. This makes it unsuitable to be run on the OSTree images, since none
of them ships Python.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since version 0.4+20150326.3, `apparmor-demo` no longer uses `audit.log` but
relies exclusively on the system journal to check for audit messages.

For this reason, truncating `audit.log` before running the test is no longer
useful, and it is still actually harmful on the images that don't ship auditd
like the minimal ones (other image types may follow).

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use the systemd journal to retrieve audit messages, as the minimal images no
longer ship auditd.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use the systemd journal to retrieve audit messages, as the minimal images no
longer ship auditd.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

With battery backed devices on networks with no properly reachable NTP servers
the boards would fail to automatically get the right time, so testers set it up
manually with `date`.

However, the hardware clock need to be configured as well for the time
configuration to survive a reboot, needed by the Apertis update manager OTA
test.

Without this, the device will fail to establish any TLS connection after the
reboot since every certificate will look like they are out of their validity
period.

This is only a workaround until we ensure NTP servers are reachable on the
testers' network, see https://phabricator.apertis.org/T6115

Fixes: https://phabricator.apertis.org/T6284



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The sdk-persistent-disk.yaml test case suggest using ade to verify that
the persistent disk is working. However ade uses the release version of
the SDK as default value for the sysroot. If two different versions of
the SDK are used, not specifying the release for ade will cause ade
update to fail. This patch adds --release to all calls of ade.

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

Replace aa_log_extract_tokens.pl by aa_log_extract_tokens.sh to remove
dependencies on auditd and libapparmor-perl packages.

Fixes: APERTIS-6233

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>