Since the rebase to Buster we are no longer shipping `clutter/test-text`,
making the test execution impossible to complete.

Replace the test program with the MiniBrowser from WebKitGTK opening
a simple WebGL scene and testing that the output is not stretched.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Our guidelines say that to contribute people should fork repositories,
but at the moment this gives them a broken pipeline.

Tweak things in a way that the main project triggers
tests/apertis-test-cases-web to collect and indirectly publish all the
release branches while forked repositories directly publish their own
branches via Pages for easier testing.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Lightweight CI runners have been introduced to run jobs which have low
resource consumption and do not need privileged containers to reduce costs
and latency, so let's use them.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The test uses `/usr/lib/chaiwala-tests/bluez/simple-agent` shipped in
the `chaiwala-tests` package, let's explicitly depend on it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Two of the test cases point to videos to show what a successful run looks
like. These were randomly stored on the old wiki machine. Rather than
having unlinked static content on the website, add these to the
`images` folder and update links.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We are going to start generating test cases for multiple generations of
Apertis to enable testing of stable releases. Add a menu to enable
browsing of the various releases.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Having the argument parsing in the atc_renderer code makes it much
harder to use it as a library. Pull the arg parsing into the
application.

As it turns out, atc didn't end up being used in this manner, but fix
it now to avoid this potentially needing to be addressed again in the
future.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Frederic Danis authored 4 years ago and Emanuele Aina committed 4 years ago

By default SabreLite board plays sound through HDMI connector.
In case tester want to output sound through headphone connector he should
configure PulseAudio for it.

After the 2 first commands (set-default-sink and set-sink-port) the sound
is heard correctly, but not after a reboot. This is due to mute state not
saved at this time, this needs the third command (after reboot) to fix it.
Further reboot will play sound correctly.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Frederic Danis authored 4 years ago and Emanuele Aina committed 4 years ago

The test currently fails due to timeout.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Use the multiple ostree preconditions support introduced in 6b2fd91e
instead of hardcoding the blurb for the second repository.

Also fix the branches and the resulting paths for the unpacked scripts.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Address a regression introduced in commit 6b2fd91e "Turn the OSTree
preconditions into a list" where the iteration on the preconditions list
was done improperly.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The LAVA lab is currently facing badnwidth issues. Switching to a
different ISP is planned, but due to the COVID impact there's no
progress on that front.

The current ade tests got badly impacted, with the download phase
consuming all the allocated time.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Baghmar Tarun authored 4 years ago and Emanuele Aina committed 4 years ago

If a test case requires multiple repositories it can now use a list of
objects as the OSTree preconditions to check them all out.

Signed-off-by: Baghmar Tarun <tarun.baghmar@in.bosch.com>

The templates hardcoded the `master` branch for the repositories
shipping the actual test scripts and binaries, but we use release-coded
branch names like `apertis/v2021dev2` there so the links in the
instructions pointed to the wrong place.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use names for the precondition variables that reflect the properties in
the testcase definitions so it's easier to spot what's going on.

This will alos make it easier to later expand them to use multiple
values.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Baghmar Tarun authored 4 years ago and Emanuele Aina committed 4 years ago

Signed-off-by: Baghmar Tarun <tarun.baghmar@in.bosch.com>

Baghmar Tarun authored 4 years ago and Emanuele Aina committed 4 years ago

1. ofono-sms-recieve
2. ofono-sms-send

Signed-off-by: Baghmar Tarun <tarun.baghmar@in.bosch.com>

Frederic Danis authored 4 years ago and Emanuele Aina committed 4 years ago

Currently `apparmor-bluez-avrcp-volume` test request to run
`bluez-avrcp-volume` test, allowing to install all dependencies for both
tests. Running solely one test ends up in errors due to missing
dependencies.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Frederic Danis authored 5 years ago and Emanuele Aina committed 4 years ago

Allow for Mildenhall components known to generate unrelated AppArmor
complaints.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Benani Sagar Kishore authored 4 years ago and Emanuele Aina committed 4 years ago

Signed-off-by: Benani Sagar Kishore (RBEI/ECO2) <SagarKishore.Benani@in.bosch.com>

Baghmar Tarun authored 4 years ago and Emanuele Aina committed 4 years ago

After the discussions following test cases will be dropped.
apparmor-tracker
folks-ofono-backend
telepathy-ring
tracker-indexing-local-storage
tracker-indexing-mass-storage
traffic-control-basic

Apertis: T5734
Signed-off-by: Tarun Baghmar <tarun.baghmar@in.bosch.com>
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Hattihal Basavana Gowda Chethan authored 4 years ago and Emanuele Aina committed 4 years ago

When some test scripts got moved from /usr/share/chaiwala-tests/ to
/usr/lib/apertis-tests/ their path in the test definitions didn't get
updated, causing the tests to fail:

    + common/run-test-in-systemd --basename --user=user -- /usr/share/chaiwala-tests/sdk/automated/test-valgrind.sh
    /var/lib/lava-1826179/1/tests/6_sdk-debug-tools-valgrind/apertis-tests/common/tee-exec: 19: /var/lib/lava-1826179/1/tests/6_sdk-debug-tools-valgrind/apertis-tests/common/tee-exec: Job for generated-test-case-test-valgrind.service failed because the control process exited with error code./usr/share/chaiwala-tests/sdk/automated/test-valgrind.sh: not found

To fix the issue, rather that using the version shipped in
/usr/lib/apertis-tests/ use the one in the git repository since we
already have it checked out.

Fixes: https://phabricator.apertis.org/T6349
Fixes: https://phabricator.apertis.org/T6350
Fixes: https://phabricator.apertis.org/T6351
Fixes: https://phabricator.apertis.org/T6352



Signed-off-by: hattihalbasavanagowda.chethan <hattihalbasavanagowda.chethan@in.bosch.com>

Rather than looping over branches (in included template from `pages`),
generate the test cases here, provide these as an artifact and deploy
in a separate job. This avoids the regeneration of pages that have
not changed.

Rather than using the page branch, use the apertis-test-cases-web as
part of a multi-project pipeline.

Whilst we are here, utilise an environment variable for the release to
make it easier to migrate this between releases.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Commit infrastructure/apertis-test@a699e7d "Check for failed systemd units"
introduced a new call to `systemctl` that gets included even in case the test
is successful.

Update the expected output to avoid any confusion.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The test is currently emitting results in this way:

    Testing supported locales:
    ar_QA.utf8: fail
    ar_SA.utf8: fail
    cs_CZ.utf8: fail
    da_DK.utf8: fail
    de_DE.utf8: fail
    el_GR.utf8: fail
    en_GB.utf8: fail
    en_US.utf8: fail
    es_AR.utf8: fail
    es_ES.utf8: fail
    es_MX.utf8: fail
    fi_FI.utf8: fail
    fr_CA.utf8: fail
    fr_FR.utf8: fail
    hu_HU.utf8: fail
    id_ID.utf8: fail
    it_IT.utf8: fail
    ms_MY.utf8: fail
    nb_NO.utf8: fail
    nl_NL.utf8: fail
    pl_PL.utf8: fail
    pt_BR.utf8: fail
    pt_PT.utf8: fail
    ro_RO.utf8: fail
    ru_RU.utf8: fail
    sk_SK.utf8: fail
    sv_SE.utf8: fail
    th_TH.utf8: fail
    tr_TR.utf8: fail
    zh_CN.utf8: fail
    zh_HK.utf8: fail
    31 test(s) failed

See https://lava.collabora.co.uk/scheduler/job/1827695#L5089



However, the current regex only matched uppercase characters,
expecting `PASSED`/`FAILED`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Benani Sagar Kishore (RBEI/ECO2) authored 5 years ago and Emanuele Aina committed 5 years ago

This commit add two test for extending the test coverage of the persistent SDK disk.
The new tests cover changes made to an arbitrary file on /etc and another on
/home/user.

Signed-off-by: Benani Sagar Kishore <SagarKishore.Benani@in.bosch.com>

Peter Senna Tschudin authored 5 years ago and Emanuele Aina committed 5 years ago

Using the same SDK image to setup old and new VM instance results in a
VirtualBox error:

	Locking of attached media failed. A possible reason is that one of the media is attached to a running VM.
	Result Code:
	VBOX_E_INVALID_OBJECT_STATE (0x80BB0007)
	Component:
	SessionMachine
	Interface:
	IMachine {85cd948e-a71f-4289-281e-0ca7ad48cd89}

This commit change the test case to clone the VM instead of creating two
identical VMs.

This patch also fixes other issues that were pointed out by:

	https://phabricator.apertis.org/T5868#195314



Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

This test case has been failing for a long a time since the required
telepathy packages are missing from the images.

The test case is also not so relevant anymore and makes current SDK lava
tests jobs to fail, so this commit removes the test case.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Most tests have been converted to use the system journal to check for audit
events, so there's no need to reset the audit.log anymore.

Resetting it is even harmful today since `auditd` is no longer shipped by
default on images, thus the step fails as the file and the directory that
should contain it do not exist.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Luis Araujo authored 5 years ago and Emanuele Aina committed 5 years ago

Except for the SDK image, this test is failing in the remaining images.

This commit removes the test case since it is also not so relevant
anymore and needs to be reviewed later.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Luis Araujo authored 5 years ago and Emanuele Aina committed 5 years ago

All telepathy folks tests currently fail to execute because the
telepathy-gabble package is not available in the images.

Support for folks and telepathy libs also needs to be revised, so
this commit disables all the folks telepathy tests in the meantime.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

The `auditd` package is being dropped from the images. The audit log can be
retrieved from the system journal, and the `auditd` daemon currently fails to
start on the ostree images.

However, the apparmor-{geoclue,ofono,tracker} testcases use the `run-aa-test`
tool which is not trivial to port away from `audit.log`, so add an explicit
installation step for the `auditd` package to paper over it.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The apparmor-tracker testcases uses the `run-aa-test` tool, which is written in
Python 3. This makes it unsuitable to be run on the OSTree images, since none
of them ships Python.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Since version 0.4+20150326.3, `apparmor-demo` no longer uses `audit.log` but
relies exclusively on the system journal to check for audit messages.

For this reason, truncating `audit.log` before running the test is no longer
useful, and it is still actually harmful on the images that don't ship auditd
like the minimal ones (other image types may follow).

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use the systemd journal to retrieve audit messages, as the minimal images no
longer ship auditd.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use the systemd journal to retrieve audit messages, as the minimal images no
longer ship auditd.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Peter Senna Tschudin authored 5 years ago and Emanuele Aina committed 5 years ago

The sdk-persistent-disk.yaml test case suggest using ade to verify that
the persistent disk is working. However ade uses the release version of
the SDK as default value for the sysroot. If two different versions of
the SDK are used, not specifying the release for ade will cause ade
update to fail. This patch adds --release to all calls of ade.

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

Frederic Danis authored 5 years ago and Emanuele Aina committed 5 years ago

Replace aa_log_extract_tokens.pl by aa_log_extract_tokens.sh to remove
dependencies on auditd and libapparmor-perl packages.

Fixes: APERTIS-6233

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

We want to ensure that things work out-of-the-box and having the test install
packages would mask issue due to missing packages in the default installation.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>