This reverts commit d8db26c9 which was
introduced to see how things would work. Apparently we still have some road to
do before being able to properly turn on the tests on all the platforms
since they were all failing.

Among others, https://phabricator.apertis.org/T6446 and
https://phabricator.apertis.org/T6444

 have been identified during this
experiment.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The board name is used in ostree-based images for pointing
the correct ostree branch name.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Use correct boot commands for Amd64 since they are used
in all AUM tests during reboot.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Enable tests for ostree and Apertis Update Manager for all supported
architectures.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add testing of signature verification of commits in offline updates.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Recent ARM internal jobs fails with a timeout on unpacking the overlay
into the image.

This commit increases the default timeout to the action transfering the
overlay in the LAVA job to 5 minutes for all jobs types.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

This test case has been failing for a long a time since the required
telepathy packages are missing from the images.

The test case is also not so relevant anymore and makes current SDK lava
tests jobs to fail, so this commit removes the test case.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Except for the SDK image, this test is failing in the remaining images.

This test is also not relevant anymore, so this commit removes it along
with the group-folks template file, so LAVA won't deploy and boot an
image in order to execute just one single test.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

All telepathy folks tests currently fail to execute because the
telepathy-gabble package is not available in the images.

Support for folks and telepathy libs also needs to be revised, so
this commit disables all the folks telepathy tests in the meantime.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Sysprof moved to a binary capture log, so the current machinery parsing
the old XML-based format no longer works.

Since this is a smoke test, just check that the capture file exist and is
not empty.

This also avoid printing large amount of binary contents to the
LAVA log, which makes the log unviewable from the UI and may cause
other issues.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Update the `run-aa-test` script to use the new shell/awk based
`aa_log_extract_tokens.sh` implementation instead of the old Perl-based one.

This affects the follwing testcases:
* apparmor-geoclue
* apparmor-ofono
* apparmor-tracker

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The actual type for "should-be-denied-but-in-complain-mode-so-go-ahead" audit
messages is `ALLOWED`, not `PERMIT`.

See https://github.com/torvalds/linux/blob/v4.19/security/apparmor/audit.c#L32



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This commit checks for any failed systemd units in the sanity-check
test.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

These tests are not meant to be run on minimal images:
* apparmor-gstreamer1-0
* apparmor-pulseaudio
* apparmor-tumbler
* apparmor-tracker

Move them out of the minimal group to the common one.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit da75fe5a "apparmor: Drop bluez/, now shipped by tests/apparmor-bluez"
broke the debian package build by removing the `apparmor/bluez` folder but not
dropping the debian package which used to ship its contents:

    dh_install --fail-missing
    dh_install: Please use dh_missing --list-missing/--fail-missing instead
    dh_install: This feature will be removed in compat 12.
    dh_install: Cannot find (any matches for) "apparmor/bluez/*.expected" (tried in ., debian/tmp)

    dh_install: apertis-tests-apparmor-bluez missing files: apparmor/bluez/*.expected
    dh_install: Cannot find (any matches for) "apparmor/bluez/*.sh" (tried in ., debian/tmp)

    dh_install: apertis-tests-apparmor-bluez missing files: apparmor/bluez/*.sh
    dh_install: Cannot find (any matches for) "apparmor/bluez/test-bluez" (tried in ., debian/tmp)

    dh_install: apertis-tests-apparmor-bluez missing files: apparmor/bluez/test-bluez
    dh_install: missing files, aborting
    make[1]: *** [debian/rules:13: override_dh_install] Error 25

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit bbf36571 "apparmor: Drop automated/, now shipped by
tests/apparmor-basic-profiles" broke `make install` and thus the debian
package build:

    dh_auto_install --destdir=debian/tmp
    	make -j1 install DESTDIR=/tmp/build-snapshot.x7qrtWRuJX/s/debian/tmp AM_UPDATE_INFO_DIR=no
    make[2]: Entering directory '/tmp/build-snapshot.x7qrtWRuJX/s'
    install -d /tmp/build-snapshot.x7qrtWRuJX/s/debian/tmp/usr/lib/apertis-tests/apparmor/
    cp -a apparmor/apparmor-profile-sanity-check.sh /tmp/build-snapshot.x7qrtWRuJX/s/debian/tmp/usr/lib/apertis-tests/apparmor/
    install -d /tmp/build-snapshot.x7qrtWRuJX/s/debian/tmp/usr/lib/apertis-tests/apparmor/
    cp -a apparmor/apparmor-profile-sanity-check.yaml /tmp/build-snapshot.x7qrtWRuJX/s/debian/tmp/usr/lib/apertis-tests/apparmor/
    install -d /tmp/build-snapshot.x7qrtWRuJX/s/debian/tmp/usr/lib/apertis-tests/
    cp -a apertis_tests_lib /tmp/build-snapshot.x7qrtWRuJX/s/debian/tmp/usr/lib/apertis-tests/
    install -d /tmp/build-snapshot.x7qrtWRuJX/s/debian/tmp/usr/lib/apertis-tests/apparmor/
    cp -a apparmor/automated /tmp/build-snapshot.x7qrtWRuJX/s/debian/tmp/usr/lib/apertis-tests/apparmor/
    cp: cannot stat 'apparmor/automated': No such file or directory
    make[2]: *** [Makefile:61: install-apparmor/automated] Error 1

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

AppArmor info are accessible through jounalctl.
Replace the aa_log_extract_tokens.pl by shell version, removing
dependencies on auditd and libapparmor-perl packages.

Fixes: APERTIS-6233

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Point the test repositories from the job templates to the new branch
apertis/v2020dev0.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

Signed-off-by: Ritesh Raj Sarraf <rrs@debian.org>

Frédéric Dalleau authored 5 years ago and Emanuele Aina committed 5 years ago

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

The image type is used in the QA report to distinguish the
test results.

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Emanuele Aina authored 5 years ago and Peter Senna Tschudin committed 5 years ago

Add a testcase to ensure that the sample image recipes can be built on the SDK
and that the commands in the documentation work correctly.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Having a standard status line at the end of the test is extremely valuable
for manual runs, as it provides a clear, unambiguous, consistent indicator
to the tester.

Add it to `run-test-in-systemd` so it can be used to wrap manual tests just
like it wraps automated tests in LAVA.

 foobar: test failed (exit=1)


 foobar: test succeeded

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The nfsroot nfsuser has been renamed user.
The nfsroot apertis-nfs machine name changed to apertis.
Only the group-nfs-tpl.yaml test changed because other
tests are still using a 18.09 release.
When we update lava, it will be time to update other tests.

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Currently the lava artefacts are built separately than the Apertis
standard images. The image testing profiles can't be used because
the lava artefacts may not be built already when the image tests
are submitted.
Add specific profiles for the lava artefacts tests.
Create one lava profile per arch which is based on the
apertis-minimal-<arch>-<board>

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The current code fails because it tries to check out the sample app code twice
in the same place, and git complains.

Ensure that the repository is deleted between runs to avoid the issue.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Rely on `set -e` to make the scripts immediately exit on failure and on
`run-test-in-systemd` as called by the YAML testcase to signal failures to LAVA
appropriately.

This simplifies the code and also avoids inconsistencies between the testcase
names used inside the scripts and by `run-test-in-systemd`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 5 years ago and Frédéric Dalleau committed 5 years ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 6 years ago and Frédéric Dalleau committed 5 years ago

The tests/apparmor-bluez repository now ships the contents of the
`bluez/` folder, drop the obsolete duplicate.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 6 years ago and Frédéric Dalleau committed 5 years ago

The tests/apparmor-basic-profiles repository now ships the contents of the
`automated/` folder, drop the obsolete duplicate.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 6 years ago and Frédéric Dalleau committed 5 years ago

The tests/apparmor repository now ships the contents of the `apparmor.sh`
script in its `run-test.sh`, drop the obsolete duplicate.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

To make the test files available to the DUT, LAVA downloads and packages them
in a tarball which gets sent to the DUT and unpacked there.

In the `group-apparmor-tpl.yaml` case the generated overlay is quite big, ~150MB,
and the time needed for the DUT to download and install it is now triggering
the default connection timeout, which is set to 2 minutes.

https://lava.collabora.co.uk/static/docs/v2/timeouts.html#job-timeouts



The main culprit for the size of the overlay is the `apparmor-gstreamer1-0`
testcase which contains a few relatively large videos. Since LAVA also packs
the `.git` folder, the amount of bytes consumed by the videos is doubled by the
presence of their git blobs, and the total weight of the testcase is around
100MB alone.

To work around the issue, increase the timeout specifically for the connection
matching the overlay download and unpack operation.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Cross building `helloworld-simple-agent` with GCC 8 from v2019dev0 against GLib
2.50.2 as shipped in 18.06 leads to failure due to `-Wextra` turning on the
new `-Wcast-function-type` warning and `AX_COMPILER_FLAGS` making it fatal by
turning `-Werror` on.

The fix for GLib landed in 2.56.0:
https://gitlab.gnome.org/GNOME/glib/commit/c1f5e52844c

In order to still build against the old 18.x sysroots, pass `--disable-Werror`
to the configure command.

See https://gitlab.gnome.org/GNOME/gnome-terminal/issues/96#note_480034


instead for the best way to silence the warning using a cast to `GCallback`
should the issue be in your own code.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

This commit adds an inline test definition in the modules tests job to
add the source packages for the target repository.

Most of the modules tests require to fetch source packages and such a
repository is not available by default in the images.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

The check-dbus-services test aims to test all services configured as bus
activatable, however such a test is proving to be too naive and requires
significant investment to keep in a passing state.
There are an increasing number of services that have needed to be
blacklisted as they don't pass with reasonable rationale:

* Some require the ability to display a GUI and recieve a response that
  isn't currently viable with the current automated test platform nor is a
  GUI present on many of the image types.
* Some require specific devices to be present for the underlying service
  to start that isn't present in on the test systems.
* Some aren't completely installed and operational in the smaller cut-down
  images.

It has been decided that this test lacks sufficent benefit and significant
overhead and is thus being removed.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Folks test cases are not so relevant anymore.

The apparmor-folks test case has not been executed for several releases
already (it was not executed for 18.12 and for 18.09 was only executed
for half the platforms with some failures too).

This commit removes the apparmor-folks tests.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>

This commit enables the tracker-indexing-local-storage test case in the
minimal images.

Signed-off-by: Luis Araujo <luis.araujo@collabora.co.uk>