Skip to content
Snippets Groups Projects
Commit 6c388475 authored by Luis Araujo's avatar Luis Araujo
Browse files

Add remaining test cases


- Add the remaining test cases
- Rename some sdk test cases to keep their name shorter

Signed-off-by: default avatarLuis Araujo <luis.araujo@collabora.co.uk>
parent 68a0bd13
No related branches found
No related tags found
No related merge requests found
Showing
with 663 additions and 7 deletions
metadata:
name: abstract-contact-addressbook-creation
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: automated
priority: medium
maintainer: "Apertis Project"
description: "A way to be able to create abstract addressbook is needed in
SyncEvolution, so that separate addressbooks can be created for
each paired device. Ensure creation and removal of databases
works properly in syncevolution."
expected:
- "Test script should show PASSED:"
- |
>create-abstract-sync-database: PASSED
install:
deps:
- syncevolution
- uuid-runtime
run:
steps:
- "common/run-test-in-systemd --user=user --timeout=900 folks/create-abstract-sync-database.sh"
parse:
fixupdict:
FAILED: fail
PASSED: pass
pattern: ^(?P<test_case_id>[\w-]+):\W+(?P<result>PASSED|FAILED)$
metadata:
name: ade-commands
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
type: functional
exec-type: automated
priority: critical
maintainer: "Apertis Project"
description: "Test ade command and sub-command with default and custom options
and arguments."
pre-conditions:
- "Network connectivity on the DUT as these tests fetch sysroot images over
the network."
- "Clone the apertis-tests repository:"
- $ git clone https://git.apertis.org/git/apertis-tests.git
- "Change directory to apertis-tests folder:"
- $ cd apertis-tests
expected:
- "The automated test should succeed:"
- |
>>>> Test setup successfully!
>>> ==== Testing ade usage with custom parameters
>>> "TEST_RESULT:pass:ade-tests-custom:"
notes:
- "Lots of ade messages will be logged on the standard output. Upon test fail,
the script will abort with:"
- |
>TEST_RESULT:fail:ade-tests-custom:
run:
steps:
- "# Run the ade-tests-custom.sh and ade-tests-defaults.sh script from the ade/
directory"
- common/run-test-in-systemd --timeout=30min --name=run-test ade/ade-tests-custom.sh
- common/run-test-in-systemd --timeout=30min --name=run-test ade/ade-tests-defaults.sh
parse:
pattern: 'TEST_RESULT:(?P<result>\w+):(?P<test_case_id>[^:]+):'
metadata:
name: apparmor-basic-profiles
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: automated
priority: critical
maintainer: "Apertis Project"
description: "Test basic apparmor profiles. Only parsing and syntax of profiles
are tested for now. Security infrastructure: parsing and syntax of
basic Apparmor profiles is part of strategic application and
services confinement."
macro_ostree_preconditions: apparmor-basic-profiles
expected:
- "The automated test should succeed:"
- |
>>>> Test setup successfully!
>>> Running test 'test_profile_parsing' ... PASS
>>> Running test 'test_profile_syntax' ... PASS
>>> All tests PASSED successfully!
- "If any test failed, they will be listed instead of the success message:"
- |
>>>> The following tests FAILED:
[list of tests]
run:
steps:
- "# Execute the following command:"
- common/run-test-in-systemd --user=user --timeout=90 ./run-test.sh
parse:
pattern: ^(?P<test_case_id>[a-zA-Z0-9_\-\./]+):\s*(?P<result>pass|fail|skip|unknown)$
metadata:
name: bluez-avrcp-volume
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: manual
priority: medium
maintainer: "Apertis Project"
description: "Test the AVRCP volume up and down commands."
resources:
- "A Bluetooth adapter"
- "An A2DP Source and AVRCP capable phone (Nokia N9 do not use an
Android 4.2 phone as their Bluetooth AVRCP support is broken)"
- "Note that you do not need to play any music on the phone; nor do you need
to have headphones or a speaker plugged into the Apertis device."
macro_install_packages_preconditions: apertis-tests python3-dbus python3-gi
expected:
- "If success, the following output should be generated by btmon:"
- |
>AV/C: Control: address 0x48 opcode 0x7c
Subunit: Panel
Opcode: Passthrough
Operation: 0x42 (VOLUME DOWN Pressed)
Length: 0x00
--
AV/C: Control: address 0x48 opcode 0x7c
Subunit: Panel
Opcode: Passthrough
Operation: 0x41 (VOLUME UP Pressed)
Length: 0x00
run:
steps:
- "Ensure PulseAudio is activated:"
- $ pactl stat
- "Run btmon before any connection happens:"
- "$ sudo btmon | grep -A4 'AV/C: Control'"
- "Start simple agent:"
- $ /usr/lib/chaiwala-tests/bluez/simple-agent -y
- "Pair both devices initiating from the phone side"
- "Pay attention to the simple-agent window as it requires input during
the pairing process."
- "In another terminal, execute (You can get the Bluetooth device address by
running hcitool scan):"
- $ /usr/lib/chaiwala-tests/bluez/test-avrcp.py -i hci0 device_address
metadata:
name: bluez-phone
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: manual
priority: critical
maintainer: "Apertis Project"
description: "Test all BlueZ features needing interaction with a phone
(Pairing, OBEX, PBAP, AVRCP, A2DP, HSP, HFP, PAN). Strategic
component: Bluetooth phone calls is part of smartphone interaction
functionality."
resources:
- "A Bluetooth adapter (if you have two plugged, please remove one to avoid
confusion)"
- "A Bluetooth-enabled phone with a SIM card and a data plan. (The Nokia N9
has been proven to work flawlessly. Nexus has shown problems due to its
new BT implementation in Android 4.2. Android 6 phones and iPhone>5 should
work well)"
macro_ostree_preconditions: bluez-phone
pre-conditions:
- "If running the test on an SDK image, kill the blueman-applet process, as
it prevents the test from installing its own pairing agent."
expected:
- "If PASSED is displayed, all of the test have passed. The output should be
similar to that:"
- |
>select_adapter
Selected /org/bluez/hci0
select_device: Discovering...
Device found: F0:39:42:86:2E:1B Charge 2
Device found: 88:83:22:2F:56:2A Galaxy A5 2016
Input device address: 88:83:22:2F:56:2A
Selected address: 88:83:22:2F:56:2A
start_agent
test_pairing_initiator
Scanning in progress...
Pairing request to 88:83:22:2F:56:2A in progress...
Device found: /org/bluez/hci0/dev_88_83_22_2F_56_2A
Device 88_83_22_2F_56_2A is paired
test_pairing_responder
Start a pairing from the phone! was it successful (y/n):
y
Device found: /org/bluez/hci0/dev_88_83_22_2F_56_2A
Device 88_83_22_2F_56_2A is paired
stop_agent
test_profiles
OBEXObjectPush
test_profile_opp_server
./bluez-phone.sh: 360: kill: Usage: kill [-s sigspec | -signum | -sigspec] [pid | job]... or
kill -l [exitstatus]
Start an OPP transfer in the phone, was it successful? (y/n)
y
AudioSource
test_profile_a2dp_src
Connecting
Play music from the device! Did you hear it (y/n):
y
Disconnecting
AV Remote Control Target
Headset Audio Gateway
PANU
NAP
test_profile_nap
NAP:service name
Handsfree Audio Gateway
SimAccess
PBAP Phonebook Access PSE
test_profile_pbap_pse
PB
Number of contacts: (uint16 119,)
ICH
Number of contacts: (uint16 87,)
OCH
Number of contacts: (uint16 321,)
MCH
Number of contacts: (uint16 92,)
CCH
Number of contacts: (uint16 500,)
MAP Message Access Server
test_profile_map_mse
()
PnPInformation
Unknown profile '00001800-0000-1000-8000-00805f9b34fb',
Unknown profile '00001801-0000-1000-8000-00805f9b34fb']>,)
test_profile_opp_client
Transfer will start
(objectpath '/org/bluez/obex/client/session8/transfer56', {'Status': <'queued'>, 'Name': <'vcard.vcf'>, 'Size': <uint64 81>, 'Filename': <'/tmp/vcard.vcf'>, 'Session': <objectpath '/org/bluez/obex/client/session8'>})
Please ensure the transfer is received on the target, was it received? (y/n)
y
PASSED
- "If FAILED is displayed, one of the test has failed and the test sequence
is interrupted."
- "Some specific settings has to be enabled on the iPhone in order to make the
MAP test pass. In Bluetooth > Settings, notifications must be enabled in
the device specific screen. The notifications can only be enabled after the
device is paired. The following procedure can be used. MAP test should pass
after that:"
- |
>1 - Run bluez-phone.sh a first time: pair, run all tests, and let MAP test fail.
2 - Go to Settings > Bluetooth. In the device speficic screen, enable notifications.
3 - Run bluez-phone.sh with the following options: -s -a <bluetooth address of target device>.
notes:
- "Errors most frequently occurs at pairing. This can be used to circumvent
pairing problems, though this does not mean the test is a full pass, as
the pairing is part of the test:"
- |
>cd ~/bluez-phone-master
armhf/bin/pair_two hci0 <bt address of the phone>
./bluez-phone.sh -a <address of the phone> -s
run:
steps:
- "The phone must be in discoverable mode for starting this test. Look for
the phone in this list, and save the BT address. It will be used while
running the test."
- "Execute the test suite inside an environment with dbus:"
- $ ./bluez-phone.sh
- "There are some options:"
- |
>-a select which device you want to pair with (specify the address)
-s skip pairing because the device is already paired. -a must be present when using this.
-x enables test debugging (only useful if a test fails)
- "Once the test begins, after Apertis finishes pairing with the phone, you
must initiate pairing from the phone and do the pairing again as part of
the test. You may need to unpair Apertis from within the phone first. The
test will display the following message when that is required:"
- |
>Start a pairing from the phone! was it successful (y/n).
- "After you've initiated the pairing from the phone, the test will continue."
- "If the pairing fails from the test, try to do the pairing separately, and
then run the test using './bluez-phone.sh -a <phone addr> -s'. To do a
separate pairing: unpair the phone, and run 'bluetoothctl'. Then, in
bluetoothctl prompt, issue the following commands in order: 'remove
<phone addr>', 'discoverable on', 'agent off', 'agent NoInputNoOutput',
'default-agent'. The either 'pairable on' to pair from phone, or 'pair
<phone addr>' to issue a pairing to the phone. '<phone addr>' is the phone
address of the form 'a1:b2:c3:d4:e5:f6'."
- "The next step is to initiate a file transfer from the phone. This can be
done by sending a contact from the phone. The test will display the
following message when that is required:"
- |
>Start an OPP transfer in the phone, was it successful? (y/n)
- "After you’ve initiated the file transfer from the phone, the test will
continue."
metadata:
name: check-dbus-services
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: sanity
exec-type: automated
priority: critical
maintainer: "Apertis Project"
description: "Service monitoring: verifying that every D-Bus service can be
activated successfully as a systemd unit is part of strategic
basic services functionality."
macro_ostree_preconditions: check-dbus-services
expected:
- "The output should be a succession of tests similar to that:"
- |
>---------------------------------------------------------------
+ gdbus call --session -d org.xfce.Xfconf -o / -m org.freedesktop.DBus.Peer.Ping
()
-> 0
+ systemctl --full --no-ask-password --no-pager --user status xfconfd.service
● xfconfd.service - XFCE configuration service
Loaded: loaded (/usr/lib/systemd/user/xfconfd.service; static; vendor preset: enabled)
Active: active (running) since Mon 2018-06-18 10:19:31 UTC; 21h ago
Main PID: 809 (xfconfd)
CGroup: /user.slice/user-1000.slice/user@1000.service/xfconfd.service
└─809 /usr/lib/x86_64-linux-gnu/xfce4/xfconf/xfconfd
Jun 18 10:19:31 apertis systemd[771]: Starting XFCE configuration service...
Jun 18 10:19:31 apertis systemd[771]: Started XFCE configuration service.
-> 0
vvvvvvvvvvvvvvvvvvvv
RESULT:pass:session_org.xfce.Xfconf.service:
^^^^^^^^^^^^^^^^^^^^
- "In this test results, the xfail are expected failures."
- |
>RESULT:xfail:session_FailToActivate: org.apertis.ApertisTests.FailToActivate should not have been activatable
- "If any test failed, they will be prefixed with:"
- |
>RESULT:fail:
run:
steps:
- common/run-test-in-systemd --user=user bin/check-dbus-services
- common/run-test-in-systemd bin/check-dbus-services
parse:
pattern: 'RESULT:(?P<result>\w+):(?P<test_case_id>[^:]+):'
# LAVA doesn't seem to have the concept of an expected failure,
# so calling it skipped is the next best thing
fixupdict:
xfail: skip
metadata:
name: folks-metacontacts-antilinking
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: automated
priority: medium
maintainer: "Apertis Project"
description: "Ensures anti-linking meta-contacts works properly."
expected:
- "The script will report PASSED or FAILED."
install:
deps:
- telepathy-ring
- telepathy-mission-control-5
- telepathy-gabble
- libfolks-eds25
- libfolks-telepathy25
- folks-tools
run:
steps:
- "# Execute the following command:"
- "common/run-test-in-systemd --user=user --timeout=900 folks/folks-metacontacts-antilinking.sh"
parse:
fixupdict:
FAILED: fail
PASSED: pass
pattern: ^(?P<test_case_id>[\w-]+):\W+(?P<result>PASSED|FAILED)$
metadata:
name: sanity-check
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: sanity
exec-type: automated
priority: critical
maintainer: "Apertis Project"
description: "This test checks that the system bus is up, that a user session is
created and that a graphical interface is successfully running in
the system, as well as other similar basic services so that a
stable and reliable environment can be confirmed before futher
tests are executed.
Basic functionality: services."
pre-conditions:
- "A second system with a terminal and git available on it. This test must be
executed before other tests and even before installing other packages in
the image, so the script to execute this test should be fetched from a
different machine."
- "Clone the apertis-tests repository in the second system:"
- $ git clone https://gitlab.apertis.org/infrastructure/apertis-tests.git
- "Copy the apertis-tests folder to the Apertis system. Since the Apertis
image runs sshd by default, you can easily copy this file using the scp
utility from the second machine, for this you will need the Apertis image
IP, which you can find running the ip command from Apertis:"
- $ ip addr
- "Replace <APERTIS_IMAGE_IP> by the found IP and copy the apertis-tests
repository using scp from the second system where the apertis-tests
repository was cloned. It would look like this:"
- $ tar cf apertis-tests.tar apertis-tests/ && scp apertis-tests.tar user@<APERTIS_IMAGE_IP>:/home/user/
- "Alternatively, you can copy the apertis-tests directory in other ways, for
example by using an external storage device, manually mounting the device
and copying the folder. It is recommended and expected that you use scp
from a second machine."
- "Once the apertis-tests folder is available in the Apertis image, unpack it
in the apertis home user directory:"
- $ tar xf apertis-tests.tar
- "Enter the apertis-tests directory"
- $ cd apertis-tests/
expected:
- "The command should report no failure and its output should be something
like this at the end:"
- |
>+ grep -E ^NAME=("?)Apertis\1$ /etc/os-release
NAME="Apertis"
+ grep -E ^ID=("?)apertis\1$ /etc/os-release
ID=apertis
+ id -u user
+ test -S /run/user/1000/wayland-0
+ set +x
# Sanity check successful
notes:
- "IMPORTANT: If this test fails for an image, NO further tests should be
executed on that image, since this invalidates all test results."
- "This test must be executed in an image before running either automated or
manual tests."
run:
steps:
- "# Execute the following command:"
- common/sanity-check
parse:
pattern: 'TEST_RESULT:(?P<result>\w+):(?P<test_case_id>[^:]+):'
metadata:
name: sdk-ade-remote-debugging
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
type: functional
exec-type: manual
priority: high
maintainer: "Apertis Project"
description: "Test remote debugging with ADE. Cross-build an example HelloWorld
agent, deploy it to the target device and debug it with proper
backtraces."
pre-conditions:
- "A SDK host."
- "An ARM i.MX6 target device reachable via SSH from the SDK host."
- "Access to the Apertis source code hosting service at:"
- "~https://gitlab.apertis.org/"
- "Username and password for the target device."
- "Check the IP address of the target device, you'll need it during
configuration."
- "gdbserver must be installed in the remote device. The debugging port (1234
by default) must be open."
- "apertis-dev-tools package is installed on the SDK."
expected:
- "The sample \"Hello World\" agent is successfully bundled and pushed to the
target device and the application bundle org.apertis.HelloWorldSimpleAgent
is available under /Applications on the target device."
- "The debug connection to gdbserver running on the target device is set up
succesfully, the execution of the debugged program can be controlled with
GDB commands and the backtrace GDB command is able to resolve debug symbols
and print meaningful stacktraces."
run:
steps:
- "Install an ARM 32bit sysroot on the host"
- $ ade sysroot install
- "Enable key-based SSH login on the target device"
- $ ssh-keygen
- $ ssh-copy-id USER@HOST
- "Clone the source code for the HelloWorld sample agent on the SDK host"
- $ git clone https://gitlab.apertis.org/sample-applications/helloworld-simple-agent
- $ cd helloworld-simple-agent
- "Configure target device for debugging"
- $ ade configure --debug --device user@$HOST
- $ ade export
- $ ade install --device user@$HOST
- "Finally set up a debug connection to the target device"
- $ ade debug --device user@$HOST
metadata:
name: sdk-code-analysis-tools-gcov-smoke-test
name: sdk-code-analysis-tools-gcov
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
......
metadata:
name: sdk-code-analysis-tools-splint-smoke-test
name: sdk-code-analysis-tools-splint
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
......
metadata:
name: sdk-dbus-tools-bustle-smoke-test
name: sdk-dbus-tools-bustle
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
......
metadata:
name: sdk-dbus-tools-d-feet-smoke-test
name: sdk-dbus-tools-d-feet
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
......
metadata:
name: sdk-debug-tools-gdb
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
type: functional
exec-type: automated
priority: medium
maintainer: "Apertis Project"
description: "Checks basic gdb functionality by stopping a program at a set
point and checking its backtrace matches expectations."
expected:
- "The test script will output a line with the test ID and the status of the
test (PASSED or FAILED). An example output is:"
- |
>sdk-debug-tools-gdb: PASSED
install:
deps:
- bash
- apertis-tests
run:
steps:
- "# Execute the following command:"
- common/run-test-in-systemd --basename --user=user -- /usr/share/chaiwala-tests/sdk/automated/test-gdb.sh
parse:
fixupdict:
FAILED: fail
PASSED: pass
FAIL: fail
PASS: pass
SKIP: skip
UNKNOWN: unknown
pattern: '^(?P<test_case_id>[a-zA-Z0-9_\-\./]+):\s*(?P<result>PASSED|PASS|pass|FAILED|FAIL|fail|SKIP|skip|UNKNOWN|unknown)$'
metadata:
name: sdk-debug-tools-strace
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
type: functional
exec-type: automated
priority: medium
maintainer: "Apertis Project"
description: "Checks basic strace functionality by checking its output when run
on a simple command line program."
expected:
- "The test script will output a line with the test ID and the status of the
test (PASSED or FAILED). An example output is:"
- |
>sdk-debug-tools-strace: PASSED
install:
deps:
- bash
- apertis-tests
run:
steps:
- "# Execute the following command:"
- common/run-test-in-systemd --basename --user=user -- /usr/share/chaiwala-tests/sdk/automated/test-strace.sh
parse:
fixupdict:
FAILED: fail
PASSED: pass
FAIL: fail
PASS: pass
SKIP: skip
UNKNOWN: unknown
pattern: '^(?P<test_case_id>[a-zA-Z0-9_\-\./]+):\s*(?P<result>PASSED|PASS|pass|FAILED|FAIL|fail|SKIP|skip|UNKNOWN|unknown)$'
metadata:
name: sdk-debug-tools-valgrind
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
type: functional
exec-type: automated
priority: medium
maintainer: "Apertis Project"
description: "Checks basic valgrind functionality by running a program with and
without memory leaks, ensuring expected results from valgrind."
expected:
- "The test script will output a line with the test ID and the status of the
test (PASSED or FAILED). An example output is:"
- |
>sdk-debug-tools-valgrind: PASSED
install:
deps:
- bash
- apertis-tests
run:
steps:
- "# Execute the following command:"
- common/run-test-in-systemd --basename --user=user -- /usr/share/chaiwala-tests/sdk/automated/test-valgrind.sh
parse:
fixupdict:
FAILED: fail
PASSED: pass
FAIL: fail
PASS: pass
SKIP: skip
UNKNOWN: unknown
pattern: '^(?P<test_case_id>[a-zA-Z0-9_\-\./]+):\s*(?P<result>PASSED|PASS|pass|FAILED|FAIL|fail|SKIP|skip|UNKNOWN|unknown)$'
metadata:
name: sdk-performance-tools-gprof-smoke-test
name: sdk-performance-tools-gprof
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
......
metadata:
name: sdk-performance-tools-sysprof-smoke-test
name: sdk-performance-tools-sysprof
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
......@@ -16,7 +16,7 @@ metadata:
test (PASSED or FAILED), with some additional information. An example
output is:"
- |
>sdk-performance-tools-sysprof-smoke-test: PASSED
>sdk-performance-tools-sysprof: PASSED
Program ran in 99% of samples (>= 95% required)
install:
......
metadata:
name: webkit2gtk-ac-scrolling
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: manual
priority: medium
maintainer: "Apertis Project"
description: "Test that actors created for representing layers scroll properly."
resources:
- "A monitor connected to the testing target."
- "A mouse or touchscreen."
macro_install_packages_preconditions: webkit2gtk-testing
expected:
- "After the test page is loaded, the border of the GraphicsLayerActor must
scroll in sync with the rest of the content as shown in the following
video:"
- "~https://www.apertis.org/static/AC-scrolling.ogv"
- "This test will be considered a PASS if it works with at least one of mouse
or touchpad because this is not a test for the mouse or touchpad working."
run:
steps:
- "Run GtkClutterLauncher telling it to use 400x400 and loading the test page:"
- $ env CLUTTER_PAINT=paint-volumes GtkClutterLauncher -g 400x400 file:///usr/share/webkit2gtk/testing/LayoutTests/compositing/scroll-painted-composited-content.html
- "Scroll the page to the left until you can see the GraphicsLayerActor
border, verify that it scrolls along with the text above it."
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment