Skip to content
Snippets Groups Projects
Commit 39d2a21e authored by Luis Araujo's avatar Luis Araujo
Browse files

Add new ported test cases


Signed-off-by: default avatarLuis Araujo <luis.araujo@collabora.co.uk>
parent 978bb0dc
No related branches found
No related tags found
No related merge requests found
Showing
with 664 additions and 1 deletion
metadata:
name: apparmor-bluez
format: "Apertis Test Definition 1.0"
image-type: target
image-arch: any
type: functional
exec-type: manual
priority: high
maintainer: "Apertis Project"
description: "Test AppArmor profile for support for SPP profile (bluez-spp) and
core functionalities (bluez-core)."
resources:
- "Two Bluetooth adapters."
macro_ostree_preconditions: apparmor-bluez
pre-conditions:
- "Please note that connman disables bluetooth by default on a fresh image."
- "Enable device:"
- "$ connmanctl enable bluetooth"
expected:
- "The following test results, which will appear in different places
throughout the log:"
- |
>R1.13a.1_bluez-spp.expected_underlying_tests: pass
R1.13a.1_bluez-spp.expected: pass
R1.13a.1_bluez-core.expected_underlying_tests: pass
R1.13a.1_bluez-core.expected: pass
notes:
- "This TC is based on other Bluez automated test cases, all of them should
PASS in order to execute them against AppArmor."
- "If a subtest for this TC FAILs, check that the related test is PASSING."
- "It is worth to investigate AppArmor-side only if the original TC PASSes and
the profiled version of it FAILs."
- "Any other scenario is either OK or worth waiting the original TC to PASS."
run:
steps:
- "Execute the following command:"
- $ ./test-bluez
metadata:
name: apparmor-chaiwala-system
format: "Apertis Test Definition 1.0"
image-type: target
image-arch: any
type: functional
exec-type: automated
priority: high
maintainer: "Apertis Project"
description: "Test that AppArmor's system wide rules works according to Apertis
security goals (Chaiwala was an older codename for Apertis)."
macro_ostree_preconditions: apparmor-chaiwala-system
expected:
- "The run-test.sh script should return 0. The return value indicates the
number of tests which failed. A similar output will be shown:"
- |
>Files retained in: /tmp/apparmor-chaiwala-system-gKUGXfrDVm-hFuzCE
/home/user/apparmor-chaiwala-system-master/goals/exec.sh: PASSED - /tmp/apparmor-chaiwala-system-gKUGXfrDVm-hFuzCE
Files retained in: /tmp/apparmor-chaiwala-system-ThPLvpkUnz-UZABGW
/home/user/apparmor-chaiwala-system-master/goals/home.sh: PASSED - /tmp/apparmor-chaiwala-system-ThPLvpkUnz-UZABGW
notes:
- "This tests only check that system wide rules (not components ones) are
correctly implemented."
run:
steps:
- "# Execute the following command:"
- common/run-test-in-systemd --timeout=90 --basename ./run-test.sh
parse:
fixupdict:
FAILED: fail
PASSED: pass
SKIP: skip
pattern: ^(?P<test_case_id>.+)\:\s(?P<result>PASS|pass|FAIL|fail|SKIP|skip|UNKNOWN|unknown)\s-
metadata:
name: apparmor-dbus
format: "Apertis Test Definition 1.0"
image-type: target
image-arch: any
type: functional
exec-type: automated
priority: critical
maintainer: "Apertis Project"
description: "Security infrastructure: AppArmor D-Bus mediation is part of
strategic application and service confinement."
macro_ostree_preconditions: apparmor-dbus
expected:
- "The test should report pass and not fail:"
- |
>dbus.expected: pass
notes:
- "The results of the tests printed to the standard output are enough to
determine whether they passed or fail."
- "If you get a failure and need to report a bug please add the apparmor logs
from sudo journalctl."
run:
steps:
- LAUNCH_DBUS="no" common/run-aa-test ./dbus.expected ./dbus
parse:
pattern: 'RESULT:(?P<result>\w+):(?P<test_case_id>[^:]+):'
# LAVA doesn't seem to have the concept of an expected failure,
# so calling it skipped is the next best thing
fixupdict:
xfail: skip
metadata:
name: apparmor-functional-demo
format: "Apertis Test Definition 1.0"
image-type: target
image-arch: any
type: functional
exec-type: automated
priority: medium
maintainer: "Apertis Project"
description: "Show some main AppArmor functionalities, like transitions, net
access or resource permissions, under the form of tests."
expected:
- "The /usr/lib/apparmor-demo/lava_run_tests script should return 0. The
return value indicates the number of tests which failed. A similar output
will be shown:"
- |
>/usr/lib/apparmor-demo/exec.sh: PASSED - /tmp/apparmor-demo/sdtest.6406-11853-I7WUoO
/usr/lib/apparmor-demo/exec_env.sh: FAILED - /tmp/apparmor-demo/sdtest.6704-7180-0VTgAJ
notes:
- "Some of the tests included are taken from AppArmor upstream, some are
locally developed."
install:
deps:
- bash
- apparmor-demo
- busybox
- apertis-tests-apparmor-report
run:
steps:
- "# Execute the following commands:"
- echo -n | sudo tee /var/log/audit/audit.log
- common/run-test-in-systemd --name=run-lava_run_tests --timeout=90 /usr/lib/apparmor-demo/lava_run_tests
- sudo cat /var/log/audit/audit.log | aa_log_extract_tokens.pl PERMITTING REJECTING
parse:
fixupdict:
FAILED: fail
PASSED: pass
SKIP: skip
pattern: ^(?P<test_case_id>.+)\.sh:\s*(?P<result>PASS|PASSED|pass|FAIL|FAILED|fail|SKIP|skip|UNKNOWN|unknown)\s*-
metadata:
name: connman-pan-tethering
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: manual
priority: medium
maintainer: "Apertis Project"
description: "Test ConnMan support for PAN connectivity."
resources:
- "A Bluetooth adapter."
- "A device supporting PAN PANU."
macro_install_packages_preconditions: apertis-tests python3-dbus python3-gi
pre-conditions:
- "Make sure an Internet connection is active on the system."
- "When testing on the SDK image, make sure blueman-applet is not running or
kill it."
expected:
- "If connections succeeds and you are able to navigate on the internet then
it works."
notes:
- "Warning: Connman changes are persistent (over reboot!). After finishing
testing, it might be wise to perform the dhcp test to ensure that the
network is in a sensible state."
- "For ALL tests the enable step will give an \"In progress\" error if the
device is already enabled. This is to be expected."
run:
steps:
- "Enable bluetooth:"
- $ connmanctl enable bluetooth
- "Start Bluetooth tethering:"
- $ connmanctl tether bluetooth on
- "Start simple agent:"
- $ /usr/lib/chaiwala-tests/bluez/simple-agent -y
- "On the other device, scan for the Bluetooth device and pair with, then
connect to the Bluetooth PAN. Most phones(?) don't actually support
tethering to another device (as they, by definition, already have an
internet connection) but connecting a computer and asking, say,
NetworkManager to connect to the newly paired bluetooth device should work
just fine (when testing in this case, ensure all other connections are
disabled!)"
metadata:
name: connman-services
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: manual
priority: critical
maintainer: "Apertis Project"
description: "Tests Network connection management: ensuring that ConnMan can
connect to actual networks is part of strategic network management
functionality."
pre-conditions:
- "The device must have wifi, otherwise skip this test."
- "Please note that connman disables wifi by default on a fresh image. To
enable:"
- $ connmanctl enable wifi
- "If it's already enabled, connmanctl will give the error
\"Error wifi: Already enabled\" that may be ignored."
expected:
- "If success \"Connected <service ID>\" will be printed by connmanctl,
otherwise \"Agent ReportError <service ID>\" will be printed."
notes:
- "Warning: Connman changes are persistent (over reboot!). After finishing
testing, it might be wise to perform the dhcp test to ensure that the
network is in a sensible state."
- "For ALL tests the enable step will give an \"In progress\" error if the
device is already enabled. This is to be expected."
- "When testing the SDK image in VirtualBox with an Asus USB-N10 Wireless
adapter, firmware-realtek needs to be installed on the host."
run:
steps:
- "Run the ConnMan CLI Interface:"
- $ connmanctl
- "When run without arguments, connmanctl will launch in interactive mode
with a \"connmanctl>\" prompt."
- "Scan wifi:"
- $ scan wifi
- "Wait for \"Scan completed for wifi\" before continuing (this may take a few
seconds and will appear above connmanctl prompt)."
- "Turn on the agent, to manage additional information if required:"
- $ agent on
- "connmanctl should return \"Agent registered\"."
- "List services available:"
- $ services
- "Select required service and use the \"service ID\" (second column) to
connect:"
- $ connect <service ID>
- "If it's connecting to a wifi service that needs additional information
connmanctl will output an \"Agent RequestInput (…)\" line followed by a
prompt for the required information, such as \"Passphrase? \"."
- "Ensure ConnMan returns \"Connected <service ID>\" before continuing."
- "Quit from connmanctl interactive mode with:"
- $ quit
- "Use the tool to test different kind of security and ConnMan technology:
Ethernet, Wi-Fi, cellular and Bluetooth."
metadata:
name: connman-usb-tethering
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: manual
priority: medium
maintainer: "Apertis Project"
description: "Test ConnMan support for USB tethering (connection side)."
resources:
- "Another device supporting USB tethering."
- "An USB cable."
macro_install_packages_preconditions: apertis-tests python-dialog dialog
pre-conditions:
- "Connect the remote device to an Internet connection."
expected:
- "The service will show as a Ethernet device, and its hash will reflect the
mac address of the Ethernet device created."
- "If connections succeeds and you are able to navigate on the internet then
it works."
notes:
- "Make sure that you have disconnected the ethernet connection to the target
before you start the tethering process."
- "If you are tethering to an android (4.1) device you need to enable USB
mass storage mode for the tethering option to become available."
run:
steps:
- "Enable USB tethering on the remote device."
- "Connect the USB cable from the remote device to the system."
- "Connect to the service:"
- $ /usr/lib/apertis-tests/connman/connman-test-service.py
metadata:
name: dbus-dos-reply-time
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: performance
exec-type: automated
priority: high
maintainer: "Apertis Project"
description: "Check the reply time of a few D-Bus method calls while dbus-daemon
is loaded by D-Bus traffic. Different patterns of D-Bus loads are
tested:
1) A connection calling a D-Bus method repetitively.
2) A process calling a D-Bus method repetitively using a
different connection each time to generate traffic related to
connections joining and leaving the bus (NameOwnerChanged
signals).
3) The same with 100 threads generating traffic.
4) Tens of threads registering hundreds of D-Bus match rules
DoS resistance is part of strategic basic services functionality."
macro_ostree_preconditions: dbus-dos-reply-time
expected:
- "There should be no error and it should show the time taken for every
command."
- |
>>>> Test setup successfully!
test_basic: command took 20ms (success), expected [0ms, 1000ms]
test_some_load: command took 21ms (success), expected [0ms, 4000ms]
test_heavy_load: command took 234ms (success), expected [0ms, 8000ms]
test_match_rules: command took 2333ms (success), expected [0ms, 8000ms]
run:
steps:
- "# Execute the following command:"
- common/run-test-in-systemd --basename --user=user --timeout=900 ./run-test.sh
parse:
fixupdict:
failure: fail
success: pass
pattern: '^(?P<test_case_id>[a-z_\-]+): command took\W(?P<measurement>[\d\.]+)ms
\((?P<result>\w+)\)'
metadata:
name: evolution-sync-bluetooth
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: manual
priority: medium
maintainer: "Apertis Project"
description: "Synchronizing Bluetooth-paired phone contacts to Evolution."
resources:
- "A Bluetooth USB adapter."
- "A Bluetooth-enabled phone."
macro_install_packages_preconditions: syncevolution syncevolution-dbus
folks-tools libfolks-eds25 apertis-tests
python 3.4 python3-gi python3-dbus
telepathy-mission-control-5
pre-conditions:
- "Start simple agent:"
- $ /usr/lib/chaiwala-tests/bluez/simple-agent -y
- "Pair both devices initiating from the phone side"
- "Pay attention to the simple-agent window as it requires input during
the pairing process."
expected:
- "syncevolution should display the number of contacts synchronized from the
phone in this last run."
- "folks-inspect should show the personas with their details grabbed from
your phone."
run:
steps:
- "Synchronize contacts from a second terminal:"
- $ /usr/lib/chaiwala-tests/contacts/sync-addressbook.sh <Bluetooth address>
- "The Bluetooth address is the MAC address of the device as seen in the
pairing window (e.g. 12:34:56:78:9A:BC)"
...@@ -6,8 +6,8 @@ metadata: ...@@ -6,8 +6,8 @@ metadata:
type: functional type: functional
exec-type: automated exec-type: automated
priority: medium priority: medium
description: "Run the GLib gio tests that do I/O on the local filesystem."
maintainer: "Apertis Project" maintainer: "Apertis Project"
description: "Run the GLib gio tests that do I/O on the local filesystem."
macro_ostree_preconditions: glib-gio-fs macro_ostree_preconditions: glib-gio-fs
......
metadata:
name: libsoup-unit
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: unit
exec-type: automated
priority: low
maintainer: "Apertis Project"
description: "Run the unit tests that ship with the library against the running
system."
expected:
- "PASSED or FAILED"
install:
deps:
- libsoup2.4-tests
- curl
run:
steps:
- "# Execute the following command:"
- common/run-test-in-systemd --name=run-test --timeout=900 env DEBUG=2 libsoup/automated/run-test.sh
parse:
pattern: ^(?P<test_case_id>[a-zA-Z0-9_\-\./]+):\s*(?P<result>pass|fail|skip|unknown)$
metadata:
name: newport
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: automated
priority: medium
maintainer: "Apertis Project"
description: "Run the Newport test suite"
expected:
- "Tests will show PASS or FAIL as result."
install:
deps:
- gnome-desktop-testing
- newport-tests
run:
steps:
- "# Execute the following command:"
- common/run-test-in-systemd --user=user --timeout=900 --name=newport gnome-desktop-testing-runner newport
parse:
fixupdict:
FAIL: fail
PASS: pass
SKIP: skip
pattern: ^(?P<result>PASS|FAIL|SKIP):\s*(?P<test_case_id>\S+)
metadata:
name: sdk-ide-build-run-program
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
type: functional
exec-type: manual
priority: medium
maintainer: "Apertis Project"
description: "Creates a simple test project and confirms that compiling and
executing work."
pre-conditions:
- "For SDK image: Remember, we're now requiring VirtualBox 4.2.2 for tests
using Guest Additions. Please, read https://wiki.apertis.org/Docs/VirtualBox_Guest_AdditionsDocs/VirtualBox_Guest_Additions to update your setup."
- "Download the virtual machine image for the latest SDK release from
https://images.apertis.org/"
expected:
- "The Console tab at the bottom should show the output of the program
(\"!!!Hello World!!!\")."
run:
steps:
- "Open Eclipse."
- "If prompted to choose a workspace, click OK."
- "Click File > New > Project"
- "Select C Project under C/C++"
- "Name the project \"Hello World\" and, under Executable, click Hello World
ANSI C Project and choose the Linux GCC toolchain."
- "Note the simple Hello World code in the main portion of the window."
- "Click Project > Build All"
- "Click Run > Run"
- "If prompted with \"Errors exist in a required project. Continue launch?\",
click Yes."
metadata:
name: sdk-performance-tools-gprof-smoke-test
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
type: functional
exec-type: automated
priority: medium
maintainer: "Apertis Project"
description: "Checks basic gprof functionality by ensuring a CPU-intensive
program shows up in its log as consuming the vast majority of the
CPU cycles."
expected:
- "The test script will output a line with the test ID and the status of the
test (PASSED or FAILED), with some additional information. An example
output is:"
- |
>sdk-performance-tools-gprof: PASSED
Program ran in 94% of samples (>= 85% required)
install:
deps:
- apertis-tests
run:
steps:
- "# Execute the following command:"
- common/run-test-in-systemd --basename --user=user -- sdk/automated/test-gprof.sh
parse:
fixupdict:
FAILED: fail
PASSED: pass
FAIL: fail
PASS: pass
SKIP: skip
UNKNOWN: unknown
pattern: '^(?P<test_case_id>[a-zA-Z0-9_\-\./]+):\s*(?P<result>PASSED|PASS|pass|FAILED|FAIL|fail|SKIP|skip|UNKNOWN|unknown)$'
metadata:
name: sdk-performance-tools-sysprof-smoke-test
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
type: functional
exec-type: automated
priority: medium
maintainer: "Apertis Project"
description: "Checks basic sysprof functionality by ensuring a CPU-intensive
program shows up in its log as consuming the vast majority of the
CPU cycles."
expected:
- "The test script will output a line with the test ID and the status of the
test (PASSED or FAILED), with some additional information. An example
output is:"
- |
>sdk-performance-tools-sysprof-smoke-test: PASSED
Program ran in 99% of samples (>= 95% required)
install:
deps:
- apertis-tests
run:
steps:
- "# Execute the following command:"
- common/run-test-in-systemd --basename -- sdk/automated/test-sysprof.sh
parse:
fixupdict:
FAILED: fail
PASSED: pass
FAIL: fail
PASS: pass
SKIP: skip
UNKNOWN: unknown
pattern: '^(?P<test_case_id>[a-zA-Z0-9_\-\./]+):\s*(?P<result>PASSED|PASS|pass|FAILED|FAIL|fail|SKIP|skip|UNKNOWN|unknown)$'
metadata:
name: sdk-vb-fullscreen
format: "Apertis Test Definition 1.0"
image-type: SDK
image-arch: amd64
type: compatibility
exec-type: manual
priority: critical
maintainer: "Apertis Project"
description: "Tests that the SDK VM can be fullscreened in VirtualBox.
Development environment: ensuring that the VirtualBox SDK provides
a seamless experience."
pre-conditions:
- "For SDK image: Remember, we're now requiring VirtualBox 4.2.2 for tests
using Guest Additions. Please, read https://wiki.apertis.org/Docs/VirtualBox_Guest_AdditionsDocs/VirtualBox_Guest_Additions to update your setup."
- "Download the virtual machine image for the latest SDK release from
https://images.apertis.org/"
expected:
- "The screen is correctly resized when switching to fullscreen."
- "test-text displays test glyphs inside the Xephyr window."
run:
steps:
- "Start the SDK virtual machine"
- "Click \"Switch to Fullscreen\" from the \"View\" menu"
- "Open a terminal"
- "Launch a program using GLX under Xephyr:"
- $ Xephyr :1 -host-cursor -screen 800x480x24 -ac &
- $ DISPLAY=:1 /usr/lib/clutter-1.0/installed-tests/clutter/test-text
metadata:
name: video-animation-on-boot
format: "Apertis Test Definition 1.0"
image-type: target
image-arch: any
type: functional
exec-type: manual
priority: high
maintainer: "Apertis Project"
description: "Test whether Plymouth is able to display a graphical theme."
resources:
- "LVDS panel or monitor."
expected:
- "Plymouth shows the fade-in theme with a chaiwala logo and a set of leaves
should fade in. Check example at:
https://wiki.apertis.org/mediawiki/images/7/7d/Boot.gif"
run:
steps:
- "Boot the image and the animation should be displayed during booting."
metadata:
name: webkit2gtk-event-handling-redesign
format: "Apertis Test Definition 1.0"
image-type: any
image-arch: any
type: functional
exec-type: manual
priority: medium
maintainer: "Apertis Project"
description: "Test WebKit2Gtk ability to scroll by touch with inertial behaviour
after the touch ended."
resources:
- "A touch-screen with multiple touch points."
macro_install_packages_preconditions: webkit2gtk-testing
expected:
- "A page with many nested scrollable elements should be rendered."
- "Dragging the right side of the page with a single touch or mouse click
should cause the page content to scroll."
- "Clicking or touching the entry should focus it and allow you to enter text."
- "Scrolling the area bellow the entry should scroll the content that is
inside that area only, revealing another text entry."
- "Clicking the entry that is inside the scrollable area should focus it and
allow you to enter text."
- "When scrolling to the bottom of the page with a quick drag and release
touch with enough momentum should cause the scroll to continue with
decreasing speed towards the bottom of the page."
- "Starting a touch drag from the green \"Touch scroll drags here should not
start\" element should not result in page scrolls."
- "Starting a mouse drag from the red \"Mouse scroll drags here should not
start\" element should not result in page scrolls."
- "Clicking one of the links at the bottom of the page should take you to the
appropriate page (www.collabora.com or planet.collabora.com)."
run:
steps:
- "Run the following command:"
- $ GtkClutterLauncher -g 700x600 file:///usr/share/webkit2gtk/testing/scroll.html
- "Wait for the page to load, there will be two in-page scrollable areas,
with a text entry in between them."
- "Scroll the main page until you align the top of the text entry with the
top of the viewport."
- "Click the entry to focus it and enter some text."
- "Scroll the scrollable area below the entry until you find another entry
inside the scrollable area."
- "Try to focus it and enter some text."
- "Scroll to the bottom of the main page by doing a quick drag and release."
- "If you have a touchscreen, try dragging from the green area."
- "If you have a mouse, try dragging from the red area."
- "Click one of the two links at the very bottom and wait for the page
to load."
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment