diff --git a/content/concepts/test_case_evaluation.md b/content/concepts/test_case_evaluation.md new file mode 100644 index 0000000000000000000000000000000000000000..2613426f52adb5c74b5577ad46d7b0634094b09e --- /dev/null +++ b/content/concepts/test_case_evaluation.md @@ -0,0 +1,194 @@ ++++ +title = "Test Case Review" +weight = 100 +outputs = [ "html", "pdf-in",] +date = "2020-09-11" ++++ + +Apertis has a slowly growing number of test cases that have been written over +the years that the Apertis project has been running. Effort has been made to +automate many of these tests, however many still remain as manually run testing +scenarios. + +Due to limited testing bandwidth, many of these manual tests are not frequently +performed. Additionally there may be automated tests that are being run that, +due to changes in the focus and direction of the Apertis project since it's +inception, are no longer particularly useful. When performed, such tests, both +manual and automated, are consuming valuable resources that could be better +utilised elsewhere. + +A review of the test cases has been carried out to identify tests that may now +be redundant and should therefore be removed. The following table shows the +analysis performed for this review along with any suggestions for change: + +| Test case | Manual/Auto | RBEI Estimated effort (min) | Status | Using Apt | Using Git | Value | Effort | Suggestion | Notes +| --------------------------------------------- | ------------- | ----------------------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ---------------------------------- +| ade-commands | automated | | Critical | | Yes | high | low | | Used by RBEI +| apertis-update-manager-api | manual | 10 | Low | | | low | high | Remove | Automated as aum-api +| apertis-update-manager-automount | manual | 10 | Medium | Yes | | high | high | | Used by PT +| apertis-update-manager-diskfull | manual | 10 | Low | | | low | high | Remove | Automated as aum-out-of-space +| apertis-update-manager-ota-api | manual | 10 | Medium | | | high | high | | +| apertis-update-manager-ota-auto | manual | 10 | Medium | | | low | high | Remove | Automated as aum-ota-auto +| apertis-update-manager-ota-diskfull | manual | 10 | Medium | | | low | high | Remove | Automated as aum-ota-out-of-space +| apertis-update-manager-ota-rollback | manual | 10 | Medium | | | high | high | | +| apertis-update-manager-ota-signed | manual | 10 | Medium | | | low | high | Remove | Automated as aum-ota-signed +| apertis-update-manager-powercut | manual | 10 | Low | | | high | high | | Although automated as aum-power-cut, that only simulates a power cut, so this test is still good confirmation +| apertis-update-manager-rollback | manual | 12 | Low | | | low | high | Remove | Automated as aum-rollback-blacklist +| apertis-update-manager-usb-unplug | manual | 12 | Medium | | | high | high | | +| apertis-update-manager | manual | 10 | Low | | | low | high | Remove | Automated as aum-offline-upgrade +| apparmor-basic-profiles | automated | | Critical | | Yes | high | low | | Basic security functionality (Still using git - needs updating) +| apparmor-bluez-avrcp-volume | manual | 05 | Medium | | | low | high | Merge | Performs "volume up" and "volume down" on paired device, merge with bluez-avrcp-volume +| apparmor-bluez-setup | manual | 05 | High | | | low | high | Merge | Merge with bluez-setup +| apparmor-bluez | manual | 05 | High | | | high | high | | Basic checks +| apparmor-chaiwala-system | automated | | High | | Yes | high | low | | Checks behaviour of chaiwala-base, inherited by some of the custom AppArmor profiles +| apparmor-dbus | automated | | Critical | | Yes | high | low | | Testing ability to contain dbus calls +| apparmor-functional-demo | automated | | Medium | Yes | Yes | high | high | Review | High effort as test is not easily understandable and thus harder to maintain. Contents needs full review +| apparmor-geoclue | automated | | Medium | Yes | Yes | medium | low | | Injects library to cause geoclue to try accessing something it shouldn't. +| apparmor-gstreamer1-0 | automated | | Medium | | Yes | high | low | | +| apparmor-ofono | manual | 08 | Medium | | | medium | medium | | Injects library to cause ofono to try accessing something it shouldn't. +| apparmor-pulseaudio | automated | | Medium | | Yes | medium | low | | Injects library to cause pulseaudio to try accessing something it shouldn't. +| apparmor-session-lockdown-no-deny | automated | | Medium | Yes | Yes | high | low | | Checks things we want are running and apparmor enforcement in place. +| apparmor-tumbler | automated | | Medium | | Yes | medium | low | | Injects library to cause tumbler to try accessing something it shouldn't. +| apparmor-utils | automated | | Medium | Yes | Yes | high | low | | Testing functionality of AppArmor utilities +| apparmor | automated | | Critical | | Yes | high | low | | Basic apparmor functionality tests +| aum-api | automated | | Critical | | Yes | high | low | | +| aum-offline-upgrade-collection_id | automated | | Critical | | Yes | high | low | | +| aum-offline-upgrade-signed | automated | | Critical | | Yes | high | low | | +| aum-offline-upgrade | automated | | Critical | | Yes | high | low | | +| aum-ota-api | automated | | Critical | | Yes | high | low | | +| aum-ota-auto | automated | | Critical | | Yes | high | low | | +| aum-ota-out-of-space | automated | | Critical | | Yes | high | low | | +| aum-ota-rollback-blacklist | automated | | Critical | | Yes | high | low | | +| aum-ota-signed | automated | | Critical | | Yes | high | low | | +| aum-out-of-space | automated | | Critical | | Yes | high | low | | +| aum-power-cut | automated | | Critical | | Yes | high | low | | +| aum-rollback-blacklist | automated | | Critical | | Yes | high | low | | +| bluez-avrcp-volume | manual | 08 | Medium | | | low | high | Merge | We have to do all this in `apparmor-bluez-avrcp-volume`. Merge the tests. +| bluez-hfp | manual | 08 | Medium | | | low | high | | Tests hands free profile +| bluez-phone | manual | 08 | Critical | | | high | high | Merge | Tests lots of bluetooth functionality, but can this not be merged with `apparmor-bluez` +| bluez-setup | manual | 08 | Critical | | | low | high | Merge | We have to do all this in `apparmor-bluez-setup`. Merge the tests. +| boot-no-crashes | automated | | High | | Yes | high | low | | Checks journal for auditd messages suggesting app crashes +| boot-performance | automated | | High | | Yes | high | low | | Dump boot performance (no fail state, but useful information) +| canterbury | automated | | Critical | | Yes | low | low | | +| cgroups-resource-control | automated | | Medium | | Yes | low | medium | Remove | There is a low chance this will catch bugs. The test is broken due to the reliance on a package no longer shipped by Debian and would need to be ported to something else +| connman-new-supplicant | automated | | Medium | | | high | low | Merge | Just checks that wpa_supplicant is on dbus (merge with connman test) +| connman-pan-network-access | manual | 08 | Medium | Yes | | high | high | | +| connman-pan-tethering | manual | 08 | Critical | Yes | | high | high | | +| connman-services | manual | 05 | Medium | | | high | high | | +| connman-usb-tethering | manual | 08 | Critical | Yes | | high | high | | +| connman | automated | | Medium | | Yes | high | low | | Tests bits are accessible on dbus +| dbus-dos-reply-time | automated | | High | | Yes | low | low | Remove | Runs dbus under various call load and time it. No longer a focus +| dbus-installed-tests | automated | | High | Yes | Yes | high | low | | Running dbus' own regression tests +| didcot | automated | | Medium | Yes | Yes | low | low | | +| disk-rootfs-fsck | automated | | High | | Yes | high | low | | +| eclipse-plugins-install-target | manual | 10 | Medium | | | low | high | Remove | +| eclipse-plugins-remote-debugging | manual | 10 | Medium | | | low | high | Remove | +| eclipse-sysroot-management | manual | 10 | Medium | Yes | | low | high | Remove | +| evolution-sync-bluetooth | manual | 05 | Medium | Yes | | low | high | Fix | Needed to verify phone book syncing +| frome | automated | | Low | Yes | Yes | low | low | | +| gettext-i18n | automated | | Medium | | Yes | low | low | Fix | Test broken, appears to have been for a while, needs fixing. +| glib-gio-fs | automated | | Medium | | Yes | high | low | | Test basic GIO functionality +| grilo | automated | | Medium | Yes | Yes | low | low | Improve | Still installing packages, needs improving (or removing) +| gstreamer1-0-decode | automated | | High | | Yes | high | low | | +| gupnp-services | automated | | Medium | | Yes | low | low | | +| hmi-audio-detail-view | manual | | Medium | | | high | high | | Focus shifting away from current HMI, but principal will remain relevant +| hmi-audio-play-pause | manual | | High | | | high | high | | Focus shifting away from current HMI, but principal will remain relevant +| hmi-video-load | manual | | High | | | high | high | | Focus shifting away from current HMI, but principal will remain relevant +| hmi-video-play-pause | manual | | High | | | high | high | | Focus shifting away from current HMI, but principal will remain relevant +| image-bootable | manual | 02 | Critical | | | high | high | | +| image-gui-start | manual | 02 | Critical | | | high | high | | +| iptables-basic | automated | | Critical | | Yes | high | low | | Test expected rules exist +| iptables-nmap | automated | 02 | High | | Yes | high | low | | +| newport | automated | | Medium | Yes | Yes | low | low | Improve | Tests installed via apt, needs improving +| nfsroot-simple-boot | automated | | High | | | high | low | | +| ofono-sms-receive | manual | 05 | Medium | | | low | high | | +| ofono-sms-send | manual | 05 | Medium | | | low | high | | +| ostree-collection-id | automated | | Critical | | Yes | high | low | | +| polkit-parsing | automated | | High | Yes | Yes | high | low | | +| rfkill-toggle | manual | 02 | Medium | | | low | high | | +| rhosydd | automated | | Medium | Yes | Yes | high | low | Improve | Installs lots via apt, including rhosydd. Not installed by default. +| ribchester | automated | | Critical | Yes | Yes | low | low | | +| sanity-check-manual | manual | 02 | Critical | | Yes | high | high | | Sanity check when performing manual tests +| sanity-check | automated | | Critical | | Yes | high | low | | +| sdk-ade-remote-debugging | manual | 08 | High | | Yes | high | high | | +| sdk-code-analysis-tools-gcov | automated | | Medium | Yes | Yes | low | low | | +| sdk-code-analysis-tools-splint | automated | | Medium | Yes | Yes | low | low | | +| sdk-cross-compilation | automated | | Medium | | Yes | high | low | | +| sdk-dbus-tools-bustle | manual | 02 | Medium | Yes | | low | high | Fix | +| sdk-dbus-tools-d-feet | manual | 02 | Medium | Yes | | low | high | Fix | +| sdk-debos-image-building | automated | | Critical | | Yes | high | low | | +| sdk-debug-tools-gdb | automated | | Medium | Yes | Yes | low | low | | +| sdk-debug-tools-strace | automated | | Medium | Yes | Yes | low | low | | +| sdk-debug-tools-valgrind | automated | | Medium | Yes | Yes | low | low | | +| sdk-flatpak-build-helloworld-app | manual | 20 | Medium | Yes | Yes | high | high | | +| sdk-flatpak-demo | automated | | Medium | Yes | Yes | high | low | | +| sdk-flatpak-helloworld-app | manual | 10 | Medium | | | high | high | | +| sdk-ide-build-run-program | manual | 02 | Medium | | | low | high | Improve | Needs updating to reflect subtle changes in new versions of Eclipse +| sdk-performance-tools-gprof | automated | | Medium | Yes | Yes | low | low | | +| sdk-performance-tools-sysprof | automated | | Medium | Yes | Yes | low | low | | +| sdk-persistent-disk-etc | manual | 15 | Critical | | Yes | high | high | | Remove if automated by [T7325](https://phabricator.apertis.org/T7325) +| sdk-persistent-disk-home-user | manual | 15 | Critical | | Yes | high | high | | Remove if automated by [T7325](https://phabricator.apertis.org/T7325) +| sdk-persistent-disk-sysroot | manual | 15 | Critical | | Yes | high | high | | Remove if automated by [T7325](https://phabricator.apertis.org/T7325) +| sdk-vb-fullscreen | manual | 05 | Critical | | Yes | high | high | | +| secure-boot-imx6 | manual | 20 | Medium | | | high | high | | +| tiny-container-system-aa-enforcement | automated | | Medium | | Yes | high | low | | +| tiny-container-system-basic | automated | | Medium | | Yes | high | low | | +| tiny-container-system-connectivity-profile | automated | | Medium | | Yes | high | low | | +| tiny-container-user-device-sharing | automated | | Medium | | Yes | high | low | | +| tiny-container-user-folder-sharing | automated | | Medium | | Yes | high | low | | +| tiny-container-user-seccomp | automated | | Medium | | Yes | high | low | | +| traprain | automated | | Medium | Yes | Yes | high | low | | +| tumbler-thumbnailing | automated | | Medium | | Yes | high | low | | +| video-animation-on-boot | manual | 03 | High | | | low | high | Update | Expecting old animation - update +| webkit2gtk-ac-3d-rendering | manual | 05 | Low | | Yes | high | high | | Test excercises a big chunk of stack, so keep +| webkit2gtk-ac-animations | manual | 05 | Medium | | Yes | low | high | Remove | No longer working on webkit +| webkit2gtk-ac-scrolling | manual | 05 | Medium | | Yes | low | high | Remove | No longer working on webkit +| webkit2gtk-drag-and-drop | manual | 05 | Low | | Yes | low | high | Remove | No longer working on webkit +| webkit2gtk-event-handling-redesign | manual | 05 | Medium | | Yes | low | high | Remove | No longer working on webkit +| webkit2gtk-gstreamer1.0 | manual | 05 | Medium | | Yes | high | high | | Test excercises a big chunk of stack, so keep +| webkit2gtk-mt-touch-events | manual | 05 | Low | | Yes | high | high | | Test excercises a big chunk of stack, so keep +| x-hw-accelerated | manual | 05 | Medium | Yes | | low | high | | Automate and remove if [T7325](https://phabricator.apertis.org/T7325) proves viable + +# General status of the test framework + +Whilst reviewing the test cases a number of additional A number of additional +points have been noted that are mostly out of scope for this review, though may +be worth concidering for future work: + +- **A lot of the tests seem to have needless levels of abstraction:** A number + of strategies have been employed over successive attempts to automate more of + the test cases. In order to avoid extra overhead, generally infrastructure + used to perform the previously automated tests has been kept and driven from + the latest mechanism. In parallel to this, work has been done to provide a + testing framework that makes steps carried out by the tests better visibly + documented (leading to the test descriptions on + [qa.apertis.org](https://qa.apertis.org). Due to may of these tests calling + previous mechanisms, these tests are not as transparent as they could be and + the (in some cases, multiple) layers of abstraction make them harder to read + and maintain. It is suggested that efforts are taken to remove these layers + of abstraction where possible. +- **It should be possible to merge some of the AppArmor tests with their + non-AppArmor counterparts:** Quite a few of the AppArmor tests run up the + same image and perform the same steps as the non-AppArmor test for the same + functionality, only differing in what they look at to deem the test a success + or failure. This adds overhead as the infrastructure is setting up and + performing the tests twice. This can be reduced by checking for both the + outcome of the test steps and the AppArmor status in a single test run + without loss of testing coverage. +- **Most tests should not be installing packages via `apt`:** A conciderable + number of the tests are installing packages via `apt` to for use in the + testing. These additions are fundamentally changing the image under test and + may affect the results. We have had a + [mechanism to combat this](https://www.apertis.org/qa/immutablerootfstests/) + for a while which needs to be universally rolled out. +- **We should update tests using `git clone` to use `wget`:** Rather than + requiring git to be install on the test image, we have recently been + downloading archives of the test harnesses with are provided by GitLab. This + only requires the ability to download a resource via HTTP, which can be + reasonably assumed to be possible in all the images rather than the ability + to perform a git clone. Tests should be updated to use this mechanism to + avoid the need for git. +- **Merge some tests together:** In some places we seem to have multiple tests, + each testing a facet of a larger set of functionality. Whilst not viable or + advisable for all the tests, merging some may be viable which would reduce + the overhead of setting up the environment individually each time.