Skip to content
Snippets Groups Projects
Commit f0a79202 authored by Martyn Welch's avatar Martyn Welch Committed by Martyn Welch
Browse files

Add test case evaluation 


An evaluation has been performed of the test cases to determine their
validity and value in respect to the current focus and direction of
Apertis. Add a document showing the analysis and recommendations reached as
a result of this evaluation.

Signed-off-by: default avatarMartyn Welch <martyn.welch@collabora.com>
parent 0304dd7b
No related branches found
No related tags found
1 merge request!74Add test case evaluation
Pipeline #160420 passed
Stage: generate-html
    Stage: generate-pdf
      Stage: deploy
        Hide whitespace changes
        Inline Side-by-side
        content/concepts/test_case_evaluation.md 0 → 100644
        + 194
        − 0
        View file @ f0a79202
        +++
        title = "Test Case Review"
        weight = 100
        outputs = [ "html", "pdf-in",]
        date = "2020-09-11"
        +++
        Apertis has a slowly growing number of test cases that have been written over
        the years that the Apertis project has been running. Effort has been made to
        automate many of these tests, however many still remain as manually run testing
        scenarios.
        Due to limited testing bandwidth, many of these manual tests are not frequently
        performed. Additionally there may be automated tests that are being run that,
        due to changes in the focus and direction of the Apertis project since it's
        inception, are no longer particularly useful. When performed, such tests, both
        manual and automated, are consuming valuable resources that could be better
        utilised elsewhere.
        A review of the test cases has been carried out to identify tests that may now
        be redundant and should therefore be removed. The following table shows the
        analysis performed for this review along with any suggestions for change:
        | Test case | Manual/Auto | RBEI Estimated effort (min) | Status | Using Apt | Using Git | Value | Effort | Suggestion | Notes
        | --------------------------------------------- | ------------- | ----------------------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ----------------------------------
        | ade-commands | automated | | Critical | | Yes | high | low | | Used by RBEI
        | apertis-update-manager-api | manual | 10 | Low | | | low | high | Remove | Automated as aum-api
        | apertis-update-manager-automount | manual | 10 | Medium | Yes | | high | high | | Used by PT
        | apertis-update-manager-diskfull | manual | 10 | Low | | | low | high | Remove | Automated as aum-out-of-space
        | apertis-update-manager-ota-api | manual | 10 | Medium | | | high | high | |
        | apertis-update-manager-ota-auto | manual | 10 | Medium | | | low | high | Remove | Automated as aum-ota-auto
        | apertis-update-manager-ota-diskfull | manual | 10 | Medium | | | low | high | Remove | Automated as aum-ota-out-of-space
        | apertis-update-manager-ota-rollback | manual | 10 | Medium | | | high | high | |
        | apertis-update-manager-ota-signed | manual | 10 | Medium | | | low | high | Remove | Automated as aum-ota-signed
        | apertis-update-manager-powercut | manual | 10 | Low | | | high | high | | Although automated as aum-power-cut, that only simulates a power cut, so this test is still good confirmation
        | apertis-update-manager-rollback | manual | 12 | Low | | | low | high | Remove | Automated as aum-rollback-blacklist
        | apertis-update-manager-usb-unplug | manual | 12 | Medium | | | high | high | |
        | apertis-update-manager | manual | 10 | Low | | | low | high | Remove | Automated as aum-offline-upgrade
        | apparmor-basic-profiles | automated | | Critical | | Yes | high | low | | Basic security functionality (Still using git - needs updating)
        | apparmor-bluez-avrcp-volume | manual | 05 | Medium | | | low | high | Merge | Performs "volume up" and "volume down" on paired device, merge with bluez-avrcp-volume
        | apparmor-bluez-setup | manual | 05 | High | | | low | high | Merge | Merge with bluez-setup
        | apparmor-bluez | manual | 05 | High | | | high | high | | Basic checks
        | apparmor-chaiwala-system | automated | | High | | Yes | high | low | | Checks behaviour of chaiwala-base, inherited by some of the custom AppArmor profiles
        | apparmor-dbus | automated | | Critical | | Yes | high | low | | Testing ability to contain dbus calls
        | apparmor-functional-demo | automated | | Medium | Yes | Yes | high | high | Review | High effort as test is not easily understandable and thus harder to maintain. Contents needs full review
        | apparmor-geoclue | automated | | Medium | Yes | Yes | medium | low | | Injects library to cause geoclue to try accessing something it shouldn't.
        | apparmor-gstreamer1-0 | automated | | Medium | | Yes | high | low | |
        | apparmor-ofono | manual | 08 | Medium | | | medium | medium | | Injects library to cause ofono to try accessing something it shouldn't.
        | apparmor-pulseaudio | automated | | Medium | | Yes | medium | low | | Injects library to cause pulseaudio to try accessing something it shouldn't.
        | apparmor-session-lockdown-no-deny | automated | | Medium | Yes | Yes | high | low | | Checks things we want are running and apparmor enforcement in place.
        | apparmor-tumbler | automated | | Medium | | Yes | medium | low | | Injects library to cause tumbler to try accessing something it shouldn't.
        | apparmor-utils | automated | | Medium | Yes | Yes | high | low | | Testing functionality of AppArmor utilities
        | apparmor | automated | | Critical | | Yes | high | low | | Basic apparmor functionality tests
        | aum-api | automated | | Critical | | Yes | high | low | |
        | aum-offline-upgrade-collection_id | automated | | Critical | | Yes | high | low | |
        | aum-offline-upgrade-signed | automated | | Critical | | Yes | high | low | |
        | aum-offline-upgrade | automated | | Critical | | Yes | high | low | |
        | aum-ota-api | automated | | Critical | | Yes | high | low | |
        | aum-ota-auto | automated | | Critical | | Yes | high | low | |
        | aum-ota-out-of-space | automated | | Critical | | Yes | high | low | |
        | aum-ota-rollback-blacklist | automated | | Critical | | Yes | high | low | |
        | aum-ota-signed | automated | | Critical | | Yes | high | low | |
        | aum-out-of-space | automated | | Critical | | Yes | high | low | |
        | aum-power-cut | automated | | Critical | | Yes | high | low | |
        | aum-rollback-blacklist | automated | | Critical | | Yes | high | low | |
        | bluez-avrcp-volume | manual | 08 | Medium | | | low | high | Merge | We have to do all this in `apparmor-bluez-avrcp-volume`. Merge the tests.
        | bluez-hfp | manual | 08 | Medium | | | low | high | | Tests hands free profile
        | bluez-phone | manual | 08 | Critical | | | high | high | Merge | Tests lots of bluetooth functionality, but can this not be merged with `apparmor-bluez`
        | bluez-setup | manual | 08 | Critical | | | low | high | Merge | We have to do all this in `apparmor-bluez-setup`. Merge the tests.
        | boot-no-crashes | automated | | High | | Yes | high | low | | Checks journal for auditd messages suggesting app crashes
        | boot-performance | automated | | High | | Yes | high | low | | Dump boot performance (no fail state, but useful information)
        | canterbury | automated | | Critical | | Yes | low | low | |
        | cgroups-resource-control | automated | | Medium | | Yes | low | medium | Remove | There is a low chance this will catch bugs. The test is broken due to the reliance on a package no longer shipped by Debian and would need to be ported to something else
        | connman-new-supplicant | automated | | Medium | | | high | low | Merge | Just checks that wpa_supplicant is on dbus (merge with connman test)
        | connman-pan-network-access | manual | 08 | Medium | Yes | | high | high | |
        | connman-pan-tethering | manual | 08 | Critical | Yes | | high | high | |
        | connman-services | manual | 05 | Medium | | | high | high | |
        | connman-usb-tethering | manual | 08 | Critical | Yes | | high | high | |
        | connman | automated | | Medium | | Yes | high | low | | Tests bits are accessible on dbus
        | dbus-dos-reply-time | automated | | High | | Yes | low | low | Remove | Runs dbus under various call load and time it. No longer a focus
        | dbus-installed-tests | automated | | High | Yes | Yes | high | low | | Running dbus' own regression tests
        | didcot | automated | | Medium | Yes | Yes | low | low | |
        | disk-rootfs-fsck | automated | | High | | Yes | high | low | |
        | eclipse-plugins-install-target | manual | 10 | Medium | | | low | high | Remove |
        | eclipse-plugins-remote-debugging | manual | 10 | Medium | | | low | high | Remove |
        | eclipse-sysroot-management | manual | 10 | Medium | Yes | | low | high | Remove |
        | evolution-sync-bluetooth | manual | 05 | Medium | Yes | | low | high | Fix | Needed to verify phone book syncing
        | frome | automated | | Low | Yes | Yes | low | low | |
        | gettext-i18n | automated | | Medium | | Yes | low | low | Fix | Test broken, appears to have been for a while, needs fixing.
        | glib-gio-fs | automated | | Medium | | Yes | high | low | | Test basic GIO functionality
        | grilo | automated | | Medium | Yes | Yes | low | low | Improve | Still installing packages, needs improving (or removing)
        | gstreamer1-0-decode | automated | | High | | Yes | high | low | |
        | gupnp-services | automated | | Medium | | Yes | low | low | |
        | hmi-audio-detail-view | manual | | Medium | | | high | high | | Focus shifting away from current HMI, but principal will remain relevant
        | hmi-audio-play-pause | manual | | High | | | high | high | | Focus shifting away from current HMI, but principal will remain relevant
        | hmi-video-load | manual | | High | | | high | high | | Focus shifting away from current HMI, but principal will remain relevant
        | hmi-video-play-pause | manual | | High | | | high | high | | Focus shifting away from current HMI, but principal will remain relevant
        | image-bootable | manual | 02 | Critical | | | high | high | |
        | image-gui-start | manual | 02 | Critical | | | high | high | |
        | iptables-basic | automated | | Critical | | Yes | high | low | | Test expected rules exist
        | iptables-nmap | automated | 02 | High | | Yes | high | low | |
        | newport | automated | | Medium | Yes | Yes | low | low | Improve | Tests installed via apt, needs improving
        | nfsroot-simple-boot | automated | | High | | | high | low | |
        | ofono-sms-receive | manual | 05 | Medium | | | low | high | |
        | ofono-sms-send | manual | 05 | Medium | | | low | high | |
        | ostree-collection-id | automated | | Critical | | Yes | high | low | |
        | polkit-parsing | automated | | High | Yes | Yes | high | low | |
        | rfkill-toggle | manual | 02 | Medium | | | low | high | |
        | rhosydd | automated | | Medium | Yes | Yes | high | low | Improve | Installs lots via apt, including rhosydd. Not installed by default.
        | ribchester | automated | | Critical | Yes | Yes | low | low | |
        | sanity-check-manual | manual | 02 | Critical | | Yes | high | high | | Sanity check when performing manual tests
        | sanity-check | automated | | Critical | | Yes | high | low | |
        | sdk-ade-remote-debugging | manual | 08 | High | | Yes | high | high | |
        | sdk-code-analysis-tools-gcov | automated | | Medium | Yes | Yes | low | low | |
        | sdk-code-analysis-tools-splint | automated | | Medium | Yes | Yes | low | low | |
        | sdk-cross-compilation | automated | | Medium | | Yes | high | low | |
        | sdk-dbus-tools-bustle | manual | 02 | Medium | Yes | | low | high | Fix |
        | sdk-dbus-tools-d-feet | manual | 02 | Medium | Yes | | low | high | Fix |
        | sdk-debos-image-building | automated | | Critical | | Yes | high | low | |
        | sdk-debug-tools-gdb | automated | | Medium | Yes | Yes | low | low | |
        | sdk-debug-tools-strace | automated | | Medium | Yes | Yes | low | low | |
        | sdk-debug-tools-valgrind | automated | | Medium | Yes | Yes | low | low | |
        | sdk-flatpak-build-helloworld-app | manual | 20 | Medium | Yes | Yes | high | high | |
        | sdk-flatpak-demo | automated | | Medium | Yes | Yes | high | low | |
        | sdk-flatpak-helloworld-app | manual | 10 | Medium | | | high | high | |
        | sdk-ide-build-run-program | manual | 02 | Medium | | | low | high | Improve | Needs updating to reflect subtle changes in new versions of Eclipse
        | sdk-performance-tools-gprof | automated | | Medium | Yes | Yes | low | low | |
        | sdk-performance-tools-sysprof | automated | | Medium | Yes | Yes | low | low | |
        | sdk-persistent-disk-etc | manual | 15 | Critical | | Yes | high | high | | Remove if automated by [T7325](https://phabricator.apertis.org/T7325)
        | sdk-persistent-disk-home-user | manual | 15 | Critical | | Yes | high | high | | Remove if automated by [T7325](https://phabricator.apertis.org/T7325)
        | sdk-persistent-disk-sysroot | manual | 15 | Critical | | Yes | high | high | | Remove if automated by [T7325](https://phabricator.apertis.org/T7325)
        | sdk-vb-fullscreen | manual | 05 | Critical | | Yes | high | high | |
        | secure-boot-imx6 | manual | 20 | Medium | | | high | high | |
        | tiny-container-system-aa-enforcement | automated | | Medium | | Yes | high | low | |
        | tiny-container-system-basic | automated | | Medium | | Yes | high | low | |
        | tiny-container-system-connectivity-profile | automated | | Medium | | Yes | high | low | |
        | tiny-container-user-device-sharing | automated | | Medium | | Yes | high | low | |
        | tiny-container-user-folder-sharing | automated | | Medium | | Yes | high | low | |
        | tiny-container-user-seccomp | automated | | Medium | | Yes | high | low | |
        | traprain | automated | | Medium | Yes | Yes | high | low | |
        | tumbler-thumbnailing | automated | | Medium | | Yes | high | low | |
        | video-animation-on-boot | manual | 03 | High | | | low | high | Update | Expecting old animation - update
        | webkit2gtk-ac-3d-rendering | manual | 05 | Low | | Yes | high | high | | Test excercises a big chunk of stack, so keep
        | webkit2gtk-ac-animations | manual | 05 | Medium | | Yes | low | high | Remove | No longer working on webkit
        | webkit2gtk-ac-scrolling | manual | 05 | Medium | | Yes | low | high | Remove | No longer working on webkit
        | webkit2gtk-drag-and-drop | manual | 05 | Low | | Yes | low | high | Remove | No longer working on webkit
        | webkit2gtk-event-handling-redesign | manual | 05 | Medium | | Yes | low | high | Remove | No longer working on webkit
        | webkit2gtk-gstreamer1.0 | manual | 05 | Medium | | Yes | high | high | | Test excercises a big chunk of stack, so keep
        | webkit2gtk-mt-touch-events | manual | 05 | Low | | Yes | high | high | | Test excercises a big chunk of stack, so keep
        | x-hw-accelerated | manual | 05 | Medium | Yes | | low | high | | Automate and remove if [T7325](https://phabricator.apertis.org/T7325) proves viable
        # General status of the test framework
        Whilst reviewing the test cases a number of additional A number of additional
        points have been noted that are mostly out of scope for this review, though may
        be worth concidering for future work:
        - **A lot of the tests seem to have needless levels of abstraction:** A number
        of strategies have been employed over successive attempts to automate more of
        the test cases. In order to avoid extra overhead, generally infrastructure
        used to perform the previously automated tests has been kept and driven from
        the latest mechanism. In parallel to this, work has been done to provide a
        testing framework that makes steps carried out by the tests better visibly
        documented (leading to the test descriptions on
        [qa.apertis.org](https://qa.apertis.org). Due to may of these tests calling
        previous mechanisms, these tests are not as transparent as they could be and
        the (in some cases, multiple) layers of abstraction make them harder to read
        and maintain. It is suggested that efforts are taken to remove these layers
        of abstraction where possible.
        - **It should be possible to merge some of the AppArmor tests with their
        non-AppArmor counterparts:** Quite a few of the AppArmor tests run up the
        same image and perform the same steps as the non-AppArmor test for the same
        functionality, only differing in what they look at to deem the test a success
        or failure. This adds overhead as the infrastructure is setting up and
        performing the tests twice. This can be reduced by checking for both the
        outcome of the test steps and the AppArmor status in a single test run
        without loss of testing coverage.
        - **Most tests should not be installing packages via `apt`:** A conciderable
        number of the tests are installing packages via `apt` to for use in the
        testing. These additions are fundamentally changing the image under test and
        may affect the results. We have had a
        [mechanism to combat this](https://www.apertis.org/qa/immutablerootfstests/)
        for a while which needs to be universally rolled out.
        - **We should update tests using `git clone` to use `wget`:** Rather than
        requiring git to be install on the test image, we have recently been
        downloading archives of the test harnesses with are provided by GitLab. This
        only requires the ability to download a resource via HTTP, which can be
        reasonably assumed to be possible in all the images rather than the ability
        to perform a git clone. Tests should be updated to use this mechanism to
        avoid the need for git.
        - **Merge some tests together:** In some places we seem to have multiple tests,
        each testing a facet of a larger set of functionality. Whilst not viable or
        advisable for all the tests, merging some may be viable which would reduce
        the overhead of setting up the environment individually each time.
        0% or .
        You are about to add 0 people to the discussion. Proceed with caution.
        Finish editing this message first!
        Please register or to comment
        Apertis Website Terms of Use Privacy Policy