Further updates to the test case review

Following discussions, tweak some of the recommendations from the test
case review.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

content/concepts/test_case_evaluation.md

@@ -31,9 +31,9 @@ analysis performed for this review along with any suggestions for change:
 | apertis-update-manager-ota-auto 		| manual        | 10				| Medium	| 		|  		| low		| high		| Remove	| Automated as aum-ota-auto
 | apertis-update-manager-ota-diskfull 		| manual        | 10				| Medium	| 		|  		| low		| high		| Remove	| Automated as aum-ota-out-of-space
 | apertis-update-manager-ota-rollback 		| manual        | 10				| Medium	| 		| 		| high		| high		|  		| 
-| apertis-update-manager-ota-signed 		| manual        | 10				| Medium	| 		|  		| low		| high		| Remove	| Automated as aum-ota-signed
+| apertis-update-manager-ota-signed 		| manual        | 10				| Medium	| 		|  		| low		| high		| 		| Automated as aum-ota-signed. Keep whilst rolling out update management over all supported platforms
 | apertis-update-manager-powercut 		| manual        | 10				| Low		| 		|  		| high		| high		| 		| Although automated as aum-power-cut, that only simulates a power cut, so this test is still good confirmation
-| apertis-update-manager-rollback 		| manual        | 12				| Low		| 		| 		| low		| high		| Remove	| Automated as aum-rollback-blacklist
+| apertis-update-manager-rollback 		| manual        | 12				| Low		| 		| 		| low		| high		| 		| Automated as aum-rollback-blacklist. Keep whilst rolling out update management over all supported platforms
 | apertis-update-manager-usb-unplug 		| manual        | 12				| Medium	| 		|  		| high		| high		|  		| 
 | apertis-update-manager			| manual	| 10				| Low		| 		|  		| low		| high		| Remove	| Automated as aum-offline-upgrade
 | apparmor-basic-profiles			| automated     |				| Critical	| 		| Yes		| high		| low		|  		| Basic security functionality (Still using git - needs updating)
@@ -64,13 +64,13 @@ analysis performed for this review along with any suggestions for change:
 | aum-power-cut					| automated     |				| Critical	| 		| Yes		| high		| low		|  		| 
 | aum-rollback-blacklist			| automated     |				| Critical	| 		| Yes		| high		| low		|  		| 
 | bluez-avrcp-volume 				| manual        | 08				| Medium	| 		|  		| low		| high		| Merge 	| We have to do all this in `apparmor-bluez-avrcp-volume`. Merge the tests.
-| bluez-hfp 					| manual        | 08				| Medium	| 		| 		| low		| high		|  		| Tests hands free profile
-| bluez-phone 					| manual        | 08				| Critical	| 		|  		| high		| high		| Merge		| Tests lots of bluetooth functionality, but can this not be merged with `apparmor-bluez`
+| bluez-hfp 					| manual        | 08				| Medium	| 		| 		| low		| high		| Merge		| Tests hands free profile, merge with `bluez-phone`.
+| bluez-phone 					| manual        | 08				| Critical	| 		|  		| high		| high		| 		| Tests lots of bluetooth functionality paired with a phone.
 | bluez-setup 					| manual        | 08				| Critical	| 		|  		| low		| high		| Merge		| We have to do all this in `apparmor-bluez-setup`. Merge the tests.
 | boot-no-crashes				| automated     |				| High		| 		| Yes		| high		| low		|   		| Checks journal for auditd messages suggesting app crashes
 | boot-performance				| automated     |				| High		| 		| Yes		| high		| low		|   		| Dump boot performance (no fail state, but useful information)
 | canterbury					| automated     |				| Critical	| 		| Yes		| low		| low		|   		| 
-| cgroups-resource-control			| automated     |				| Medium	| 		| Yes		| low		| medium	| Remove	| There is a low chance this will catch bugs. The test is broken due to the reliance on a package no longer shipped by Debian and would need to be ported to something else
+| cgroups-resource-control			| automated     |				| Medium	| 		| Yes		| low		| medium	| Remove	| Cgroups are now a widely used part of the Linux infrastructure and not the new technology it was when this test was added. This cursory testing is unlikely to catch any real bugs. The test is broken due to the reliance on a package no longer shipped by Debian and would need to be ported to something else.
 | connman-new-supplicant			| automated     |				| Medium	| 		| 		| high		| low		| Merge		| Just checks that wpa_supplicant is on dbus (merge with connman test)
 | connman-pan-network-access 			| manual        | 08				| Medium	| Yes		|  		| high		| high		|   		| 
 | connman-pan-tethering 			| manual        | 08				| Critical	| Yes		|  		| high		| high		|   		| 
@@ -81,10 +81,10 @@ analysis performed for this review along with any suggestions for change:
 | dbus-installed-tests				| automated     |				| High		| Yes		| Yes		| high		| low		|   		| Running dbus' own regression tests
 | didcot					| automated     |				| Medium	| Yes		| Yes		| low		| low		|   		| 
 | disk-rootfs-fsck				| automated     |				| High		| 		| Yes		| high		| low		|   		| 
-| eclipse-plugins-install-target 		| manual        | 10				| Medium	| 		| 		| low		| high		| Remove	| 
-| eclipse-plugins-remote-debugging 		| manual        | 10				| Medium	| 		| 		| low		| high		| Remove	| 
-| eclipse-sysroot-management 			| manual        | 10				| Medium	| Yes		| 		| low		| high		| Remove	| 
-| evolution-sync-bluetooth 			| manual        | 05				| Medium	| Yes		| 		| low		| high		| Fix		| Needed to verify phone book syncing
+| eclipse-plugins-install-target 		| manual        | 10				| Medium	| 		| 		| low		| high		| Remove	| Eclipse is no longer in Debian and survives in Apertis as a quite stale package which is likely to break in the next rebase. These tests are broken as the custom integrations broke on the last rebase. It is unlikely that Eclipse will stay in Apertis much longer and even less likely that the plugins will be fixed, so remove the tests.
+| eclipse-plugins-remote-debugging 		| manual        | 10				| Medium	| 		| 		| low		| high		| Remove	| As eclipse-plugins-install-target
+| eclipse-sysroot-management 			| manual        | 10				| Medium	| Yes		| 		| low		| high		| Remove	| As eclipse-plugins-install-target
+| evolution-sync-bluetooth 			| manual        | 05				| Medium	| Yes		| 		| low		| high		| Remove	| The technologies behind this functionality have become unsupported over time. Implementing such functionality now may require a substantially different approach and thus a different testing strategy.
 | frome						| automated     |				| Low		| Yes		| Yes		| low		| low		| 		| 
 | gettext-i18n					| automated     |				| Medium	| 		| Yes		| low		| low		| Fix		| Test broken, appears to have been for a while, needs fixing.
 | glib-gio-fs					| automated     |				| Medium	| 		| Yes		| high		| low		|   		| Test basic GIO functionality