Skip to content
Snippets Groups Projects
Commit 9c703f20 authored by Samir Kumar Sarangi's avatar Samir Kumar Sarangi Committed by Emanuele Aina
Browse files

Remove redundant AUM testcases


The manual testcases below have an automated counterpart, so let's drop
them as they are redundant:

* apertis-update-manager-api
* apertis-update-manager-diskfull
* apertis-update-manager-ota-auto
* apertis-update-manager-ota-diskfull
* apertis-update-manager

Apertis: T7104

Signed-off-by: default avatarSamir <external.SamirKumar.Sarangi@in.bosch.com>
parent 3afc472c
No related branches found
No related tags found
No related merge requests found
metadata:
name: apertis-update-manager-api
format: "Apertis Test Definition 1.0"
image-types:
minimal: [ armhf ]
image-deployment:
- OSTree
type: functional
exec-type: manual
priority: low
maintainer: "Apertis Project"
description: "Test the apertis-update-manager automatic update.
The automated version of this test: https://qa.apertis.org/aum-api.html"
resources:
- "A static encrypted update bundle file of the same architecture, variant and version as the testing image"
- "A Fat32 USB flash drive, preloaded with the encrypted update bundle at the root of the disk"
- "The latest static encrypted update file can be downloaded at the same location than the target image. It has the same basename, and a '.delta.enc' extension"
- "The static encrypted update file should be copied to the flash drive using the name 'static-update.bundle.enc'."
- "A PC must be connected to DUT serial port"
expected:
- "CLI utility `updatectl` is able to interact with Apertis update manager"
- "The update was properly applied"
run:
steps:
- "Check the initial deployment"
- $ sudo ostree admin status
- "Prepare the copy of commit and deploy to allow the upgrade to the same version"
- "Command below shows you an initial commit ID, for instance"
- |
$ export BOOTID=$(sudo ostree admin status | sed -n -e 's/^\* apertis \([0-9a-f]*\)\.[0-9]$/\1/p'); echo $BOOTID
- "Get the Collection ID and ref"
- $ export CID=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 1 -d ' '); echo COLLECTION_ID=$CID
- $ export REF=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 2 -d ' '); echo REF=$REF
- "Create the list of files to skip and enshure there are some files in these directories"
- $ ls -1d /usr/share/locale /usr/share/man /usr/share/zoneinfo > /tmp/skip
- $ du -sh /usr/share/locale /usr/share/man /usr/share/zoneinfo
- "Create the commit with changed timestamp and skipped list from above to allow upgrade with recent update file"
- |
$ export NEWID=$(sudo ostree commit --orphan --tree=ref=$BOOTID --add-metadata-string=ostree.collection-binding=$CID --bind-ref=$REF --timestamp="1 year ago" --skip-list=/tmp/skip); echo "New commit: $NEWID"
- "Deploy the prepared commit"
- $ sudo ostree admin upgrade --allow-downgrade --deploy-only --override-commit=$NEWID --reboot
- "Wait until the system is booted again and check the deployment"
- $ sudo ostree admin status
- "The booted commit (started with '*') must have ID which we prepare and the initial commit ID should be marked as '(rollback)'"
- "Check booted deployment have no file objects which we skip"
- $ du -sh /usr/share/locale /usr/share/man /usr/share/zoneinfo
- "Remove the initial deployment"
- $ sudo ostree admin undeploy 1
- "Reboot the system"
- "Check the current deployment"
- $ sudo ostree admin status
- "Start the user interface agent with mode preventing automatic system reboot after update"
- $ sudo updatectl --register-upgrade-handler &
- "Plug the USB flash drive into the device"
- "The update starts automatically"
- "After the update, the device does *not* reboot automatically"
- "Check that the user interface agent reports the pending update"
- |
>** Message: Upgrade status: Checking
** Message: An upgrade is pending
- "Remove the USB flash drive"
- "Check if there is pending deployment and reboot the DUT"
- $ sudo ostree admin status
- $ sudo reboot
- "Check the current deployment has been updated and that the rollback entry points to the prepared deployment"
- $ sudo ostree admin status
- "Start the user interface agent"
- $ sudo updatectl &
- "Plug the USB flash drive with the same update file into the device"
- "Check that the user interface agent reports the system is up to update"
- |
>** Message: Upgrade status: Checking
** Message: System is up to date
metadata:
name: apertis-update-manager-diskfull
format: "Apertis Test Definition 1.0"
image-types:
minimal: [ armhf ]
image-deployment:
- OSTree
type: functional
exec-type: manual
priority: low
maintainer: "Apertis Project"
description: "Ensure that failures due to disk full errors during updates have no effect on the system
The automated version of this test: https://qa.apertis.org/aum-out-of-space.html"
resources:
- "A static encrypted update bundle file of the same architecture, variant and version as the testing image"
- "A Fat32 USB flash drive, preloaded with the encrypted update bundle at the root of the disk"
- "The latest static encrypted update file can be downloaded at the same location than the target image. It has the same basename, and a '.delta.enc' extension"
- "The static encrypted update file should be copied to the flash drive using the name 'static-update.bundle.enc'."
- "A PC must be connected to DUT serial port"
expected:
- "The update wasn't applied"
- "System boots using the initial deployment"
run:
steps:
- "Check the initial deployment"
- $ sudo ostree admin status
- "Prepare the copy of commit and deploy to allow the upgrade to the same version"
- "Command below shows you an initial commit ID, for instance"
- |
$ export BOOTID=$(sudo ostree admin status | sed -n -e 's/^\* apertis \([0-9a-f]*\)\.[0-9]$/\1/p'); echo $BOOTID
- "Get the Collection ID and ref"
- $ export CID=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 1 -d ' '); echo COLLECTION_ID=$CID
- $ export REF=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 2 -d ' '); echo REF=$REF
- "Create the list of files to skip and enshure there are some files in these directories"
- $ ls -1d /usr/share/locale /usr/share/man /usr/share/zoneinfo > /tmp/skip
- $ du -sh /usr/share/locale /usr/share/man /usr/share/zoneinfo
- "Create the commit with changed timestamp and skipped list from above to allow upgrade with recent update file"
- |
$ export NEWID=$(sudo ostree commit --orphan --tree=ref=$BOOTID --add-metadata-string=ostree.collection-binding=$CID --bind-ref=$REF --timestamp="1 year ago" --skip-list=/tmp/skip); echo "New commit: $NEWID"
- "Deploy the prepared commit"
- $ sudo ostree admin upgrade --allow-downgrade --deploy-only --override-commit=$NEWID --reboot
- "Wait until the system is booted again and check the deployment"
- $ sudo ostree admin status
- "The booted commit (started with '*') must have ID which we prepare and the initial commit ID should be marked as '(rollback)'"
- "Check booted deployment have no file objects which we skip"
- $ du -sh /usr/share/locale /usr/share/man /usr/share/zoneinfo
- "Remove the initial deployment"
- $ sudo ostree admin undeploy 1
- "Reboot the system"
- "Check the current deployment"
- $ sudo ostree admin status
- "Need to monitor the journal log"
- $ sudo journalctl -ef --unit apertis-update-manager &
- "Check the free space on rootfs, calculate the size of file to leave insufficient free space for upgrade to happen
(this prevents the unexpected reboot during the test caused by other services).
For instance, in the case below, NUM value would be 1451. It is the result of the calculation 1823-348-24,
there 24 is a magic constant, so that almost, but not all free space is used.
The free space can be calculated with command line `sudo df -B 1M / | grep / | awk '{print $2-$3-24}'`"
- $ sudo df -B 1M /
- |
>Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mmcblk0p2 1823 348 1365 21% /
- $ sudo dd if=/dev/zero of=/var/bigfile bs=1M count=<NUM> ; echo "dd completed!"
- Wait until 'dd' completes (it should take time), you will see the output similar to
- |
>1521483776 bytes (1.5 GB) copied, 82.1661 s, 18.5 MB/s
dd completed!
- "Plug the USB flash drive in the device"
- "The update should fail with error message"
- |
>Dec 19 22:53:33 apertis apertis-update-[348]: Ostree upgrade failed: min-free-space-percent '3%' would be exceeded, 16.8 MB available
- "Remove the USB flash drive"
- Set reserved space for ostree to '0'
- $ sudo ostree config set core.min-free-space-percent "0"
- Restart the Apertis Update Manager
- $ sudo systemctl restart apertis-update-manager
- "Plug the USB flash drive in the device"
- "The update should fail with some error message related to disk space, for instance:"
- |
>Dec 19 22:55:10 apertis apertis-update-[319]: Ostree upgrade failed: mkdir(boot/loader.0/entries): Input/output error
Dec 19 22:43:46 apertis apertis-update-[794]: Ostree upgrade failed: Error writing to file descriptor: No space left on device
- "Remove the USB flash drive"
- "Remove temporary file"
- $ sudo rm -f /var/bigfile
- "Reboot the system"
- $ sudo reboot
- "Check that the status is the same than at the beginning of the test"
- $ sudo ostree admin status
metadata:
name: apertis-update-manager-ota-auto
format: "Apertis Test Definition 1.0"
image-types:
minimal: [ armhf ]
image-deployment:
- OSTree
type: functional
exec-type: manual
priority: medium
maintainer: "Apertis Project"
description: "Test the apertis-update-manager automatic updates over the air."
pre-conditions:
- >
This test requires a properly configured time source: when testing
devices not carrying a battery-backed real time clock in a network which
prevents connections to NTP servers (and only in that case) manually
ensure that the time on the device is set appropriately and that it is
propagated to the hardware clock so it is stay set on the next reboot if
the power is not plugged off.
- $ sudo timedatectl --adjust-system-clock set-time "2019-08-21 18:49:03" # use the appropriate date
- >
If direct access to repository with updates for DUT is restricted and proxy server should be used,
then need to add the address of this proxy for OSTree on DUT by command:
- $ sudo ostree config set 'remote "origin"'.proxy "http://10.168.128.45:3128"
resources:
- "The DUT u-boot environment must be clean: in u-boot, run: `env default -a` followed by `saveenv`"
- "A PC must be connected to DUT serial port"
- "The DUT must be connected to network"
expected:
- "The OTA update was properly applied"
run:
steps:
- "Check the initial deployment"
- $ sudo ostree admin status
- "Prepare the copy of commit and deploy to allow the upgrade to the same version"
- "Command below shows you an initial commit ID, for instance"
- |
$ export BOOTID=$(sudo ostree admin status | sed -n -e 's/^\* apertis \([0-9a-f]*\)\.[0-9]$/\1/p'); echo $BOOTID
- "Get the Collection ID and ref"
- $ export CID=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 1 -d ' '); echo COLLECTION_ID=$CID
- $ export REF=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 2 -d ' '); echo REF=$REF
- "Create the commit with changed timestamp to allow upgrade with recent update file"
- |
$ export NEWID=$(sudo ostree commit --orphan --tree=ref=$BOOTID --add-metadata-string=ostree.collection-binding=$CID --bind-ref=$REF --timestamp="1 year ago"); echo "New commit: $NEWID"
- "Deploy the prepared commit"
- $ sudo ostree admin upgrade --allow-downgrade --deploy-only --override-commit=$NEWID --reboot
- "Wait until the system is booted again and check the deployment"
- $ sudo ostree admin status
- "The booted commit (started with '*') must have ID which we prepare and the initial commit ID should be marked as '(rollback)'"
- "Remove the initial deployment"
- $ sudo ostree admin undeploy 1
- "Check the current deployment"
- $ sudo ostree admin status
- "Enable automatic OTA updates in configuration file '/etc/apertis-update-manager.ini'"
- $ sudo sed -i -e "s/^UpdatesFromNetwork=.*/UpdatesFromNetwork=true/" /etc/apertis-update-manager.ini
- "Configuration file '/etc/apertis-update-manager.ini' must have property 'UpdatesFromNetwork' set to true"
- $ sudo grep UpdatesFromNetwork /etc/apertis-update-manager.ini
- "Remove blacklist file if it exists"
- $ sudo rm -f /var/aum_blacklist.conf
- "Reboot the system"
- $ sudo reboot
- "Wait while the system booted successfully. Need to wait a short time (30-120 sec) depending on your network. The update must be downloaded and the system should be rebooted second time automatically.
The current status of upgrade could be checked with journalctl log -- it should not contain error messages about the upgrade. The listing below is just for example:"
- $ sudo journalctl -ef --unit apertis-update-manager
- |
>Jun 01 11:54:26 apertis apertis-update-[355]: Ostree upgrade progress: 1 metadata, 0 content objects fetched
Jun 01 11:54:26 apertis apertis-update-[355]: Cannot check the ID in black list: No such file or directory
Jun 01 11:57:10 apertis apertis-update-managerd[355]: libostree pull from 'origin' for apertis/v2019pre/amd64-uefi/minimal complete
security: GPG: disabled http: TLS
non-delta: meta: 23 content: 39
transfer: secs: 163 size: 32.2 MB
Jun 01 11:57:10 apertis apertis-update-[355]: New upgrade downloaded! Deploying..
Jun 01 11:57:10 apertis apertis-update-managerd[355]: Copying /etc changes: 13 modified, 0 removed, 1 added
Jun 01 11:57:10 apertis apertis-update-[355]: Couldn't unlink ostree-0-1.conf
Jun 01 11:57:11 apertis apertis-update-managerd[355]: Transaction complete; bootconfig swap: yes; deployment count change: 1
Jun 01 11:57:11 apertis apertis-update-[355]: Ostree upgrade ready, system should be rebooted
Jun 01 11:57:11 apertis apertis-update-[355]: Rebooting to apply pending update
- "Check the current deployment has been updated and that the rollback entry points to the prepared deployment"
- $ sudo ostree admin status
- "Check the journal log should mention that the system check the update availability and report it is already up to date"
- $ sudo journalctl -ef --unit apertis-update-manager
- |
>Jun 01 22:35:32 apertis apertis-update-[365]: Ostree upgrade poll starting
Jun 01 22:35:34 apertis apertis-update-[365]: Ostree upgrade progress: 1 metadata, 0 content objects fetched
Jun 01 22:35:34 apertis apertis-update-[365]: Ostree already up to date
metadata:
name: apertis-update-manager-ota-diskfull
format: "Apertis Test Definition 1.0"
image-types:
minimal: [ armhf ]
image-deployment:
- OSTree
type: functional
exec-type: manual
priority: medium
maintainer: "Apertis Project"
description: "Ensure that failures due to disk full errors during network updates have no effect on the system"
pre-conditions:
- >
This test requires a properly configured time source: when testing
devices not carrying a battery-backed real time clock in a network which
prevents connections to NTP servers (and only in that case) manually
ensure that the time on the device is set appropriately and that it is
propagated to the hardware clock so it is stay set on the next reboot if
the power is not plugged off.
- $ sudo timedatectl --adjust-system-clock set-time "2019-08-21 18:49:03" # use the appropriate date
- >
If direct access to repository with updates for DUT is restricted and proxy server should be used,
then need to add the address of this proxy for OSTree on DUT by command:
- $ sudo ostree config set 'remote "origin"'.proxy "http://10.168.128.45:3128"
resources:
- "The DUT u-boot environment must be clean: in u-boot, run: `env default -a` followed by `saveenv`"
- "A PC must be connected to DUT serial port"
- "The DUT must be connected to network"
expected:
- "The update wasn't applied"
- "System boots using the initial deployment"
run:
steps:
- "Check the initial deployment"
- $ sudo ostree admin status
- "Prepare the copy of commit and deploy to allow the upgrade to the same version"
- "Command below shows you an initial commit ID, for instance"
- $ export BOOTID=$(sudo ostree admin status | sed -n -e 's/^\* apertis \([0-9a-f]*\)\.[0-9]$/\1/p'); echo $BOOTID
- "Get the Collection ID and ref"
- $ export CID=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 1 -d ' '); echo COLLECTION_ID=$CID
- $ export REF=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 2 -d ' '); echo REF=$REF
- "Create the list of files to skip and enshure there are some files in these directories"
- $ ls -1d /usr/share/locale /usr/share/man /usr/share/zoneinfo > /tmp/skip
- $ du -sh /usr/share/locale /usr/share/man /usr/share/zoneinfo
- "Create the commit with changed timestamp and skipped list from above to allow upgrade with recent update file"
- |
$ export NEWID=$(sudo ostree commit --orphan --tree=ref=$BOOTID --add-metadata-string=ostree.collection-binding=$CID --bind-ref=$REF --timestamp="1 year ago" --skip-list=/tmp/skip); echo "New commit: $NEWID"
- "Deploy the prepared commit"
- $ sudo ostree admin upgrade --allow-downgrade --deploy-only --override-commit=$NEWID --reboot
- "Wait until the system is booted again and check the deployment"
- $ sudo ostree admin status
- "The booted commit (started with '*') must have ID which we prepare and the initial commit ID should be marked as '(rollback)'"
- "Check booted deployment have no file objects which we skip"
- $ du -sh /usr/share/locale /usr/share/man /usr/share/zoneinfo
- "Remove the initial deployment"
- $ sudo ostree admin undeploy 1
- "Reboot the system"
- "Check the current deployment"
- $ sudo ostree admin status
- "Need to monitor the journal log"
- $ sudo journalctl -ef --unit apertis-update-manager &
- "Check the free space on rootfs, calculate the size of file to leave insufficient free space for upgrade to happen
(this prevents the unexpected reboot during the test caused by other services).
For instance, in the case below, NUM value would be 1451. It is the result of the calculation 1823-348-24,
there 24 is a magic constant, so that almost, but not all free space is used.
The free space can be calculated with command line `sudo df -B 1M / | grep / | awk '{print $2-$3-24}'`"
- $ sudo df -B 1M /
- |
>Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mmcblk0p2 1823 348 1365 21% /
- $ sudo dd if=/dev/zero of=/var/bigfile bs=1M count=<NUM> ; echo "dd completed!"
- Wait until 'dd' completes (it should take time), you will see the output similar to
- |
>1521483776 bytes (1.5 GB) copied, 82.1661 s, 18.5 MB/s
dd completed!
- "Enable network updates with CLI tool"
- $ sudo updatectl --check-network-updates
- "The update should fail with error message"
- |
>** Message: 20:59:57.249: Network connected: Yes
** Message: 20:59:57.251: Auto update status: active
** Message: 20:59:57.251: Ostree upgrade poll starting
** Message: 20:59:58.697: Ostree upgrade failed: min-free-space-percent '3%' would be exceeded
- "Set reserved space for ostree to '0'"
- $ sudo ostree config set core.min-free-space-percent "0"
- "Restart the Apertis Update Manager"
- $ sudo systemctl restart apertis-update-manager
- "Enable network updates with CLI tool"
- $ sudo updatectl --check-network-updates
- "The update should fail with some error message related to disk space, for instance:"
- |
>** Message: 21:02:31.972: Network connected: Yes
** Message: 21:02:31.974: Auto update status: active
** Message: 21:02:31.974: Ostree upgrade poll starting
** Message: 21:02:33.800: Ostree upgrade progress: 2 metadata, 0 content objects fetched; 284 B transferred in 1 seconds
** (process:532): WARNING **: 21:02:33.801: Cannot check the ID in black list: No such file or directory
** Message: 21:02:41.054: Ostree upgrade failed: Error writing to file descriptor: No space left on device
- "Remove temporary file"
- $ sudo rm -f /var/bigfile
- "Reboot the system"
- $ sudo reboot
- "Check that there is only one deployment"
- $ sudo ostree admin status
metadata:
name: apertis-update-manager
format: "Apertis Test Definition 1.0"
image-types:
minimal: [ armhf ]
image-deployment:
- OSTree
type: functional
exec-type: manual
priority: low
maintainer: "Apertis Project"
description: "Test the manual update with apertis-update-manager.
The automated version of this test: https://qa.apertis.org/aum-offline-upgrade.html"
resources:
- "A PC must be connected to DUT serial port"
- "A static update bundle file of the same architecture, variant and version as the testing image"
- "The latest static update file can be downloaded at the same location than the target image. It has the same basename, and a '.delta' extension"
expected:
- "The update was properly applied"
run:
steps:
- "Check the initial deployment"
- $ sudo ostree admin status
- "Prepare the copy of commit and deploy to allow the upgrade to the same version"
- "Command below shows you an initial commit ID, for instance"
- |
$ export BOOTID=$(sudo ostree admin status | sed -n -e 's/^\* apertis \([0-9a-f]*\)\.[0-9]$/\1/p'); echo $BOOTID
- "Get the Collection ID and ref"
- $ export CID=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 1 -d ' '); echo COLLECTION_ID=$CID
- $ export REF=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 2 -d ' '); echo REF=$REF
- "Create the list of files to skip and enshure there are some files in these directories"
- $ ls -1d /usr/share/locale /usr/share/man /usr/share/zoneinfo > /tmp/skip
- $ du -sh /usr/share/locale /usr/share/man /usr/share/zoneinfo
- "Create the commit with changed timestamp and skipped list from above to allow upgrade with recent update file"
- |
$ export NEWID=$(sudo ostree commit --orphan --tree=ref=$BOOTID --add-metadata-string=ostree.collection-binding=$CID --bind-ref=$REF --timestamp="1 year ago" --skip-list=/tmp/skip); echo "New commit: $NEWID"
- "Deploy the prepared commit"
- $ sudo ostree admin upgrade --allow-downgrade --deploy-only --override-commit=$NEWID --reboot
- "Wait until the system is booted again and check the deployment"
- $ sudo ostree admin status
- "The booted commit (started with '*') must have ID which we prepare and the initial commit ID should be marked as '(rollback)'"
- "Check booted deployment have no file objects which we skip"
- $ du -sh /usr/share/locale /usr/share/man /usr/share/zoneinfo
- "Remove the initial deployment"
- $ sudo ostree admin undeploy 1
- "Reboot the system"
- "Check the current deployment"
- $ sudo ostree admin status
- "Remove blacklist file if it exists"
- $ sudo rm -f /var/aum_blacklist.conf
- "From the host, copy the static delta update to the target using scp"
- $ scp /path/to/static/delta user@target:update.bundle
- "The update does not start automatically, start it manually"
- $ sudo updatectl --apply-static-delta /home/user/update.bundle
- "After the update, the device will reboot automatically"
- "Check the current deployment has been updated and that the rollback entry points to the prepared deployment"
- $ sudo ostree admin status
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment