Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Frederic Danis authored 3 years ago and Emanuele Aina committed 3 years ago

OSTree branch installed on RPi 4b boards use
`origin:apertis/v2023dev1/arm64-rpi64/fixedfunction` instead of
`origin:apertis/v2023dev1/arm64-uboot/fixedfunction` . The later is generic
for R-Car and RPi 4b boards.

This commit adds a function to return the expected branch, and adapt it
for the RPi64.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Add boot-delay-install.sh to install the boot-delay systemd service

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Rollback tests need to reset the DUT before boot is completed.
This commit add a service delaying start of AUM and user logging, and
displaying a message allowing a tester to reset the board.
This service should be run before systemd-bless-boot.service which rename
the EFI loader entry discarding the bootcount.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This reverts commit 8c71fb5c.

Filesystem corruption during image creation has been fixed in
pkg/user-mode-linux

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Workaround: manually unmount the boot partition before the board is reset
by Lava to prevent unexpected entries in /boot/lost+found which can
prevent subsequent boots.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Since addition of EFI support, the bootcounter variables are shared between
the different bootloader check parts.
Variables set using read function in pipe subshell are not available
outside of this subshell, and so unset for the rest of the function.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

aum-rollback-blacklist test fail on UEFI AMD64 targets with log:
  + check_bootcounter   00 01
  + + read -r m ver cnt uflag
  od -t x1 -An /boot/uboot.cnt
  od: '/boot/uboot.cnt': No such file or directory (os error 2)

For EFI, the bootcounter is stored in the loader entry filename, i.e.
/boot/efi/loader/entries/ostree-0-2[+bootleft[-bootcount]].conf.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

AUM api test failed due to `No space left on device` error when copying
HMI encrypted bundle to `/var/image_file`.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

As described in 15b4dc92 rust-coreutils df does not support block size,
so switch to busybox df.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Since rust-coreutils df lacks of several features like block size switch
to busybox df.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Since rust-coreutils does not not provide dd yet, use busybox dd.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Apertis now uses rust-coreutils, which is not fully compatible with
GNU coreutils. Fix the usage of tail according to rust-coreutils
requirements.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

`uutils/coreutils` lacks dd.
We can replace that call with `busybox dd`

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

In case if bootloader behave incorrectly and ignore the upgrade state
it can trigger rollback even after successful upgrade.
With working `apertis-update-complete` service this state is masked
since AUM rewrite the bootcounter file on every successful boot.

However in case if the system is interrupted 3 times in a raw before
re-marking update as "successful", the bootloader may start the rollback
procedure. For instance in case of sequential power failure or user's
interruption.

Added the test emulating failure of the boot process prior starting the
mark service after successful upgrade by failing this service. Check if
no rollback happens after several boots in a raw.

See: https://phabricator.apertis.org/T7239



Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Test the boot counter state to ensure it has upgrade
status set during upgrade and unset after successful upgrade.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

AUM crash is detected as unexpected AUM restart. However due
races on startup the cursor for logs check may catch the expected
restart and treat it as unexpected.
Reset the log cursor to check events occurred during update only.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

If we expect the failure of `apply_update_sync` we should treat
only "1" return status as expected value.
Return status "99" is reserved on unexpected failure of AUM itself and
must indicate the failing test.
This helps to avoid masking of AUM internal problems.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

If AUM is crashed by any reason (SIGSEGV for example) the function
`apply_update_sync` don't detect that behavior and exit with error code
after timeout (~8 min). This is leading to false positive result if we
are expecting the update failure.

Added the check of AUM restart by systemd and return code `99` for that case.
Different error code allow to determine if the upgrade is failed as
expected or we have bug in AUM code. Reduces the test time in case of
AUM crash.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Function to reset the boot counter
Supporting old and new formats
Accept 2 arguments:
 bootcounter: number, 0 if skipped
 upgrade flag (could be omitted):
   1: set upgrade state (default)
   0: disable upgrade state

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

New `uboot.cnt` file format appears in u-boot 2019.01+dfsg-7co7.
Allow to read legacy format.
If "magic" and "version" are not requested explicitly the function
accept any supported `uboot.cnt` format.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Fix the check of the uboot boot counter and the test logic.
Added some useful debug output.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Function to validate values from /boot/uboot.cnt

accept 4 arguments:
  magic: bd
  version: 1
  bootcount: any number
  upgrade bit:
    0: not in upgrade
    1: in upgrade

In case if any argument is not defined i.e. passed empty value ""
it is meaning no need to validate that argument

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Commit 8f826964 contain the fix of ostree commit ID parsing.
Before that commit the ID contained additional suffix (".0") and
thus the comparison with `COMMIT_AFTER` always shows a difference
and the test result was __always__ false positive: "pass".

After applying the 8f826964 commit we shed light to other issue --
we saved a wrong ostree commit ID for later usage.
In case if we prepare the "outdated" commit -- we must use it's ID
as "starting" for the test.

Moved the preparation of "outdated" ostree commit into the common
function `prepare_outdated_commit()` and store the proper ostree
commit ID as starting point.

Additional side-effect fixed for OTA upgrades -- before current fix
OTA tests were unstable, sometimes allowing to pass the check even
with wrong ID written as starting point.
This is caused by long queues in LAVA nowadays -- the build system may
prepare and submit the new version while the test is waiting in queue.
In that case the system do the real upgrade to the OS version newer
than we have during the test submit.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

All scripts are started by root user hence the `sudo` call is not
needed. It comes from manual test cases by copy-paste method.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Removed unneeded `sudo` call from the library.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Added automatic test allowing to check the offline upgrade from
different branch (previous release in case of Apertis).

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Function `enable_aum_debug` enable debug messages and restart AUM.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

There were an arror in commit ID parsing from the output of:
  $ ostree admin status
  * apertis 334f57354221f0800b008d043c5c50683e83e8b6afa5880e95449a980132fa83.0
      origin refspec: origin:apertis/v2021pre/armhf-uboot/minimal
    apertis bab35cb1fe04e2fc11c3b462ebac0e38640173ebcff55ca326f838296a0121f5.0 (rollback)
      origin refspec: origin:apertis/v2020/armhf-uboot/minimal

Some test contains incorrect boot ID detection, including the suffix
`.0` which is an error. Hence fixing the issue in the initial script and
all copy-paste copies.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

This reverts commit d9ee56d0.

After updating `coreutils-gplv2` package to version 6.9-0co17, it
contains `mktemp` compatible with a previous options used for `mktemp`.
Reverting the commit allows to have additional point for checking the
compatibility of `coreutils-gplv2` with more recent versions of coreutils.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Incompatible options used for `mktemp` call after shipping `mktemp`
from `coreutils-gplv2` package. Fix options for temporary PEM file
creation.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

aum-ota-signed test fails with the following traces:
+ testname=test-sign-no_signature
+ apply_update_sync -o
+ local RESULT=1
+ date +%Y-%m-%d %H:%M:%S
+ local BEFORE_UP_DATE=2020-07-14 08:02:30
…
Jul 14 08:02:40 apertis apertis-update-[667]: Ostree upgrade failed: Can't verify commit: signature: ed25519: no keys loaded: signed with unknown key
+ journalctl --since 2020-07-14 08:02:30 --unit apertis-update-manager
+ grep -qE Ostree already up to date
+ journalctl --since 2020-07-14 08:02:30 --unit apertis-update-manager
+ grep -qE Ostree upgrade failed
+ echo update failed
update failed
…
+ echo test-sign-no_signature: pass
test-sign-no_signature: pass
+ testname=test-sign-update
+ mkdir -p /etc/ostree/trusted.ed25519.d
+ cp -av apertis.ed25519 /etc/ostree/trusted.ed25519.d/
`apertis.ed25519' -> `/etc/ostree/trusted.ed25519.d/apertis.ed25519'
+ apply_update_sync -o
+ local RESULT=1
+ date +%Y-%m-%d %H:%M:%S
+ local BEFORE_UP_DATE=2020-07-14 08:02:40
+ + grep -qE Ostree already up to datejournalctl
 --since 2020-07-14 08:02:40 --unit apertis-update-manager
+ + journalctl --since 2020-07-14 08:02:40 --unit apertis-update-manager
grep -qE Ostree upgrade failed
+ echo update failed
update failed
+ RESULT=1
+ break
+ kill 707
+ kill 701
+ return 1
+ error_occured
+ set +x
Job for generated-test-case-offline-upgrade.service failed because the control process exited with error code.
See \"systemctl status generated-test-case-offline-upgrade.service\" and \"journalctl -xe\" for details.
Command exited with non-zero status 1
real	0m 16.77s
user	0m 0.03s
sys	0m 0.02s
<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=offline-upgrade RESULT=fail>

When apply_update_sync is called multiple time too quickly it can see
messages from previous call and base its result on them.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

New version of apertis-update-manager expects offline updates to be
encrypted.
This commit adds a third parameter to get_static_delta() to allow to
retrieve encrypted static bundle.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This new tests ensure:
- delta files with missing signature are not applied
- delta files with mismatching signature are not applied

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Copy ed25519 helper function from ostree test to update-manager-common.sh,
this will allow to share it between all test files.
Generateed25519 keys to sign and verify static delta superblock during
the test. Save the public key to its own file.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Next version of AUM will request signed static delta superblock.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Use the common cleanup function for all tests.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Move function `check_current_status()` into common library.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Move status definitions and GETPROP "macros" into the
sourced lib.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Add the test for checking non-successful update during OTA update.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>