This is mainly to avoid the message:
> cat: /var/run-phase: No such file or directory (os error 2)"

Which is expected at the first step and can be confusing.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

This is to simplify the execution of the test
aum-rollback-bootcount. Instead of using manual commands,
we can simply use this script.

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

We are seeing failures in the log that appear like this:

+ journalctl --after-cursor s=3fe1fe422f3f482191a361280eed2fdd;i=3b5;b=d8d8e27b9e244b3a83c385fe9dee1358;m=5be7e37;t=617e797638e8f;x=967810e36b39afc --unit apertis-update-manager
+ grep -qE Ostree upgrade failed
+ echo update failed
update failed
+ RESULT=1
+ break

It suggests that the board's journal may contain some information about
the failure, but we currently aren't seeing this in the LAVA logs.

On failure print out the logs that are being used to determine the
failure to see if we can understand the issue better.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Walter Lozano authored 1 year ago and Dylan Aïssi committed 1 year ago

In order to improve readability and maintainability move code that is
repeated to a new set of functions.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 1 year ago and Dylan Aïssi committed 1 year ago

While running the out-of-space test a dummy file is created to reduce
the space available. Unfortunately, in some cases the space available
is still enough to upgrade the system leading to sporadic failures.

The reason behind this are a few:
- The delta file is dinamically created and is very small, so with very
little space the update can be applied
- The way of calculating the space required for a file to fill the disk
free space is not accurate as it relies in the output of df

In order to make the test more reliable, instead of trying to compute
the free space allow dd to fill the disk and then truncate the file
allowing only a very limited space.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 1 year ago and Dylan Aïssi committed 1 year ago

In order for the test update_does_not_start_if_disk_too_low make sense
OStree should be configured with an appropiate value. The previous
implementation relied in default values that can changed from
configuration to configuration, so in order to test the functionality
force a well known value.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

During the power cut test several upgrades are triggered and a delay is
used to trigger a power cut from LAVA. This delay should only allow the
upgrade to start but not to finish, as the purpose of the test is to ensure
that in a power cut AUM does not apply the update.

However, with modern boards delays bigger than 3 sec make the system to
complete the upgrade and thus making the test to fail.

Workaround this issue by limiting the max delay used. This approach is not
perfect as it is still not deterministic, but at least should make the test
more stable. A better approach, taking into account the progress in the update
should be implemented once this support is included.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Utility xxd is used to tweak the bootcount during tests. Since the version
in busybox is enough let's use it instead of installing a new package.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

With the latest approach, from time to time, the AUM tests does not catch
the upgrade and reports the error. This happens due to the fact that
the upgrade is done right after the restart and cursor is updated in the loop.

To avoid the issue detect change the way AUM crashed are caught taking into
account the number of restarts in the log.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

After the rebase on top of Bookworm the string printed by systemd when
a service is restarted changed, including both service name and description.
Fix the log parsing taking this into account.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

In some cases the logs for AUM restart are delayed causing false test
failures. In order to avoid the issue, wait to the log before continue
with the rest of the tests.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

If the cursor is not saved in an entry matching the filtering, -u in
this case, later when logs are searched with a filtering enabled, cursor
is first move to the first match. With this behavior, using --after-cursor
will probably skip a valid entry.

To properly save the cursor to later retrieve new logs the same filtering
should be used to avoid losing logs if the cursor is not pointing to a
filtered entry.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Under some scenarios, the logs for the AUM restart take longer to appear,
causing that the cursor is saved before them. Under this situation the test
might think that AUM was restarted after a crash.

In order to have a reliable way of check logs flush them after restart AUM.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Frederic Danis authored 3 years ago and Emanuele Aina committed 3 years ago

OSTree branch installed on RPi 4b boards use
`origin:apertis/v2023dev1/arm64-rpi64/fixedfunction` instead of
`origin:apertis/v2023dev1/arm64-uboot/fixedfunction` . The later is generic
for R-Car and RPi 4b boards.

This commit adds a function to return the expected branch, and adapt it
for the RPi64.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Add boot-delay-install.sh to install the boot-delay systemd service

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Rollback tests need to reset the DUT before boot is completed.
This commit add a service delaying start of AUM and user logging, and
displaying a message allowing a tester to reset the board.
This service should be run before systemd-bless-boot.service which rename
the EFI loader entry discarding the bootcount.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This reverts commit 8c71fb5c.

Filesystem corruption during image creation has been fixed in
pkg/user-mode-linux

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Workaround: manually unmount the boot partition before the board is reset
by Lava to prevent unexpected entries in /boot/lost+found which can
prevent subsequent boots.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Since addition of EFI support, the bootcounter variables are shared between
the different bootloader check parts.
Variables set using read function in pipe subshell are not available
outside of this subshell, and so unset for the rest of the function.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

aum-rollback-blacklist test fail on UEFI AMD64 targets with log:
  + check_bootcounter   00 01
  + + read -r m ver cnt uflag
  od -t x1 -An /boot/uboot.cnt
  od: '/boot/uboot.cnt': No such file or directory (os error 2)

For EFI, the bootcounter is stored in the loader entry filename, i.e.
/boot/efi/loader/entries/ostree-0-2[+bootleft[-bootcount]].conf.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

AUM api test failed due to `No space left on device` error when copying
HMI encrypted bundle to `/var/image_file`.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

As described in 15b4dc92 rust-coreutils df does not support block size,
so switch to busybox df.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Since rust-coreutils df lacks of several features like block size switch
to busybox df.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Since rust-coreutils does not not provide dd yet, use busybox dd.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Apertis now uses rust-coreutils, which is not fully compatible with
GNU coreutils. Fix the usage of tail according to rust-coreutils
requirements.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

`uutils/coreutils` lacks dd.
We can replace that call with `busybox dd`

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

In case if bootloader behave incorrectly and ignore the upgrade state
it can trigger rollback even after successful upgrade.
With working `apertis-update-complete` service this state is masked
since AUM rewrite the bootcounter file on every successful boot.

However in case if the system is interrupted 3 times in a raw before
re-marking update as "successful", the bootloader may start the rollback
procedure. For instance in case of sequential power failure or user's
interruption.

Added the test emulating failure of the boot process prior starting the
mark service after successful upgrade by failing this service. Check if
no rollback happens after several boots in a raw.

See: https://phabricator.apertis.org/T7239



Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Test the boot counter state to ensure it has upgrade
status set during upgrade and unset after successful upgrade.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

AUM crash is detected as unexpected AUM restart. However due
races on startup the cursor for logs check may catch the expected
restart and treat it as unexpected.
Reset the log cursor to check events occurred during update only.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

If we expect the failure of `apply_update_sync` we should treat
only "1" return status as expected value.
Return status "99" is reserved on unexpected failure of AUM itself and
must indicate the failing test.
This helps to avoid masking of AUM internal problems.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

If AUM is crashed by any reason (SIGSEGV for example) the function
`apply_update_sync` don't detect that behavior and exit with error code
after timeout (~8 min). This is leading to false positive result if we
are expecting the update failure.

Added the check of AUM restart by systemd and return code `99` for that case.
Different error code allow to determine if the upgrade is failed as
expected or we have bug in AUM code. Reduces the test time in case of
AUM crash.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Function to reset the boot counter
Supporting old and new formats
Accept 2 arguments:
 bootcounter: number, 0 if skipped
 upgrade flag (could be omitted):
   1: set upgrade state (default)
   0: disable upgrade state

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

New `uboot.cnt` file format appears in u-boot 2019.01+dfsg-7co7.
Allow to read legacy format.
If "magic" and "version" are not requested explicitly the function
accept any supported `uboot.cnt` format.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Fix the check of the uboot boot counter and the test logic.
Added some useful debug output.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Function to validate values from /boot/uboot.cnt

accept 4 arguments:
  magic: bd
  version: 1
  bootcount: any number
  upgrade bit:
    0: not in upgrade
    1: in upgrade

In case if any argument is not defined i.e. passed empty value ""
it is meaning no need to validate that argument

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Commit 8f826964 contain the fix of ostree commit ID parsing.
Before that commit the ID contained additional suffix (".0") and
thus the comparison with `COMMIT_AFTER` always shows a difference
and the test result was __always__ false positive: "pass".

After applying the 8f826964 commit we shed light to other issue --
we saved a wrong ostree commit ID for later usage.
In case if we prepare the "outdated" commit -- we must use it's ID
as "starting" for the test.

Moved the preparation of "outdated" ostree commit into the common
function `prepare_outdated_commit()` and store the proper ostree
commit ID as starting point.

Additional side-effect fixed for OTA upgrades -- before current fix
OTA tests were unstable, sometimes allowing to pass the check even
with wrong ID written as starting point.
This is caused by long queues in LAVA nowadays -- the build system may
prepare and submit the new version while the test is waiting in queue.
In that case the system do the real upgrade to the OS version newer
than we have during the test submit.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

All scripts are started by root user hence the `sudo` call is not
needed. It comes from manual test cases by copy-paste method.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Removed unneeded `sudo` call from the library.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Added automatic test allowing to check the offline upgrade from
different branch (previous release in case of Apertis).

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>