High level tracking of AUM issues

changed the description

v2022

aum-offline-upgrade-branch all
v2022pre not available -> https://gitlab.apertis.org/infrastructure/apertis-issues/-/issues/418

aum-offline-upgrade bcm2711-rpi-4-b-cbg-0
bootloader-interrupt timed out after 895 seconds -> sd card issue in bcm2711-rpi-4-b-cbg-0?
https://lava.collabora.dev/scheduler/job/11239653

aum-out-of-space bcm2711-rpi-4-b-cbg-0
bootloader-interrupt timed out after 895 seconds -> sd card issue in bcm2711-rpi-4-b-cbg-0?
https://lava.collabora.dev/scheduler/job/11239665

apparmor* imx6q-sabrelite-lava-cbg-1
dd timeout -> sd card error on imx6q-sabrelite-lava-cbg-1 
https://lava.collabora.dev/scheduler/job/11241106

v2023

aum-ota-out-of-space r8a7796-m3ulcb-cbg-1
ext4 fs
https://lava.collabora.dev/scheduler/job/11241388

aum-offline-upgrade bcm2711-rpi-4-b-cbg-0
bootloader-interrupt timed out after 895 seconds -> sd card issue in bcm2711-rpi-4-b-cbg-0?
https://lava.collabora.dev/scheduler/job/11241164

aum-offline-upgrade-branch aaeon-UPN-EHLX4RE-A10-0864-cbg-0
login timeout -> https://gitlab.apertis.org/infrastructure/apertis-issues/-/issues/339
https://lava.collabora.dev/scheduler/job/11241106

aum-ota-api aaeon-UPN-EHLX4RE-A10-0864-cbg-2
login timeout -> https://gitlab.apertis.org/infrastructure/apertis-issues/-/issues/339
https://lava.collabora.dev/scheduler/job/11241145

aum-ota-out-of-space aaeon-UPN-EHLX4RE-A10-0864-cbg-1
login timeout -> https://gitlab.apertis.org/infrastructure/apertis-issues/-/issues/339
https://lava.collabora.dev/scheduler/job/11241147

v2024dev3

aum-ota-signed all v2024dev3
need to investigate

aum-offline-upgrade imx6q-sabrelite-lava-cbg-1 
dd timeout -> sd card error on imx6q-sabrelite-lava-cbg-1 
https://lava.collabora.dev/scheduler/job/11239153

aum-offline-upgrade-branch bcm2711-rpi-4-b-cbg-2
booting error
https://lava.collabora.dev/scheduler/job/11239350

aum-rollback-blacklist bcm2711-rpi-4-b-cbg-0
bootloader-interrupt timed out after 895 seconds -> sd card issue in bcm2711-rpi-4-b-cbg-0?
https://lava.collabora.dev/scheduler/job/11239358

aum-offline-upgrade-branch aaeon-UPN-EHLX4RE-A10-0864-cbg-2 
login timeout -> https://gitlab.apertis.org/infrastructure/apertis-issues/-/issues/339
https://lava.collabora.dev/scheduler/job/11239308

added area:aum priority:high release:v2022 release:v2023 status:inprogress + 1 deleted label and removed status:unconfirmed label

removed priority:needs triage label

assigned to @wlozano

A new round of checks:

Several tests fail on imx6q-sabrelite-lava-cbg-4, it looks like somehow this board has issues with the bootcount https://lava.collabora.dev/scheduler/job/11616592
Several tests that rely on bootcount fail in v2022 and v2023 for Up Squared board
- #206
- #211 (closed)
Upgrade branch fail in v2022 as there is no previous branch to use a base
- #364 (closed)

After investigating a bit more the failures in imx6q-sabrelite-lava-cbg-4 logs like this were found

Couldn't find partition mmc 0:1
Error selecting device
File System is consistent
file found, deleting
File System is consistent

The first two entries seem to point to the fact that u-boot is unable to read the mmc and later it stores a bootcount without any issues (no error log). This causes bootcount to be reset and making ostree tests to fail. I guess this could be caused by a very slow mmc and this u-boot version does not handle it properly?

As reference U-boot code involved

https://gitlab.apertis.org/pkg/u-boot/-/blob/apertis/2021.10+dfsg-1+apertis10/drivers/bootcount/bootcount_ext.c?ref_type=tags#L52

During the regular check it was noted that aum-offline-upgrade-branch #364 (closed) always fails on RPi4 for v2022. This issue is probably related to the fact the we used v2021 as base release where support for RPi4 was no completed. For this reason and for since EOL v2022 is near this issue won't be fixed.

The error shown is

Oct 17 02:03:05 apertis apertis-update-[1059]: Ostree upgrade failed: Installing kernel: regfile copy: No space left on device

https://lava.collabora.dev/scheduler/job/11794963#L8325

mentioned in issue #418 (closed)

Another interesting pattern is that out-of-space tests tend to fail in amd64. Somehow the upgrade applies even if the space is limited. This needs to be investigated.

#411 (closed) #371 (closed)

As v2022 has reached its EOL several AUM issues that only affected that release have been closed.

Several AUM failures have been triggered by problems with sdcard in bcm2711-rpi-4-b-cbg-1

`bmaptool: [91mERROR[0m: cannot open destination file '/dev/disk/by-id/mmc-SD_0x0000000c':` `[](https://lava.collabora.dev/scheduler/job/12836086#L1557)Setting prompt string to ['copying time: [0-9ms\\.\\ ]+, copying speed [0-9\\.]+ MiB\\/sec']` `[](https://lava.collabora.dev/scheduler/job/12836086#L1558)Changing prompt to ['copying time: [0-9ms\\.\\ ]+, copying speed [0-9\\.]+ MiB\\/sec']` `[](https://lava.collabora.dev/scheduler/job/12836086#L1559)dd-image: Wait for prompt ['copying time: [0-9ms\\.\\ ]+, copying speed [0-9\\.]+ MiB\\/sec'] (timeout 00:59:59)` `[](https://lava.collabora.dev/scheduler/job/12836086#L1560)[Errno 30] Read-only file system: '/dev/disk/by-id/mmc-SD_0x0000000c'`

https://lava.collabora.dev/scheduler/job/12836086#L1560

+ echo Expected bootcounter: bd, 01, 00, 00
Expected bootcounter: bd, 01, 00, 00
+ echo     Read bootcounter: bd, 01, 01, 01
    Read bootcounter: bd, 01, 01, 01
+ return 1
+ error_occured
+ set +x

https://lava.collabora.dev/scheduler/job/12808166#L14797