Up Squared 6000 board fails to boot randomly in LAVA

added priority:needs triage release:v2022 release:v2023 status:unconfirmed + 1 deleted label

changed the description

fwiw this might be the serial console setup in lava going too fast overflowing the dut

I thought about that but the echo is correct:

set root='(hd0,gpt1)'
grub> set root='(hd0,gpt1)'
[...]
error: disk `(hd0,gpt' not found.

https://lava.collabora.dev/scheduler/job/10564034#L6695

@wlozano Do you have links to other failures like this ? Is it always on aaeon-UPN-EHLX4RE-A10-0864-cbg-2 ? I can't list the jobs of aaeon-UPN-EHLX4RE-A10-0864-cbg-2 on LAVA (error 500), even when logged in.

I have checked and this also rarely happens for stable releases in other boards. This is an example for v2023 in a different board.

https://lava.collabora.dev/scheduler/job/10599965#L10672

error: disk `(hd0,gpt' not found.

Why do you say that the echo is correct? The error seems to show that only part of the command was processed, missing "1)"

The first line set root='(hd0,gpt1)' is what is sent to the DUT by lava over the serial port. The second line grub> set root='(hd0,gpt1)' is what the DUT outputs on the serial port to show what was typed (the console echo). The command seems complete there, so I'm not sure what can cause grub to miss the end of the value.

Also, because of the ', the grub console would probably fail as there would be no closing '.

Because it fails on other devices the same way, maybe the serial port is not the culprit, but a bug in the grub console ? That seems unlikely as well.

yeah indeed if the ' was missing grub would complain (or really it goes into multi-line input waiting for the final '), which would cause lava to timeout waiting for the prompt... So apparently "sometimes" 2 bytes are missed on the input even if it echo's... which is weird.

Especially as for the initial network boot the inputs are way way longer, so if it's a problem of e.g. overflows we'd see it there as well..

@detlev @wlozano thusfar our sample size is 2, which is a bit small; Could you analyse a bunch more failing test jobs to work out if:

It's always the "boot-to-disk" custom commands that have this issue (e.g. not the initial flashing phase)
The error is always the same (e.g. always missing the 1))

That might help us pin down if it's somewhat specific to setting env vars or...

added priority:high label and removed priority:needs triage label

assigned to @detlev

I did some analysis in the grub code but all looks quite good. I thought there could be an issue with the simple quotes ' somehow messing up the strlen of the value (2 chars missing, 2 quotes) but my flex/yacc/bison knowledge is not good enough for something so subtle.

I couldn't find other jobs with this issue. @wlozano do you keep a list of these failures ?

Same failures here:

https://lava.collabora.dev/scheduler/job/10759540 (aaeon-UPN-EHLX4RE-A10-0864-cbg-0)
https://lava.collabora.dev/scheduler/job/10741425 (aaeon-UPN-EHLX4RE-A10-0864-cbg-1)
https://lava.collabora.dev/scheduler/job/10700253 (aaeon-UPN-EHLX4RE-A10-0864-cbg-0)
https://lava.collabora.dev/scheduler/job/10666156 (aaeon-UPN-EHLX4RE-A10-0864-cbg-3)

Exactly the same each time: the 1) is missing in the variable but the echo is correct when it is being set.

According to grubs code, the message error: disk `(hd0,gpt' not found. is only shown when parsing the (hd0 part. If that succeeds, it goes on to the partition probing.

So the real error here seems to be that the disk hd0 was not found. The fact that `(hd0,gpt is shown in the message can be another issue, maybe just related to the print function and the data is actually correct.

That could be a timing issue in grub (not a hardware issue as it started happening on different boards at the same time)

To be sure of that, I would suggest adding a printenv (or equivalent) in the LAVA grub boot script to check that the variable is indeed correct the next time this happens. (a simple change to get a better understanding)

@sjoerd do you know what kind of disk hd0 is supposed to be in LAVA on these boards ? is It a USB Mass storage ?

hd0 should be the onboard emmc there aren't any extra storage devices plugged in on these afaik; You could add ls -l to the custom grub commands to see if it that provides us a bit more background of what's going on here.. To show the env var you'd add echo ${root}

For reference though if either the partition or the hd doesn't exist you don't get those errors:

grub> set root=(hd1,gpt1)                                                                
grub> chainloader /efi/boot/bootx64.efi                                                    
error: disk `hd1,gpt1' not found. 
grub> set root=(hd0,gpt4)                                        
grub> chainloader /efi/boot/bootx64.efi                                                      
error: disk `hd0,gpt4' not found.

The only way to get error: disk (hd0,gpt' not found.seems to be when actually inputting(hd0,gpt`

It's interesting to see that in your snippet, the opening parenthesis is not shown in the error message, but it is shown in the failing lava job.

What version of grub did you run that with ? I see that Debian uses 2.06 (in bookworm) but lava seems to use 2.11

Note that you do get (hd0,gpt if you input that; so it only ends up stripping the parentheses iff there is a pair. For grub i'm using 2.11 same as our lava setups -- the build comes from https://gitlab.collabora.com/lava/grub

added status:inprogress label

removed status:unconfirmed label

@sjoerd I ran 150 jobs on the upsquared 6000 on staging and none failed when using the chainloader. I also tried 50 jobs with a character_delay of 20 ms and none failed either. So that doesn't tell us that the character_delay is the solution, but that tells us that the character_delay doesn't hurt.

Does it make sense to add it in the production template ? (Maybe 5 ms would be enough though)

Sorry, I don't understand what do you mean with "using the chainloader". Regarding the comment of a delay of 5 ms I don't fully understand the consequences, but it looks OK to me.

There is no point at throwing things at a machine if before the change we couldn't reproduce the issue either..

Well, couldn't reproduce on staging. It still does happen on prod. The additional 50 with delay were to confirm it doesn't break anything.

@detlev is there any update on this issue?

I have seen other hangs, but I'm not sure if they are related to this issue

https://lava.collabora.dev/scheduler/job/11352076#L12954

A different issue but still around Up Squared 6000 boards

#430 (closed)

This issue still appears with a probability of .05, in different boards and different releases:

A full report up_squared_6000_unknown_failures.txt

This still happens

Usually the error is triggered after flashing the image and rebooting. However, https://lava.collabora.dev/scheduler/job/11623895#L1305 the case is slightly different, it booted once and after a reboot it failed.

Moving to normal priority since this issue has not been reproduced outside of LAVA.

added priority:normal label and removed priority:high label

unassigned @detlev

removed 1 deleted label

added 1 deleted label

Issue still valid

https://lava.collabora.dev/scheduler/job/12241527

Issue still present

https://lava.collabora.dev/scheduler/job/12239558

Issue still present

https://lava.collabora.dev/scheduler/job/12626141

added priority:needs triage label and removed priority:normal label

assigned to @sudarshan

Issue is not reproducible with apt/ostree boot-up image: https://images.apertis.org/weekly/v2025dev1/20240131.0017/

Test cases are also getting passed: https://qa.apertis.org/submit_report/v2025dev1/20240131.0017/hmi-amd64-uefi-public/ostree

@zbu1kor this issue seems to be related to the LAVA setup, did you try to reproduce this on LAVA?

changed title from Up Squared 6000 board fails to boot randomly to Up Squared 6000 board fails to boot randomly in LAVA

added status:needinfo label and removed status:inprogress label

Taking a look at this, if I look at jobs 12885140 and 12885141. These appear to both be runs against apertis_v2025dev2-hmi-amd64-uefi_20240228.0317, testing different things but the part of the test we care about seems basically the same. One fails the other doesn't.

After both echos of grub> set root='(hd0,gpt1)'\n from the target board, there is a string of mainly ASCII escape sequences. These differ subtly, one ends (the working one):

\e[9;25H\e[9;25H1\e[9;26H\e[9;26H)\e[9;27H\e[9;27H'\e[9;28H\n

the other (the failing one):

\e[9;25H\e[9;25H'\e[9;26H\n

Decoding those, the former is positioning the cursor at successive locations between columns 25 and 28 on row 9 and printing 1)', the other prints ' at row 9, column 25.

I'm assuming that this data is primarily to output into a buffer for a screen? But shows it's somehow lost the 1), but keeps the '!!!

that's grub over a serial line so it's expecting vt100 on the other side; the loss of characters can be both loss on input and output still (likely input).

Looked into dropping to "dumb" mode on the serial however the vt100 seems to be hard coded for a given target/platform set and the following message found in the grub source suggests pursuing this further may not be very fruitful:

/* FIXME: The dumb interface is not supported yet. */

I've run 200 boot runs on the staging LAVA instance against the UP 6000 there. Of the 200 runs there were 3 failures.

I've tweaked the boot character delay to 10ms and submitted 400 more runs to see what the impact is.

400 runs over the weekend, there were 41 failures and the failure subtly changed to:

error: disk `hd0,t1' not found.

This seems to be more missing characters and significantly more failures.

Ran 400 iterations of the following script locally overnight, using Qemu with the Grub binary from LAVA and Apertis v2023 fixedfunction image. Zero failures occurred, but then this is a highly contrived serial interface.

#!/usr/bin/env python3

import pexpect
import sys

failures = 0
count = 0

while count < 400:
	print("Loop %d" % count)
	child = pexpect.spawn("qemu-system-x86_64 -m 2G -nographic -bios /usr/share/ovmf/OVMF.fd -kernel core-20220705.efi apertis_v2023-fixedfunction-amd64-uefi_v2023.4.img") 
	#child.logfile = sys.stdout.buffer
	fout = open("logs/run_%d.log" % count, "wb")
	child.logfile = fout
	try: 
		child.expect("grub> ")
		child.sendline("set root='(hd0,gpt1)'")
		child.expect("grub> ")
		child.sendline("chainloader /efi/boot/bootx64.efi")
		child.expect("grub> ")
		child.sendline("boot")
		child.expect("login: ", timeout=120)
		child.sendline("user")
		child.expect("Password: ")
		child.sendline("user")
		child.sendline("sudo poweroff")
	except:
		print("Failure detected")
		failures = failures + 1

	fout.close()
	count = count + 1

print("\nCompleted with %d failures in %d runs\n" %(failures, count))