Adopt exponential backoff when retrying

The current naive approach at retrying is not exactly gentle: if the server is overloaded we retry right ahead making the problem just worse.

See https://encore.dev/blog/retries for a fun explanation of how the naive approach can be catastrophic.

Fortunately the tenacity library allow us to add a bounded exponential backoff while also making the code easier to understand.

This should make the dashboard behave much better toward GitLab and OBS when they hit some issue.

Task: https://phabricator.apertis.org/T10169

changed the description

added 1 commit

• 9fc43368 - Adopt exponential backoff when retrying

Compare with previous version

changed the description

approved this merge request

Merging as @koushik.tm approved it and testing showed it to cope with failures much more sensibly than the current code.

merged

mentioned in merge request !181 (merged)