Speedup test_urls (!278) · Merge requests · docs / apertis-website

Merged Emanuele Aina requested to merge wip/em/speedup-test-urls into master 3 years ago

Jul 09, 2021

Avoid a bunch of wiki.gnome.org redirects · e616a5c1

Emanuele Aina authored 3 years ago


Skip another set of redirects so it's less likely that validating links
makes us fall into rate limiting.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

e616a5c1

Update old live.gnome.org links to wiki.gnome.org · 775efb1d

Emanuele Aina authored 3 years ago


Avoid some redirects which makes us incur into rate limiting when
validating links.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

775efb1d

gitlab-ci: Make URL checking run in paralle with everything else · 1fe7d1f5

Emanuele Aina authored 3 years ago


There's little point in gating all the other jobs on a relatively slow
job as the URL checker, so make it run in parallel with everything else
without blocking anything.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

1fe7d1f5

test_urls: Emojis! · 4ff935a8

Emanuele Aina authored 3 years ago


Make spotting failures in the output more easy with colored icons.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

4ff935a8

test_urls: Ratelimit some origins · bddc195e

Emanuele Aina authored 3 years ago


The GNOME wiki is unhappy about our requests, be nice.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

bddc195e

test_urls: Ensure there are enough connections in the pool · aa93fae0
Emanuele Aina authored 3 years ago
```
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>
```
aa93fae0
test_urls: Block waiting for connections in the pool · 56d3229e
Emanuele Aina authored 3 years ago
```
Avoid `Pool is closed` exceptions.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>
```
56d3229e

test_urls: Use a threadpool for the HTTP requests · e1e4a0df

Emanuele Aina authored 3 years ago


Speed up the querying by using a threadpool.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

e1e4a0df

test_urls: Use a session to retry requests · 5dc9655b

Emanuele Aina authored 3 years ago


We often get DNS resolution or connection failures, try a few time
before giving up.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

5dc9655b

test_urls: Always print the HTTP status code for each URL · cd82ee49

Emanuele Aina authored 3 years ago


It can be useful to figure out what goes wrong.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

cd82ee49

test_urls: Print exceptions when validating URLs · 88c2c3be

Emanuele Aina authored 3 years ago


Print the HTTP status code and the returned exception when checking
each URL, to better understand what went wrong.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

88c2c3be

test_urls: Use a set to avoid O(N) insertion · d4981381

Emanuele Aina authored 3 years ago


Not that it makes any sensible difference, but still. :)

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

d4981381

Speedup test_urls

Merge request reports

Activity