Skip to content

Reduce parallelism when fetching licensing data

Emanuele Aina requested to merge wip/em/enhance-calm-when-fetching-licenses into master

With 30 concurrent threads we were ovehelming both our client-side HTTP connection pool and the GitLab server itself.

In the best case, we only got many warnings on the client side like:

INFO:root:Fetching component and license_report for 5745 projects
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: gitlab-apertispro.boschdevcloud.com

In other cases we got GitLab failing with many exceptions:

gitlab.exceptions.GitlabHttpError: 500

Some of those errors mapped to server side logs indicating exhaustion of the database connection slots:

connection to server at "10.0.69.74", port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

Merge request reports