ci-package-builder: Monitor build results from OBS
Stream the build log and propagate the build results from OBS, so that developers get integrated feeback directly on their GitLab pipeline.
The approach chosen has the drawback of keeping a full worker slot busy doing almost nothing, simply streaming the log from OBS and propagating the result.
Builds for larger packages can take more than 7h, and it is not acceptable to keep a normal runner slot busy for that.
The best way to handle such issue is to deploy a separate runner that only runs these monitoring jobs using the appropriate tags. Since each job only consumes a negligible amount of resources a large amount of slots can be configured, in a way that long builds do not block other jobs.
Sample pipelines here:
Each upload triggers at most 3 jobs. With three concurrent releases we may easily end up with 9 jobs per package update. Most of them do not take that long, but linux, webkit or llvm are good examples of very very long ones.
The processes are quite lightweight, so I would not see much of an issue with setting 100 slots on the runner to handle more than 10 concurrent package updates, and see how well or badly it works.
Using AWS would be very wasteful, spinning up VMs just to stay nearly idle, so I ended up using the graviera host by configuring a new runner instance with 100 slots but restricted to the monitoring tag. The runner also has a memory limit of 20M with no swap set on the containers it creates.