Filter out packages not in Apertis during the packaging-data-fetch-sources-upstream job
Fetching upstream sources data for all Debian packages (~ 38100 pkgs)
and not only those in Apertis (~ 5600 pkgs) generates big YAML files
containing useless data. Processing these files lead to frequent
out-of-memory issues. To reduce the memory consumption, we can filter out
all packages not available in Apertis based on the cache file.
This job produces a 42 MB YAML file, merging this file with others YAML
files leads to out-of-memory issues in downstream jobs. Filtering the
generated YAML file at this step reduces the size to only 8 MB.
Signed-off-by:
Dylan Aïssi <dylan.aissi@collabora.com>
Loading
Please register or sign in to comment