Skip to content
Snippets Groups Projects
  • Dylan Aïssi's avatar
    38e0fc03
    Filter out packages not in Apertis during the packaging-data-fetch-sources-upstream job · 38e0fc03
    Dylan Aïssi authored
    
    Fetching upstream sources data for all Debian packages (~ 38100 pkgs)
    and not only those in Apertis (~ 5600 pkgs) generates big YAML files
    containing useless data. Processing these files lead to frequent
    out-of-memory issues. To reduce the memory consumption, we can filter out
    all packages not available in Apertis based on the cache file.
    
    This job produces a 42 MB YAML file, merging this file with others YAML
    files leads to out-of-memory issues in downstream jobs. Filtering the
    generated YAML file at this step reduces the size to only 8 MB.
    
    Signed-off-by: default avatarDylan Aïssi <dylan.aissi@collabora.com>
    38e0fc03
    History
    Filter out packages not in Apertis during the packaging-data-fetch-sources-upstream job
    Dylan Aïssi authored
    
    Fetching upstream sources data for all Debian packages (~ 38100 pkgs)
    and not only those in Apertis (~ 5600 pkgs) generates big YAML files
    containing useless data. Processing these files lead to frequent
    out-of-memory issues. To reduce the memory consumption, we can filter out
    all packages not available in Apertis based on the cache file.
    
    This job produces a 42 MB YAML file, merging this file with others YAML
    files leads to out-of-memory issues in downstream jobs. Filtering the
    generated YAML file at this step reduces the size to only 8 MB.
    
    Signed-off-by: default avatarDylan Aïssi <dylan.aissi@collabora.com>