Apertis CI robot authored 1 day ago and Dylan Aïssi committed 1 day ago

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The computation of the license and copyright information is done at
image build time, which seemed good enough in early times, and gave
us the flexibility to improve the logic in the image pipeline which
runs on daily basis instead of at package build time. Unfortunately,
through time, we understood that this approach had its limitations.

First, compilation units provide information which we needed to map
to files in the source tree. To overcome this difficulty, some
workarounds were placed but those were not perfect.

Second, when Rust and Go packages were supported it was clear that
we needed to propagate lots of information to the image pipeline,
such as copyright reports of the build dependencies and mappings
between files and packages.

Now, that additional support was added to allow improving the
metadata, with information about non binary artifacts and shared
libraries there is no point in propagating all the data to the image
pipeline. Under these circumstances it is better to also compute
the license and copyright information at build time. This also aligns
well with the idea of having a SBOM at package level, which can
be useful.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 5 months ago and Dylan Aïssi committed 1 month ago

The inclusion of the package metadata provides useful information, however,
its format and data does not cover all the needs.

First, we need a reliable way of mapping compilation units to source files,
since the each package does its own tweaks to build binaries causing debug
information to report not very useful paths.

Second, having a map from external sources to packages is useful, but we need
to identify the file in the source tree to as well to be able to extract
license and copyright information.

And last but not least, the format of the metadata does not align with the
SBOM reports we generate, as for each use case new metadata was added,
without trying to provide more value.

For these reasons, rework the code to provide a new metadata format which
aligns with the SBOM generated at image build time, listing each artifact
and the sources used to build them.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 2 months ago and Dylan Aïssi committed 1 month ago

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Walter Lozano authored 2 months ago and Dylan Aïssi committed 1 month ago

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Package rust-coreutils is special as it uses a different folder for the
registry, causing dh_setup_copyright to fail at finding the package
reference files belong to.

To avoid this issue, take into account this fact and replace the names.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

The script dh_setup_copyright tries to find the package each reference
file belongs to, however, in some cases this is not possible and there
is now warning about that.

Add a warning about the unknown file to allow further investigations.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Signed-off-by: Dylan Aïssi <dylan.aissi@collabora.com>

The previous version of this code was `O(m*n)` where `m` = the number of
files in the source tree and `n` = the number of files referenced by
the binaries. In most cases, those numbers are quite small, but on large
packages they can grow incredibly large. For instance, for rustc's main
binary, `m > 300k` and `n > 33k`, resulting in each outer loop iteration
taking an average of ~1.9s. That would result in a runtime of over 17
hours, which is a rather absurd bump to the build time.

Instead, we can reorganize the code so that the source tree contents are
stored in a hash, indexed by basename. That turns the entire inner
matching loop into a single hash lookup, bringing the outer loop runtime
to a worst-case single-digit number of milliseconds.

Fixes: infrastructure/apertis-issues#595

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Otherwise they'll be printed following the internal order of the hash,
which isn't reproducible across runs.

Fixes: infrastructure/apertis-issues#594

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

The previous one didn't account for distro names other than "apertis".

https://phabricator.apertis.org/T10413



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

In addition to scanning binary files, now we also try to match files
directly copied to the install prefix to their original location, which
will allow the BOM generation scripts in the images to track their
copyright properly.

Because dh_setup_copyright now takes into account non-binary files, it
cannot run as part of elf-tools anymore, so this also moves it into the
root sequence.

https://phabricator.apertis.org/T7616



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

find_package_providing_path would try to look up the source package for
every incoming file, even if the file doesn't actually exist on disk.
This is largely useless, because dpkg-query only finds matches for
*installed* packages anyway, so just skip calling it at all for
non-existent files.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

https://github.com/rust-lang/rust/pull/92024 appends a build-specific
suffix to each compilation unit name in order to ensure they're all
unique. Thus, in order to get the original filenames, we need to strip
that suffix off.

In order to ensure the SBOM contains the right paths, we also write the
modified sources back to the bin2sources file.

infrastructure/apertis-issues#456



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

The OBS builders don't have any source lists set up, so `apt-cache
showsrc` never finds any results. Instead, we can rely on dpkg-query to
extract the info, along with some adjustments to get the correct version
value.

https://phabricator.apertis.org/T9756



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

This uses ldd to scan the shared libraries that the package depends on
and save their source packages to the metadata.

https://phabricator.apertis.org/T9756



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

dwarf2sources now writes information on "external files" that aren't CUs
but are referenced by the binary, which lets us save the source packages
that these external files came from. Note that the copyrights are not
imported, because the current use case for this isn't tied to licensing.

The number of external files means that querying the packages and their
original source packages turns out to be rather slow. However, in
general, multiple external files from a single package will be used,
and the same source packages will be looked up repeatedly. Thus, we can
just cache the full list of files for a package and binaries for a
source package, then reuse that list repeatedly.

https://phabricator.apertis.org/T9756



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Apertis CI robot authored 2 years ago and Ritesh Raj Sarraf committed 2 years ago

Signed-off-by: Apertis CI robot <noreply+not_a_robot@apertis.org>

Ritesh Raj Sarraf authored 2 years ago and Ritesh Raj Sarraf committed 2 years ago

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

dwarf2sources failures are supposed to be ignored (hence the use of
doit_noerror when invoking the command), but the recently added steps to
extract copyright information with fail the build if the dwarf2sources
result file is not found. In that case, just skip extraction altogether.

https://phabricator.apertis.org/T8985



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

This will copy the copyright and copyright_report files for any packages
whose files are found inside the resulting binaries, making them
available at `/usr/share/doc/PACKAGE/PACKAGE_external_copyrights_ARCH`.

https://phabricator.apertis.org/T8554



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

It's expanding from being just a dwarf2sources wrapper, so rename it
accordingly.

https://phabricator.apertis.org/T8554



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Signed-off-by: Andre Moreira Magalhaes <andre.magalhaes@collabora.com>

The bin2sources file may differ between architectures which may cause failures
when trying to install the same packages for multiple archs (multiarch support)
given they are installed in a arch independent directory.

Fix that by adding the architecture as suffix to the file.

Signed-off-by: Andre Moreira Magalhaes <andre.magalhaes@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>