Small fix, rewrite the Package builds section

This MR:

• removes the use of snapshotted
• rewrites the Package builds section adding more details about package builds using the SDK, devroots and snapshots of APT archives. It mentions the differences between building on OBS and on the SDK and differences between native and emulated builds.

content/architecture/long-term-reproducibility.md

@@ -308,14 +308,55 @@ the remaining planned improvements.
 
 ## Package builds
 
-Package builds happen on OBS which does not have snapshotting capabilities and
-always builds every package on a clean, isolated environment built using the
-latest package versions for each channel.
-
-Since the purposes taken in account in this document do not involve large scale
-package rebuilds, it is recommended to use the SDK images and the devroots in
-combination with APT archives snapshots to rebuild packages in an
-environment closely matching a past build.
+Apertis uses [OBS](https://openbuildservice.org/) for building packages,
+however since the purpose of the long term reproducibility does not involve
+large scale package rebuilds, we recommend using the [Apertis
+SDK](designs/software-development-kit/) for building packages. In addition to
+the SDK a [devroot](architecture/sysroots-and-devroots/#devroot) and a snapshot
+of the APT repositories are also required.
+
+We recommend the SDK release to match the release of Apertis in the target
+device, and it is important that both the release and architecture of the
+devroot and of the snapshot of the APT repositories matches the target device.
+As an example, if the target is an `ARM64` device, the devroot and the snapshot
+of the APT repositories should also be for `ARM64`.
+
+See the [devroot](architecture/sysroots-and-devroots/#devroot) for more
+information.
+
+For the long term reproducibility we recommend the product team to have a
+backup copy of:
+ * The Apertis SDK: Matching the Apertis release in use by the target device.
+ * The Apertis devroot: Matching the Apertis release and the architecture of
+   the target device.
+
+At the time of writing OBS supports native and emulated builds, and we have
+successfully deployed emulated builds on the Bosch Developer Cloud. While
+emulated builds are expected to work in most cases, they may require
+changes to the packages being build using emulation, and they may produce
+different results when compared to native builds.
+
+For more information about emulated and native builds see
+[Build infrastructure on Intel x86-64](designs/x86-build-infrastructure/).
+
+Building packages on the SDK uses emulation, and while this approach is
+currently expected to be satisfactory, it may produce different results when
+compared to native builds. In the unlikely event of emulated builds not being
+satisfactory, we describe a mitigation on the section Future work on package
+builds.
+
+### Future work on package builds
+
+We recommend using the Apertis SDK for building packages in the context of the
+long term reproducibility to enable product teams to build packages without
+depending on custom OBS deployments.
+
+However using OBS for building packages in the context of the long term
+reproducibility may offer advantages, when large package rebuilds are needed or
+when native builds are required.
+
+We will consider for future work automating the OBS configuration to support
+building packages for the long term reproducibility.
 
 # Recommendations for product teams
 

[Emanuele Aina]

This document is not really about the Apertis images themselves, but about the product-team specific recipes. They should have their own SDK and devroots.

What about this?

-For the long term reproducibility we recommend the product team to have a
-backup copy of:
- * The Apertis SDK: Matching the Apertis release in use by the target device.
- * The Apertis devroot: Matching the Apertis release and the architecture of
-   the target device.
+For long term reproducibility we recommend product teams to either store
+and use the SDK image and devroots matching the original build that is
+being reproduced, or use the procedure described in this document to
+reproduce the SDK image and devroots as a first step.

[Peter Senna Tschudin]

Not sure I follow. Suggestion is to replace what is between lines 327 and 331 with the new text? Also which previously described mechanism?

[Emanuele Aina]

Yup, sorry.

The point is that product teams will have their own SDK images, so either they store them somewhere or they reproduce them in the same way as the target images.

(I've edited the suggestion)

[Emanuele Aina]

In theory, packages should not even notice they are built on emulated devices. For the vast majority of packages that's the case. There may be bugs and corner cases, but I don't think it's worth mentioning it here.

-At the time of writing OBS supports native and emulated builds, and we have
-successfully deployed emulated builds on the Bosch Developer Cloud. While
-emulated builds are expected to work in most cases, they may require
-changes to the packages being build using emulation, and they may produce
-different results when compared to native builds.
-
-For more information about emulated and native builds see
-[Build infrastructure on Intel x86-64](designs/x86-build-infrastructure/).

[Peter Senna Tschudin]

I took a different perspective: similarity between builds. While we do not guarantee bit-by-bit similarity between builds, I expect that the build results will vary between native and emulated builds. So the idea here is make the reader aware of this here.

[Emanuele Aina]

No, really, emulated vs. native probably introduces less variance than the filesystem ordering. It's just that by being much slower it can cause unit tests to fail if they have some kind of race condition.

[Peter Senna Tschudin]

@em here is a concrete example(even if between KVM and LXC in this case). I did not have this particular case in mind, but I expect that there is plenty of room for such corner cases that will produce different results when building on hardware and when building on an emulator.

[Emanuele Aina]

If the package detects the current CPU features and selects options based on that it's going to be funny anyway when it ends up being built on our ARM64 worker once and then it gets rebuilt on a Packet cloud worker when Sjoerd turns them on (or vice-versa).

One thing that is much more likely to cause issues is the kernel version on the workers since some testcase exercise features that are not available there as we ship newer kernels.

All of them are valid concerns (and are really bugs in the packaging), I just don't feel this is the right place to point them out.

[Peter Senna Tschudin]

@em I added this section based on the request from @sudarshan here Elaborate the Risk and mitigation of SDK and devroot usage on latest release to reproduce the build setup. What do you suggest to address the request?

[Emanuele Aina]

To be honest, I'm not sure what is being requested. Why "on latest release" when the whole point is to reproduce the build environment of an older release?

[Emanuele Aina]

I'd set that aside for the moment and merge the other changes while we can clarify that point with @sudarshan. :)