Small fix, rewrite the Package builds section
This MR:
- removes the use of
snapshotted - rewrites the Package builds section adding more details about package builds using the SDK, devroots and snapshots of APT archives. It mentions the differences between building on OBS and on the SDK and differences between native and emulated builds.
Merge request reports
Activity
added 1 commit
- 06bd1484 - contribution_process.md: Add contribution template
316 of the APT repositories are also required. 317 318 We recommend the SDK release to match the release of Apertis in the target 319 device, and it is important that both the release and architecture of the 320 devroot and of the snapshot of the APT repositories matches the target device. 321 As an example, if the target is an `ARM64` device, the devroot and the snapshot 322 of the APT repositories should also be for `ARM64`. 323 324 See the [devroot](architecture/sysroots-and-devroots/#devroot) for more 325 information. 326 327 For the long term reproducibility we recommend the product team to have a 328 backup copy of: 329 * The Apertis SDK: Matching the Apertis release in use by the target device. 330 * The Apertis devroot: Matching the Apertis release and the architecture of 331 the target device. -
This document is not really about the Apertis images themselves, but about the product-team specific recipes. They should have their own SDK and devroots.
What about this?
327 For the long term reproducibility we recommend the product team to have a 328 backup copy of: 329 * The Apertis SDK: Matching the Apertis release in use by the target device. 330 * The Apertis devroot: Matching the Apertis release and the architecture of 331 the target device. 327 For long term reproducibility we recommend product teams to either store 328 and use the SDK image and devroots matching the original build that is 329 being reproduced, or use the procedure described in this document to 330 reproduce the SDK image and devroots as a first step. Edited by Emanuele Aina -
Yup, sorry.
The point is that product teams will have their own SDK images, so either they store them somewhere or they reproduce them in the same way as the target images.
(I've edited the suggestion)
Edited by Emanuele Aina
325 information. 326 327 For the long term reproducibility we recommend the product team to have a 328 backup copy of: 329 * The Apertis SDK: Matching the Apertis release in use by the target device. 330 * The Apertis devroot: Matching the Apertis release and the architecture of 331 the target device. 332 333 At the time of writing OBS supports native and emulated builds, and we have 334 successfully deployed emulated builds on the Bosch Developer Cloud. While 335 emulated builds are expected to work in most cases, they may require 336 changes to the packages being build using emulation, and they may produce 337 different results when compared to native builds. 338 339 For more information about emulated and native builds see 340 [Build infrastructure on Intel x86-64](designs/x86-build-infrastructure/). -
In theory, packages should not even notice they are built on emulated devices. For the vast majority of packages that's the case. There may be bugs and corner cases, but I don't think it's worth mentioning it here.
333 At the time of writing OBS supports native and emulated builds, and we have 334 successfully deployed emulated builds on the Bosch Developer Cloud. While 335 emulated builds are expected to work in most cases, they may require 336 changes to the packages being build using emulation, and they may produce 337 different results when compared to native builds. 338 339 For more information about emulated and native builds see 340 [Build infrastructure on Intel x86-64](designs/x86-build-infrastructure/). changed this line in version 4 of the diff
@em here is a concrete example(even if between KVM and LXC in this case). I did not have this particular case in mind, but I expect that there is plenty of room for such corner cases that will produce different results when building on hardware and when building on an emulator.
Edited by Peter Senna Tschudin-
If the package detects the current CPU features and selects options based on that it's going to be funny anyway when it ends up being built on our ARM64 worker once and then it gets rebuilt on a Packet cloud worker when Sjoerd turns them on (or vice-versa).
One thing that is much more likely to cause issues is the kernel version on the workers since some testcase exercise features that are not available there as we ship newer kernels.
All of them are valid concerns (and are really bugs in the packaging), I just don't feel this is the right place to point them out.
-
@em I added this section based on the request from @sudarshan here
Elaborate the Risk and mitigation of SDK and devroot usage on latest release to reproduce the build setup. What do you suggest to address the request? -
I'd set that aside for the moment and merge the other changes while we can clarify that point with @sudarshan. :)
added 7 commits
-
54799472...008a78a6 - 5 commits from branch
master - 1edcaa33 - Drop the use of snapshotted
- 9f10f5e3 - Rewrite the Package builds section
-
54799472...008a78a6 - 5 commits from branch
137 137 To further make build environments more reproducible, care can be taken to 138 138 make their own build process as reproducible as possible. 139 139 The same concerns affecting the main build recipes affect the recipes for the 140 Docker images, from storing pipelines in git, to relying only on snapshotted -
(this is meant for the commit message but gitlab doesn't support that yet)..
This patch removes the use of the work snapshotted.; Surely you meant removing the "world" snapshotted". Would be nice if you could add a bit of a veneer of rationale though (why not what!).. e.g. "snapshotted is not a realy word; remove it and make the sentence more natural" or somesuch
309 309 ## Package builds 310 310 311 Package builds happen on OBS which does not have snapshotting capabilities and 312 always builds every package on a clean, isolated environment built using the 313 latest package versions for each channel. 314 315 Since the purposes taken in account in this document do not involve large scale 316 package rebuilds, it is recommended to use the SDK images and the devroots in 317 combination with APT archives snapshots to rebuild packages in an 318 environment closely matching a past build. 311 Apertis uses [OBS](https://openbuildservice.org/) for building packages, 312 however since the purpose of the long term reproducibility does not involve 313 large scale package rebuilds, we recommend using the [Apertis 314 SDK]({{< ref "software-development-kit.md" >}}) for building packages. In addition to 315 the SDK a [devroot]({{< ref "sysroots-and-devroots/#devroot" >}}) and a snapshot 316 of the APT repositories are also required. -
What about this?
Apertis uses [OBS](https://openbuildservice.org/) to build packages in a predictable and clean environment, but at the moment there no automatic integration with the APT snapshots mechanism and packages are built only against the latest state of each repository. For this reason product teams have two different options to build their packages to ship fixes on top of an image build reproduced using the approach described in this document. For small hotfixes that involve no more than a couple of packages the [SDK]({{< ref "software-development-kit.md" >}}) and [devroot]({{< ref "sysroots-and-devroots/#devroot" >}}) matching the original build can be used to reproduce the right build environment and then manually build the involved packages. In this case, the developers have to take care of ensuring the build environment is clean. OBS can instead be used in case more packages are involved or creating a clean build environment manually is problematic, by asking the OBS administrators to set up dedicated projects for the activity. A Download-on-Demand (DoD) project needs to be set up sourcing packages from the APT snapshot matching the original build, and then a dedicated project host the packages to be built, using the newly-created DoD as a dependency.
316 of the APT repositories are also required. 317 318 We recommend the SDK release to match the release of Apertis in the target 319 device, and it is important that both the release and architecture of the 320 devroot and of the snapshot of the APT repositories matches the target device. 321 As an example, if the target is an `ARM64` device, the devroot and the snapshot 322 of the APT repositories should also be for `ARM64`. 323 324 See the [devroot]({{< ref "sysroots-and-devroots/#devroot" >}}) for more 325 information. 326 327 For the long term reproducibility we recommend the product team to have a 328 backup copy of: 329 * The Apertis SDK: Matching the Apertis release in use by the target device. 330 * The Apertis devroot: Matching the Apertis release and the architecture of 331 the target device. 322 of the APT repositories should also be for `ARM64`. 323 324 See the [devroot]({{< ref "sysroots-and-devroots/#devroot" >}}) for more 325 information. 326 327 For the long term reproducibility we recommend the product team to have a 328 backup copy of: 329 * The Apertis SDK: Matching the Apertis release in use by the target device. 330 * The Apertis devroot: Matching the Apertis release and the architecture of 331 the target device. 332 333 At the time of writing OBS supports native and emulated builds, and we have 334 successfully deployed emulated builds on the Bosch Developer Cloud. While 335 emulated builds are expected to work in most cases, they may require 336 changes to the packages being build using emulation, and they may produce 337 different results when compared to native builds. 337 different results when compared to native builds. 338 339 For more information about emulated and native builds see 340 [Build infrastructure on Intel x86-64]({{< ref "x86-build-infrastructure.md" >}}). 341 342 Building packages on the SDK uses emulation, and while this approach is 343 currently expected to be satisfactory, it may produce different results when 344 compared to native builds. In the unlikely event of emulated builds not being 345 satisfactory, we describe a mitigation on the section 346 [Future work on package builds]({{< ref "long-term-reproducibility.md#future-work-on-package-builds" >}}) 347 348 ### Future work on package builds 349 350 We recommend using the Apertis SDK for building packages in the context of the 351 long term reproducibility to enable product teams to build packages without 352 depending on custom OBS deployments. -
Why would they be "custom" deployments? And what do you mean with custom? Does that mean their own OBS setup or ?
Probably would be best to clarify that a bit.. That said for teams needing long-term reproducible builds you'd expect them to only have very specific gold images so manually configuring obs to have projects for those doesn't seem overly cumbersome (especially not at the current scale).
341 342 Building packages on the SDK uses emulation, and while this approach is 343 currently expected to be satisfactory, it may produce different results when 344 compared to native builds. In the unlikely event of emulated builds not being 345 satisfactory, we describe a mitigation on the section 346 [Future work on package builds]({{< ref "long-term-reproducibility.md#future-work-on-package-builds" >}}) 347 348 ### Future work on package builds 349 350 We recommend using the Apertis SDK for building packages in the context of the 351 long term reproducibility to enable product teams to build packages without 352 depending on custom OBS deployments. 353 354 However using OBS for building packages in the context of the long term 355 reproducibility may offer advantages, when large package rebuilds are needed or 356 when native builds are required. requested review from @em
-
@peter, would you have the chance to pick this one up again and drive it to the completion line, or should we officially reassign it? :D
removed review request for @em
assigned to @wlozano