Skip to content
Snippets Groups Projects
Commit afc12ef6 authored by Martyn Welch's avatar Martyn Welch Committed by Frederic Danis
Browse files

Improve spelling and formatting of the long term reproducibility document 


The long term reproducibility document doesn't utilise the formatting
available in Hugo overly well and contains a number of uses of UK English
and terms that aren't widely used.

Tweak the document to use US spelling and more common terms. Make use of
Hugo formatting capabilities where sensible.

Signed-off-by: default avatarMartyn Welch <martyn.welch@collabora.com>
parent 864f413f
No related branches found
No related tags found
1 merge request!155Start making some spelling improvements on website
Hide whitespace changes
Inline Side-by-side
content/architecture/long-term-reproducibility.md
+ 98
− 87
View file @ afc12ef6
......@@ -13,7 +13,7 @@ date = "2020-01-29"
# Background
One of the main goals for Apertis is to provide teams the tools to support
their products for long lifecycles needed in many industries, from civil
their products for long life cycles needed in many industries, from civil
infrastructure to automotive.
This document discusses some of the challenges related to long-term support and
......@@ -21,9 +21,10 @@ how Apertis addresses them, with particular interest in reliably reproducing
builds over a long time span.
Apertis addresses that need by providing stable release channels as a platform
for products with a clear trade-off between updateness and stability. Apertis
encourages products to track these channels closely to deploy updates on a
regular basis to ensure important fixes reach devices in a timely manner.
for products with a clear trade-off between leading-edge functionality and
stability. Apertis encourages products to track these channels closely to
deploy updates on a regular basis to ensure important fixes reach devices in a
timely manner.
Stable release channels are supported for at least two years, and product teams
have three quarters of overlap to rebase to the next release before the old one
......@@ -41,13 +42,13 @@ involve things like timestamps or items being listed differently in places
where order is not significant, cause builds to not be bit-by-bit identical
while the runtime behavior is not affected.
# Apertis artefacts and release channels
# Apertis artifacts and release channels
As described in the [release flow]( {{< ref "release-flow.md" >}} ) document, at any given time Apertis
has multiple active release channels to both provide a stable foundation for
product teams and also give them full visibility on the latest developments.
Each release channel has its own artefacts, the main one being the
Each release channel has its own artifacts, the main one being the
[deployable images]({{< ref "images.md" >}}) targeting the [reference
hardware platforms](https://www.apertis.org/reference_hardware/), which get
built by mixing:
......@@ -55,9 +56,9 @@ built by mixing:
* reproducible build environments
* build recipes
* packages
* external artefacts
* external artifacts
These inputs are also artefacts themselves in moderately complex ways:
These inputs are also artifacts themselves in moderately complex ways:
* build environments are built by mixing dedicated recipes and packages
* packages are themselves built using dedicated reproducible build environments
......@@ -78,7 +79,7 @@ repositories and to any external data.
To track which input got used during the build process the pipeline stores
an identifier for each of them to uniquely identify them. For instance,
the pipeline saves all the git commit hashes, Docker image hashes, and
the pipeline saves all the Git commit hashes, Docker image hashes, and
package versions in the output metadata.
![](/images/pipeline-input-output-tracking.svg)
......@@ -109,11 +110,11 @@ external resources to keep the impact of the environment as low as possible.
For the most critical components, even the container images themselves are
created using Apertis resources, minimizing the reliance on any external
service and artefacts.
service and artifacts.
For instance, the `apertis-v2020-image-builder` container image provides
the reproducible environment to run the pipelines building the reference
image artefacts for the v2020 release, and the
image artifacts for the v2020 release, and the
`apertis-v2020-package-source-builder` container image is used to convert the
source code stored in GitLab in a format suitable for building on OBS.
......@@ -137,7 +138,7 @@ but not exactly the original one gives acceptable results.
To further make build environments more reproducible, care can be taken to
make their own build process as reproducible as possible.
The same concerns affecting the main build recipes affect the recipes for the
Docker images, from storing pipelines in git, to relying only on snapshotted
Docker images, from storing pipelines in Git, to relying only on snapshotted
package archives, to taking extra care on third-party downloads, and the
following sections address those concerns for both the build environments
and the main build process.
......@@ -146,15 +147,15 @@ and the main build process.
The process to the reference images is described by textual, YAML-based
[Debos recipes](https://gitlab.apertis.org/infrastructure/apertis-image-recipes/)
git repository, with a different branch for each release channel.
Git repository, with a different branch for each release channel.
The textual, YAML-based GitLab-CI pipeline definitions then control how the
recipes are invoked and combined.
Relying on git for the definition of the build pipelines make preserving old
Relying on Git for the definition of the build pipelines make preserving old
versions and tracking changes over time trivial.
Rebuilding the `v2020` artefacts locally is then a matter of checking out the
Rebuilding the `v2020` artifacts locally is then a matter of checking out the
recipes in the `apertis/v2020` branch and launching `debos` from a container
based on the `apertis-v2020-image-builder` container image.
......@@ -165,16 +166,16 @@ easily with any desired customization under the control of the developer.
The large majority of the software components shipped in Apertis are
packaged using the Debian packaging format, with the source code stored in
GitLab that OBS uses to generate pre-built binaries to be published in a
GitLab that OBS uses to generate prebuilt binaries to be published in a
APT-compatible repository.
Separate git branches and OBS projects are used to track packages and versions
Separate Git branches and OBS projects are used to track packages and versions
across different parallel releases, see the[release flow]( {{< ref "release-flow.md" >}} ) document for
more details.
For instance, for the v2020 stable release:
* the `apertis/v2020` git branch tracks the source revisions to be landed in the
* the `apertis/v2020` Git branch tracks the source revisions to be landed in the
main OBS project
* the `apertis:v2020:{target,development,sdk}` projects build the stable
packages
......@@ -189,13 +190,13 @@ and updates that also address important but non security-related issues which
usually benefit from more testing.
For security updates:
* the git branch is `apertis/v2020-security`
* the Git branch is `apertis/v2020-security`
* the OBS projects are `apertis:v2020:security:{target,development,sdk}`
* `deb https://repositories.apertis.org/apertis/ v2020-security target development sdk`
is the APT repository
Similarly, for the general updates:
* the git branch is `apertis/v2020-updates`
* the Git branch is `apertis/v2020-updates`
* the OBS projects are `apertis:v2020:updates:{target,development,sdk}`
* `deb https://repositories.apertis.org/apertis/ v2020-updates target development sdk`
is the APT repository
......@@ -233,51 +234,51 @@ the full history of each archive.
More advanced use-cases can be addressed using the optional
[Aptly HTTP API](https://www.aptly.info/doc/api/).
## External artefacts
## External artifacts
While the packaging pipeline effectively forbids any reliance on external
artefacts, the other pipelines in some case include components not under the
artifacts, the other pipelines in some case include components not under the
previously mentioned systems to track per-release resources.
For instance, the recipes for the HMI-enabled images include a set of
example media files retrieved from a `multimedia-demo.tar.gz` file hosted on
an Apertis webserver.
an Apertis web server.
Another example is given by the `apertis-image-builder` recipe checking out
Debos directly from the master branch on GitHub.
In both cases, any change on the external resources impacts directly all the
release channels when building the affected artefacts.
release channels when building the affected artifacts.
A minimal solution for `multimedia-demo.tar.gz` would be to put a version in its
URL, so that recipes can be updated to download new versions without affecting
older recipes. Even better, its contents could be put in a version tracking
tool, for instance using the Git-LFS support available on GitLab.
tool, for instance using the Git LFS support available on GitLab.
In the Debos case it would be sufficient to encode in the recipe a specific
revision to be checked out. A more robust solution would be to use the packaged
version shipped in the Apertis repositories.
## Main artefacts and metadata
## Main artifacts and metadata
The purpose of the previously described software items is to generate a set of
artefacts, such as those described on the [images]( {{< ref "images.md" >}} )
page. With the artefacts themselves a few metadata entries are generated to
artifacts, such as those described on the [images]( {{< ref "images.md" >}} )
page. With the artifacts themselves a few metadata entries are generated to
help tracking what has been used during the build.
In particular, the `pkglist` files capture the full list of packages installed
on each artefacts along their version. The `filelist` files instead provide
basic information about the actual files in each artefacts.
on each artifacts along their version. The `filelist` files instead provide
basic information about the actual files in each artifacts.
With the information contained in the `pkglist` files it is possible to find
the exact binary package version installed and from there find the
corresponding commit for the sources stored in GitLab by looking at the
matching git tag.
matching Git tag.
The `build-env.txt` file instead captures metadata about the build environment.
For instance, here's a sample from the pipeline that
[built the v2021dev3.0 release](https://images.apertis.org/release/v2021dev3/v2021dev3.0/meta/build-env.txt):
[built the `v2021dev3.0` release](https://images.apertis.org/release/v2021dev3/v2021dev3.0/meta/build-env.txt):
PIPELINE_VERSION=20200921.1223
DOCKER_IMAGE=registry.gitlab.apertis.org/infrastructure/apertis-docker-images/v2021dev3-image-builder@sha256:50724ec3105f9ea840fa70b536768148722ae59e09b7861a9051ad1397b57f64
......@@ -313,7 +314,7 @@ always builds every package on a clean, isolated environment built using the
latest package versions for each channel.
Since the purposes taken in account in this document do not involve large scale
package rebuilds, it is recommended to use the SDK images and the devroots in
package rebuilds, it is recommended to use the SDK images and the deviants in
combination with the snapshotted APT archives to rebuild packages in an
environment closely matching a past build.
......@@ -369,17 +370,17 @@ Due to the way APT works, the increase in storage costs for the snapshot is
small, as the duplication is limited to the index files, while the package
contents are deduplicated.
## Version control external artefacts
## Version control external artifacts
External artefacts like the sample multimedia files need to be versioned just
like all the other components. Using Git-LFS and git tags would give fine
External artifacts like the sample multimedia files need to be versioned just
like all the other components. Using Git LFS and Git tags would give fine
control to the build recipe over what gets downloaded.
## Link to the tagged sources
The package name and package version as captured in the `pkglist` files are
sufficient to identify the exact sources used to generate the packages
installed on each artefacts, as they can be used to identify an exact commit.
installed on each artifacts, as they can be used to identify an exact commit.
However, the process can be further automated by providing explicit hyperlinks
to the tagged revision on GitLab.
......@@ -388,7 +389,7 @@ to the tagged revision on GitLab.
## Reproduce the build
1. Open the folder containing the build artefacts, for instance
1. Open the folder containing the build artifacts, for instance
[`v2021dev3.0/`](https://images.apertis.org/release/v2021dev3/v2021dev3.0/)
1. Find the `build-env.txt` metadata, for instance
[`meta/build-env.txt`](https://images.apertis.org/release/v2021dev3/v2021dev3.0/meta/build-env.txt)
......@@ -401,7 +402,7 @@ to the tagged revision on GitLab.
[new branch](https://docs.gitlab.com/ee/gitlab-basics/create-branch.html) in
the recipes repository pointing to the commit saved in the `RECIPES_COMMIT`
field of `build-env.txt`, for instance commit
[b4f1c5c85bd4603f2d9158f513c142a77a3c65c3](https://gitlab.apertis.org/infrastructure/apertis-image-recipes/commit/b4f1c5c85bd4603f2d9158f513c142a77a3c65c3)
[`b4f1c5c85bd4603f2d9158f513c142a77a3c65c3`](https://gitlab.apertis.org/infrastructure/apertis-image-recipes/commit/b4f1c5c85bd4603f2d9158f513c142a77a3c65c3)
1. Go to `Pipelines` → `Run Pipeline` page on GitLab to
[execute a CI pipeline](https://docs.gitlab.com/ee/ci/pipelines.html#manually-executing-pipelines)
1. [Configure a variable](https://docs.gitlab.com/ee/ci/variables/README.html#create-a-custom-variable-in-the-ui)
......@@ -411,7 +412,7 @@ to the tagged revision on GitLab.
it is recommended to set a custom one
1. Run the pipeline
When the pipeline completes, the produced artefacts should closely match the
When the pipeline completes, the produced artifacts should closely match the
original ones, albeit not being bit-by-bit identical.
## Customizing the build
......@@ -440,12 +441,12 @@ OpenSSL?
It is important for product teams to consider their future requirements at the
point they make a release. To ensure bug and security fixes can be deployed
with minimal impact on users a number of artefacts need to be preserved from
with minimal impact on users a number of artifacts need to be preserved from
the initial release:
1. The image recipes
1. The Docker images used as build environment
1. The APT repositories
1. External artefacts
1. External artifacts
## Getting started with Apertis: one year before release 1.0.0
......@@ -459,7 +460,7 @@ the product team control over importing changes from Apertis, such as package
updates. We recommend using release channels for that.
A product team can have multiple release channels, each reflecting what is
deployed for an specific product. And because release channels are independent
deployed for a specific product. And because release channels are independent
and parallel deliveries, a single product may even have multiple release
channels, for instance a stable channel and a development one.
......@@ -474,9 +475,9 @@ each release channel has its own set of projects on OBS. However only the compon
that the product team need to customize have to be branched or forked. To
maximize reuse, it is expected that the bulk of packages used by every product
team will come directly from the main Apertis release channels.
1. What: Create a dedicated release channel
1. Where: GitLab and OBS
1. How: Create release channel branches in each git repository that diverges
1. **What**: Create a dedicated release channel
1. **Where**: GitLab and OBS
1. **How**: Create release channel branches in each Git repository that diverges
from the ones provided by Apertis; set up OBS projects matching those
release channels to build the packages
......@@ -499,8 +500,10 @@ this example we are going to recommend the product team to create a new APT
suite and a few APT components, and host them on the Apertis infrastructure. We
will call the new suite cluster-v1. The list of APT repositories will then
be:
deb https://repositories.apertis.org/apertis/ v2020 target development sdk
deb https://repositories.apertis.org/automotive/ cluster-v1 target
```
deb https://repositories.apertis.org/apertis/ v2020 target development sdk
deb https://repositories.apertis.org/automotive/ cluster-v1 target
```
For reference, in [APT
terminology](https://manpages.debian.org/testing/apt/sources.list.5.en.html)
......@@ -508,8 +511,8 @@ both `v2020` and `cluster-v1` are suites or distributions, and `target`,
`development`, and `sdk` are components.
The steps are:
1. What: Create new APT suite and APT components for the product team
1. Where to host: Apertis infrastructure
1. **What**: Create new APT suite and APT components for the product team
1. **Where to host**: Apertis infrastructure
## Creating the list of golden components: the day of the release 1.0.0
......@@ -520,22 +523,26 @@ recommend careful evaluation as most tools allow tags to be modified after
creation. Modifying tags can lead to problems that are difficult to debug.
The image recipe is usually a small set of files that are stored in a single
git repository. Collect the hash of the latest commit of the recipe repository.
1. What: Image recipe
1. Where: Apertis GitLab
1. How: Collect the git hash of the latest commit of the recipe files
Git repository. Collect the hash of the latest commit of the recipe repository.
1. **What**: Image recipe
1. **Where**: Apertis GitLab
1. **How**: Collect the Git hash of the latest commit of the recipe files
The Docker containers used for building are stored in GitLab Container
Registry. The Registry also allow to identify containers by hashes. A note of
caution: There are expiration policies and clean-up tools for deleting old
versions of containers. Make sure the golden containers are protected against
clean-up and expiration.
1. What: Docker containers used for building: apertis-v2020-image-builder and
apertis-v2020-package-source-builder
1. Where: GitLab Container Registry
1. How: On the GitLab Container Registry collect the hash for each container used
for building
1. Do not forget: Make sure the expiration policy and clean-up routines will
Registry. The Registry also allow to identify containers by hashes.
{{% notice warning %}}
There are expiration policies and clean-up tools for deleting old versions of
containers. Make sure the golden containers are protected against clean-up and
expiration.
{{% /notice %}}
1. **What**: Docker containers used for building: `apertis-v2020-image-builder`
and `apertis-v2020-package-source-builder`
1. **Where**: GitLab Container Registry
1. **How**: On the GitLab Container Registry collect the hash for each
container used for building
1. **Do not forget**: Make sure the expiration policy and clean-up routines will
not delete the golden containers
From the perspective of APT clients, such as the tools used to create Apertis
......@@ -543,42 +550,46 @@ images, APT repositories are simply a collection of static files served through
the web. The recommended method for creating the golden set of APT repositories
is to create snapshots using `aptly`. Aptly is used by Debian upstream and is
capable of making efficient use of disk space for snapshots. aptly snapshots
are identified by tags. Something in the lines of `aptly snapshot create v1.0.0
from mirror target`, and repeat the command for `target`, `development`, `sdk`,
and `cluster-v1`.
are identified by tags. Something along the lines of:
aptly snapshot create v1.0.0 from mirror target
Repeat the command for `target`, `development`, `sdk`, and `cluster-v1`.
It is important to mention that the product team needs to create a snapshot
**every time a package is updated**. This is the only way to keep track the
full history of the APT archive.
1. What: APT repositories:
1. **What**: APT repositories:
```
deb https://repositories.apertis.org/apertis/ v2020 target development sdk
deb https://repositories.apertis.org/automotive/ cluster-v1 target
1. Where: aptly
1. How: create a snapshot for each repository using aptly
1. Do not forget: create a snapshot for every package update
External artefacts should be avoided, but some times they are required. An
example of external artefacts are the multimedia files Apertis uses for
testing. Those files are currently simply hosted on a webserver which creates
```
1. **Where**: aptly
1. **How**: create a snapshot for each repository using aptly
1. **Do not forget**: create a snapshot for every package update
External artifacts should be avoided, but some times they are required. An
example of external artifacts are the multimedia files Apertis uses for
testing. Those files are currently simply hosted on a web server which creates
two problems: no versioning information, and no long term guarantee of
availability.
To address this issue we recommend creating a repository on GitLab, and copy
all external artefacts to it. This gives the benefit of using the well defined
all external artifacts to it. This gives the benefit of using the well defined
processes around versioning and tracking that are already used by the other
components. For large files we recommend using git-lfs.
1. What: External artefacts: files that are needed during the build but that are
not in git repositories
1. Where: A new repository in GitLab
1. How: Create a GitLab repository for external artefacts, add files, use
git-lfs for large files, and collect the hash pointing to the correct
components. For large files we recommend using Git LFS.
1. **What**: External artifacts: files that are needed during the build but that are
not in Git repositories
1. **Where**: A new repository in GitLab
1. **How**: Create a GitLab repository for external artifacts, add files, use
Git LFS for large files, and collect the hash pointing to the correct
version of files
Notice that the main idea is to collect hashes for the various resources used
for building. The partial exception are external resources, but our suggestion
is to also create a git repository for hosting the external artefacts and then
collect and use the git hash as a pointer to the correct version of the
is to also create a Git repository for hosting the external artifacts and then
collect and use the Git hash as a pointer to the correct version of the
content.
At the time of writing there is work planned to automate the collection of
......@@ -621,16 +632,16 @@ The `build-env.txt` produced by the build pipeline should capture all the
information needed to reproduce it as closely as possible:
1. Retrieve the `build-env.txt` from the golden build
1. On Gitlab
1. On GitLab
[create a new branch](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-new-branch-from-a-projects-dashboard)
on the previously identified recipe repository. The branch should point to the
golden commit which should be captured in the `RECIPES_COMMIT` field.
1. [Execute a CI pipeline](https://docs.gitlab.com/ee/ci/pipelines.html#manually-executing-pipelines)
on the newly created branch, reproducing or customising the original build
on the newly created branch, reproducing or customizing the original build
environment by creating a variable called `BUILD_ENV_OVERRIDE` into which the
contents from `build-env.txt` should be pasted, modifying it as desired.
When the pipeline completes, the produced artefacts should closely match the
When the pipeline completes, the produced artifacts should closely match the
original ones, albeit not being bit-by-bit identical.
## Customizing the build
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
Apertis Website Terms of Use Privacy Policy