Image building infrastructure documentation

Document the high-level concept and infrastructure setup required for
debos based image building

Apertis: https://phabricator.apertis.org/T4237Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.com>
Differential Revision: https://phabricator.apertis.org/D7511

docs/image-build-infrastructure.md

+---
+title: Image building infrastructure
+short-description: Overview of the image build infrastructure for Apertis
+authors:
+    - name: Sjoerd Simons
+...
+
+# Apertis image build infrastructure overview
+
+## Introduction
+
+The Apertis infrastructure supports continuous building of reference images,
+hwpacks and ospacks. This document explains the infrastructure setup,
+configuration and concepts.
+
+## Technology overview
+
+To build the various packs (hardware, os) as well as images, Apertis uses
+[Debos](https://github.com/go-debos/debos), a flexible tool to configure the
+build of Debian-based operating systems. Debos uses tools like `debootstrap`
+already present in the environment and relies on virtualisation to securely do
+privileged operations without requiring root access.
+
+For orchestrating Apertis uses the well-known [Jenkins](https://jenkins.io)
+automation server. Following current best practices the Apertis image build
+jobs use Jenkins pipelines (introduced in Jenkins 2.0) to drive the build
+process as well as doing the actual build inside
+[Docker images](https://jenkins.io/doc/book/pipeline/docker/) to allow for
+complete control of the job specific build-environment without relying on
+job-specific Jenkins slave configuration. As an extra bonus the Docker images
+used by Jenkins can be re-used by developers for local testing in the same
+environment.
+
+For each Apertis release there are two relevant Jenkins jobs to build images;
+The first job builds a Docker image which defines the build environment and
+uploads the resulting image to the Apertis Docker registry. This is defined in
+the
+[apertis-docker-images git repository](https://git.apertis.org/cgit/apertis-docker-images.git/).
+The second job defines the build steps for the various ospacks, hardware packs and
+images which are run in the Docker image build by the previous job; it also
+uploads the results to images.apertis.org.
+
+## Jenkins master setup
+
+Instructions to install Jenkins can be can be found on the
+[Jenkins download page](https://jenkins.io/download/). Using the Long-Term
+support version of Jenkins is recommended. For the Apertis infrastructure
+Jenkins master is being run on Debian 9.3 (stretch).
+
+The plugins that are installed on the master can be found in the [plugins
+appendix][Appendix: List of plugins installed on the Jenkins master]
+
+## Jenkins slave setup
+
+Each Jenkins slave should be installed on a separate machine (or VM) in line
+with the Jenkins best practices. As the image build environment is contained in
+a Docker image, the Jenkins slave requires only a few tools to be installed.
+Apart from running a Jenkins slave itself, the following requirements must be
+satisfied on slave machines:
+* git client installed on the slave
+* Docker installed on the slave and usable by the Jenkins slave user
+* /dev/kvm accessible by the Jenkins slave user (for hw acceleration support in
+  the image builder)
+
+For the last requirement on Debian systems this can be achieved by dropping a
+file called /etc/udev/rules.d/99-kvm-perms.rules in place with the following
+content.
+```
+SUBSYSTEM=="misc", KERNEL=="kvm", GROUP="kvm", MODE="0666"
+```
+
+Documentation for installing Docker on Debian can be found as  part of the
+[Docker documentation](https://docs.docker.com/install/linux/docker-ce/debian/).
+To allow Docker to be usable by Jenkins, the Jenkins slave user should be
+configured as part of the `docker` group.
+
+Documentation on how to setup Jenkins slaves can be found as part of the
+[Jenkins documentation](https://wiki.jenkins.io/display/JENKINS/Distributed+builds).
+
+## Docker registry setup
+
+To avoid building Docker images for every image build round and to make it
+easier for Jenkins and developers to share the same Docker environment for
+build testing, it is recommended to run a Docker registry. The
+[Docker registry documentation](https://docs.docker.com/registry/deploying/)
+contains information on how to setup a registry.
+
+## Docker images for the build environment
+
+The Docker images defining the environment for building the images can be found
+in the
+[apertis-docker-images git repository](https://git.apertis.org/cgit/apertis-docker-images.git/).
+
+The toplevel Jenkinsfile is setup to build a Docker image
+based on the [Dockerfile](https://docs.docker.com/engine/reference/builder/)
+defined in the Apertis-image-builder directory and upload the result to the
+public Apertis Docker registry.
+
+For Apertis derivatives this file should be adjusted to upload the Docker image
+to the Docker registry of the derivative.
+
+## Image building process
+
+The image recipes and configuration can be found in the
+[apertis-image-recipes git repository](https://git.Apertis.org/cgit/apertis-image-recipes.git/).
+As with the Docker images, the top-level `Jenkinsfile` defines
+the Jenkins job. For each image type to be built a parallel job is started
+which runs the image-building toolchain in the Docker-defined environment.
+
+The various recipes provide the configuration for debos, documentation about
+the available actions can be found in the
+[Debos documentation](https://godoc.org/github.com/go-debos/debos/actions).
+
+## Appendix: List of plugins installed on the Jenkins master
+At the time of this writing the following plugins are installed on the Apertis
+Jenkins master:
+
+* ace-editor
+* ant
+* antisamy-markup-formatter
+* apache-httpcomponents-client-4-api
+* artifactdeployer
+* authentication-tokens
+* blueocean
+* blueocean-autofavorite
+* blueocean-bitbucket-pipeline
+* blueocean-commons
+* blueocean-config
+* blueocean-core-js
+* blueocean-dashboard
+* blueocean-display-url
+* blueocean-events
+* blueocean-github-pipeline
+* blueocean-git-pipeline
+* blueocean-i18n
+* blueocean-jira
+* blueocean-jwt
+* blueocean-personalization
+* blueocean-pipeline-api-impl
+* blueocean-pipeline-editor
+* blueocean-pipeline-scm-api
+* blueocean-rest
+* blueocean-rest-impl
+* blueocean-web
+* bouncycastle-api
+* branch-api
+* build-flow-plugin
+* buildgraph-view
+* build-name-setter
+* build-pipeline-plugin
+* build-token-root
+* cloudbees-bitbucket-branch-source
+* cloudbees-folder
+* cobertura
+* command-launcher
+* conditional-buildstep
+* copyartifact
+* credentials
+* credentials-binding
+* cvs
+* display-url-api
+* docker-commons
+* docker-custom-build-environment
+* docker-workflow
+* durable-task
+* email-ext
+* embeddable-build-status
+* envinject
+* envinject-api
+* external-monitor-job
+* favorite
+* git
+* git-client
+* github
+* github-api
+* github-branch-source
+* github-organization-folder
+* git-server
+* git-tag-message
+* handlebars
+* htmlpublisher
+* hudson-pview-plugin
+* icon-shim
+* jackson2-api
+* javadoc
+* jenkins-design-language
+* jira
+* jquery
+* jquery-detached
+* jsch
+* junit
+* ldap
+* mailer
+* mapdb-api
+* matrix-auth
+* matrix-project
+* mattermost
+* maven-plugin
+* mercurial
+* metrics
+* modernstatus
+* momentjs
+* multiple-scms
+* pam-auth
+* parameterized-trigger
+* phabricator-plugin
+* pipeline-build-step
+* pipeline-github-lib
+* pipeline-graph-analysis
+* pipeline-input-step
+* pipeline-milestone-step
+* pipeline-model-api
+* pipeline-model-declarative-agent
+* pipeline-model-definition
+* pipeline-model-extensions
+* pipeline-rest-api
+* pipeline-stage-step
+* pipeline-stage-tags-metadata
+* pipeline-stage-view
+* plain-credentials
+* pollscm
+* promoted-builds
+* publish-over
+* publish-over-ssh
+* pubsub-light
+* repo
+* resource-disposer
+* run-condition
+* scm-api
+* script-security
+* sse-gateway
+* ssh-agent
+* ssh-credentials
+* ssh-slaves
+* structs
+* subversion
+* timestamper
+* token-macro
+* translation
+* variant
+* versionnumber
+* view-job-filters
+* windows-slaves
+* workflow-aggregator
+* workflow-api
+* workflow-basic-steps
+* workflow-cps
+* workflow-cps-global-lib
+* workflow-durable-task-step
+* workflow-job
+* workflow-multibranch
+* workflow-scm-step
+* workflow-step-api
+* workflow-support
+* ws-cleanup

sitemap.txt

@@ -12,6 +12,7 @@ index.md
 	geolocation-and-navigation.md
 	global-search.md
 	hardkeys.md
+	image-build-infrastructure.md
 	internationalization.md
 	inter-domain-communication.md
 	list.md