From 3af6ceb19c84f2a96e3ab562f0090289a3111fd2 Mon Sep 17 00:00:00 2001
From: mtt2hi <thomas.mittelstaedt@de.bosch.com>
Date: Thu, 20 May 2021 06:15:07 +0000
Subject: [PATCH] Initial version von vendorkernelguide.md

Signed-off-by: mtt2hi <thomas.mittelstaedt@de.bosch.com>
---
 content/guides/vendorkernelguide.md | 385 ++++++++++++++++++++++++++++
 1 file changed, 385 insertions(+)
 create mode 100644 content/guides/vendorkernelguide.md

diff --git a/content/guides/vendorkernelguide.md b/content/guides/vendorkernelguide.md
new file mode 100644
index 000000000..3d1d1947d
--- /dev/null
+++ b/content/guides/vendorkernelguide.md
@@ -0,0 +1,385 @@
++++
+date = "2020-05-12"
+weight = 100
+
+title = "Evaluating Vendor Kernels for Use In Apertis"
++++
+
+Often target platforms are not supported by Apertis. This may be due to the way
+the Apertis components are built/configured or due to these components lacking
+support for the required hardware. Whilst they may lack support in Apertis, these
+boards frequently have reference images based on another Linux variant with the
+associated source code being available. This support is often provided in a
+Yocto-based deliverable. The support generally provides good coverage of
+the vendors hardware, however the quality of implementation can vary
+quite dramatically.
+
+When evaluating a platform for use in a product, it is important to evaluate both 
+the technical and business related factors to determine whether it represents a cost effective solution. 
+This document aims to:
+
+- highlight deficiencies commonly found in vendor hardware support, 
+- provide some ways in which the quality of vendor provided support can be quantified,
+- suggestions to help with evaluating the changes to be integrated into Apertis, and
+
+
+# Hardware packages and OSpacks
+
+[Hardware packages]({{< ref "hwpack-requirements.md#hwpack" >}}) (HWpacks) provide
+tailored Linux kernels, bootloaders and any specific drivers required to enable Apertis
+[OSpacks]({{< ref "hwpack-requirements.md#ospack" >}}) to be run on target boards.
+Apertis provides hardware packages that enable it to be used on several
+[reference boards]({{< ref "reference_hardware" >}}) across a number of architectures.
+
+This document focuses on the Linux kernel as it is expected that a version of the Linux kernel will be required by all Apertis users. Whilst a Bootloader will be required for a device, these can vary far more and thus do not provide such a good example. 
+
+# Common vendor kernel issues
+
+Whilst the software provided by the vendor typically does a good job of
+showcasing their hardware it is typically not provided in a form that
+would be considered to follow best practice. Common issues are outlined
+below.
+
+## Limited enablement of hardware and core functionality in provided configuration
+The configuration provided when building the kernel and other low-level
+components is generally tailored to just provide the support required to
+demo the vendors hardware. As a result this configuration is generally
+inappropriate for direct use in products as they will typically require
+drivers for hardware not provided on the vendors reference hardware and
+require additional functionality to be enabled to support application
+specific workloads.
+
+## Disregard for common code
+The vendor is typically only interested in providing working support for
+their hardware and this is frequently done in the most expedient manner
+possible. As a result both the parameters for and logic of common code, which
+exist to support multiple existing pieces of hardware, are modified without
+regard for the continued operation of the other hardware. This is problematic
+when attempting to utilise a common source tree for multiple projects,
+utilizing different hardware platforms, where the changes made for one
+platform break support implemented for others.
+
+This can sometimes result in build failures when attempting to enable
+hardware not found in the vendor provided configuration (see
+"[Limited Enablement of Hardware and Core Functionality In Provided Configuration]({{< ref "#limited-enablement-of-hardware-and-core-functionality-in-provided-configuration" >}})").
+
+## No "clean" patch series implementing the vendors changes
+Frequently the vendor will provide a git tree containing the kernel
+source and any target specific changes. The commits from which these
+changes are comprised are not usually provided as a clean set of changes on top
+of the mainline version (such as would be achieved by performing `git rebase`
+when shifting focus to a newer version of the underlying component), but are
+scattered through the commit history as subsequent mainline versions have been
+added through the use of `git merge`. This can make it difficult to generate
+a clean set of patches to apply the vendor BSP to the mainline kernel source
+of the same version and to port support to a newer kernel version to take advantage of security and bug fixes.
+
+## Unrelated changes
+The changes to the source code might not only be changes for the required
+processor/target board, but also other changes like:
+
+- Realtime patches
+- Some improvements from perspective of vendor not directly related to hardware support.
+- Addition of hardware support for other hardware provided by the vendor but which
+  is not of consequence for the projects target platform 
+
+These changes may be scattered throughout the commit history (see
+[No "clean" patch series implementing the vendors changes]({{< ref "#no-clean-patch-series-implementing-the-vendors-changes" >}})
+and typically lack useful commit messages to 
+enable the intention behind any given change from being understood. This makes resolving 
+conflicting changes between the newer mainline and BSP specific changes difficult to rectify.
+
+This leads to problems when upgrading with newer versions of the mainline kernel.
+
+
+## Compatibility with ospacks
+The Apertis operating system packages (ospacks) rely on the kernel with which
+they are paired to provide certain functionality. This functionality is
+expected to be provided using the common APIs found in the mainline kernel.
+Vendors sometimes either modify/extend or utilize custom APIs to expose
+functionality supported by their product. This combined with limitations in
+functionality enabled in the vendors kernel config can lead to the kernel not
+providing the functionality expected by the ospack, either in a form it can
+use or at all.
+
+## Lack of Mainline Support for Vendors Hardware
+The Apertis project recommends hardware support is submitted for inclusion in
+the mainline kernel. This results in:
+
+- Greater testing of common code.
+- Ability to easily generate a single binary, supporting multiple target boards
+  with a common architecture.
+- Best support for security updates due to being able to efficiently track the
+  active upstream code changes.
+- Best compatibility with Apertis.
+
+Many vendors do not actively work with the mainline kernel developers to get support
+for their hardware into the mainline kernel. The use of a vendor specific kernel
+reduces the upfront time and effort required to build a solution, but results in a
+significant increase in long term maintenance effort. Whilst more effort is required
+to achieve mainline kernel support, this effort frequently results in a higher
+quality of support being achieved than what would be expected from a vendor kernel.
+
+There are often a number of changes that need to be made before the mainline
+kernel supports a target board. Sometimes the mainline kernel can be used
+directly though typically the following additions are required (in increasing
+effort):
+
+- The kernel configuration has to be adopted
+- A new device tree (.DTB) has to be added
+- Board specific drivers have to be added
+- Addition of support for unsupported system on chip.
+
+The board specific drivers are available in the vendor specific kernel. With
+some kernel knowledge, the drivers can usually be ported to the mainline kernel
+(though this can be difficult for low quality vendor kernels). If the platform
+is important to you and the vendor has no interest in upstreaming support
+themselves, taking on the challenge of upstreaming support yourself may be
+beneficial in the long term.
+
+# Evaluating the quality of the vendor BSP
+With the increasing complexity and requirement for devices to be internet enabled,
+the old "embedded" approach of freezing and hardening a BSP no longer works. Any
+issues that are not detected during the hardening process may result in serious
+security vulnerabilities being left present in shipped devices. As a result there
+is an increasing requirement to perform "in the field" updates to shipped devices
+so that upgraded kernels and other system components to resolve security issues.
+
+Some vendors do not yet provide on going support for newer
+kernels, nor work to include support in mainline kernels. Some will use
+development practices that hinder the ability of a product team to perform these
+updates themselves when necessary. It is therefore imperative that the quality of
+the vendors software support be evaluated when selecting hardware for a project as this can
+drastically effect the economic viability of a vendors offering over the lifetime
+of a product.
+
+## Understanding Kernel Versioning
+Each mainline kernel is defined by its version number. This is typically
+in the form of 2 or 3 dot separated numbers, where these numbers are:
+
+    <major version>.<minor version>.<stable release>
+
+This may have other parts appended to it after a dash with these extra
+parts being used to denote "release candidate" kernels or vendor specific
+kernels.
+
+{{% notice note %}}
+This description holds true back to the beginning of the `3.x` series,
+kernels prior to that used a different convention, though we would not
+expect kernels of that vintage to be being used in active development
+owing to them being over 10 years old and thus will not cover that here.
+{{% /notice %}}
+
+The mainline kernel is released as a 2-number version, such as `5.3`. The
+minor version is incremented with each official release. Every so often
+the major version number is incremented and the minor is returned to zero.
+There is no defined point at which this occurs and it is at Linus Torvalds'
+discretion as to when this happens.
+
+Once a release is made, this is considered stable. Bugfixes will be applied
+to this until the next kernel release. Each bugfix release is released with
+a 3 number version, with the "stable release" part incrementing for each
+release. Some kernels will be supported for an extended period and are know
+as longterm stable (LTS) releases. Theses are the kernels that are used by
+Apertis.
+
+{{% notice info %}}
+More information regarding kernel release can be found on
+[kernel.org](https://www.kernel.org/releases.html).
+{{% /notice %}} 
+
+## Determine what kernel upgrades can be expected by the vendor
+The vendor support for upgrades to newer versions varies a lot. 
+Some vendors will do a single code drop with no support provided beyond that. 
+Others provide upgrades, but without offering an upgrade plan. Most of the time
+it will be months between updates, during which time the vendor kernel will lack
+security and bug fixes that may be found in the mainline and longterm stable kernels.
+
+## Age and maintenance state of the related mainline kernel
+Each vendor kernel is based on a mainline kernel. It is important to find out
+which kernel the vendors support is based on and determine it's age and it's
+support status. Wikipedia maintains a
+[full list](https://en.wikipedia.org/wiki/Linux_kernel_version_history) of
+release kernel versions, noting both their age and anticipated support "end
+of life" (EOL) dates, whilst kernel.org lists the
+[projected EOL dates](https://www.kernel.org/releases.html) for still supported
+LTS releases along with detailing the [latest release](https://www.kernel.org/)
+for each supported version. 
+
+When shipping a product, we do not want to be following the mainline
+development as a source of bug and security fixes as this will result in
+significant changes being pulled in when updating the kernel, which is likely
+to result in both unwanted changes to the features and performance on the
+target product and is likely to result in difficulties porting any vendor or
+product specific kernel modifications.
+
+As stable kernel releases aim to only include security and bug fixes, this
+results in significantly less changes to the code base, reducing the risk of
+feature or performance change and reducing or eliminating issues with porting
+modifications. From a long term maintenance perspective it is therefore
+beneficial for a product to use a vendor that bases their vendor kernel on
+the recent LTS kernel (one with sufficient time before it reaches it's end of
+life, as determined by the project time frames), even more so if they actively
+track the stable point releases.
+
+## Is a rebase to a longterm stable kernel needed?
+Since development of the mainline kernel is always going on, the quality of these
+parts varies dependant on maturity. Also security issues like vulnerabilties
+are handled during this development process with security fixes backported
+to the kernels supported stable releases.
+
+Apertis tracks this development effort to ensure the bug fixes, security fixes and
+feature improvements made during the mainline development process are available in
+Apertis, with each Apertis release including the latest
+[Longterm Stable kernel](https://www.kernel.org/releases.html) with point
+releases tracking changes on that branch. Any vendor BSP integrated into
+Apertis should also be able to update regularly to include these fixes and
+improvements, however many vendors do not provide kernel updates leaving it to
+their users to either perform these updates themselves or remain on older
+kernels versions with known security vulnerabilites.
+
+## Is the vendor working to integrate support into the mainline kernel?
+
+Even vendors that provide good on going support for their hardware via vendor trees lag behind 
+the fixes provided in the mainstream kernel, as they require these releases to be made before 
+porting any of their changes. Additionally, projects with long expected lifetimes may exceed 
+the period for which a vendor is willing to provide support.
+
+If a vendor "upstreams" support for their device, integrating that support right into the 
+mainline kernel, the vendor kernel is typically no longer required. A correctly configured 
+mainline kernel can be used, removing any lag and typically enabling the device to be used 
+long after the vendor stops actively supporting it without significant addition effort and cost.
+
+## Are there qualified test results for vendor BSP ?
+Some vendors provide tests and results for their specific images to check hardware and kernel features. 
+These tests usually cover specific hardware functions and less common features of the kernel. 
+So additional tests have to be planned and organized, probably got from mainline kernel. 
+Since the image is a new binary (compared to images of mainline kernel), these tests have to be redone best 
+at automatic test systems like LAVA (see also [Apertis integration testing with LAVA]({{< ref "lava-apertis-testing" >}})). 
+
+## How is the responsibility to maintain the vendor BSP organized ?
+Developing and purchasing a product includes product liability for the company over product lifetime. 
+This includes aspects for security (loss of personal data) and safety (risk to customers physically). 
+The Linux kernel potentially plays a very important role in both, thus ensuring bugs and
+security issues get fixed promptly and reliably is critical. These updates either have to be
+done by the product manufacturer or the supplier of the vendor providing the kernel support.
+As vendors will typically stop providing support after a defined period, the timing of this
+needs to be understood in relation to the products life time. If the vendors support period
+would end before that of the product, then it needs to be understood (and factored in) that
+at such a time it becomes the responsibility of the product team to organize such
+maintenance support.
+
+## Evaluate the Quality of Vendor Added Drivers
+Any drivers added by a vendor can be found by analyzing a diff between the
+vendor's kernel sources against the related mainline kernel source. Drivers
+should be fairly self contained under `drivers/`, with minor changes in:
+
+- Startup files
+- Makefile(s)
+- Kconfig(s)
+
+A deep review of implementation and testing of the relevant kernel driver is
+recommended. This is a time consuming task.
+
+# Practical checks 
+The following practical checks can be made to help determine the quality of the
+vendor supplied kernel source.
+
+## Check the Ability to Build Mainline Supported Devices
+Well integrated vendor support will result in a kernel that can still be built
+for other machines which share the same architecture and which are defined in
+the mainline kernel. Attempting to build using the configs provided in the
+`configs/` folder for your target architecture (such as `arch/arm/configs` for
+32-bit ARM devices). Build problems may indicate incomplete or unintended
+changes to common parts of the code.
+
+## Check the Ability to Build for Machines With Different Architectures
+This is similar to testing configs for other machines from your target
+architecture, but can point to additional issues where changes have been made
+to very common code in architecture specific ways. This will require a
+toolchain supporting the relevant architecture. Build problems may indicate
+incomplete or unintended changes to common parts of the code.
+
+## Attempting to build a multi-target kernel binary
+As long as sufficient similarities exist, it is possible to build a single
+kernel binary that can be used with multiple different machines with different
+peripherals and even processors.
+
+The kernel provides configs that build a kernel binary which can run on
+multiple different devices. An example is
+`arch/arm/configs/multi_v7_defconfig`, which will build a kernel for many ARMv7
+compatible devices. If such a configuration exists that may be compatible with
+the device your interested in, it may be instructive to attempt to add the
+target device to such a config and build it.
+
+The need to build separate kernel binaries may indicate, that some driver
+interfaces are not implemented optimally. This will lead to reduced test depth
+and increased build efforts.
+
+## Determining quantity and quality of vendor kernel mondifications
+It is typically not possible to utilise the git log to compare a vendor
+kernel to the mainline kernel from which it was built due to the previously
+mentioned development practices. As a result it is typical to need to utilise
+standard "diff" tools to make the comparison.
+
+The differences highlighted by these tools need to be examined to identify the
+changes relating to target hardware. It would be typical to see:
+
+- Hardware drivers and related header files
+- Files to integrate hardware drivers like Kconfig(s) or Makefile(s)
+- Addition of hooks to common files
+- Expanded support in shared architecture files.
+
+Thus the majority of the changes required to support a target device should be
+expected to be limited to those files providing drivers and support for the
+applicable architecture.
+
+It is important to identify changes in other areas, including those not related
+to the target hardware, such as:
+
+- Changes of significant complexity to common headers and source files
+- Addition of unrelated support, such as those provided by the real-time patch set
+- Unintended changes, e.g. changes of header files with contents of older
+kernel versions
+- Unrelated improvements (at least from viewpoint of the supplier of the vendor kernel)
+- Reformatting indenting and code style of entire files, masking subtle changes
+
+The bigger the changes unrelated to the target hardware and the more spread the
+changes are, the greater the effort that will be required to upgrade to newer
+kernel versions.
+
+## Attempt to build a kernel using the Debian config with Debian patches applied
+When adding kernel support for a device to Apertis, it is advised to base this
+on the Apertis kernel to enable you to easier take advantage of security
+updates. If the vendor kernel and the Apertis kernel are both based on a
+similar kernel, in order to test compatibility between the changes introduced
+by the vendor and those applied to the Apertis kernel it can be instructive to
+apply the Apertis patches to the vendor kernel and attempt to build with the
+Apertis kernel configuration (which is likely to have support for a broader
+number of features enabled).
+
+As is
+[customary with Debian packaging](https://wiki.debian.org/SimplePackagingTutorial),
+changes to an upstream source and any other files (such as the component
+configuration file) are stored in the `.debian.tar.xz` file. The relevant files
+for Apertis can be found on the
+[Apertis package repository](https://repositories.apertis.org/apertis/pool/target/l/linux/).
+
+Extracting this archive will create a `debian/` folder. The patches applied by
+Apertis can be found in `debian/patches` (the `series` file provides the order
+and enables the series to be quickly applied with tools such as `quilt`). The
+Debian configuration is split in to a generic portion (in
+`debian/config/config`), architecture configuration (in
+`debian/config/kernelarch-<arch>`) and more fine grained architecture config
+(in `debian/config/<arch>`), with a "full" configuration comprised of the
+options found in one of each of these sets.
+
+{{% notice note %}} While Debian builds kernels for many different
+architectures (and thus configuration files are present for many
+architectures), Apertis limits it's builds to `armhf`, `arm64` and `amd64` and
+thus it is advised to limit testing to those architectures.  {{% /notice %}}
+
+Applying these patches and building the kernel with this kernel configuration
+can highlight problems in common parts, which may make upgrades difficult
+without loss of functionality.
+
-- 
GitLab