Skip to content
Snippets Groups Projects
Commit 3af6ceb1 authored by Thomas Mittelstädt's avatar Thomas Mittelstädt
Browse files

Initial version von vendorkernelguide.md 


Signed-off-by: default avatarmtt2hi <thomas.mittelstaedt@de.bosch.com>
parent 2b2a5037
No related branches found
No related tags found
1 merge request!238Initial version von vendorkernelguide.md
Pipeline #252867 passed
Stage: generate-html
Stage: generate-pdf
Stage: deploy
Hide whitespace changes
Inline Side-by-side
content/guides/vendorkernelguide.md 0 → 100644
+ 385
− 0
View file @ 3af6ceb1
+++
date = "2020-05-12"
weight = 100
title = "Evaluating Vendor Kernels for Use In Apertis"
+++
Often target platforms are not supported by Apertis. This may be due to the way
the Apertis components are built/configured or due to these components lacking
support for the required hardware. Whilst they may lack support in Apertis, these
boards frequently have reference images based on another Linux variant with the
associated source code being available. This support is often provided in a
Yocto-based deliverable. The support generally provides good coverage of
the vendors hardware, however the quality of implementation can vary
quite dramatically.
When evaluating a platform for use in a product, it is important to evaluate both
the technical and business related factors to determine whether it represents a cost effective solution.
This document aims to:
- highlight deficiencies commonly found in vendor hardware support,
- provide some ways in which the quality of vendor provided support can be quantified,
- suggestions to help with evaluating the changes to be integrated into Apertis, and
# Hardware packages and OSpacks
[Hardware packages]({{< ref "hwpack-requirements.md#hwpack" >}}) (HWpacks) provide
tailored Linux kernels, bootloaders and any specific drivers required to enable Apertis
[OSpacks]({{< ref "hwpack-requirements.md#ospack" >}}) to be run on target boards.
Apertis provides hardware packages that enable it to be used on several
[reference boards]({{< ref "reference_hardware" >}}) across a number of architectures.
This document focuses on the Linux kernel as it is expected that a version of the Linux kernel will be required by all Apertis users. Whilst a Bootloader will be required for a device, these can vary far more and thus do not provide such a good example.
# Common vendor kernel issues
Whilst the software provided by the vendor typically does a good job of
showcasing their hardware it is typically not provided in a form that
would be considered to follow best practice. Common issues are outlined
below.
## Limited enablement of hardware and core functionality in provided configuration
The configuration provided when building the kernel and other low-level
components is generally tailored to just provide the support required to
demo the vendors hardware. As a result this configuration is generally
inappropriate for direct use in products as they will typically require
drivers for hardware not provided on the vendors reference hardware and
require additional functionality to be enabled to support application
specific workloads.
## Disregard for common code
The vendor is typically only interested in providing working support for
their hardware and this is frequently done in the most expedient manner
possible. As a result both the parameters for and logic of common code, which
exist to support multiple existing pieces of hardware, are modified without
regard for the continued operation of the other hardware. This is problematic
when attempting to utilise a common source tree for multiple projects,
utilizing different hardware platforms, where the changes made for one
platform break support implemented for others.
This can sometimes result in build failures when attempting to enable
hardware not found in the vendor provided configuration (see
"[Limited Enablement of Hardware and Core Functionality In Provided Configuration]({{< ref "#limited-enablement-of-hardware-and-core-functionality-in-provided-configuration" >}})").
## No "clean" patch series implementing the vendors changes
Frequently the vendor will provide a git tree containing the kernel
source and any target specific changes. The commits from which these
changes are comprised are not usually provided as a clean set of changes on top
of the mainline version (such as would be achieved by performing `git rebase`
when shifting focus to a newer version of the underlying component), but are
scattered through the commit history as subsequent mainline versions have been
added through the use of `git merge`. This can make it difficult to generate
a clean set of patches to apply the vendor BSP to the mainline kernel source
of the same version and to port support to a newer kernel version to take advantage of security and bug fixes.
## Unrelated changes
The changes to the source code might not only be changes for the required
processor/target board, but also other changes like:
- Realtime patches
- Some improvements from perspective of vendor not directly related to hardware support.
- Addition of hardware support for other hardware provided by the vendor but which
is not of consequence for the projects target platform
These changes may be scattered throughout the commit history (see
[No "clean" patch series implementing the vendors changes]({{< ref "#no-clean-patch-series-implementing-the-vendors-changes" >}})
and typically lack useful commit messages to
enable the intention behind any given change from being understood. This makes resolving
conflicting changes between the newer mainline and BSP specific changes difficult to rectify.
This leads to problems when upgrading with newer versions of the mainline kernel.
## Compatibility with ospacks
The Apertis operating system packages (ospacks) rely on the kernel with which
they are paired to provide certain functionality. This functionality is
expected to be provided using the common APIs found in the mainline kernel.
Vendors sometimes either modify/extend or utilize custom APIs to expose
functionality supported by their product. This combined with limitations in
functionality enabled in the vendors kernel config can lead to the kernel not
providing the functionality expected by the ospack, either in a form it can
use or at all.
## Lack of Mainline Support for Vendors Hardware
The Apertis project recommends hardware support is submitted for inclusion in
the mainline kernel. This results in:
- Greater testing of common code.
- Ability to easily generate a single binary, supporting multiple target boards
with a common architecture.
- Best support for security updates due to being able to efficiently track the
active upstream code changes.
- Best compatibility with Apertis.
Many vendors do not actively work with the mainline kernel developers to get support
for their hardware into the mainline kernel. The use of a vendor specific kernel
reduces the upfront time and effort required to build a solution, but results in a
significant increase in long term maintenance effort. Whilst more effort is required
to achieve mainline kernel support, this effort frequently results in a higher
quality of support being achieved than what would be expected from a vendor kernel.
There are often a number of changes that need to be made before the mainline
kernel supports a target board. Sometimes the mainline kernel can be used
directly though typically the following additions are required (in increasing
effort):
- The kernel configuration has to be adopted
- A new device tree (.DTB) has to be added
- Board specific drivers have to be added
- Addition of support for unsupported system on chip.
The board specific drivers are available in the vendor specific kernel. With
some kernel knowledge, the drivers can usually be ported to the mainline kernel
(though this can be difficult for low quality vendor kernels). If the platform
is important to you and the vendor has no interest in upstreaming support
themselves, taking on the challenge of upstreaming support yourself may be
beneficial in the long term.
# Evaluating the quality of the vendor BSP
With the increasing complexity and requirement for devices to be internet enabled,
the old "embedded" approach of freezing and hardening a BSP no longer works. Any
issues that are not detected during the hardening process may result in serious
security vulnerabilities being left present in shipped devices. As a result there
is an increasing requirement to perform "in the field" updates to shipped devices
so that upgraded kernels and other system components to resolve security issues.
Some vendors do not yet provide on going support for newer
kernels, nor work to include support in mainline kernels. Some will use
development practices that hinder the ability of a product team to perform these
updates themselves when necessary. It is therefore imperative that the quality of
the vendors software support be evaluated when selecting hardware for a project as this can
drastically effect the economic viability of a vendors offering over the lifetime
of a product.
## Understanding Kernel Versioning
Each mainline kernel is defined by its version number. This is typically
in the form of 2 or 3 dot separated numbers, where these numbers are:
<major version>.<minor version>.<stable release>
This may have other parts appended to it after a dash with these extra
parts being used to denote "release candidate" kernels or vendor specific
kernels.
{{% notice note %}}
This description holds true back to the beginning of the `3.x` series,
kernels prior to that used a different convention, though we would not
expect kernels of that vintage to be being used in active development
owing to them being over 10 years old and thus will not cover that here.
{{% /notice %}}
The mainline kernel is released as a 2-number version, such as `5.3`. The
minor version is incremented with each official release. Every so often
the major version number is incremented and the minor is returned to zero.
There is no defined point at which this occurs and it is at Linus Torvalds'
discretion as to when this happens.
Once a release is made, this is considered stable. Bugfixes will be applied
to this until the next kernel release. Each bugfix release is released with
a 3 number version, with the "stable release" part incrementing for each
release. Some kernels will be supported for an extended period and are know
as longterm stable (LTS) releases. Theses are the kernels that are used by
Apertis.
{{% notice info %}}
More information regarding kernel release can be found on
[kernel.org](https://www.kernel.org/releases.html).
{{% /notice %}}
## Determine what kernel upgrades can be expected by the vendor
The vendor support for upgrades to newer versions varies a lot.
Some vendors will do a single code drop with no support provided beyond that.
Others provide upgrades, but without offering an upgrade plan. Most of the time
it will be months between updates, during which time the vendor kernel will lack
security and bug fixes that may be found in the mainline and longterm stable kernels.
## Age and maintenance state of the related mainline kernel
Each vendor kernel is based on a mainline kernel. It is important to find out
which kernel the vendors support is based on and determine it's age and it's
support status. Wikipedia maintains a
[full list](https://en.wikipedia.org/wiki/Linux_kernel_version_history) of
release kernel versions, noting both their age and anticipated support "end
of life" (EOL) dates, whilst kernel.org lists the
[projected EOL dates](https://www.kernel.org/releases.html) for still supported
LTS releases along with detailing the [latest release](https://www.kernel.org/)
for each supported version.
When shipping a product, we do not want to be following the mainline
development as a source of bug and security fixes as this will result in
significant changes being pulled in when updating the kernel, which is likely
to result in both unwanted changes to the features and performance on the
target product and is likely to result in difficulties porting any vendor or
product specific kernel modifications.
As stable kernel releases aim to only include security and bug fixes, this
results in significantly less changes to the code base, reducing the risk of
feature or performance change and reducing or eliminating issues with porting
modifications. From a long term maintenance perspective it is therefore
beneficial for a product to use a vendor that bases their vendor kernel on
the recent LTS kernel (one with sufficient time before it reaches it's end of
life, as determined by the project time frames), even more so if they actively
track the stable point releases.
## Is a rebase to a longterm stable kernel needed?
Since development of the mainline kernel is always going on, the quality of these
parts varies dependant on maturity. Also security issues like vulnerabilties
are handled during this development process with security fixes backported
to the kernels supported stable releases.
Apertis tracks this development effort to ensure the bug fixes, security fixes and
feature improvements made during the mainline development process are available in
Apertis, with each Apertis release including the latest
[Longterm Stable kernel](https://www.kernel.org/releases.html) with point
releases tracking changes on that branch. Any vendor BSP integrated into
Apertis should also be able to update regularly to include these fixes and
improvements, however many vendors do not provide kernel updates leaving it to
their users to either perform these updates themselves or remain on older
kernels versions with known security vulnerabilites.
## Is the vendor working to integrate support into the mainline kernel?
Even vendors that provide good on going support for their hardware via vendor trees lag behind
the fixes provided in the mainstream kernel, as they require these releases to be made before
porting any of their changes. Additionally, projects with long expected lifetimes may exceed
the period for which a vendor is willing to provide support.
If a vendor "upstreams" support for their device, integrating that support right into the
mainline kernel, the vendor kernel is typically no longer required. A correctly configured
mainline kernel can be used, removing any lag and typically enabling the device to be used
long after the vendor stops actively supporting it without significant addition effort and cost.
## Are there qualified test results for vendor BSP ?
Some vendors provide tests and results for their specific images to check hardware and kernel features.
These tests usually cover specific hardware functions and less common features of the kernel.
So additional tests have to be planned and organized, probably got from mainline kernel.
Since the image is a new binary (compared to images of mainline kernel), these tests have to be redone best
at automatic test systems like LAVA (see also [Apertis integration testing with LAVA]({{< ref "lava-apertis-testing" >}})).
## How is the responsibility to maintain the vendor BSP organized ?
Developing and purchasing a product includes product liability for the company over product lifetime.
This includes aspects for security (loss of personal data) and safety (risk to customers physically).
The Linux kernel potentially plays a very important role in both, thus ensuring bugs and
security issues get fixed promptly and reliably is critical. These updates either have to be
done by the product manufacturer or the supplier of the vendor providing the kernel support.
As vendors will typically stop providing support after a defined period, the timing of this
needs to be understood in relation to the products life time. If the vendors support period
would end before that of the product, then it needs to be understood (and factored in) that
at such a time it becomes the responsibility of the product team to organize such
maintenance support.
## Evaluate the Quality of Vendor Added Drivers
Any drivers added by a vendor can be found by analyzing a diff between the
vendor's kernel sources against the related mainline kernel source. Drivers
should be fairly self contained under `drivers/`, with minor changes in:
- Startup files
- Makefile(s)
- Kconfig(s)
A deep review of implementation and testing of the relevant kernel driver is
recommended. This is a time consuming task.
# Practical checks
The following practical checks can be made to help determine the quality of the
vendor supplied kernel source.
## Check the Ability to Build Mainline Supported Devices
Well integrated vendor support will result in a kernel that can still be built
for other machines which share the same architecture and which are defined in
the mainline kernel. Attempting to build using the configs provided in the
`configs/` folder for your target architecture (such as `arch/arm/configs` for
32-bit ARM devices). Build problems may indicate incomplete or unintended
changes to common parts of the code.
## Check the Ability to Build for Machines With Different Architectures
This is similar to testing configs for other machines from your target
architecture, but can point to additional issues where changes have been made
to very common code in architecture specific ways. This will require a
toolchain supporting the relevant architecture. Build problems may indicate
incomplete or unintended changes to common parts of the code.
## Attempting to build a multi-target kernel binary
As long as sufficient similarities exist, it is possible to build a single
kernel binary that can be used with multiple different machines with different
peripherals and even processors.
The kernel provides configs that build a kernel binary which can run on
multiple different devices. An example is
`arch/arm/configs/multi_v7_defconfig`, which will build a kernel for many ARMv7
compatible devices. If such a configuration exists that may be compatible with
the device your interested in, it may be instructive to attempt to add the
target device to such a config and build it.
The need to build separate kernel binaries may indicate, that some driver
interfaces are not implemented optimally. This will lead to reduced test depth
and increased build efforts.
## Determining quantity and quality of vendor kernel mondifications
It is typically not possible to utilise the git log to compare a vendor
kernel to the mainline kernel from which it was built due to the previously
mentioned development practices. As a result it is typical to need to utilise
standard "diff" tools to make the comparison.
The differences highlighted by these tools need to be examined to identify the
changes relating to target hardware. It would be typical to see:
- Hardware drivers and related header files
- Files to integrate hardware drivers like Kconfig(s) or Makefile(s)
- Addition of hooks to common files
- Expanded support in shared architecture files.
Thus the majority of the changes required to support a target device should be
expected to be limited to those files providing drivers and support for the
applicable architecture.
It is important to identify changes in other areas, including those not related
to the target hardware, such as:
- Changes of significant complexity to common headers and source files
- Addition of unrelated support, such as those provided by the real-time patch set
- Unintended changes, e.g. changes of header files with contents of older
kernel versions
- Unrelated improvements (at least from viewpoint of the supplier of the vendor kernel)
- Reformatting indenting and code style of entire files, masking subtle changes
The bigger the changes unrelated to the target hardware and the more spread the
changes are, the greater the effort that will be required to upgrade to newer
kernel versions.
## Attempt to build a kernel using the Debian config with Debian patches applied
When adding kernel support for a device to Apertis, it is advised to base this
on the Apertis kernel to enable you to easier take advantage of security
updates. If the vendor kernel and the Apertis kernel are both based on a
similar kernel, in order to test compatibility between the changes introduced
by the vendor and those applied to the Apertis kernel it can be instructive to
apply the Apertis patches to the vendor kernel and attempt to build with the
Apertis kernel configuration (which is likely to have support for a broader
number of features enabled).
As is
[customary with Debian packaging](https://wiki.debian.org/SimplePackagingTutorial),
changes to an upstream source and any other files (such as the component
configuration file) are stored in the `.debian.tar.xz` file. The relevant files
for Apertis can be found on the
[Apertis package repository](https://repositories.apertis.org/apertis/pool/target/l/linux/).
Extracting this archive will create a `debian/` folder. The patches applied by
Apertis can be found in `debian/patches` (the `series` file provides the order
and enables the series to be quickly applied with tools such as `quilt`). The
Debian configuration is split in to a generic portion (in
`debian/config/config`), architecture configuration (in
`debian/config/kernelarch-<arch>`) and more fine grained architecture config
(in `debian/config/<arch>`), with a "full" configuration comprised of the
options found in one of each of these sets.
{{% notice note %}} While Debian builds kernels for many different
architectures (and thus configuration files are present for many
architectures), Apertis limits it's builds to `armhf`, `arm64` and `amd64` and
thus it is advised to limit testing to those architectures. {{% /notice %}}
Applying these patches and building the kernel with this kernel configuration
can highlight problems in common parts, which may make upgrades difficult
without loss of functionality.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
Apertis Website Terms of Use Privacy Policy