isa/citadel
1
0
Fork 0
forked from brl/citadel
Code Pull Requests Activity
f956d2e3d6
citadel/poky/documentation/kernel-dev/kernel-dev-concepts-appx.xml

622 lines
31 KiB
XML
Raw Normal View History

Squashed 'poky/' content from commit 4469acdf1d git-subtree-dir: poky git-subtree-split: 4469acdf1d0338220f3fe2ecb5e079eea6fda375
2017-12-04 16:35:29 -05:00
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
<appendix id='kernel-dev-concepts-appx'>
<title>Advanced Kernel Concepts</title>
<section id='kernel-big-picture'>
<title>Yocto Project Kernel Development and Maintenance</title>
<para>
Kernels available through the Yocto Project (Yocto Linux kernels),
like other kernels, are based off the Linux kernel releases from
<ulink url='http://www.kernel.org'></ulink>.
At the beginning of a major Linux kernel development cycle, the
Yocto Project team chooses a Linux kernel based on factors such as
release timing, the anticipated release timing of final upstream
<filename>kernel.org</filename> versions, and Yocto Project
feature requirements.
Typically, the Linux kernel chosen is in the final stages of
development by the Linux community.
In other words, the Linux kernel is in the release candidate
or "rc" phase and has yet to reach final release.
But, by being in the final stages of external development, the
team knows that the <filename>kernel.org</filename> final release
will clearly be within the early stages of the Yocto Project
development window.
</para>
<para>
This balance allows the Yocto Project team to deliver the most
up-to-date Yocto Linux kernel possible, while still ensuring that
the team has a stable official release for the baseline Linux
kernel version.
</para>
<para>
As implied earlier, the ultimate source for Yocto Linux kernels
are released kernels from <filename>kernel.org</filename>.
In addition to a foundational kernel from
<filename>kernel.org</filename>, the available Yocto Linux kernels
contain a mix of important new mainline developments, non-mainline
developments (when no alternative exists), Board Support Package
(BSP) developments, and custom features.
These additions result in a commercially released Yocto
Project Linux kernel that caters to specific embedded designer
needs for targeted hardware.
</para>
<para>
You can find a web interface to the Yocto Linux kernels in the
<ulink url='&YOCTO_DOCS_REF_URL;#source-repositories'>Source Repositories</ulink>
at
<ulink url='&YOCTO_GIT_URL;'></ulink>.
If you look at the interface, you will see to the left a
grouping of Git repositories titled "Yocto Linux Kernel".
Within this group, you will find several Linux Yocto kernels
developed and included with Yocto Project releases:
<itemizedlist>
<listitem><para>
<emphasis><filename>linux-yocto-4.1</filename>:</emphasis>
The stable Yocto Project kernel to use with the Yocto
Project Release 2.0.
This kernel is based on the Linux 4.1 released kernel.
</para></listitem>
<listitem><para>
<emphasis><filename>linux-yocto-4.4</filename>:</emphasis>
The stable Yocto Project kernel to use with the Yocto
Project Release 2.1.
This kernel is based on the Linux 4.4 released kernel.
</para></listitem>
<listitem><para>
<emphasis><filename>linux-yocto-4.6</filename>:</emphasis>
A temporary kernel that is not tied to any Yocto Project
release.
</para></listitem>
<listitem><para>
<emphasis><filename>linux-yocto-4.8</filename>:</emphasis>
The stable yocto Project kernel to use with the Yocto
Project Release 2.2.
</para></listitem>
<listitem><para>
<emphasis><filename>linux-yocto-4.9</filename>:</emphasis>
The stable Yocto Project kernel to use with the Yocto
Project Release 2.3.
This kernel is based on the Linux 4.9 released kernel.
</para></listitem>
<listitem><para>
<emphasis><filename>linux-yocto-4.10</filename>:</emphasis>
The default stable Yocto Project kernel to use with the
Yocto Project Release 2.3.
This kernel is based on the Linux 4.10 released kernel.
</para></listitem>
<listitem><para>
<emphasis><filename>linux-yocto-4.12</filename>:</emphasis>
The default stable Yocto Project kernel to use with the
Yocto Project Release 2.4.
This kernel is based on the Linux 4.12 released kernel.
</para></listitem>
<listitem><para>
<emphasis><filename>yocto-kernel-cache</filename>:</emphasis>
The <filename>linux-yocto-cache</filename> contains
patches and configurations for the linux-yocto kernel
tree.
This repository is useful when working on the linux-yocto
kernel.
For more information on this "Advanced Kernel Metadata",
see the
"<link linkend='kernel-dev-advanced'>Working With Advanced Metadata (<filename>yocto-kernel-cache</filename>)</link>"
Chapter.
</para></listitem>
<listitem><para>
<emphasis><filename>linux-yocto-dev</filename>:</emphasis>
A development kernel based on the latest upstream release
candidate available.
</para></listitem>
</itemizedlist>
<note><title>Notes</title>
Long Term Support Initiative (LTSI) for Yocto Linux
kernels is as follows:
<itemizedlist>
<listitem><para>
For Yocto Project releases 1.7, 1.8, and 2.0,
the LTSI kernel is
<filename>linux-yocto-3.14</filename>.
</para></listitem>
<listitem><para>
For Yocto Project releases 2.1, 2.2, and 2.3,
the LTSI kernel is <filename>linux-yocto-4.1</filename>.
</para></listitem>
<listitem><para>
For Yocto Project release 2.4, the LTSI kernel is
<filename>linux-yocto-4.9</filename>
</para></listitem>
<listitem><para>
<filename>linux-yocto-4.4</filename> is an LTS
kernel.
</para></listitem>
</itemizedlist>
</note>
</para>
<para>
Once a Yocto Linux kernel is officially released, the Yocto
Project team goes into their next development cycle, or upward
revision (uprev) cycle, while still continuing maintenance on the
released kernel.
It is important to note that the most sustainable and stable way
to include feature development upstream is through a kernel uprev
process.
Back-porting hundreds of individual fixes and minor features from
various kernel versions is not sustainable and can easily
compromise quality.
</para>
<para>
During the uprev cycle, the Yocto Project team uses an ongoing
analysis of Linux kernel development, BSP support, and release
timing to select the best possible <filename>kernel.org</filename>
Linux kernel version on which to base subsequent Yocto Linux
kernel development.
The team continually monitors Linux community kernel development
to look for significant features of interest.
The team does consider back-porting large features if they have a
significant advantage.
User or community demand can also trigger a back-port or creation
of new functionality in the Yocto Project baseline kernel during
the uprev cycle.
</para>
<para>
Generally speaking, every new Linux kernel both adds features and
introduces new bugs.
These consequences are the basic properties of upstream
Linux kernel development and are managed by the Yocto Project
team's Yocto Linux kernel development strategy.
It is the Yocto Project team's policy to not back-port minor
features to the released Yocto Linux kernel.
They only consider back-porting significant technological
jumps &dash; and, that is done after a complete gap analysis.
The reason for this policy is that back-porting any small to
medium sized change from an evolving Linux kernel can easily
create mismatches, incompatibilities and very subtle errors.
</para>
<para>
The policies described in this section result in both a stable
and a cutting edge Yocto Linux kernel that mixes forward ports of
existing Linux kernel features and significant and critical new
functionality.
Forward porting Linux kernel functionality into the Yocto Linux
kernels available through the Yocto Project can be thought of as
a "micro uprev."
The many “micro uprevs” produce a Yocto Linux kernel version with
a mix of important new mainline, non-mainline, BSP developments
and feature integrations.
This Yocto Linux kernel gives insight into new features and
allows focused amounts of testing to be done on the kernel,
which prevents surprises when selecting the next major uprev.
The quality of these cutting edge Yocto Linux kernels is evolving
and the kernels are used in leading edge feature and BSP
development.
</para>
</section>
<section id='yocto-linux-kernel-architecture-and-branching-strategies'>
<title>Yocto Linux Kernel Architecture and Branching Strategies</title>
<para>
As mentioned earlier, a key goal of the Yocto Project is
to present the developer with a kernel that has a clear and
continuous history that is visible to the user.
The architecture and mechanisms, in particular the branching
strategies, used achieve that goal in a manner similar to
upstream Linux kernel development in
<filename>kernel.org</filename>.
</para>
<para>
You can think of a Yocto Linux kernel as consisting of a
baseline Linux kernel with added features logically structured
on top of the baseline.
The features are tagged and organized by way of a branching
strategy implemented by the Yocto Project team using the
Source Code Manager (SCM) Git.
<note><title>Notes</title>
<itemizedlist>
<listitem><para>
Git is the obvious SCM for meeting the Yocto Linux
kernel organizational and structural goals described
in this section.
Not only is Git the SCM for Linux kernel development in
<filename>kernel.org</filename> but, Git continues to
grow in popularity and supports many different work
flows, front-ends and management techniques.
</para></listitem>
<listitem><para>
You can find documentation on Git at
<ulink url='http://git-scm.com/documentation'></ulink>.
You can also get an introduction to Git as it
applies to the Yocto Project in the
"<ulink url='&YOCTO_DOCS_REF_URL;#git'>Git</ulink>"
section in the Yocto Project Reference Manual.
The latter reference provides an overview of
Git and presents a minimal set of Git commands
that allows you to be functional using Git.
You can use as much, or as little, of what Git
has to offer to accomplish what you need for your
project.
You do not have to be a "Git Expert" in order to
use it with the Yocto Project.
</para></listitem>
</itemizedlist>
</note>
</para>
<para>
Using Git's tagging and branching features, the Yocto Project
team creates kernel branches at points where functionality is
no longer shared and thus, needs to be isolated.
For example, board-specific incompatibilities would require
different functionality and would require a branch to
separate the features.
Likewise, for specific kernel features, the same branching
strategy is used.
</para>
<para>
This "tree-like" architecture results in a structure that has
features organized to be specific for particular functionality,
single kernel types, or a subset of kernel types.
Thus, the user has the ability to see the added features and the
commits that make up those features.
In addition to being able to see added features, the user
can also view the history of what made up the baseline
Linux kernel.
</para>
<para>
Another consequence of this strategy results in not having to
store the same feature twice internally in the tree.
Rather, the kernel team stores the unique differences required
to apply the feature onto the kernel type in question.
<note>
The Yocto Project team strives to place features in the tree
such that features can be shared by all boards and kernel
types where possible.
However, during development cycles or when large features
are merged, the team cannot always follow this practice.
In those cases, the team uses isolated branches to merge
features.
</note>
</para>
<para>
BSP-specific code additions are handled in a similar manner to
kernel-specific additions.
Some BSPs only make sense given certain kernel types.
So, for these types, the team creates branches off the end
of that kernel type for all of the BSPs that are supported on
that kernel type.
From the perspective of the tools that create the BSP branch,
the BSP is really no different than a feature.
Consequently, the same branching strategy applies to BSPs as
it does to kernel features.
So again, rather than store the BSP twice, the team only
stores the unique differences for the BSP across the supported
multiple kernels.
</para>
<para>
While this strategy can result in a tree with a significant number
of branches, it is important to realize that from the developer's
point of view, there is a linear path that travels from the
baseline <filename>kernel.org</filename>, through a select
group of features and ends with their BSP-specific commits.
In other words, the divisions of the kernel are transparent and
are not relevant to the developer on a day-to-day basis.
From the developer's perspective, this path is the "master" branch
in Git terms.
The developer does not need to be aware of the existence of any
other branches at all.
Of course, value exists in the having these branches in the tree,
should a person decide to explore them.
For example, a comparison between two BSPs at either the commit
level or at the line-by-line code <filename>diff</filename> level
is now a trivial operation.
</para>
<para>
The following illustration shows the conceptual Yocto
Linux kernel.
<imagedata fileref="figures/kernel-architecture-overview.png" width="6in" depth="7in" align="center" scale="100" />
</para>
<para>
In the illustration, the "Kernel.org Branch Point" marks the
specific spot (or Linux kernel release) from which the
Yocto Linux kernel is created.
From this point forward in the tree, features and differences
are organized and tagged.
</para>
<para>
The "Yocto Project Baseline Kernel" contains functionality that
is common to every kernel type and BSP that is organized
further along in the tree.
Placing these common features in the tree this way means
features do not have to be duplicated along individual
branches of the tree structure.
</para>
<para>
From the "Yocto Project Baseline Kernel", branch points represent
specific functionality for individual Board Support Packages
(BSPs) as well as real-time kernels.
The illustration represents this through three BSP-specific
branches and a real-time kernel branch.
Each branch represents some unique functionality for the BSP
or for a real-time Yocto Linux kernel.
</para>
<para>
In this example structure, the "Real-time (rt) Kernel" branch has
common features for all real-time Yocto Linux kernels and
contains more branches for individual BSP-specific real-time
kernels.
The illustration shows three branches as an example.
Each branch points the way to specific, unique features for a
respective real-time kernel as they apply to a given BSP.
</para>
<para>
The resulting tree structure presents a clear path of markers
(or branches) to the developer that, for all practical
purposes, is the Yocto Linux kernel needed for any given set of
requirements.
<note>
Keep in mind the figure does not take into account all the
supported Yocto Linux kernels, but rather shows a single
generic kernel just for conceptual purposes.
Also keep in mind that this structure represents the Yocto
Project
<ulink url='&YOCTO_DOCS_REF_URL;#source-repositories'>Source Repositories</ulink>
that are either pulled from during the build or established
on the host development system prior to the build by either
cloning a particular kernel's Git repository or by
downloading and unpacking a tarball.
</note>
</para>
<para>
Working with the kernel as a structured tree follows recognized
community best practices.
In particular, the kernel as shipped with the product, should be
considered an "upstream source" and viewed as a series of
historical and documented modifications (commits).
These modifications represent the development and stabilization
done by the Yocto Project kernel development team.
</para>
<para>
Because commits only change at significant release points in the
product life cycle, developers can work on a branch created
from the last relevant commit in the shipped Yocto Project Linux
kernel.
As mentioned previously, the structure is transparent to the
developer because the kernel tree is left in this state after
cloning and building the kernel.
</para>
</section>
<section id='kernel-build-file-hierarchy'>
<title>Kernel Build File Hierarchy</title>
<para>
Upstream storage of all the available kernel source code is
one thing, while representing and using the code on your host
development system is another.
Conceptually, you can think of the kernel source repositories
as all the source files necessary for all the supported
Yocto Linux kernels.
As a developer, you are just interested in the source files
for the kernel on which you are working.
And, furthermore, you need them available on your host system.
</para>
<para>
Kernel source code is available on your host system several
different ways:
<itemizedlist>
<listitem><para>
<emphasis>Files Accessed While using <filename>devtool</filename>:</emphasis>
<filename>devtool</filename>, which is available with the
Yocto Project, is the preferred method by which to
modify the kernel.
See the
"<link linkend='kernel-modification-workflow'>Kernel Modification Workflow</link>"
section.
</para></listitem>
<listitem><para>
<emphasis>Cloned Repository:</emphasis>
If you are working in the kernel all the time, you probably
would want to set up your own local Git repository of the
Yocto Linux kernel tree.
For information on how to clone a Yocto Linux kernel
Git repository, see the
"<link linkend='preparing-the-build-host-to-work-on-the-kernel'>Preparing the Build Host to Work on the Kernel</link>"
section.
</para></listitem>
<listitem><para>
<emphasis>Temporary Source Files from a Build:</emphasis>
If you just need to make some patches to the kernel using
a traditional BitBake workflow (i.e. not using the
<filename>devtool</filename>), you can access temporary
kernel source files that were extracted and used during
a kernel build.
</para></listitem>
</itemizedlist>
</para>
<para>
The temporary kernel source files resulting from a build using
BitBake have a particular hierarchy.
When you build the kernel on your development system, all files
needed for the build are taken from the source repositories
pointed to by the
<ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
variable and gathered in a temporary work area where they are
subsequently used to create the unique kernel.
Thus, in a sense, the process constructs a local source tree
specific to your kernel from which to generate the new kernel
image.
</para>
<para>
The following figure shows the temporary file structure
created on your host system when you build the kernel using
Bitbake.
This
<ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
contains all the source files used during the build.
<imagedata fileref="figures/kernel-overview-2-generic.png"
width="6in" depth="5in" align="center" scale="100" />
</para>
<para>
Again, for additional information on the Yocto Project kernel's
architecture and its branching strategy, see the
"<link linkend='yocto-linux-kernel-architecture-and-branching-strategies'>Yocto Linux Kernel Architecture and Branching Strategies</link>"
section.
You can also reference the
"<link linkend='using-devtool-to-patch-the-kernel'>Using <filename>devtool</filename> to Patch the Kernel</link>"
and
"<link linkend='using-traditional-kernel-development-to-patch-the-kernel'>Using Traditional Kernel Development to Patch the Kernel</link>"
sections for detailed example that modifies the kernel.
</para>
</section>
<section id='determining-hardware-and-non-hardware-features-for-the-kernel-configuration-audit-phase'>
<title>Determining Hardware and Non-Hardware Features for the Kernel Configuration Audit Phase</title>
<para>
This section describes part of the kernel configuration audit
phase that most developers can ignore.
For general information on kernel configuration including
<filename>menuconfig</filename>, <filename>defconfig</filename>
files, and configuration fragments, see the
"<link linkend='configuring-the-kernel'>Configuring the Kernel</link>"
section.
</para>
<para>
During this part of the audit phase, the contents of the final
<filename>.config</filename> file are compared against the
fragments specified by the system.
These fragments can be system fragments, distro fragments,
or user-specified configuration elements.
Regardless of their origin, the OpenEmbedded build system
warns the user if a specific option is not included in the
final kernel configuration.
</para>
<para>
By default, in order to not overwhelm the user with
configuration warnings, the system only reports missing
"hardware" options as they could result in a boot
failure or indicate that important hardware is not available.
</para>
<para>
To determine whether or not a given option is "hardware" or
"non-hardware", the kernel Metadata in
<filename>yocto-kernel-cache</filename> contains files that
classify individual or groups of options as either hardware
or non-hardware.
To better show this, consider a situation where the
<filename>yocto-kernel-cache</filename> contains the following
files:
<literallayout class='monospaced'>
yocto-kernel-cache/features/drm-psb/hardware.cfg
yocto-kernel-cache/features/kgdb/hardware.cfg
yocto-kernel-cache/ktypes/base/hardware.cfg
yocto-kernel-cache/bsp/mti-malta32/hardware.cfg
yocto-kernel-cache/bsp/fsl-mpc8315e-rdb/hardware.cfg
yocto-kernel-cache/bsp/qemu-ppc32/hardware.cfg
yocto-kernel-cache/bsp/qemuarma9/hardware.cfg
yocto-kernel-cache/bsp/mti-malta64/hardware.cfg
yocto-kernel-cache/bsp/arm-versatile-926ejs/hardware.cfg
yocto-kernel-cache/bsp/common-pc/hardware.cfg
yocto-kernel-cache/bsp/common-pc-64/hardware.cfg
yocto-kernel-cache/features/rfkill/non-hardware.cfg
yocto-kernel-cache/ktypes/base/non-hardware.cfg
yocto-kernel-cache/features/aufs/non-hardware.kcf
yocto-kernel-cache/features/ocf/non-hardware.kcf
yocto-kernel-cache/ktypes/base/non-hardware.kcf
yocto-kernel-cache/ktypes/base/hardware.kcf
yocto-kernel-cache/bsp/qemu-ppc32/hardware.kcf
</literallayout>
The following list provides explanations for the various
files:
<itemizedlist>
<listitem><para>
<filename>hardware.kcf</filename>:
Specifies a list of kernel Kconfig files that contain
hardware options only.
</para></listitem>
<listitem><para>
<filename>non-hardware.kcf</filename>:
Specifies a list of kernel Kconfig files that contain
non-hardware options only.
</para></listitem>
<listitem><para>
<filename>hardware.cfg</filename>:
Specifies a list of kernel <filename>CONFIG_</filename>
options that are hardware, regardless of whether or not
they are within a Kconfig file specified by a hardware
or non-hardware Kconfig file (i.e.
<filename>hardware.kcf</filename> or
<filename>non-hardware.kcf</filename>).
</para></listitem>
<listitem><para>
<filename>non-hardware.cfg</filename>:
Specifies a list of kernel <filename>CONFIG_</filename>
options that are not hardware, regardless of whether or
not they are within a Kconfig file specified by a
hardware or non-hardware Kconfig file (i.e.
<filename>hardware.kcf</filename> or
<filename>non-hardware.kcf</filename>).
</para></listitem>
</itemizedlist>
Here is a specific example using the
<filename>kernel-cache/bsp/mti-malta32/hardware.cfg</filename>:
<literallayout class='monospaced'>
CONFIG_SERIAL_8250
CONFIG_SERIAL_8250_CONSOLE
CONFIG_SERIAL_8250_NR_UARTS
CONFIG_SERIAL_8250_PCI
CONFIG_SERIAL_CORE
CONFIG_SERIAL_CORE_CONSOLE
CONFIG_VGA_ARB
</literallayout>
The kernel configuration audit automatically detects these
files (hence the names must be exactly the ones discussed here),
and uses them as inputs when generating warnings about the
final <filename>.config</filename> file.
</para>
<para>
A user-specified kernel Metadata repository, or recipe space
feature, can use these same files to classify options that are
found within its <filename>.cfg</filename> files as hardware
or non-hardware, to prevent the OpenEmbedded build system from
producing an error or warning when an option is not in the
final <filename>.config</filename> file.
</para>
</section>
</appendix>
<!--
vim: expandtab tw=80 ts=4
-->
Copy Permalink