478 lines
23 KiB
XML
478 lines
23 KiB
XML
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
|
|
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
|
|
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
|
|
|
|
<chapter id='ref-bitbake'>
|
|
|
|
<title>BitBake</title>
|
|
|
|
<para>
|
|
BitBake is a program written in Python that interprets the
|
|
<link linkend='metadata'>Metadata</link> used by
|
|
the OpenEmbedded build system.
|
|
At some point, developers wonder what actually happens when you enter:
|
|
<literallayout class='monospaced'>
|
|
$ bitbake core-image-sato
|
|
</literallayout>
|
|
</para>
|
|
|
|
<para>
|
|
This chapter provides an overview of what happens behind the scenes from BitBake's perspective.
|
|
</para>
|
|
|
|
<note>
|
|
BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships.
|
|
As such, it has no real knowledge of what the tasks being executed actually do.
|
|
BitBake just considers a list of tasks with dependencies and handles
|
|
<link linkend='metadata'>Metadata</link>
|
|
consisting of variables in a certain format that get passed to the tasks.
|
|
</note>
|
|
|
|
<section id='ref-bitbake-parsing'>
|
|
<title>Parsing</title>
|
|
|
|
<para>
|
|
BitBake parses configuration files, classes, and <filename>.bb</filename> files.
|
|
</para>
|
|
|
|
<para>
|
|
The first thing BitBake does is look for the
|
|
<filename>bitbake.conf</filename> file.
|
|
This file resides in the
|
|
<link linkend='source-directory'>Source Directory</link>
|
|
within the <filename>meta/conf/</filename> directory.
|
|
BitBake finds it by examining its
|
|
<link linkend='var-BBPATH'><filename>BBPATH</filename></link> environment
|
|
variable and looking for the <filename>meta/conf/</filename>
|
|
directory.
|
|
</para>
|
|
|
|
<para>
|
|
The <filename>bitbake.conf</filename> file lists other configuration
|
|
files to include from a <filename>conf/</filename>
|
|
directory below the directories listed in <filename>BBPATH</filename>.
|
|
In general, the most important configuration file from a user's perspective
|
|
is <filename>local.conf</filename>, which contains a user's customized
|
|
settings for the OpenEmbedded build environment.
|
|
Other notable configuration files are the distribution
|
|
configuration file (set by the
|
|
<filename><link linkend='var-DISTRO'>DISTRO</link></filename> variable)
|
|
and the machine configuration file
|
|
(set by the
|
|
<filename><link linkend='var-MACHINE'>MACHINE</link></filename> variable).
|
|
The <filename>DISTRO</filename> and <filename>MACHINE</filename> BitBake environment
|
|
variables are both usually set in
|
|
the <filename>local.conf</filename> file.
|
|
Valid distribution
|
|
configuration files are available in the <filename>meta/conf/distro/</filename> directory
|
|
and valid machine configuration
|
|
files in the <filename>meta/conf/machine/</filename> directory.
|
|
Within the <filename>meta/conf/machine/include/</filename>
|
|
directory are various <filename>tune-*.inc</filename> configuration files that provide common
|
|
"tuning" settings specific to and shared between particular architectures and machines.
|
|
</para>
|
|
|
|
<para>
|
|
After the parsing of the configuration files, some standard classes are included.
|
|
The <filename>base.bbclass</filename> file is always included.
|
|
Other classes that are specified in the configuration using the
|
|
<filename><link linkend='var-INHERIT'>INHERIT</link></filename>
|
|
variable are also included.
|
|
Class files are searched for in a <filename>classes</filename> subdirectory
|
|
under the paths in <filename>BBPATH</filename> in the same way as
|
|
configuration files.
|
|
</para>
|
|
|
|
<para>
|
|
After classes are included, the variable
|
|
<filename><link linkend='var-BBFILES'>BBFILES</link></filename>
|
|
is set, usually in
|
|
<filename>local.conf</filename>, and defines the list of places to search for
|
|
<filename>.bb</filename> files.
|
|
By default, the <filename>BBFILES</filename> variable specifies the
|
|
<filename>meta/recipes-*/</filename> directory within Poky.
|
|
Adding extra content to <filename>BBFILES</filename> is best achieved through the use of
|
|
BitBake layers as described in the
|
|
"<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
|
|
section of the Yocto Project Development Tasks Manual.
|
|
</para>
|
|
|
|
<para>
|
|
BitBake parses each <filename>.bb</filename> file in <filename>BBFILES</filename> and
|
|
stores the values of various variables.
|
|
In summary, for each <filename>.bb</filename>
|
|
file the configuration plus the base class of variables are set, followed
|
|
by the data in the <filename>.bb</filename> file
|
|
itself, followed by any inherit commands that
|
|
<filename>.bb</filename> file might contain.
|
|
</para>
|
|
|
|
<para>
|
|
Because parsing <filename>.bb</filename> files is a time
|
|
consuming process, a cache is kept to speed up subsequent parsing.
|
|
This cache is invalid if the timestamp of the <filename>.bb</filename>
|
|
file itself changes, or if the timestamps of any of the include,
|
|
configuration files or class files on which the
|
|
<filename>.bb</filename> file depends change.
|
|
</para>
|
|
|
|
<note>
|
|
<para>
|
|
You need to be aware of how BitBake parses curly braces.
|
|
If a recipe uses a closing curly brace within the function and
|
|
the character has no leading spaces, BitBake produces a parsing
|
|
error.
|
|
If you use a pair of curly brace in a shell function, the
|
|
closing curly brace must not be located at the start of the line
|
|
without leading spaces.
|
|
</para>
|
|
|
|
<para>
|
|
Here is an example that causes BitBake to produce a parsing
|
|
error:
|
|
<literallayout class='monospaced'>
|
|
fakeroot create_shar() {
|
|
cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
|
|
usage()
|
|
{
|
|
echo "test"
|
|
###### The following "}" at the start of the line causes a parsing error ######
|
|
}
|
|
EOF
|
|
}
|
|
</literallayout>
|
|
Writing the recipe this way avoids the error:
|
|
<literallayout class='monospaced'>
|
|
fakeroot create_shar() {
|
|
cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
|
|
usage()
|
|
{
|
|
echo "test"
|
|
######The following "}" with a leading space at the start of the line avoids the error ######
|
|
}
|
|
EOF
|
|
}
|
|
</literallayout>
|
|
</para>
|
|
</note>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-providers'>
|
|
<title>Preferences and Providers</title>
|
|
|
|
<para>
|
|
Once all the <filename>.bb</filename> files have been
|
|
parsed, BitBake starts to build the target (<filename>core-image-sato</filename>
|
|
in the previous section's example) and looks for providers of that target.
|
|
Once a provider is selected, BitBake resolves all the dependencies for
|
|
the target.
|
|
In the case of <filename>core-image-sato</filename>, it would lead to
|
|
<filename>packagegroup-core-x11-sato</filename>,
|
|
which in turn leads to recipes like <filename>matchbox-terminal</filename>,
|
|
<filename>pcmanfm</filename> and <filename>gthumb</filename>.
|
|
These recipes in turn depend on <filename>glibc</filename> and the toolchain.
|
|
</para>
|
|
|
|
<para>
|
|
Sometimes a target might have multiple providers.
|
|
A common example is "virtual/kernel", which is provided by each kernel package.
|
|
Each machine often selects the best kernel provider by using a line similar to the
|
|
following in the machine configuration file:
|
|
</para>
|
|
|
|
<literallayout class='monospaced'>
|
|
PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
|
|
</literallayout>
|
|
|
|
<para>
|
|
The default <filename><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></filename>
|
|
is the provider with the same name as the target.
|
|
</para>
|
|
|
|
<para>
|
|
Understanding how providers are chosen is made complicated by the fact
|
|
that multiple versions might exist.
|
|
BitBake defaults to the highest version of a provider.
|
|
Version comparisons are made using the same method as Debian.
|
|
You can use the
|
|
<filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
|
|
variable to specify a particular version (usually in the distro configuration).
|
|
You can influence the order by using the
|
|
<filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></filename>
|
|
variable.
|
|
By default, files have a preference of "0".
|
|
Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
|
|
package unlikely to be used unless it is explicitly referenced.
|
|
Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used.
|
|
<filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
|
|
<filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package
|
|
versions until they have undergone sufficient testing to be considered stable.
|
|
</para>
|
|
|
|
<para>
|
|
In summary, BitBake has created a list of providers, which is prioritized, for each target.
|
|
</para>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-dependencies'>
|
|
<title>Dependencies</title>
|
|
|
|
<para>
|
|
Each target BitBake builds consists of multiple tasks such as
|
|
<filename>fetch</filename>, <filename>unpack</filename>,
|
|
<filename>patch</filename>, <filename>configure</filename>,
|
|
and <filename>compile</filename>.
|
|
For best performance on multi-core systems, BitBake considers each task as an independent
|
|
entity with its own set of dependencies.
|
|
</para>
|
|
|
|
<para>
|
|
Dependencies are defined through several variables.
|
|
You can find information about variables BitBake uses in the
|
|
BitBake documentation, which is found in the
|
|
<filename>bitbake/doc/manual</filename> directory within the
|
|
<link linkend='source-directory'>Source Directory</link>.
|
|
At a basic level, it is sufficient to know that BitBake uses the
|
|
<filename><link linkend='var-DEPENDS'>DEPENDS</link></filename> and
|
|
<filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename>
|
|
variables when calculating dependencies.
|
|
</para>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-tasklist'>
|
|
<title>The Task List</title>
|
|
|
|
<para>
|
|
Based on the generated list of providers and the dependency information,
|
|
BitBake can now calculate exactly what tasks it needs to run and in what
|
|
order it needs to run them.
|
|
The build now starts with BitBake forking off threads up to the limit set in the
|
|
<filename><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></filename> variable.
|
|
BitBake continues to fork threads as long as there are tasks ready to run,
|
|
those tasks have all their dependencies met, and the thread threshold has not been
|
|
exceeded.
|
|
</para>
|
|
|
|
<para>
|
|
It is worth noting that you can greatly speed up the build time by properly setting
|
|
the <filename>BB_NUMBER_THREADS</filename> variable.
|
|
See the
|
|
"<ulink url='&YOCTO_DOCS_QS_URL;#qs-building-images'>Building Images</ulink>"
|
|
section in the Yocto Project Quick Start for more information.
|
|
</para>
|
|
|
|
<para>
|
|
As each task completes, a timestamp is written to the directory specified by the
|
|
<filename><link linkend='var-STAMP'>STAMP</link></filename> variable.
|
|
On subsequent runs, BitBake looks within the <filename>build/tmp/stamps</filename>
|
|
directory and does not rerun
|
|
tasks that are already completed unless a timestamp is found to be invalid.
|
|
Currently, invalid timestamps are only considered on a per
|
|
<filename>.bb</filename> file basis.
|
|
So, for example, if the configure stamp has a timestamp greater than the
|
|
compile timestamp for a given target, then the compile task would rerun.
|
|
Running the compile task again, however, has no effect on other providers
|
|
that depend on that target.
|
|
This behavior could change or become configurable in future versions of BitBake.
|
|
</para>
|
|
|
|
<note>
|
|
Some tasks are marked as "nostamp" tasks.
|
|
No timestamp file is created when these tasks are run.
|
|
Consequently, "nostamp" tasks are always rerun.
|
|
</note>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-runtask'>
|
|
<title>Running a Task</title>
|
|
|
|
<para>
|
|
Tasks can either be a shell task or a Python task.
|
|
For shell tasks, BitBake writes a shell script to
|
|
<filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script.
|
|
The generated shell script contains all the exported variables, and the shell functions
|
|
with all variables expanded.
|
|
Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
|
|
Looking at the expanded shell functions in the run file and the output in the log files
|
|
is a useful debugging technique.
|
|
</para>
|
|
|
|
<para>
|
|
For Python tasks, BitBake executes the task internally and logs information to the
|
|
controlling terminal.
|
|
Future versions of BitBake will write the functions to files similar to the way
|
|
shell tasks are handled.
|
|
Logging will be handled in a way similar to shell tasks as well.
|
|
</para>
|
|
|
|
<para>
|
|
Once all the tasks have been completed BitBake exits.
|
|
</para>
|
|
|
|
<para>
|
|
When running a task, BitBake tightly controls the execution environment
|
|
of the build tasks to make sure unwanted contamination from the build machine
|
|
cannot influence the build.
|
|
Consequently, if you do want something to get passed into the build
|
|
task's environment, you must take a few steps:
|
|
<orderedlist>
|
|
<listitem><para>Tell BitBake to load what you want from the environment
|
|
into the data store.
|
|
You can do so through the <filename>BB_ENV_EXTRAWHITE</filename>
|
|
variable.
|
|
For example, assume you want to prevent the build system from
|
|
accessing your <filename>$HOME/.ccache</filename> directory.
|
|
The following command tells BitBake to load
|
|
<filename>CCACHE_DIR</filename> from the environment into the data
|
|
store:
|
|
<literallayout class='monospaced'>
|
|
export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR"
|
|
</literallayout></para></listitem>
|
|
<listitem><para>Tell BitBake to export what you have loaded into the
|
|
environment store to the task environment of every running task.
|
|
Loading something from the environment into the data store
|
|
(previous step) only makes it available in the datastore.
|
|
To export it to the task environment of every running task,
|
|
use a command similar to the following in your
|
|
<filename>local.conf</filename> or distro configuration file:
|
|
<literallayout class='monospaced'>
|
|
export CCACHE_DIR
|
|
</literallayout></para></listitem>
|
|
</orderedlist>
|
|
</para>
|
|
|
|
<note>
|
|
A side effect of the previous steps is that BitBake records the variable
|
|
as a dependency of the build process in things like the shared state
|
|
checksums.
|
|
If doing so results in unnecessary rebuilds of tasks, you can whitelist the
|
|
variable so that the shared state code ignores the dependency when it creates
|
|
checksums.
|
|
For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename>
|
|
example in the "<link linkend='checksums'>Checksums (Signatures)</link>" section.
|
|
</note>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-commandline'>
|
|
<title>BitBake Command Line</title>
|
|
|
|
<para>
|
|
Following is the BitBake help output:
|
|
</para>
|
|
|
|
<screen>
|
|
$ bitbake --help
|
|
Usage: bitbake [options] [recipename/target ...]
|
|
|
|
Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
|
|
It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
|
|
will provide the layer, BBFILES and other configuration information.
|
|
|
|
Options:
|
|
--version show program's version number and exit
|
|
-h, --help show this help message and exit
|
|
-b BUILDFILE, --buildfile=BUILDFILE
|
|
Execute tasks from a specific .bb recipe directly.
|
|
WARNING: Does not handle any dependencies from other
|
|
recipes.
|
|
-k, --continue Continue as much as possible after an error. While the
|
|
target that failed and anything depending on it cannot
|
|
be built, as much as possible will be built before
|
|
stopping.
|
|
-a, --tryaltconfigs Continue with builds by trying to use alternative
|
|
providers where possible.
|
|
-f, --force Force the specified targets/task to run (invalidating
|
|
any existing stamp file).
|
|
-c CMD, --cmd=CMD Specify the task to execute. The exact options
|
|
available depend on the metadata. Some examples might
|
|
be 'compile' or 'populate_sysroot' or 'listtasks' may
|
|
give a list of the tasks available.
|
|
-C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
|
|
Invalidate the stamp for the specified task such as
|
|
'compile' and then run the default task for the
|
|
specified target(s).
|
|
-r PREFILE, --read=PREFILE
|
|
Read the specified file before bitbake.conf.
|
|
-R POSTFILE, --postread=POSTFILE
|
|
Read the specified file after bitbake.conf.
|
|
-v, --verbose Output more log message data to the terminal.
|
|
-D, --debug Increase the debug level. You can specify this more
|
|
than once.
|
|
-n, --dry-run Don't execute, just go through the motions.
|
|
-S, --dump-signatures
|
|
Don't execute, just dump out the signature
|
|
construction information.
|
|
-p, --parse-only Quit after parsing the BB recipes.
|
|
-s, --show-versions Show current and preferred versions of all recipes.
|
|
-e, --environment Show the global or per-package environment complete
|
|
with information about where variables were
|
|
set/changed.
|
|
-g, --graphviz Save dependency tree information for the specified
|
|
targets in the dot syntax.
|
|
-I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
|
|
Assume these dependencies don't exist and are already
|
|
provided (equivalent to ASSUME_PROVIDED). Useful to
|
|
make dependency graphs more appealing
|
|
-l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
|
|
Show debug logging for the specified logging domains
|
|
-P, --profile Profile the command and save reports.
|
|
-u UI, --ui=UI The user interface to use (e.g. knotty and taskexp).
|
|
-t SERVERTYPE, --servertype=SERVERTYPE
|
|
Choose which server to use, process or xmlrpc.
|
|
--revisions-changed Set the exit code depending on whether upstream
|
|
floating revisions have changed or not.
|
|
--server-only Run bitbake without a UI, only starting a server
|
|
(cooker) process.
|
|
-B BIND, --bind=BIND The name/address for the bitbake server to bind to.
|
|
--no-setscene Do not run any setscene tasks. sstate will be ignored
|
|
and everything needed, built.
|
|
--remote-server=REMOTE_SERVER
|
|
Connect to the specified server.
|
|
-m, --kill-server Terminate the remote server.
|
|
--observe-only Connect to a server as an observing-only client.
|
|
</screen>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-fetchers'>
|
|
<title>Fetchers</title>
|
|
|
|
<para>
|
|
BitBake also contains a set of "fetcher" modules that allow
|
|
retrieval of source code from various types of sources.
|
|
For example, BitBake can get source code from a disk with the metadata, from websites,
|
|
from remote shell accounts, or from Source Code Management (SCM) systems
|
|
like <filename>cvs/subversion/git</filename>.
|
|
</para>
|
|
|
|
<para>
|
|
Fetchers are usually triggered by entries in
|
|
<filename><link linkend='var-SRC_URI'>SRC_URI</link></filename>.
|
|
You can find information about the options and formats of entries for specific
|
|
fetchers in the BitBake manual located in the
|
|
<filename>bitbake/doc/manual</filename> directory of the
|
|
<link linkend='source-directory'>Source Directory</link>.
|
|
</para>
|
|
|
|
<para>
|
|
One useful feature for certain Source Code Manager (SCM) fetchers
|
|
is the ability to "auto-update" when the upstream SCM changes
|
|
version.
|
|
Since this ability requires certain functionality from the SCM,
|
|
not all systems support it.
|
|
Currently Subversion, Bazaar and to a limited extent, Git support
|
|
the ability to "auto-update".
|
|
This feature works using the <filename><link linkend='var-SRCREV'>SRCREV</link></filename>
|
|
variable.
|
|
See the
|
|
"<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-appdev-srcrev'>Using an External SCM</ulink>"
|
|
section in the Yocto Project Development Tasks Manual for more
|
|
information.
|
|
</para>
|
|
|
|
</section>
|
|
|
|
</chapter>
|
|
<!--
|
|
vim: expandtab tw=80 ts=4 spell spelllang=en_gb
|
|
-->
|