# 2018-05 Yocto Presented by [Nathan Genetzky](http://nathan.genetzky.us). ___ ## What will be covered in this talk? 1. What is the build environment for Yocto? 2. What inputs are supplied for a build? 3. What are the possible outputs of the build? 4. What steps are involved to produce these outputs? ___ ## What will be *not* covered in this talk? 1. How to setup a computer for the first Yocto build [[1](https://www.yoctoproject.org/docs/2.4.2/yocto-project-qs/yocto-project-qs.html#yocto-project-qs-intro)] 2. How to create a custom distro or board support package [[2](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#metadata-machine-configuration-and-policy-configuration)] ___ ## Projects associated with Yocto 1. OpenEmbedded(oe): Metadata (recipes) for majority of system [[1](https://www.yoctoproject.org/software-item/openembedded-core/)] 2. Bitbake: Build system [[2](http://www.yoctoproject.org/docs/2.4.2/bitbake-user-manual/bitbake-user-manual.html)] 3. Poky: Reference distribution and reference bsp [[3](https://www.yoctoproject.org/software-item/poky/)] 4. CROPS: Docker containers and plugins for Yocto. [[4](https://github.com/crops/)] 5. Toaster: Web GUI for build system [[5](http://www.yoctoproject.org/docs/2.4.2/toaster-manual/toaster-manual.html)] 6. Extensible SDK (ESDK) [[6](https://www.yoctoproject.org/docs/2.4.2/sdk-manual/sdk-manual.html)] --- # Structure - Overview ___ ## Input 1. Bitbake reads a series of conf files to determine which other conf files, layers, and global variables to use. 2. Bitbake uses variables from these conf files to determine where to look for recipes, recipes appends, classes and includes. ___ ## Caching - [Downloads](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#structure-build-downloads) stores the output of the 'fetch' task and is used as an input to the 'unpack' task. - [Shared State Cache](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#structure-build-sstate-cache) stores the results of tasks which than can be used instead of rerunning the task in some scenarios. Read the manual for more information about [Shared State](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#shared-state). ___ ## Output 1. Everything under the [build directory](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#structure-build) should be considered disposable. 2. The configuration files in build/conf should be generated before bitbake is run. 3. By default the caching directories mentioned above are also in the build directory. --- # Input - Bitbake ___ ## Bitbake BitBake is a generic task execution engine that allows shell and Python tasks to be run efficiently and in parallel while working within complex inter-task dependency constraints. ___ ![user-configuration](https://www.yoctoproject.org/docs/latest/mega-manual/figures/user-configuration.png) [Copyright © 2010-2018 Linux Foundation](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html) ___ ## env - init-build-env - BBPATH: List of base paths that bitbake will search. - BB_ENV_EXTRAWHITE: Only certain variables affect build. - BUILDDIR: Many variables are defined relative to this path. - PATH: Add bitbake/bin and poky/scripts. - PWD: Change Directory into $BUILDDIR [demo/oe-init-build-env](/demo/oe-init-build-env) ___ ## conf - bblayers Configure inputs to Bitbake - BBPATH: List of base paths that bitbake will search. - BBFILES: List of recipe files used by BitBake. - BBLAYERS: List of paths to layers. ```python # ${BUILDDIR}/conf/bblayer.conf BBPATH = "${TOPDIR}" BBFILES ?= "" BBLAYERS ?= " \ /tmp/poky/meta \ /tmp/poky/meta-poky \ /tmp/poky/meta-yocto-bsp \ " ``` ___ ## conf - layer Configure additional inputs to Bitbake - BBPATH: List of base paths that bitbake will search. - BBFILES: List of recipe files used by BitBake. ```python # ${LAYERDIR}/conf/layer.conf BBPATH .= ":${LAYERDIR}" BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \ ${LAYERDIR}/recipes-*/*/*.bbappend" ``` --- # Input - Variables ___ ## conf - bitbake.conf 1. Defines 90% of the "magic" variables. 2. Configures sane defaults. 3. Includes other 'conf' files. This exert shows how the other 'conf' are included. ```python # meta/conf/bitbake.conf include conf/site.conf include conf/auto.conf include conf/local.conf include conf/machine/${MACHINE}.conf include conf/machine-sdk/${SDKMACHINE}.conf include conf/distro/${DISTRO}.conf include conf/distro/defaultsetup.conf ``` ___ ## conf - local.conf Please edit me! This file allows you to quickly and drastically change the build. Note: All edits should be disposable, but consider designing a 'local.conf.sample' to use as a template (see TEMPLATECONF). ```python # conf/local.conf MACHINE ??= "qemux86" DISTRO ?= "poky" EXTRA_IMAGE_FEATURES ?= "debug-tweaks" ``` ___ ## conf - distro/machine Out of scope for this talk. See [Metadata, Machine Configuration, and Policy Configuration](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#metadata-machine-configuration-and-policy-configuration). --- # Input - Recipes ___ ## recipe - bb (1/2) Package recipes provide: 1. Metadata about packages. 2. Instructions to fetch, build and install packages. 3. Additional tasks related to packages. ```bash # Typical recipe path: ${LAYERDIR}/recipes-${SECTION}/${PN}/${PN}_${PV}.bb # Typical packages: (in ${TOPDIR}/tmp/deploy/deb/${PACKAGE_ARCH}/) ${PN}_${PV}-${PR}-${DPKG_ARCH}.deb ${PN}-dev_${PV}-${PR}_${DPKG_ARCH}.deb ${PN}-dbg_${PV}-${PR}_${DPKG_ARCH}.deb ``` [[1](https://www.yoctoproject.org/docs/2.5/bitbake-user-manual/bitbake-user-manual.html#recipes)] ___ ## recipe - bb (2/2) Fundamental Blocks: [variables](https://www.yoctoproject.org/docs/2.5/bitbake-user-manual/bitbake-user-manual.html#basic-syntax) , [functions](https://www.yoctoproject.org/docs/2.5/bitbake-user-manual/bitbake-user-manual.html#functions) , and [tasks](https://www.yoctoproject.org/docs/2.5/bitbake-user-manual/bitbake-user-manual.html#tasks) Features that make recipes awesome: 1. Flexible variable manipulation and expansion. 2. Functions can be written in shell or python. 3. Inline python to generate recipe code. [[1](https://www.yoctoproject.org/docs/2.5/bitbake-user-manual/bitbake-user-manual.html#inline-python-variable-expansion)] 4. Dependency management between tasks. [[2](https://www.yoctoproject.org/docs/2.5/bitbake-user-manual/bitbake-user-manual.html#dependencies)] ___ ## recipe - bbappend 1. Append additional code to existing recipe. 2. Most commonly used with recipe from another layer. 3. '%' can be used as a wildcard as part of ${PV}. ```bash # Typical recipe append paths: ${LAYERDIR}/recipes-${SECTION}/${PN}/${PN}_%.bbappend ${LAYERDIR}/recipes-${SECTION}/${PN}/${PN}_${PV}.bbappend ``` ___ ## recipe - bbclass 1. Inheritance for recipes. 2. Every recipe inherits `base.bbclass`. 3. Located in: `${LAYERDIR}/classes/` 4. Used with: `inherit my_class_name` [[1](https://www.yoctoproject.org/docs/2.5/mega-manual/mega-manual.html#structure-meta-classes)] [[2](https://www.yoctoproject.org/docs/2.5/mega-manual/mega-manual.html#ref-classes)] [[3](https://www.yoctoproject.org/docs/2.5/bitbake-user-manual/bitbake-user-manual.html#inherit-configuration-directive)] ___ ## recipe - inc 1. Code sharing between recipes. 2. Optional with: `include relpath/to.inc` 3. Required with: `require relpath/to.inc` 4. Often located near recipes they are used for. [[1](https://www.yoctoproject.org/docs/2.5/bitbake-user-manual/bitbake-user-manual.html#include-directive)] [[2](https://www.yoctoproject.org/docs/2.5/bitbake-user-manual/bitbake-user-manual.html#require-directive)] --- # Output ___ ## Output - Downloads <img src="https://www.yoctoproject.org/docs/latest/mega-manual/figures/source-input.png" width="400" height "400" /> [Copyright © 2010-2018 Linux Foundation](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html) ___ ## Output - Packages <img src="https://www.yoctoproject.org/docs/latest/mega-manual/figures/package-feeds.png" width="400" height "400" /> [Copyright © 2010-2018 Linux Foundation](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html) ___ ## Output - Images <img src="https://www.yoctoproject.org/docs/latest/mega-manual/figures/images.png" width="400" height "400" /> [Copyright © 2010-2018 Linux Foundation](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html) --- # Bitbaking - base ___ ## Bitbake - tasks 1. Fetch - get the sourcecode, 2. Extract - unpack the sources 3. Patch - apply patches for bug fixes and new capability 4. Configure - set up your environment specifications 5. Build - compile and link 6. Install - copy files to target directories 7. Package - bundle files for installation. [[1](https://www.yoctoproject.org/docs/what-i-wish-id-known/)] ___ ## Bitbaking - fetch+unpack <img src="https://www.yoctoproject.org/docs/latest/mega-manual/figures/source-fetching.png" width="400" height "400" /> [Copyright © 2010-2018 Linux Foundation](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html) ___ ## Bitbaking - patch <img src="https://www.yoctoproject.org/docs/latest/mega-manual/figures/patching.png" width="400" height "400" /> [Copyright © 2010-2018 Linux Foundation](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html) ___ ## Bitbaking - config+compile+install <img src="https://www.yoctoproject.org/docs/latest/mega-manual/figures/configuration-compile-autoreconf.png" width="400" height "400" /> [Copyright © 2010-2018 Linux Foundation](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html) ___ ## Bitbaking - package+package_split <img src="https://www.yoctoproject.org/docs/latest/mega-manual/figures/analysis-for-package-splitting.png" width="400" height "400" /> [Copyright © 2010-2018 Linux Foundation](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html) --- # Bitbaking - pypi+setuptools ___ ## pypi - metadata 1. `${PN}` and `${PV}` are extracted from recipe name 2. pypi extracts `${PYPI_PACKAGE}` from `${PN}`. ```python # 72 character description of package. SUMMARY = "" # Update based on LICENSE, PKG_INFO, or setup.py. LICENSE = "MIT" # Runtime Dependencies based on requirements.txt or setup.py: RDEPENDS_${PN} += " ${PYTHON_PN}-json " ``` ___ ## pypi - fetch+unpack 1. fetch: `https://pypi.org/project/${PYPI_PACKAGE}/` -> `${DL_DIR}` 2. unpack: `${DL_DIR}` -> `${S}` ```python # Short (~25 LOC) class from meta-openembedded/meta-python. inherit pypi # Update with file that indicates or contains the LICENSE. LIC_FILES_CHKSUM = "file://PKG-INFO;md5=49903ea9c586b357ceb682e761818804" # Update with hash of the package downloaded from pypi. SRC_URI[md5sum] = "116d9dd1115054674c4eb2b13a09aea2" SRC_URI[sha256sum] = "96bd60c51a34d7844b89197d7f19a855e249635a4e23dbcf5b892a10e301c032" ``` ___ ## setuptools - compile+install 1. Typically use `inc` to support Python 2 and 3. 2. compile: `python setup.py build` 3. install: `python setup.py install` ```python # python-tinydb_${PV}.bb require python-tinydb.inc inherit setuptools # python3-tinydb_${PV}.bb require python-tinydb.inc inherit setuptools3 ``` --- # Yocto - WHAT I WISH I’D KNOWN [[1](https://www.yoctoproject.org/docs/what-i-wish-id-known/)] ___ ## Layer Index 1. Get to know the layer index ([layerindex](http://layers.openembedded.org/layerindex/branch/master/layers/)) 2. Use existing BSP layers from silicon vendors when possible ([layerindex/machines](http://layers.openembedded.org/layerindex/branch/master/machines/?q=&browse=1)) 3. Find recipes ([layerindex/recipes](https://layers.openembedded.org/layerindex/branch/master/recipes/)) ___ ## Dependency Graphs Know that you can generate a dependency graph and learn how to do it - [Generating Dependency Graphs](http://www.yoctoproject.org/docs/2.5/bitbake-user-manual/bitbake-user-manual.html#generating-dependency-graphs) - [How to view or work with Graphviz Dot files](https://github.com/rakhimov/cppdep/wiki/How-to-view-or-work-with-Graphviz-Dot-files) ```bash # Generate bitbake core-image-minimal -g # Convert dot into PDF dot -Tpdf 'recipe-depends.dot' -o 'recipe-depends.pdf' # Grep can quickly reveal dependencies grep 'readline.do_build' 'task-depends.dot' ``` ___ ## Building more than just images 1. Build just a package, and install manually. [[1](https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/package-manager-white-paper.pdf)] 2. [Running Specific Tasks](https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#dev-debugging-taskrunning) ___ ## Recipe vs Package An ambiguous definition: Package vs Recipe - `bitbake` a recipe to create one or more packages. ```python # PN is derived from recipe filename PN = "readline" # recipe has build dependency on glib recipe DEPENDS = " glib " # readline-dev package has runtime dependency on readline package RDEPENDS_${PN}-dev = " ${PN} " ``` ___ ## Packaged in rootfs 1. [Buildhistory](https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#maintaining-build-output-quality) 2. Work directory for an image: `${WORKDIR}/rootfs` ___ ## Buildhistory [Buildhistory](https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#maintaining-build-output-quality) contains so much information. 1. find out what packages are created by a recipe 2. find out what files are in a package 3. find out what files are in an image ___ ## Add package to an image 1. Add to local.conf 2. Add to image recipe 3. Add to packagegroup recipe 4. Add to RDEPENDs of another recipe (if applicable) ```python # local.conf or my-image.bb IMAGE_INSTALL_append = " package-name " # my-packagegroup.bb or my-recipe-that-uses-it.bb RDEPENDS_${PN} = " package-name " ``` ___ ## Add ssh server to image [Adding packages to your OS image](https://wiki.yoctoproject.org/wiki/Cookbook:Example:Adding_packages_to_your_OS_image) ___ ## Bitbake Environment generate and understand bitbake environment 1. Use '`--environment`' and read the output. 2. Use '`-c devpyshell`' and python ```python # Save environment for recipe into file. bitbake core-image-minimal -e > core-image-minimal.env.bb # Grep can quickly show how environment was set grep '^MACHINE_ARCH' core-image-minimal.env.bb -B 3 ``` ```python bitbake core-image-minimal -c devpyshell d.getVar('MACHINE_FEATURES').split() ```