Finding your way through Yocto

One of the strengths of the Yocto Project is the documentation. Not only do they provide the wonderful all-in-one mega-manual, which works great as a reference to search when developing with Yocto, they also provide more focused documentation such as the Development Tasks Manual which explains many of the common tasks involved in developing a Yocto-based embedded system. Another good source of information for beginners and advanced users alike is the official YouTube channel.

However, all this information can be confusing when just starting out, so what are the first steps that need to be taken to configure a working Yocto-based embedded system?

The definition of a hardened Linux system can vary depending on the requirements of the product, but here we will consider hardening as increasing the security of a system. This means not including any unnecessary software, as well as carefully choosing which software to include based on its complexity, quality, and security awareness, not just its offered features. Configuring the included software to only provide the necessary functionality also contributes to a hardened system. Standards for the development of secure embedded systems, such as IEC 62443, IEC 62304 and EU Cyber Resilience Act (CRA), specify concrete design requirements. A hardened Linux distribution is often a minimal one, where only the necessary features are available and nothing more.

As an example, glibc is a C library which supports a lot of use cases but that means it is large and highly complex. muslc on the other hand is more minimal, meaning it’s smaller and less complex than glibc, resulting in a smaller attack surface. The number of CVEs against glibc vs muslc, 144 vs 5 according to www.cvedetails.com, reflects the difference in size and complexity. That’s not to say muslc is better software than glibc, but when the extra functionality provided by glibc isn’t strictly necessary, muslc may be the better choice when putting together a hardened system.

When configuration options to turn on or off certain features exist for a piece of software, Yocto can make use of these to allow further hardening. Continuing with the libc example from above, if glibc is chosen as the libc, it’s very unlikely that all features offered by glibc will be needed, for example profiling can be disabled through –disable-profile, as the feature to generate profiling data is unlikely to be needed, especially in a production environment.

Another good example is the the configuration of the Linux kernel. It can be tempting to turn on many unnecessary configuration options in order to provide more functionality out of the box, such as optional hardware support or additional file systems. However, this results in more code being included in the compiled kernel unnecessarily, which in turn increases the cyber security surface of the Linux kernel.

When an image is built, a .manifest file is created listing the packages included in the rootfs. The .manifest doesn’t include all the packages used during the build of the image, nor the locations from which the source code was fetched, among other information useful for supply chain management.

The Yocto Project provides a way to generate a Software Bill of Materials (SBOM), which, for example, can be used to meet open source compliance requirements. It does this using the SPDX standard, an open standard developed by the SPDX Project, another Linux Foundation project.

The SBOM information isn’t generated during a build by default, but can easily be enabled by adding INHERIT += “create-spdx” to local.conf or to a custom distro configuration file. This causes the build to inherit the create-spdx class and will then generate SPDX files for all recipes included in the build, as well as the generated images and runtime dependencies. If the SPDX output will be read by human eyes, also adding “SPDX_PRETTY=1” in the chosen configuration file will make the output more readable.

An SPDX file for a recipe contains the download source of the software, the version of the software, the license of the software, and the dependencies needed for the build, among other things. The dependencies listed include the compiler and other tools used during building, as well as libraries needed during build time and run time. This makes it easy to tell not only which software is included in an image, but also why. 

For example, a rootfs recipe will usually just specify the software packages required to provide the functionality for the embedded system, such as graphics libraries, networking tools, or custom applications. One or more of these may rely on a compression library, such as zlib, but it would be unusual to include zlib directly in the rootfs recipe. From the generated SBOM it is possible to see which software has the dependency on zlib. Furthermore, information about zlib is also generated, so it can be seen where the zlib source code comes from, which version is used, and even if it pulls in other dependencies.

The SBOM can be used also as the starting point for an in-depth security monitoring of the embedded Linux system, including classification of and possibly fixes for vulnerabilities in constituent packages. More information on CVE Security Management and Vulnerability handling can be found on the emlix website.

There is a lot of overlap between maintainability and hardening embedded Linux systems. Both benefit from only including the minimum number of packages necessary. Both benefit from high quality software. The planning and implementation of maintainability is considered key to reducing life cycle costs (maintenance and security updates) for Yocto-based industrial products. In order for a Yocto BSP to be maintainable there are many good practices which should be followed. In this section we will highlight a few of these.

Use existing, well-maintained layers and recipes where possible. Recipes already exist for much of the software that is commonly included into an embedded Linux system. The benefit of relying on these rather than writing new recipes is that when the software changes upstream, the necessary changes to the recipe will be made by the recipe maintainers, thus reducing your maintenance burden.

Use appends files to customise existing recipes. If an existing recipe doesn’t quite fulfil the needs of the project it can be extended or modified by adding a custom .bbappend file. A Yocto recipe may by default enable all of the available features a piece of software offers, not all of them being necessary for all systems. A .bbappend file can be used to reduce this configuration to what’s strictly needed, hardening the system in the process. The version of the software which will be built can also be changed in the .bbappend file, allowing a newer or, as sometimes required, older version to be used. Using appends files rather than writing recipes from scratch allow a project to benefit from the upstream maintenance effort put into upstream recipes while still having full control over how its included packages are configured and built. Appends files may need to be updated when an upstream recipe changes, but the amount of effort required is far lower than keeping a bespoke recipe updated.

Keep custom recipes and appends in a project specific layer. This has two advantages. Keeping the changes separate from the upstream layers makes it much easier to see what the project specific behaviour is and it also means that when an update of the Yocto layers takes place, the changes don’t get lost.

Regularly syncing with upstream layers is also important for maintainability. Upstream syncs often bring fixes and additional features from the upstream maintainers and may even make custom overrides done in .bbappend files redundant. This raises the question, “How often is there a new release?”

Brief aside: Release cadence

The Yocto Project does a major release every 6 months, roughly in April and October. Each major release has an associated codename as well as a release number, e.g. gatesgarth 3.2 or kirkstone 4.0. Codenames are used because the release number may conflict with the versioning scheme from a layer, software package, or company.

Each release comes with strong testing and quality assurance guarantees, including testing on a variety of platforms, to ensure that no regressions creep in on update. All the testing is publicly visible and can also be used to validate projects with configurations not covered by the upstream testing.

The Yocto Project typically supports a major release with bug fixes and security fixes for seven months, but every fourth release is selected for long term support. These Long Term Support (LTS) releases are supported for four years. This allows for a more stable base while still receiving important fixes. The overlap between LTS releases is 2 years, more than enough time to prepare and carry out the upgrade. 

The current LTS releases are kirkstone 4.0, released in 2022, and scarthgap 5.0, released in 2024.