Accepting Constraints in Build Systems

As Embedded Systems become more complex, the complexity of the process to build the software for these systems also increases. As humans, our ability to deal with complexity is limited, so we develop tools and processes to manage the complexity. In the end, these tools and processes are about constraints and patterns. A well-designed tool or process encourages you to do things in a way that is consistent and maintainable, which leads to reliable and predictable results.


Git, Versioning, and Branching for Embedded Linux Development

When building a product using Linux, versioning and branching of your software is an important consideration. Everyone’s needs are different depending on the size of the team, culture, and testing requirements, so there is no one size that fits all. However, after working on a number of different projects for a dozen or so different companies, there are several practices that are often used.


Separation of Source and Build Directories

As we work with larger and more complex systems (i.e. Linux), more and more of our time is spent on integration and pulling different pieces together.  We often need to debug or understand code we did not write — especially in build systems.  To work effectively in this scenario you must be able to quickly search through a lot of source code.  Therefore, we are always looking for ways to make this more efficient.


Why Git?

Some time back, I gave a presentation that included an overview of the Git version control system.  I still occasionally get asked why Git should be used instead of Subversion, as it seems harder at first.  Most developers don’t really understand Git until they have used it for awhile, and then they will have an “aha moment.”  There are 3 features of Git that are especially interesting to me:

  1. many repositories (vs. one large repository)
  2. distributed development
  3. cheap branches

Best practices for using CMake in OpenEmbedded/Yocto

I’ve already written about using autotools and qmake in OE.  With recent projects, we’re using CMake to build most C/C++ components.  Like any good tool, CMake has a learning curve, but it seems to do its job quite well.  Below are examples of a CMake build file, and corresponding OE recipe.


OpenEmbedded Build Template

How does one set an OpenEmbedded/Yocto/Poky/Angstrom build? There are many options. Some include:

(I’m sure there are many others, feel free to add in comments …)

Over the past years, we’ve supported a number of customers using OpenEmbedded to develop products using various SOC’s. We also try to keep builds going for a number of different systems so that we can continuously evaluate the state of OpenEmbedded and related Embedded Linux technologies. We also needed a standard way to set up builds for customers (some don’t have a lot of experience with OE and Git) that is simple and consistent. What we ended up with is the BEC OpenEmbedded build template.

The goal is to have a quick entry point into OpenEmbedded that includes the necessary layers for a number of different machines, and automates a number of routine tasks such as installing images to a SD card, setting up a development feed server, etc. The build template is only updated when the build is stable and tested on a number of machines, so it provides a series of stable snapshots of OpenEmbedded and associated layers.

This build template currently tracks the master branches for all the layers used. This gives us a platform to track the latest OE changes. With most projects, that ability to use the features in the latest versions of software outweighs the stability benefits of OpenEmbedded release branches. There are times when the OpenEmbedded project goes through invasive changes (such as the systemd integration), and using the master branches is not practical, so in this case we simply use the last stable snapshot that builds and works. In most cases if there are issues, simply report or fix the issue and wait a week.

Perhaps the most controversial decision is the use of Git submodules for including OpenEmbedded layers. The Internet is full of rants against Git submodules. For heavy developer use, submodules may not be optimal. However, from a user perspective, Git submodules provide a simple mechanism for including external repositories in a project. If most of the submodules (OE layers) won’t be touched (typical OE user scenario), submodules work very well. The fact that Git locks down submodules to a specific commit ensures you are getting exactly what you think you are getting (vs a branch that may have been rebased, modified etc). If the git hash matches, you can be pretty sure it is the same as the last time you built it. This is an important factor in production build systems where you want to be sure of what you are building. Google repo is another option under consideration, but there are still some trade-offs to work through.

In the end, build systems are very personal, and must be customized for your product and development team. The number one requirement for a an Embedded Linux build system is that you can get repeatable builds with a single command. There must be no manual steps where human error can be introduced. This is just one way to accomplish this goal.


OS Containers for Build Systems

Since I’ve been running archlinux on some of my systems, one thing I’ve found useful is systemd-nspawn. systemd-nspawn containers (or chroots on non-systemd systems) give you a quick way to install a Linux distribution, that can run inside an existing Linux system.

Some cases where systemd-nspawn containers (referred to as containers in this document) are useful:

  1. At one point, OpenEmbedded would not build with GCC 4.8 (this is no longer the case with recent versions of OE).  So a Debian or Ubuntu OS container was a quick way to get builds going again.
  2. For product build systems (may live for many years), typically OE will eventually break as you upgrade the workstation distribution.  For projects that need a long-lived OpenEmbedded build system, setting it up in a chroot makes a lot of sense.
  3. Someone might be having a compile or build problem with a distribution you don’t currently have installed.  With containers, you can quickly set up a test distribution to reproduce problems.
  4. I’ve had cases where I need an older version of Qt for a project, but my workstation includes a newer version.  Again, setting up a OS container is sometimes simpler than getting two versions of Qt to dwell together peaceably in the same distribution.
  5. Backing up or replicating your entire build system is very easy — simply rsync the OS container directory to another machine.

So the solution is to select a relatively stable, long-lived distribution to host your product builds. Debian is good choice.  Because the container is simply a directory in the host workstation filesystem, you can use host workstation tools (editors, git, etc) directly in the container filesystem.  The only thing you need to use the chroot for is the actual building.  If you make sure the user ID is the same between your workstation and nspawn container, then permissions are seamless — you can easily access files in the container from the context of your host workstation.

To set up a nspawn-container:

  1. Install debootstrap.  On arch systems, this needs to be obtained for the AUR.
  2. host: sudo debootstrap –arch=amd64 wheezy ~/debian-wheezy/
  3. host: sudo systemd-nspawn -D ~/debian-wheezy/
  4. container: apt-get update && apt-get install ssh
  5. container: edit /etc/ssh/sshd_config, and set port to something other than 22 (23 in this example)
  6. container: /etc/init.d/ssh start

(This systemd-nspawn man page gives examples for setting up other distributions.)

To set up a user in your container:

  1. host: id
  2. will return something like: uid=1000(cbrake) gid=100(users) …
  3. container: adduser –uid 1000 -gid 100 cbrake
  4. host: ssh-copy-id -p 23 localhost (will copy public key to container)

Now, on the host system, you can simply “ssh -p23 localhost” any time you want to log into the container.  Soft links between the project workspace on the host system, and the container can also make shifting between the two easier.

An alternative way to start the container once its set up is:

sudo systemd-nspawn -D ~/debian-wheezy /sbin/init

Its also handy to make the shell prompt in the container slightly different than the host OS so that you can easily tell the difference.  To accomplish this, add the following to ~/.profile in the container OS:

export PS1=”[\u@wheezy \w]\$ “

To create a service that starts your container, put something like the following in /lib/systemd/system/debian-wheezy.service

Description=Debian Wheezy
ExecStart=/usr/bin/systemd-nspawn -D /scratch/debian-wheezy/ /sbin/init 3

Hopefully this gives you a quick overview of how OS containers can be set up, and used in your OpenEmbedded build systems.


Git submodules can now track branches

As of version 1.8.2, Git submodules can now track branches instead of specific commits.  This is good news as in many cases, this is exactly the behavior we want.  However, Git submodules are still not as flexible as Google repo, but since submodules are built into Git, the submodule command is a good solution in many cases.

The “git submodule update –remote” command is the key to tracking branches with submodules.   The following is from the Git man pages:

This option is only valid for the update command. Instead of using the superproject’s recorded SHA-1 to update the submodule, use the status of the submodule’s remote tracking branch. The remote used is branch’s remote (branch.<name>.remote), defaulting to origin. The remote branch used defaults to master, but the branch name may be overridden by setting the submodule.<name>.branch option in either .gitmodules or .git/config (with .git/config taking precedence).

This works for any of the supported update procedures (–checkout, –rebase, etc.). The only change is the source of the target SHA-1. For example, submodule update –remote –merge will merge upstream submodule changes into the submodules, while submodule update –merge will merge superproject gitlink changes into the submodules.

In order to ensure a current tracking branch state, update –remote fetches the submodule’s remote repository before calculating the SHA-1. If you don’t want to fetch, you should use submodule update –remote –no-fetch.

So, if you already have a Git submodule set up, its a simple matter to run git submodule update –remote to update the submodule to the latest master branch.  If you want a different branch, simple edit .gitconfig.

[submodule "meta-bec"]
   path = meta-bec
   url =
   branch = test

Now, if you run git submodule update –remote, Git will update the meta-bec submodule to the latest on the test branch.

This functionality is purely a convenience feature in the submodule update command.  In the actual repository, Git still stores submodules pointed to a particular commit.  The same thing could be accomplished with something like git foreach “git fetch && git checkout test”.  The branch option in .gitmodules functions more as documentation and convenience.  It is very handy to be able to look at .gitmodules and quickly determine that submodule X is tracking branch Y.  Normally, this would have to be documented elsewhere, or figured out in some other way.  Also, for build systems where you want the build to always track the production branches of various projects, update –remote gives you a convenient way to update the build tree.


Git and Distributed Development

This is part of an ongoing series of articles on the Git version control system.

The “many repository” paradigm has been partly driven by the distributed development paradigm.  Git solves the problem of multiple developers working in multiple repositories very well.  Because we want to use and customize projects like the Linux kernel, U-boot, and OpenEmbedded in our projects, then we naturally find ourself in the situation where we need to manage multiple repositories.  Yes, you can check the Linux kernel into your company Subversion repository, but you are much better off long term if you bite the bullet and implement your own Git infrastructure.

As we consider the product development process, we need to consider the life cycle of a product.  Most products live for at least several years, and will go through several software iterations.  If we can update the software components we use, then we can add value to the product in the form of new or updated drivers to support new peripherals, new libraries, performance improvements, etc.  But, we are dealing with millions of lines of source code, so we must have an efficient way to deal with software projects of this size.  The below Figure 2 illustrates how you might organize a typical project.  Notice we can pull updates from the U-boot and Kernel source trees at any time in the development process, and merge the changes with our own modifications.  We might have an outside team working an application, and we then easily synchronize the repositories when it makes sense.

There are many other design flows possible.  Once you have the ability to support multiple branches and repositories easily, it becomes trivial to implement a staging/testing repository for a QA processes, maintenance repositories for supporting old releases, etc.

Even at a personal developer level, Git’s distributed capabilities offers many advantages.  Each Git workspace is actually a full Git repository.  This means you can check changes in locally, re-organize your changes, be able to track changes when off-line, etc.  For this reason, many developers are now using git-svn when they need to work with Subversion repositories.  With git-svn, you have all the benefits of using git locally, and yet you can easily synchronize with Subversion repositories.  And this leads us to our next topic: cheap branches (coming soon).


Git and Why Multiple Repositories

This is part of an ongoing series of articles on the Git version control system.

This article discusses the trend in software configuration management toward multiple repositories, rather than one large repository.  In the past when many companies used Subversion or comparable systems, there was typically one huge company repository (I’ve seen them in the 10’s of GB in size) that held a hierarchical tree of all the company source code.  This worked fairly well, and is comfortable in that the organization is very similar to a file system directory structure.  However, this model is not very flexible in that it does not have a consistent way to re-use components between different projects.  Some people simply copy source code.  Some have a “common” project that is included in all their other projects.  Subversion externals can be used.  With Git, typically a separate repository is created for each software component.  There are perhaps several reasons for this, but one reason is that Git simply does not scale to huge multi-GByte repositories.  However, this turns out to be a blessing in disguise as I think it is a better model in many cases.  What we end up with is more of a catalog of software components rather than a rigid hierarchy.

There is much emphasis these days on modular, re-usable software components (Object Oriented Programming, plugins, etc.).  Why not keep things modular at the repository level?  Another example of this type of organization is the Internet itself.  It is not a hierarchy of information, but rather a flat system that is organized by hyperlinks.

One of the benefits of organizing your source code this way is that it encourages clean boundaries between software components.  Each software component needs to stand on its own without being propped up by header files located in an unrelated source tree 3 levels up in the directory hierarchy.  This type of organization forces us to make better use of standard build system practices.

How do you implement this type of repository infrastructure?  Build systems such as OpenEmbedded, Gentoo, and the Android build system manage this fairly well.  However, Git also includes a feature named “submodules” that provides a mechanism for one repository to reference source code from another.  What you end up using really depends on your needs, and what you are trying to accomplish.

A screencast is also available that covers this topic.


Git Overview Screencast

This screencast (use Firefox to view the screencast) provides an overview of the Git version control system.  There are 3 features of Git that are especially interesting:

  1. many repositories (vs. one large repository)
  2. distributed development
  3. cheap branches

The fundamental driver for better tools is increasing system complexity.  More and more we are required to manage and integrate more third party software, work in distributed teams, and more effectively re-use the software we do have.  Git is a tool that helps you accomplish these goals.


OpenEmbedded srctree and gitver

Recently an OpenEmbedded class name srctree became usable.  The srctree.bbclass enables operation inside of an existing source tree for a project, rather than using the fetch/unpack/patch idiom.  The srctree.bbclass in combination with the OpenEmbedded gitver.bbclass and git submodules provides a very interesting way to build custom software with OpenEmbedded.

One of the classic problems with OpenEmbedded is how do application and kernel developers use it.  While OpenEmbedded excels at automating image builds, it is less friendly when used as a a cross development tool.  Historically there are several options for iterative development:

  1. develop in the working directory: cd <tmp>/work/arm…/<my recipe>; ../temp/run.do_compile …
  2. a variation of #1: bitbake -c devshell <my recipe>
  3. manually set up an environment that uses the toolchain generated by OE.  As example see this script in the BEC OE template.
  4. a variation of #3: export a SDK that includes a toolchain and libs

While the above solutions work OK, the process can be a little cumbersome.  Unless your OE recipe pulls software directly from the TIP of a SVN repository, you may have to manually update the recipe after you make changes, create patch files, etc.  There is also the problem that if your recipe fetches the latest from SVN, it drastically slows down the recipe parsing as it has to check the repository for a new version every time the recipe is processed.

The optimal solution would to be to simply check a software component out of a version control system, and build it directly using OpenEmbedded.  Icing on the cake would be if the package generated would automatically extract version information from the version control system.  This would facilitate iterative development for software components that need to be cross compiled.

Although srctree can be used with any directory of source code, it really works best with a git repository.  The gitver.bbclass provides a GITVER variable which is a (fairly) sane version, for use in ${PV}, extracted from the ${S} git checkout, assuming it is one (text from recipe).  gitver uses the ‘git describe’ command to extract the last tag and uses that for the version.

The best way to illustrate the use of these tools is an example:

The easiest way to try this is clone the above project in your openembedded/recipes directory:

$ cd openembedded/recipes
$ git clone
$ cd autotools-demo
$ git describe
$ git tag -l

Notice that git describe simple returns the latest tag.   The recipe can be located in the same directory as the source code and has the following contents:

# recipe to build the autotools-demo in the working tree

inherit srctree autotools gitver

PV = "${GITVER}"

Can’t get much easier than that!  If you build the recipe, you end up with a package:

$ bitbake autotools-demo
$ ls tmp/deploy/glibc/ipk/armv5te/autotools-demo_1.1-r0.6_armv5te.ipk

Now, what happens if you make changes and commit them?

$ cd .../autotools-demo
$ (make a change and commit)
$ git describe
1.1-1-gfbc1ecc (notice the count and hash automatically appended)
$ (make another change and commit)
$ git describe
1.1-2-g7ad3715 (notice the count is now 2)

If we bitbake the recipe now, we end up with a packaged named:


The gitver class in OpenEmbedded automatically takes care of creating a usable PV (package version) that always increments.

So in summary, srctree and gitver give developers a convenient way to handle custom components that change often in an Embedded Linux build without increasing parse times, requiring manual tweaks to version numbers, or creating a separate workspace for each version of the application that is built.  As practices such as continuous integration become more common, OpenEmbedded features like this are increasingly needed.  An added benefit is that the OpenEmbedded recipe can now be stored in the same location as the source code.  Perhaps in the future, most applications will include an OpenEmbedded recipe as part of their source code and git submodules could be used to simple populate the components you want to use.

2017-11-08 update: A tool named devtool is now the preferred way to do much of the above.


Installing OMAP3 images on a SD card

This article and screen-cast is a continuation of that last couple posts describing the BEC OE build template.  The purpose again for a build system is to automate tedious manual tasks, and in doing so, we end up documenting how the build system works.  Having a good build system is important during product development so that you have an easy, repeatable process for building and deploying images.  One of these tasks is setting up a SD card and installing images to the SD card that an OMAP system can boot from.  This screencast goes over the script and makefile targets that are used to automate this process.

In summary, the steps to set up a SD card and install an image using the build template are:

  1. cd <build template directory>
  2. determine the SD card device name (/dev/sdX)
  3. unmount any file systems mounted on the SD card
  4. sudo ./scripts/ /dev/sdX  (make sure you have the right device!!!)
  5. make omap-install-boot
  6. make omap-install-<image name>-rootfs

Previous articles in this series:


OpenEmbedded Build Template

Setting up an OpenEmbedded build is a fairly simple process if you carefully follow the instructions.  There are also a number of scripts available that automate the setup such as the OpenEmbedded Tools for Eclipse, the Angstrom setup scripts, the KaeilOS Openembedded Manager, and I’m sure there are many more (feel free to add in comments section).  As we have helped a number of clients use OpenEmbedded in commercial projects, we have the following requirements for a build template:

  1. The build needs to be completely automated such that only one step is required to build images.
  2. We need to be able to lock down versions of various components.
  3. Needs to be as simple as possible.

As we continue to think about creating a build template, the primary reason for creating one is to get repeatable builds with very little effort.  The build template serves as documentation; how did we build the 1.10 release.  A good build system allows any developer to create any release on about any machine at any time.  A good build system does not depend on the “golden build machine.”  We have been using makefile wrappers around OpenEmbedded for several years now to accomplish much of this.  These wrappers were typically stored in a Subversion repository with the rest of the project files.  However, now that most open source projects have moved to Git repositories, there are significant advantages to using Git for the build system.  Git has a feature called submodules which allows you to embed a git repository inside a source tree, always pointed at a particular commit.  In concept, this is perfect.  In practice, Git submodules are sometimes difficult to use, but considering the functionality provided, I think it is a reasonable trade-off.

The use of Git encourages a different development paradigm.  Typically when people use Subversion within a company, there is one repository that stores everything.  Subversion is actually quite good at this.  I’ve run into a number of companies who have repositories many GB in size that have been running for many years.  The stability is remarkable.  Git does not scale in this way, so people tend to set up a separate repository for each component, application, etc.  While there may be some disadvantages to this model, there are also significant advantages, and we can learn some of these by observing Open Source Software (OSS) development.  Just as we strive for modularity in software development where each class can stand on its own and be generally useful, OSS maintains this dynamic at the project level.  Each OSS project moves at its own pace, largely decoupled from other projects.  This dynamic forces a level of organization that might be neglected otherwise.  Likewise, in any project, if each major component is stored in its own source code repository, there will also tend to be less coupling, willy-nilly including of header files from other components, etc.  It helps people think in a more modular manner at the component level.  A component may be the Linux kernel, bootloader, each application in the system, etc.  A similar concept may be thinking about storing documents in hierarchical filesystem directory structure versus documents in a Wiki, which are stored in a flat structure and then organized by cross linking.  Yes, there are times to use a directory structure, but flexibility of a linked organization system (the essence of the Internet) is hard to beat.  Git submodules (or similar concepts) provides a similar linking method of organization.

With this in mind, we look at a build system as a way to collect a number of independent components and build them in a repeatable fashion.  While this may seem obvious to anyone somewhat familiar with OpenEmbedded, our vision is to extend this concept to easily facilitate product development so that we can move quickly and efficiently, but yet in a controlled and reliable way.  The first step is a new build template that is available here.  The following video provides a short overview of how to use the build template (can be viewed in Firefox).

The functionality provided so far is fairly simple and provides a way to quickly set up a build environment.  With Git submodules, the versions of Bitbake and the OpenEmbedded are locked down.  Submodules are also used to provide an array of components to choose from, and the user initializes only the submodules they are interested in using.  The first example of this is providing both the recipe meta data from the OpenEmbedded project or the modified version from Gumstix.  In the future, examples will be provided for doing custom application and kernel development using the OE srctree and gitver classes.  Feedback is welcome; please let me know if you have questions or suggestions.


Notification at the end of builds

I do quite a few OpenEmbedded project builds during the course of a week.  This process usually takes 3-5 minutes.  That is just enough time to get distracted doing something else and forget about the build until an hour later when you realize — oops, I was supposed to send out a release email once the build was finished and uploaded.  It occured to me that it would be nice if my computer played a distinct sound at the end of the build.  With Linux this is incredibly easy:

  • wget
  • sudo mv Ding_Dong_Bell.ogg /usr/local/
  • sudo aptitude install cplay
  • sudo echo “cplay /usr/local/Ding_Dong_Bell.ogg” > /usr/local/bin/bell
  • sudo chmod 775 /usr/local/bin/bell

Now, when I have a long build, I simply do something like:

bitbake my-image; bell

There is probably something better than cplay that does not start a UI — any suggestions?