Why Docker? When using OE to build software for products, we often run into the scenario where software needs to be built using the same version of OpenEmbedded over the course of several years. Production builds need to be predictable. We’ve also observed that old versions of OE often break as new Linux distros come out. This is just the result of the complexity of building tool chains. Additionally, for predictable builds you really don’t want to be changing the build OS. This requirement automatically rules out Arch Linux, Debian Unstable, Gentoo, etc as production build machines. Additionally, having developers debug OE build issues on varying workstation distributions is frustrating and time consuming.
Go has a pretty neat development environment, and its helpful to set up a standard GOPATH on your workstation up front. This is what I do:
- mkdir ~/go
- add the following to .bashrc (or some file that configures your env on login)
- export GOPATH=~/go
- export PATH=$GOPATH/bin:$PATH
Now, after you log in, you can do things like:
In the quest for technologies that work well for embedded Linux systems, I recently gave Go another try. The last time I tried this was very early on and there were some floating point issues on ARM that appear to be fixed now. Having spent a few days porting an existing application to Go, there is a lot to like about Go.
With the explosion of IOT (Internet of things), there are now more technologies we can use to build systems. In reality, we’ve been doing IOT for years. We’ve been networking devices for a long time. We’ve been collecting data from remote nodes. This is nothing new, but what the IOT movement brings to the table is technologies that are much lower cost, and more standardized. Two of these technologies are MQTT and CoAP. Both very interesting, and very useful. Recently, I helped a system manufacturer think through the architecture of a system with the following requirements:
Recently, I sent a survey to the OpenEmbedded and Yocto mail lists asking a series of questions about how they use OE. 38 people responded. The average number of years using OE is 4.8 years.
One of the common things we do during Linux kernel development is move a series of patches from one kernel version to a similar version (say Linux 4.1 to 4.1.12). This is required as new stable versions of particular kernel version are released. One approach is to merge, but then your changes are mixed in with upstream commits and are more difficult to manage. Git rebase offers a convenient way to move a set of patches. In the following example we have a series of changes we made (or patches we applied) on top of the 4.1 kernel.
Recently, I needed to enable the eject command in BusyBox for an OpenEmbedded (Yocto) based project. Below is a way to do this in a fairly painless way:
- bitbake -c menuconfig busybox (enable the eject command in the config and save)
- bitbake -c diffconfig busybox (this generates a config fragment, note the fragment file location)
- recipetool appendsrcfile -w [path to layer] busybox [path to fragment generated in step #2]
As I work at the intersection of hardware and software, I spend a fair amount of time reading schematics for printed circuit boards (PCBs). I’m astounded at how sloppy most schematics are, and have come up with a few suggestions to consider.
Is the size of the libraries required for C++ (vs C) a concern in Embedded Linux systems? Most Embedded Linux systems likely include some C++ code, so this is probably not even a decision we need to make in many cases. However, often there is a need for a small initramfs that is used as part of the boot process (perhaps for software updates) before switching to the main root file system. In this case, it makes sense to keep the initramfs as small as possible, and we might be concerned here with the size of C++ libraries.
Recently I was asked why use systemd vs sysvinit in embedded systems? There are many discussions on this, and really most of the reasons people use it for servers and desktops are also valid for embedded systems. Lennart Poettering’s articles explain very well why you might want to consider systemd. A few things that rank high on my list:
With the BEC OE build template, you can easily set up an opkg feed server that serves up packages from your build directory. This allows you to easily install new packages during development, without generating and reflashing an entire image. To use:
- edit local.sh, and define MACHINE_IP to point to your target machine, and re-run “source envsetup.sh”. Alternatively, you can export the MACHINE_IP variable in your environment.
- run: oe_setup_feed_server
- run: oe_feed_server
On target system:
- opkg update
- opkg install <some package>
Opkg will install dependencies, so if you have a complex package, this is so much easier than copying over an opkg file manually, and figuring out you need 6 other packages as dependencies.
How does one set an OpenEmbedded/Yocto/Poky/Angstrom build? There are many options. Some include:
(I’m sure there are many others, feel free to add in comments …)
Over the past years, we’ve supported a number of customers using OpenEmbedded to develop products using various SOC’s. We also try to keep builds going for a number of different systems so that we can continuously evaluate the state of OpenEmbedded and related Embedded Linux technologies. We also needed a standard way to set up builds for customers (some don’t have a lot of experience with OE and Git) that is simple and consistent. What we ended up with is the BEC OpenEmbedded build template.
The goal is to have a quick entry point into OpenEmbedded that includes the necessary layers for a number of different machines, and automates a number of routine tasks such as installing images to a SD card, setting up a development feed server, etc. The build template is only updated when the build is stable and tested on a number of machines, so it provides a series of stable snapshots of OpenEmbedded and associated layers.
This build template currently tracks the master branches for all the layers used. This gives us a platform to track the latest OE changes. With most projects, that ability to use the features in the latest versions of software outweighs the stability benefits of OpenEmbedded release branches. There are times when the OpenEmbedded project goes through invasive changes (such as the systemd integration), and using the master branches is not practical, so in this case we simply use the last stable snapshot that builds and works. In most cases if there are issues, simply report or fix the issue and wait a week.
Perhaps the most controversial decision is the use of Git submodules for including OpenEmbedded layers. The Internet is full of rants against Git submodules. For heavy developer use, submodules may not be optimal. However, from a user perspective, Git submodules provide a simple mechanism for including external repositories in a project. If most of the submodules (OE layers) won’t be touched (typical OE user scenario), submodules work very well. The fact that Git locks down submodules to a specific commit ensures you are getting exactly what you think you are getting (vs a branch that may have been rebased, modified etc). If the git hash matches, you can be pretty sure it is the same as the last time you built it. This is an important factor in production build systems where you want to be sure of what you are building. Google repo is another option under consideration, but there are still some trade-offs to work through.
In the end, build systems are very personal, and must be customized for your product and development team. The number one requirement for a an Embedded Linux build system is that you can get repeatable builds with a single command. There must be no manual steps where human error can be introduced. This is just one way to accomplish this goal.
Since I’ve been running archlinux on some of my systems, one thing I’ve found useful is systemd-nspawn. systemd-nspawn containers (or chroots on non-systemd systems) give you a quick way to install a Linux distribution, that can run inside an existing Linux system.
Some cases where systemd-nspawn containers (referred to as containers in this document) are useful:
- At one point, OpenEmbedded would not build with GCC 4.8 (this is no longer the case with recent versions of OE). So a Debian or Ubuntu OS container was a quick way to get builds going again.
- For product build systems (may live for many years), typically OE will eventually break as you upgrade the workstation distribution. For projects that need a long-lived OpenEmbedded build system, setting it up in a chroot makes a lot of sense.
- Someone might be having a compile or build problem with a distribution you don’t currently have installed. With containers, you can quickly set up a test distribution to reproduce problems.
- I’ve had cases where I need an older version of Qt for a project, but my workstation includes a newer version. Again, setting up a OS container is sometimes simpler than getting two versions of Qt to dwell together peaceably in the same distribution.
- Backing up or replicating your entire build system is very easy — simply rsync the OS container directory to another machine.
So the solution is to select a relatively stable, long-lived distribution to host your product builds. Debian is good choice. Because the container is simply a directory in the host workstation filesystem, you can use host workstation tools (editors, git, etc) directly in the container filesystem. The only thing you need to use the chroot for is the actual building. If you make sure the user ID is the same between your workstation and nspawn container, then permissions are seamless — you can easily access files in the container from the context of your host workstation.
To set up a nspawn-container:
- Install debootstrap. On arch systems, this needs to be obtained for the AUR.
- host: sudo debootstrap –arch=amd64 wheezy ~/debian-wheezy/
- host: sudo systemd-nspawn -D ~/debian-wheezy/
- container: apt-get update && apt-get install ssh
- container: edit /etc/ssh/sshd_config, and set port to something other than 22 (23 in this example)
- container: /etc/init.d/ssh start
(This systemd-nspawn man page gives examples for setting up other distributions.)
To set up a user in your container:
- host: id
- will return something like: uid=1000(cbrake) gid=100(users) …
- container: adduser –uid 1000 -gid 100 cbrake
- host: ssh-copy-id -p 23 localhost (will copy public key to container)
Now, on the host system, you can simply “ssh -p23 localhost” any time you want to log into the container. Soft links between the project workspace on the host system, and the container can also make shifting between the two easier.
An alternative way to start the container once its set up is:
sudo systemd-nspawn -D ~/debian-wheezy /sbin/init
Its also handy to make the shell prompt in the container slightly different than the host OS so that you can easily tell the difference. To accomplish this, add the following to ~/.profile in the container OS:
export PS1=”[\u@wheezy \w]\$ “
To create a service that starts your container, put something like the following in /lib/systemd/system/debian-wheezy.service
[Unit]Description=Debian Wheezy[Service]ExecStart=/usr/bin/systemd-nspawn -D /scratch/debian-wheezy/ /sbin/init 3KillMode=process[Install]WantedBy=multi-user.target
Hopefully this gives you a quick overview of how OS containers can be set up, and used in your OpenEmbedded build systems.
As of version 1.8.2, Git submodules can now track branches instead of specific commits. This is good news as in many cases, this is exactly the behavior we want. However, Git submodules are still not as flexible as Google repo, but since submodules are built into Git, the submodule command is a good solution in many cases.
The “git submodule update –remote” command is the key to tracking branches with submodules. The following is from the Git man pages:
This option is only valid for the update command. Instead of using the superproject’s recorded SHA-1 to update the submodule, use the status of the submodule’s remote tracking branch. The remote used is branch’s remote (branch.<name>.remote), defaulting to origin. The remote branch used defaults to master, but the branch name may be overridden by setting the submodule.<name>.branch option in either .gitmodules or .git/config (with .git/config taking precedence).
This works for any of the supported update procedures (–checkout, –rebase, etc.). The only change is the source of the target SHA-1. For example, submodule update –remote –merge will merge upstream submodule changes into the submodules, while submodule update –merge will merge superproject gitlink changes into the submodules.
In order to ensure a current tracking branch state, update –remote fetches the submodule’s remote repository before calculating the SHA-1. If you don’t want to fetch, you should use submodule update –remote –no-fetch.
So, if you already have a Git submodule set up, its a simple matter to run git submodule update –remote to update the submodule to the latest master branch. If you want a different branch, simple edit .gitconfig.
[submodule "meta-bec"] path = meta-bec url = email@example.com:cbrake/meta-bec.git branch = test
Now, if you run git submodule update –remote, Git will update the meta-bec submodule to the latest on the test branch.
This functionality is purely a convenience feature in the submodule update command. In the actual repository, Git still stores submodules pointed to a particular commit. The same thing could be accomplished with something like git foreach “git fetch && git checkout test”. The branch option in .gitmodules functions more as documentation and convenience. It is very handy to be able to look at .gitmodules and quickly determine that submodule X is tracking branch Y. Normally, this would have to be documented elsewhere, or figured out in some other way. Also, for build systems where you want the build to always track the production branches of various projects, update –remote gives you a convenient way to update the build tree.