Categories
Uncategorized

Drone for Continuous Integration

Recently, we have been evaluating Continuous Integration (CI) systems for a variety of projects (both OSS and customer). There are many OSS options to chose from. Because we already use Docker containers for Yocto/OE builds, Concourse and Drone made the short list. Both communities seem responsive and helpful.

Categories
Uncategorized

Accepting Constraints in Build Systems

As Embedded Systems become more complex, the complexity of the process to build the software for these systems also increases. As humans, our ability to deal with complexity is limited, so we develop tools and processes to manage the complexity. In the end, these tools and processes are about constraints and patterns. A well-designed tool or process encourages you to do things in a way that is consistent and maintainable, which leads to reliable and predictable results.

Categories
Uncategorized

Getting started with Embedded Linux

Recently I was asked by a developer, who has done windows development for 10 years, how to get started with Embedded Linux. Embedded Linux covers a lot of ground and includes a broad range of components/skills to put together an entire system. Below are a few suggestions.

Categories
Uncategorized

Git, Versioning, and Branching for Embedded Linux Development

When building a product using Linux, versioning and branching of your software is an important consideration. Everyone’s needs are different depending on the size of the team, culture, and testing requirements, so there is no one size that fits all. However, after working on a number of different projects for a dozen or so different companies, there are several practices that are often used.

Categories
Uncategorized

Separation of Source and Build Directories

As we work with larger and more complex systems (i.e. Linux), more and more of our time is spent on integration and pulling different pieces together.  We often need to debug or understand code we did not write — especially in build systems.  To work effectively in this scenario you must be able to quickly search through a lot of source code.  Therefore, we are always looking for ways to make this more efficient.

Categories
Uncategorized

Using Docker for OE/Yocto builds

Why Docker?  When using OE to build software for products, we often run into the scenario where software needs to be built using the same version of OpenEmbedded over the course of several years.  Production builds need to be predictable.  We’ve also observed that old versions of OE often break as new Linux distros come out.  This is just the result of the complexity of building tool chains.  Additionally, for predictable builds you really don’t want to be changing the build OS.  This requirement automatically rules out Arch Linux, Debian Unstable, Gentoo, etc as production build machines.  Additionally, having developers debug OE build issues on varying workstation distributions is frustrating and time consuming.

Categories
Uncategorized

OpenEmbedded / Yocto Usage Survey

Recently, I sent a survey to the OpenEmbedded and Yocto mail lists asking a series of questions about how they use OE.  38 people responded.  The average number of years using OE is 4.8 years.

Categories
Uncategorized

Best practices for using CMake in OpenEmbedded/Yocto

I’ve already written about using autotools and qmake in OE.  With recent projects, we’re using CMake to build most C/C++ components.  Like any good tool, CMake has a learning curve, but it seems to do its job quite well.  Below are examples of a CMake build file, and corresponding OE recipe.

Categories
Uncategorized

Setting up an OpenEmbedded Package Feed Server

With the BEC OE build template, you can easily set up an opkg feed server that serves up packages from your build directory.  This allows you to easily install new packages during development, without generating and reflashing an entire image.  To use:

On workstation:

  • edit local.sh, and define MACHINE_IP to point to your target machine, and re-run “source envsetup.sh”.  Alternatively, you can export the MACHINE_IP variable in your environment.
  • run: oe_setup_feed_server
  • run: oe_feed_server

On target system:

  • opkg update
  • opkg install <some package>

Opkg will install dependencies, so if you have a complex package, this is so much easier than copying over an opkg file manually, and figuring out you need 6 other packages as dependencies.

Categories
Uncategorized

OpenEmbedded Build Template

How does one set an OpenEmbedded/Yocto/Poky/Angstrom build? There are many options. Some include:

(I’m sure there are many others, feel free to add in comments …)

Over the past years, we’ve supported a number of customers using OpenEmbedded to develop products using various SOC’s. We also try to keep builds going for a number of different systems so that we can continuously evaluate the state of OpenEmbedded and related Embedded Linux technologies. We also needed a standard way to set up builds for customers (some don’t have a lot of experience with OE and Git) that is simple and consistent. What we ended up with is the BEC OpenEmbedded build template.

The goal is to have a quick entry point into OpenEmbedded that includes the necessary layers for a number of different machines, and automates a number of routine tasks such as installing images to a SD card, setting up a development feed server, etc. The build template is only updated when the build is stable and tested on a number of machines, so it provides a series of stable snapshots of OpenEmbedded and associated layers.

This build template currently tracks the master branches for all the layers used. This gives us a platform to track the latest OE changes. With most projects, that ability to use the features in the latest versions of software outweighs the stability benefits of OpenEmbedded release branches. There are times when the OpenEmbedded project goes through invasive changes (such as the systemd integration), and using the master branches is not practical, so in this case we simply use the last stable snapshot that builds and works. In most cases if there are issues, simply report or fix the issue and wait a week.

Perhaps the most controversial decision is the use of Git submodules for including OpenEmbedded layers. The Internet is full of rants against Git submodules. For heavy developer use, submodules may not be optimal. However, from a user perspective, Git submodules provide a simple mechanism for including external repositories in a project. If most of the submodules (OE layers) won’t be touched (typical OE user scenario), submodules work very well. The fact that Git locks down submodules to a specific commit ensures you are getting exactly what you think you are getting (vs a branch that may have been rebased, modified etc). If the git hash matches, you can be pretty sure it is the same as the last time you built it. This is an important factor in production build systems where you want to be sure of what you are building. Google repo is another option under consideration, but there are still some trade-offs to work through.

In the end, build systems are very personal, and must be customized for your product and development team. The number one requirement for a an Embedded Linux build system is that you can get repeatable builds with a single command. There must be no manual steps where human error can be introduced. This is just one way to accomplish this goal.

Categories
Uncategorized

OS Containers for Build Systems

Since I’ve been running archlinux on some of my systems, one thing I’ve found useful is systemd-nspawn. systemd-nspawn containers (or chroots on non-systemd systems) give you a quick way to install a Linux distribution, that can run inside an existing Linux system.

Some cases where systemd-nspawn containers (referred to as containers in this document) are useful:

  1. At one point, OpenEmbedded would not build with GCC 4.8 (this is no longer the case with recent versions of OE).  So a Debian or Ubuntu OS container was a quick way to get builds going again.
  2. For product build systems (may live for many years), typically OE will eventually break as you upgrade the workstation distribution.  For projects that need a long-lived OpenEmbedded build system, setting it up in a chroot makes a lot of sense.
  3. Someone might be having a compile or build problem with a distribution you don’t currently have installed.  With containers, you can quickly set up a test distribution to reproduce problems.
  4. I’ve had cases where I need an older version of Qt for a project, but my workstation includes a newer version.  Again, setting up a OS container is sometimes simpler than getting two versions of Qt to dwell together peaceably in the same distribution.
  5. Backing up or replicating your entire build system is very easy — simply rsync the OS container directory to another machine.

So the solution is to select a relatively stable, long-lived distribution to host your product builds. Debian is good choice.  Because the container is simply a directory in the host workstation filesystem, you can use host workstation tools (editors, git, etc) directly in the container filesystem.  The only thing you need to use the chroot for is the actual building.  If you make sure the user ID is the same between your workstation and nspawn container, then permissions are seamless — you can easily access files in the container from the context of your host workstation.

To set up a nspawn-container:

  1. Install debootstrap.  On arch systems, this needs to be obtained for the AUR.
  2. host: sudo debootstrap –arch=amd64 wheezy ~/debian-wheezy/
  3. host: sudo systemd-nspawn -D ~/debian-wheezy/
  4. container: apt-get update && apt-get install ssh
  5. container: edit /etc/ssh/sshd_config, and set port to something other than 22 (23 in this example)
  6. container: /etc/init.d/ssh start

(This systemd-nspawn man page gives examples for setting up other distributions.)

To set up a user in your container:

  1. host: id
  2. will return something like: uid=1000(cbrake) gid=100(users) …
  3. container: adduser –uid 1000 -gid 100 cbrake
  4. host: ssh-copy-id -p 23 localhost (will copy public key to container)

Now, on the host system, you can simply “ssh -p23 localhost” any time you want to log into the container.  Soft links between the project workspace on the host system, and the container can also make shifting between the two easier.

An alternative way to start the container once its set up is:

sudo systemd-nspawn -D ~/debian-wheezy /sbin/init

Its also handy to make the shell prompt in the container slightly different than the host OS so that you can easily tell the difference.  To accomplish this, add the following to ~/.profile in the container OS:

export PS1=”[\u@wheezy \w]\$ “

To create a service that starts your container, put something like the following in /lib/systemd/system/debian-wheezy.service

[Unit]
Description=Debian Wheezy
[Service]
ExecStart=/usr/bin/systemd-nspawn -D /scratch/debian-wheezy/ /sbin/init 3
KillMode=process
[Install]
WantedBy=multi-user.target

Hopefully this gives you a quick overview of how OS containers can be set up, and used in your OpenEmbedded build systems.

Categories
Uncategorized

OpenEmbedded Source Mirrors

When using OpenEmbedded for product development, there are several reasons you may want to consider setting up a source mirror for your OpenEmbedded build:

  • over time, sources disappear from download locations
  • various servers for source packages may be off-line at the time a build is run
  • some servers may be very slow, which slows down your build
  • occasionally the checksums of a source package will change

For a production build system, you want the build to be reliable and consistent, so this means not depending on 3rd party web sites/servers for a clean build to complete.  Fortunately, OpenEmbedded makes it easy to set up a source mirror with the PREMIRRORS variable. When bitbake tries to fetch source code, it tries PREMIRRORS, the upstream source, and then MIRRORS.  There are several advantages to using a PREMIRROR variable over a MIRROR for your source mirror:

  • your source mirror will be used first, thus slow web sites are not an issue
  • if the checksums of the package change, the build will not fail because its still using the original source package from the mirror.  You are guaranteed to be always using the same source package.

Setting up a source mirror is as simple as copying the contents of your downloads directory to a web server, and then populating the following variable in local.conf:

PREMIRRORS_prepend = "\
     git://.*/.* http://my-server/sources/ \n \
     ftp://.*/.* http://my-server/sources/ \n \
     http://.*/.* http://my-server/sources/ \n \
     https://.*/.* http://my-server/sources/ \n"

The Poky reference manual has more details.

Categories
Uncategorized

A quick way to set up an OpenEmbedded feed server

During development with OpenEmbedded (oe-core, meta-oe, meta-angstrom), I often find it useful to set up a feed server so that packages can quickly be installed on the target system without manually copying them over or building a new image.  One way to do this is copy your deploy/ipk directory to an existing web server (perhaps Apache running on your workstation), or configure Apache to point at your OE build directory, etc.  But, it might be more convenient if your build system could directly create an opkg feed with no extra configuration.  In the below example, we use node.js + express.js to create a feed server (basically just a web server that serves up the ipk files).

tools/feed-server/app.js:

// nodejs script to start a http server with the feed from this directory
// default is port 4000
var express = require('express')
var app = express()
app.use('/', express.static(__dirname + '/../../build/tmp-angstrom_next-eglibc/deploy/ipk/'))
app.use('/', express.directory(__dirname + '/../../build/tmp-angstrom_next-eglibc/deploy/ipk/'))
console.log("feed server started on port 4000")
app.listen(4000)

The express.directory function is used to create a directory listing that can be browsed.  With most other web servers, at least this much code is required just for configuration.  With node.js, this is the code to create an entire server from scratch!  This node.js app can be started with a bash function in the environment:

function oe_feed_server()
{
  cd $OE_BASE
  bitbake package-index
  node tools/feed-server/app.js
  cd -
}

The package-index target is used to rebuild the index files that list the available packages.

On the target system, the /etc/opkg configuration files must be modified to point to the feed server, and then you can run:

opkg update; opkg install <package>

An exmaple build environement with this integrated is located:

https://github.com/cbrake/oe-build-core

https://github.com/cbrake/oe-build-core/commit/23352b9a43c60d67070abe5ac001aba9a9ac5cc4

https://github.com/cbrake/oe-build-core/commit/6baceb8b1e4477ccfd03aa553a5fcf501b398196

It might be argued that it is just as easy or easier to set up a more conventional web server.  However, the benefit of node.js is that it is a full blown programming environment.  You can quickly extend it to provide a web interface, perhaps add functionality to automatically push updated opkg configuration files to the target system that point to your feed server, etc.  It is much more than just a web server.

Categories
Uncategorized

OpenEmbedded: configuring openssh to allow a blank password

Noticed the following when browsing around in the OpenEmbedded sources the other day:

ROOTFS_POSTPROCESS_COMMAND += "openssh_allow_empty_password ;"

This allows a blank password for development, which is conveient for running ssh/scp commands to the device.  The above can be placed in an image recipe.

The command modifies the PermitEmptyPasswords config option in /etc/ssh/sshd_config or /etc/default/dropbear.

Categories
Uncategorized

Setting the root password in an OpenEmbedded image

During development, often a blank root password is used for the embedded Linux target system.  However, when deploying an embedded Linux system, often there is a requirement to set the root password to something non-obvious.  One way to do this is boot the system, and change the password using the passwd command.  Then copy the password hash from the /etc/shadow file into the below line in your image recipe.

ROOTFS_POSTPROCESS_COMMAND += "\
sed 's%^root:[^:]*:%root:password_hash_from_etc_shadow:%' \
< ${IMAGE_ROOTFS}/etc/shadow \
> ${IMAGE_ROOTFS}/etc/shadow.new;\
mv ${IMAGE_ROOTFS}/etc/shadow.new ${IMAGE_ROOTFS}/etc/shadow ;"

The ROOTFS_POSTPROCESS_COMMAND is useful for simple modifications like this to the rootfs image.