Bitbake has a new way of displaying build status

Now instead of displaying a scrolling log, bitbake will display a simple output that lists which tasks it is working on at the moment:

Currently 4 running tasks (185 of 3093):
0: gmp-native-5.0.5-r0 do_configure (pid 22919)
1: lzo-native-2.06-r1 do_configure (pid 27103)
2: expat-native-2.1.0-r0 do_compile (pid 7463)
3: ncurses-native-5.9-r10.1 do_compile (pid 9820)

This really allows for a clear view of how the parallel threads option (BB_NUMBER_THEADS) in bitbake works.


Yocto and OpenEmbedded

Recently, I attended an Embedded Linux summit hosted by the Linux Foundation to discuss the Yocto project. Overall, I thought the meeting was very positive and constructive. Having met and discussed at length the methods and goals of the Linux Foundation with some of their people, I’m impressed with their approach. They are there to help Linux succeed in the Embedded space in any way they can.

It is my understanding that the Yocto project has the goal of making Linux much more accessible and usable in embedded systems, and improve efficiencies. While the OpenEmbedded project has served many of us well for many years, we all can readily understand there are some deficiencies. These can be overcome by experienced developers, but there are a number of areas that can obviously be improved. Not only are we concerned with making it easier for new users to use Embedded Linux, I think there are areas where we can drastically improve the efficiency of implementing embedded Linux systems for those who are experienced. It was stated once that tools implemented must be useful to both the new developer as well as the experienced developer.

It should be noted that building an embedded Linux system is an inherently complex undertaking, and although we can improve tools and processes to make it more efficient, and somewhat easier, in the end it is still a very complex problem, and will require considerable skill to be an effective embedded Linux developer. There is no substitute for experience, and developer skill. Just as we would not slap a fancy GUI on top of a robotic surgery instrument, and tell a novice to have at it, likewise it is still going to require considerable engineering skill to be effective in developing Embedded Linux systems. But, if we improve the base technologies, and tools, we will spend less time messing with these and doing the same things over and over, and will have more resources toward implementing new things.

One example of the pain experienced in a typical OpenEmbedded system is getting Oprofile to run. Oprofile requires that ARM system be built with frame pointers (OE defaults to not), and that you have symbols in your binaries. Figuring out how to build a reasonably sized image with the needed symbols might be a 1-2 day process for me. Then there is the issue of kernel symbols, etc. I’m sure many other developers go through the same steps, but as many of us are paid to develop products on tight schedules, we don’t have the time to polish the Oprofile integration and tools.

As an extension of this, the application development story with OpenEmbedded is not all that great. Yes, we can generate a binary SDK, but again it may take a couple days of messing around to figure out the process, and get all the necessary libraries, etc. Then you still have the problem that application developers want Eclipse/Qt Creator integration, etc. Again, this can all be done, but takes time, and many people are doing the same things over and over again.

The Yocto project seems to have two primary thrusts:

  1. stabilize a subset of the OpenEmbedded (called Poky) metadata and make it very high quality.
  2. improve tools associated with the build process, and general embedded Linux development.

One thing the OpenEmbedded project has historically lacked in the past is industry funding and ongoing developer commitment. The OE project is characterized by a rotating group of developers. When someone needs OE for a project, they jump in and do a lot of work on OE, then they disappear again once their need is met. We have very few developers who work on the project consistently for long periods of time. This has been positive in some ways in that we have a very large number of contributors, and a very open commit policy. There are so many patches coming in right now, we can’t even keep up with processing them. Week after week, we have a healthy number of committers and changesets. The community behind OE is rather astounding, and it is amazing how OE succeeds almost in spite of itself as a self organizing project without much organization.

In the past the OpenEmbedded and Poky projects have existed as two independent trees, and things were shared back and forth manually. This works fairly well as recipes are self contained, and generally can simply be copied from one tree to another. However, some of the core recipes are fairly complex, and if they start to diverge, then sharing information can get more difficult.

It seems the vision that is emerging is that Poky could become the core component of OpenEmbedded. Poky would contain a very controlled subset of core recipes that must remain at a very high quality level. OpenEmbedded could then expand on this (through the use of layers) to provide the breadth and scope it has traditionally provided.

We may bemoan the lack of quality in the OpenEmbedded project as it has 1000’s of recipes, and many of them have rotted, etc. But as a consultant helping people implement products, I still find considerable value in this. For example, one of the products I support uses the HPLIP drivers from HP. Yes, the recipe is broke every time I go to use the latest version, but with a little work I can get it working again. Having something to start with provides value. The same is true for the geos mapping libraries. Very few people are going to use geos, so it will never be in a Poky type repository, but some of us do use geos, so having a common place like OpenEmbedded to put recipes like this is very important. Using Poky as the core of OpenEmbedded seems like a win-win. We are relieved of some of the burden of maintaining core components (like compilers, C library, tool integration, sdk generation, etc), but we can still have a very open culture, and provide a wide scope of platform, library, and application support that we have historically provided.

Richard Purdie is poised to become the Linus Torvalds of Yocto, and if OpenEmbedded choses to base on Poky, then the Torvalds of the OpenEmbedded core. I am personally fine with this as I don’t know anyone else who has contributed so much time and has the same level of skill in the OE/Poky area. He has proven over time to be levelheaded and able to deal with difficult situations. Also, as a Linux Foundation fellow, he is positioned to be company neutral, which is very important for someone in this position.

Yocto is using a “pull” model similar to the Linux kernel for accepting changes. It is planned to put maintainers in place for various subsystems. With the goal of providing a very high quality subset of embedded Linux technologies, it seems this makes a lot of sense. If OpenEmbedded choses to base on Poky, there is no reason OpenEmbedded can’t continue to still use the push model for its layer that has worked well in the past. But, as we see patches languishing in the patchwork, perhaps a pull model might actually be more efficient at processing incoming patches so they don’t get lost. This is also an area where layers might actually help in that we have dedicated people committed to processing patches for each layer. One of the problems with the OpenEmbedded project with its 1000’s of recipes, the structure is very flat, and its fairly difficult to divide it up into areas of responsibility.

So going forward, it seems that if OpenEmbedded can somehow use Poky as a core technology (just like it uses bitbake, etc), then we could have the best of both worlds. With Poky, we get strong industry commitment and dedicated people working on making the core pieces better, and the tools better (50 engineers are already dedicated). This is something those of us developing products don’t really have time for and we are not currently funded to do this. With OpenEmbedded we can still keep our broad involvement, and vast recipe and target collection. Yes, the quality of OpenEmbedded will be lower, but it still provides a different type of value. Poky provides the known stable core, and OpenEmbedded is just that–it is “open”, and provides the broad menu of components that can be used. Over time this could evolve into something different, but for now it seems a good place to start.

The last thing to consider is the OpenEmbedded brand. It is recognizable from the name what it is (unlike Yocto, and Poky). It has had broad exposure for many years. It has historically been very vendor neutral with very little corporate direction or influence. From this perspective, it seems the Yocto project can benefit greatly from having an association with the OpenEmbedded project. This topic was discussed at the summit, and there was general consensus on the strong awareness of the OpenEmbedded brand, as well as an openness to how branding for the Yocto core build system might look.


OpenEmbedded srctree and gitver

Recently an OpenEmbedded class name srctree became usable.  The srctree.bbclass enables operation inside of an existing source tree for a project, rather than using the fetch/unpack/patch idiom.  The srctree.bbclass in combination with the OpenEmbedded gitver.bbclass and git submodules provides a very interesting way to build custom software with OpenEmbedded.

One of the classic problems with OpenEmbedded is how do application and kernel developers use it.  While OpenEmbedded excels at automating image builds, it is less friendly when used as a a cross development tool.  Historically there are several options for iterative development:

  1. develop in the working directory: cd <tmp>/work/arm…/<my recipe>; ../temp/run.do_compile …
  2. a variation of #1: bitbake -c devshell <my recipe>
  3. manually set up an environment that uses the toolchain generated by OE.  As example see this script in the BEC OE template.
  4. a variation of #3: export a SDK that includes a toolchain and libs

While the above solutions work OK, the process can be a little cumbersome.  Unless your OE recipe pulls software directly from the TIP of a SVN repository, you may have to manually update the recipe after you make changes, create patch files, etc.  There is also the problem that if your recipe fetches the latest from SVN, it drastically slows down the recipe parsing as it has to check the repository for a new version every time the recipe is processed.

The optimal solution would to be to simply check a software component out of a version control system, and build it directly using OpenEmbedded.  Icing on the cake would be if the package generated would automatically extract version information from the version control system.  This would facilitate iterative development for software components that need to be cross compiled.

Although srctree can be used with any directory of source code, it really works best with a git repository.  The gitver.bbclass provides a GITVER variable which is a (fairly) sane version, for use in ${PV}, extracted from the ${S} git checkout, assuming it is one (text from recipe).  gitver uses the ‘git describe’ command to extract the last tag and uses that for the version.

The best way to illustrate the use of these tools is an example:

The easiest way to try this is clone the above project in your openembedded/recipes directory:

$ cd openembedded/recipes
$ git clone
$ cd autotools-demo
$ git describe
$ git tag -l

Notice that git describe simple returns the latest tag.   The recipe can be located in the same directory as the source code and has the following contents:

# recipe to build the autotools-demo in the working tree

inherit srctree autotools gitver

PV = "${GITVER}"

Can’t get much easier than that!  If you build the recipe, you end up with a package:

$ bitbake autotools-demo
$ ls tmp/deploy/glibc/ipk/armv5te/autotools-demo_1.1-r0.6_armv5te.ipk

Now, what happens if you make changes and commit them?

$ cd .../autotools-demo
$ (make a change and commit)
$ git describe
1.1-1-gfbc1ecc (notice the count and hash automatically appended)
$ (make another change and commit)
$ git describe
1.1-2-g7ad3715 (notice the count is now 2)

If we bitbake the recipe now, we end up with a packaged named:


The gitver class in OpenEmbedded automatically takes care of creating a usable PV (package version) that always increments.

So in summary, srctree and gitver give developers a convenient way to handle custom components that change often in an Embedded Linux build without increasing parse times, requiring manual tweaks to version numbers, or creating a separate workspace for each version of the application that is built.  As practices such as continuous integration become more common, OpenEmbedded features like this are increasingly needed.  An added benefit is that the OpenEmbedded recipe can now be stored in the same location as the source code.  Perhaps in the future, most applications will include an OpenEmbedded recipe as part of their source code and git submodules could be used to simple populate the components you want to use.

2017-11-08 update: A tool named devtool is now the preferred way to do much of the above.


Installing OMAP3 images on a SD card

This article and screen-cast is a continuation of that last couple posts describing the BEC OE build template.  The purpose again for a build system is to automate tedious manual tasks, and in doing so, we end up documenting how the build system works.  Having a good build system is important during product development so that you have an easy, repeatable process for building and deploying images.  One of these tasks is setting up a SD card and installing images to the SD card that an OMAP system can boot from.  This screencast goes over the script and makefile targets that are used to automate this process.

In summary, the steps to set up a SD card and install an image using the build template are:

  1. cd <build template directory>
  2. determine the SD card device name (/dev/sdX)
  3. unmount any file systems mounted on the SD card
  4. sudo ./scripts/ /dev/sdX  (make sure you have the right device!!!)
  5. make omap-install-boot
  6. make omap-install-<image name>-rootfs

Previous articles in this series:


Creating a Custom OpenEmbedded Image

In this article screencast, we’ll demonstrate how to create a custom Linux OS image using the OpenEmbedded build system.  This demonstration builds on the earlier article about using the BEC OE build template.  The OpenEmbedded build system is similar to Linux distributions in that you can select from a wide array of components to install.  One of the big differences is you can select these components when building the OS image, instead of after you have installed the “standard” image.  This screencast demonstrates how to set up a custom image (can be viewed with Firefox).

A related article also covers this procedure.

In summary, the steps include:

  1. Create an image recipe (typically in your meta data overlay).  An example is located here.  An existing image can be referenced for simplicity.
  2. Figure out what packages need to be added.  This can be done by browsing the recipe tree to see what is available.
  3. Bitbake the recipe.
  4. Look at what packages were generated.  In this example they are found in the build/angstrom-2008.1/tmp/deploy/glibc/ipk/armv7a directory.
  5. Add the relevant package names to your image recipe.

As the OpenEmbedded project includes over 6000 recipes, it gives you a big head start in including standard components in your Embedded Linux build.  Leveraging these existing components is why we use Linux, and having a way to easily build and deploy these components to your target hardware is what OpenEmbedded is all about.


OpenEmbedded Build Template

Setting up an OpenEmbedded build is a fairly simple process if you carefully follow the instructions.  There are also a number of scripts available that automate the setup such as the OpenEmbedded Tools for Eclipse, the Angstrom setup scripts, the KaeilOS Openembedded Manager, and I’m sure there are many more (feel free to add in comments section).  As we have helped a number of clients use OpenEmbedded in commercial projects, we have the following requirements for a build template:

  1. The build needs to be completely automated such that only one step is required to build images.
  2. We need to be able to lock down versions of various components.
  3. Needs to be as simple as possible.

As we continue to think about creating a build template, the primary reason for creating one is to get repeatable builds with very little effort.  The build template serves as documentation; how did we build the 1.10 release.  A good build system allows any developer to create any release on about any machine at any time.  A good build system does not depend on the “golden build machine.”  We have been using makefile wrappers around OpenEmbedded for several years now to accomplish much of this.  These wrappers were typically stored in a Subversion repository with the rest of the project files.  However, now that most open source projects have moved to Git repositories, there are significant advantages to using Git for the build system.  Git has a feature called submodules which allows you to embed a git repository inside a source tree, always pointed at a particular commit.  In concept, this is perfect.  In practice, Git submodules are sometimes difficult to use, but considering the functionality provided, I think it is a reasonable trade-off.

The use of Git encourages a different development paradigm.  Typically when people use Subversion within a company, there is one repository that stores everything.  Subversion is actually quite good at this.  I’ve run into a number of companies who have repositories many GB in size that have been running for many years.  The stability is remarkable.  Git does not scale in this way, so people tend to set up a separate repository for each component, application, etc.  While there may be some disadvantages to this model, there are also significant advantages, and we can learn some of these by observing Open Source Software (OSS) development.  Just as we strive for modularity in software development where each class can stand on its own and be generally useful, OSS maintains this dynamic at the project level.  Each OSS project moves at its own pace, largely decoupled from other projects.  This dynamic forces a level of organization that might be neglected otherwise.  Likewise, in any project, if each major component is stored in its own source code repository, there will also tend to be less coupling, willy-nilly including of header files from other components, etc.  It helps people think in a more modular manner at the component level.  A component may be the Linux kernel, bootloader, each application in the system, etc.  A similar concept may be thinking about storing documents in hierarchical filesystem directory structure versus documents in a Wiki, which are stored in a flat structure and then organized by cross linking.  Yes, there are times to use a directory structure, but flexibility of a linked organization system (the essence of the Internet) is hard to beat.  Git submodules (or similar concepts) provides a similar linking method of organization.

With this in mind, we look at a build system as a way to collect a number of independent components and build them in a repeatable fashion.  While this may seem obvious to anyone somewhat familiar with OpenEmbedded, our vision is to extend this concept to easily facilitate product development so that we can move quickly and efficiently, but yet in a controlled and reliable way.  The first step is a new build template that is available here.  The following video provides a short overview of how to use the build template (can be viewed in Firefox).

The functionality provided so far is fairly simple and provides a way to quickly set up a build environment.  With Git submodules, the versions of Bitbake and the OpenEmbedded are locked down.  Submodules are also used to provide an array of components to choose from, and the user initializes only the submodules they are interested in using.  The first example of this is providing both the recipe meta data from the OpenEmbedded project or the modified version from Gumstix.  In the future, examples will be provided for doing custom application and kernel development using the OE srctree and gitver classes.  Feedback is welcome; please let me know if you have questions or suggestions.


OpenEmbedded development activity

Ever since I have been sending out weekly change logs, I have been impressed by the consistent amount of development activity in the OpenEmbedded project.  Every week there are consistently over a dozen developers making changes.  Developers come and go, but the contribution level always seems healthy.  While this amount of development leads to some amount of churn and issues, this amount of development is also required to keep pace with all the new developments in the OSS space.  Not having to wait 2 years for the next big release is one of the big advantages of using Linux and OpenEmbedded.  Having access to the latest technology provides a competitive advantage for many products, and makes dealing with the occasional issues that come up an acceptable trade-off.  Coupled with a flexible and consistent build system, OpenEmbedded is the vehicle to build advanced products.  Yes, its hard.  Yes, you better budget for some system software development time.  Yes, you should get some help if you’re not an embedded Linux expert.  Yes, you can start with some canned vendor “BSP” that seems to work and is seemingly the low-cost way to go, but eventually you’ll hit a brick wall where you need some bit of functionality that is not there, and then you’ll start up the exponential curve of dumping lots of time into trying to make something work with little progress forward.  It is much better to count the cost up front, and do things right.


Notification at the end of builds

I do quite a few OpenEmbedded project builds during the course of a week.  This process usually takes 3-5 minutes.  That is just enough time to get distracted doing something else and forget about the build until an hour later when you realize — oops, I was supposed to send out a release email once the build was finished and uploaded.  It occured to me that it would be nice if my computer played a distinct sound at the end of the build.  With Linux this is incredibly easy:

  • wget
  • sudo mv Ding_Dong_Bell.ogg /usr/local/
  • sudo aptitude install cplay
  • sudo echo “cplay /usr/local/Ding_Dong_Bell.ogg” > /usr/local/bin/bell
  • sudo chmod 775 /usr/local/bin/bell

Now, when I have a long build, I simply do something like:

bitbake my-image; bell

There is probably something better than cplay that does not start a UI — any suggestions?


Best practices for kernel development with OpenEmbedded

A common question is how do you do kernel development with OpenEmbedded?  Typically, OpenEmbedded builds a kernel by checking the source out of a git repository, or by applying patch files to a released version of the kernel.  See the many recipes for examples.  This works very well for a Linux distribution build system, but is less than ideal for ongoing kernel development where you will have many iterations over a period of time. The OE work directory is considered transient, and should not be used for any ongoing development.  Fortunately, the kernel build system is self contained, so it is a very simple matter to reference the toolchain built by OpenEmbedded and work outside the OpenEmbedded build system.

Use Git

Trust me on this one, just use it!  It will make your life easier.  There is a learning curve to any powerful tool.  Another way to state this is if a tool does not do much, then it would be easy to learn.  So expect to spend a little time learning to use git.  With modern Linux kernel development, much of your time is spent integrating work that others are doing.  For example, if you are using the OMAP3 CPU, you might want to start with linux-omap branch, integrate some patches from the OE beagleboard build, merge the OMAP PM tree, and then merge some bits from the Linux-wireless tree.  This is the way embedded Linux development is done today.  Git makes it possible to have many parallel developments going at their own pace, including your own.  Even in your own development, you will find significant advantages to being able to quickly create and merge branches, revise commits, etc.  There are several articles about git on this site.

Importing an OE kernel into git

If you are starting with an existing OE kernel tree, then you typically need to set up an equivalent tree in your local git workspace.  Typically you just look at the recipe to see how to manually reconstruct it.  For example, lets consider the SRC_URI from the palm-omap1 recipe:

SRC_URI = "${KERNELORG_MIRROR}/pub/linux/kernel/v2.6/linux-2.6.22.tar.bz2 \
 ;patch=1 \

To set this kernel up in git, we might do the following:

  1. git clone git://
  2. cd linux-2.6
  3. git checkout -b my_branch v2.6.22
  4. wget
  5. bzip2 -d patch-2.6.22-omap1.bz2
  6. git am patch-2.6.22-omap1

If the git am fails, then you might need to add a subject and author line to the patch in the following form and then retry git am.

From: Name <>
Subject: omap1 patch

Now when you do a git log, you will see the above patch applied.  git apply can also be used to apply patches, but I prefer git-am as it automatically handles new files and fills in the commit message and author.

Setting up the toolchain

Now that you have a kernel, you must set up a toolchain so you can build it.  I typically set up my kernel directory next to my OE build directory in my project space, so I can then source the following script to set up the toolchain:

CROSS_COMPILER_PATH=`cd ../oe/build/angstrom-2008.1/tmp/cross/armv7a/bin; pwd`
BUILD_ARCH=`uname -m`
OE_STAGING_PATH=`cd ../oe/build/angstrom-2008.1/tmp/staging/${BUILD_ARCH}-linux/usr/bin; pwd`
STAGING_KERNEL_DIR=`cd linux-2.6; pwd`
export ARCH=arm
export CROSS_COMPILE=arm-angstrom-linux-gnueabi-

Adjust to whatever you are building for.  Now you can simply type make.  The reason the OE_STAGING_PATH is added to the PATH in the above example is so that the uboot mkimage utility can be used directly from the OE build if your build target is make uImage.

Installing modules

Sometimes you need to re-install all modules on the target because you changed kernel version, etc.  Typically OE packages up all modules and adds the one you specify to the rootfs.  Because we are building the kernel outside the OE directory, OE can no longer do this.  However, its still very easy to install modules from your kernel development directory:

rm -rf modules_install; INSTALL_MOD_PATH=modules_install make modules_install
rsync -av modules_install/lib/modules/2.6.22 root@

The first command installs the modules to a temporary directory, and the second command rsync’s the modules to your target system.

Creating a kernel recipe with your changes

Once you get to the point where you want to create a recipe with your changes, you can easily export a patch, or series of patches using git diff or git format-patch.  Simply add the patch files to your kernel recipe SRC_URI.  You can also teach bitbake to fetch your kernel source directly from a git repository, but I’ve generally found the patch mechanism to be adequate and easier in most cases.  As an example, you can create one large patch with all your changes that can be applied on the 2.6.22 released kernel:

git diff HEAD..v2.6.22 > my_kernel_changes.patch

Other ways …

This is just one flow and there are many variations.  There is also a new mechanism in OpenEmbedded called srctree that allows you to use OpenEmbedded in an external source tree.  srctree has the potential to further streamline this type of development.

As with any task, you want to use the tools that make sense for the task.  OpenEmbedded makes sense for building Linux distributions, but it is not a kernel development tool.  Git and the kernel build system make sense for kernel development.


Best practices for building Gtk+ applications with OpenEmbedded

I recently wrote an article about best practices building Qt applications with OpenEmbedded, and it occured to me that I should write an equivalent article for Gtk+ applications.  The same points apply — put your application source in a SCM system, and put the install logic in the application source (read the above article).  The difference is that Gtk applications typically use autotools where Qt applications use qmake to build the application.  This article details how a minimal GTK+ application should be set up and built using OpenEmbedded.

Application Source

I created a sample GTK hello application located at:  This is a SVN repository, so you can simply “svn co” the above URI to check out the code.  If you are running Ubuntu, you can install the necessary tools to build a native Gtk+ application by:

  • sudo apt-get install libgtk2.0-dev build-essential autoconf automake pkg-config

To build on your x86 Linux PC, run the following steps:

This compiles and installs the application in to the “install” directory.  If you look in this directory, you will notice the application binary is installed in the “install/bin” directory.  Typically, the install directory is set to /usr/bin, but in this example we set it to install so we don’t need to run “make install” as root, but yet we can verify the install logic works properly.

OpenEmbedded Recipe

Now that you have verified the application builds and installs properly on a x86 PC, it is trivial to build the application in OpenEmbedded.  Create a recipe in your OE recipes directory named with the following contents:

DESCRIPTION = "Sample Gtk+ Hello application, used to demonstrate build system"
AUTHOR = "Cliff Brake <>"

SRCREV = "17"
PV = "0.0+svn${SRCREV}"
PR = "r0"

DEPENDS = "gtk+"

SRC_URI = "svn://;module=gtk_hello;proto=http;rev=17"

S = "${WORKDIR}/gtk_hello/"

inherit autotools

Now, run: bitbake gtk-hello.  That is it!  Building Linux applications is easy if you simply use the tools, whether it be autotools, qmake, etc.  Too often there is the tendency to set up your own compile steps with ${CC} variables, etc.  While this seems to be the simple approach at first glance (autotools is too hard), it quickly becomes unmaintainable and in the end is a lot more work than simply learning the basics of the industry standard tools.  See previous autotools article for more information.


Best practices for building Qt applications with OpenEmbedded

This article describes how to cross compile a Qt application (named qt_tutorial) with OpenEmbedded, and several best practices you should consider.  OpenEmbedded currently includes fairly good support for building Qt — both Qt Embedded and Qt X11.   OE also includes a number of qt classes that make building Qt applications easy.  One of the main considerations with embedded Linux application development is that you keep the build system flexible so that you can easily build on a PC or for your embedded hardware.  This means that hand crafted Makefiles with hardcoded paths to cross compilers do not quality.

Put Your Application in a SCM system

No matter what type of application you are building, it is a good idea to put your source in a SCM or revision control system and fetch it directly using OE.  OpenEmbedded supports fetching sources from a number of revision control systems including Subversion, Git, Bazaar, Mercurial, CVS, etc.   The not so obvious reason we do this is so you can easily check out the source code and build it on a PC as well as your target system in OE.  In this example we fetch the application source from a SVN repository:

SRC_URI = "svn://;module=qt_tutorial;proto=http"

For the above repository, the direct URI to the source would be:

Put the Install logic in the Application Source

Most Linux applications support “make install”, and this is the case with autotools, and qmake (Qt’s build tool).  We could put logic in the OE recipe to install the application something like the following:

do_install() {
	install -d ${D}/${bindir}
	install -m 0755  ${S}/qt_tutorial ${D}/${bindir}

But, the problem with this approach is you can’t install the application in other environments (like a native x86 PC build) unless you are building with OE.  So a better approach is to put the logic to install the application in the application source, so that in can be installed in both your PC environment, and your OpenEmbedded build.  To accomplish this, you can set up the project as follows

application qmake project file (

TARGET = qt_tutorial

# Input
HEADERS += lcdrange.h
SOURCES += lcdrange.cpp main.cpp

target.path = /usr/bin
INSTALLS += target

OpenEmbedded recipe (

inherit qt4e

PV = "1.0+svnr${SRCREV}"
PR = "r1"

SRC_URI = "svn://;module=qt_tutorial;proto=http"

S = ${WORKDIR}/qt_tutorial

FILES_${PN}-dbg += "${bindir}/.debug"

do_install() {
	export INSTALL_ROOT=${D}
	make install

Now, the same mechanism is used to install the application on both a PC native build, as well an OpenEmbedded build.  If you look in the Makefile generated by qmake, you see the following:

install_target: first FORCE
	@$(CHK_DIR_EXISTS) $(INSTALL_ROOT)/usr/bin/ || $(MKDIR) $(INSTALL_ROOT)/usr/bin/

install:  install_target  FORCE

INSTALL_ROOT can be set to force the entire system to be installed in a subdirectory.  This is required for build systems that generate packages, like OpenEmbedded.  To build this example, put the file in your OE recipes tree, and run: bitbake qt-tutorial.  This will fetch the source code and build a package.  To run the tutorial, install the package on a system that includes Qt Embedded, and then run: qt_tutorial -qws.


How to capture source changes to an OpenEmbedded package

One task that is a often confusing to new OpenEmbedded users is how to capture changes to the source code for a package/recipe.  First, lets review the progression in tools use to capture source code changes.

Progression of tools
Figure 1: Progression of tools

In days of yore, developers often used gnu diff to capture modifications to a source tree.  The typical practice was to create two source trees: a pristine, and a dirty tree.  Modifications were made in the dirty tree, and periodically gnu diff was run between the two trees to capture the changes.  Then came quilt which allowed you to capture changes without a pristine tree — kind of a cheap version control system.  The typical flow was:

  • cd <source dir>
  • quilt new my_changes.patch
  • quilt add file1.c
  • edit file1.c
  • quilt refresh (now my_changes.patch records the edits to file1.c)
  • debug
  • edit file2.c
  • (oops forget to quilt add file2.c — now what ….)
  • restore file2.c from pristine source
  • quilt add file2.c
  • edit file2.c
  • quilt refresh
  • (now my_changes.patch records edits to file1.c and file2.c)

As you can see, things are much better than plain diff, but you still have to keep track of the files you edit.  With git, we can do essentially the same things, only git keeps track of the files you edit, plus it is easier to organize patch files.  The flow with git is something like this:

  • cd <source dir>
  • git init
  • git add *
  • git commit (now the entire source tree is recorded by git)
  • edit file1.c
  • edit file2.c ….
  • git status (shows you files that were modified)
  • git add file1.c file2.c
  • git commit
  • git log (shows you changes you have made)
  • git format-patch HEAD~1 (outputs the patch created by the last commit)

Once your source code is in git, then you can do all kinds of fancy things like merging, re-ordering, and re-naming patch files with the git-rebase –interactive command.

One of the uses of this technique is when you need to record a small change to a package source in a build system such as OpenEmbedded, and apply that change for subsequent builds.  In this case you might do something like the following:

  • bitbake -c devshell <recipe_to_patch>
  • in devshell capture changes in git as shown above to create my_changes.patch
  • cp my_changes.patch <oe_tree>/recipes/<recipe_to_change>/files/
  • edit <oe_tree>/recipes/<recipe_to_change>/<recipe_to_change>.bb
    • tell bitbake to apply the patch when building the recipe: SRC_URI += “file://my_changes.patch;patch=1”
    • increment the PR variable so that bitbake knows to rebuild the recipe.

Figure 2 illustrates the above steps.

Flow to capture packages source changes in OpenEmbedded
Figure 2: Flow to capture package source changes in OpenEmbedded

While the above steps might seem overly complex, you must consider what you are trying to do.  Building a Linux distribution is not a simple task.  We are dealing with large amounts of source code from many different projects.  Putting all this source code in one tree is not an option, because then we would be defeating the very reasons open source is attractive in the first place: flexibility, largly decoupled projects, advanced technology, fast bug fixes, many options to choose from, developer support, etc. Each open source project must remain a self contained project that is largly decoupled from other projects so it can develop at its own pace.  It is for these reasons that open source technologies are used to give products a competitive edge.  So, like any complex process or task, there must be some understanding of how it works, an appreciation for what it can do for you, and an investment in tools and techniques required to be effective.


Embedded Linux versus Windows CE

Occasionally I am asked how Embedded Linux compares with Windows CE.  I have spent the past 5 years doing mostly embedded Linux development, and the previous 5 years doing mostly WinCE development with a few exceptions, so my thoughts are no doubt a little biased toward what I understand best.  So take this with a grain of salt 🙂  In my experience, the choice is often made largely on perception and culture, rather than concrete data.  And, making a choice based on concrete data is difficult when you consider the complexity of a modern OS, all the issues associated with porting it to custom hardware, and unknown future requirements.  Even from an application perspective, things change over the life of a project.  Requirements come and go.  You find yourself doing things you never thought you would, especially if they are possible.  The ubiquitous USB and network ports open a lot of possibilities — for example adding Cell modem support or printer support. Flash based storage makes in-field software updates the standard mode of operation.  And in the end, each solution has its strengths and weaknesses — there is no magic bullet that is the best in all cases.

When considering Embedded Linux development, I often use the iceberg analogy; what you see going into a project is the part above the water.  These are the pieces your application interacts with, drivers you need to customize, the part you understand.  The other 90% is under water, and herein lies a great deal of variability.  Quality issues with drivers or not being able to find a driver for something you may want to support in the future can easily swamp known parts of the project.  There are very few people who have a lot of experience with both WinCE and Linux solutions, hence the tendency to go with what is comfortable (or what managers are comfortable with), or what we have experience with.  Below are thoughts on a number of aspects to consider:


Questions in this realm include CPU support, driver quality, in field software updates, filesystem support, driver availability, etc.  One of the changes that has happened in the past two years, is CPU vendors are now porting Linux to their new chips as the first OS.  Before, the OS porting was typically done by Linux software companies such as MontaVista, or community efforts.  As a result, the Linux kernel now supports most mainstream embedded cpus with few additional patches.  This is radically different than the situation 5 years ago.  Because many people are using the same source code, issues get fixed, and often are contributed back to the mainstream source.  With WinCE, the BSP/driver support tends to be more of a reference implementation, and then OEM/users take it, fix any issues, and that is where the fixes tend to stay.

From a system perspective, it is very important to consider flexibility for future needs.  Just because it is not a requirement now does not mean it will not be a requirement in the future.  Obtaining driver support for a peripheral may be nearly impossible, or be too large an effort to make it practical.

Most people give very little thought to the build system, or never look much beyond the thought that “if there is a nice gui wrapped around the tool, it must be easy”.  OpenEmbedded is very popular way to build embedded Linux products, and has recently been endorsed as the technology base of MontaVista’s Linux 6 product, and is generally considered “hard to use” by new users.  While WinCE build tools look simpler on the surface (the 10% above water), you still have the problem of what happens when I need to customize something, implement complex features such as software updates, etc.  To build a production system with production grade features, you still need someone on your team who understands the OS and can work at the detail level of both the operating system, and the build system.  With either WinCE or Embedded Linux, this generally means companies either need to have experienced developers in house, or hire experts to do portions of the system software development.  System software development is not the same as application development, and is generally not something you want to take on with no experience unless you have a lot of time.  It is quite common for companies to hire expert help for the first couple projects, and then do follow-on projects in-house.  Another feature to consider is parallel build support.  With quad core workstations becoming the standard, is it a big deal that a full build can be done in 1.2 hours versus 8?  How flexible is the build system at pulling and building source code from various sources such as diverse revision control systems, etc.

Embedded processors are becoming increasingly complex.  It is no longer good enough to just have the cpu running.  If you consider the OMAP3 cpu family from TI, then you have to ask the following questions: are there libraries available for the 3D acceleration engine, and can I even get them without committing to millions of units per year?  Is there support for the DSP bridge?  What is the cost of all this?  On a recent project I was involved in, a basic WinCE BSP for the Atmel AT91SAM9260 cost $7000.  In terms of developer time, this is not much, but you have to also consider the on-going costs of maintenance, upgrading to new versions of the operating system, etc.


Both Embedded Linux and WinCE support a range of application libraries and programming languages.  C and C++ are well supported.  Most business type applications are moving to C# in the WinCE world.  Linux has Mono, which provides extensive support for .NET technologies and runs very well in embedded Linux systems.  There are numerous Java development environments available for Embedded Linux.  One area where you do run into differences is graphics libraries.  Generally the Microsoft graphical APIs are not well supported on Linux, so if you have a large application team that are die-hard windows GUI programmers, then perhaps WinCE makes sense.  However, there are many options for GUI toolkits that run on both Windows PCs and Embedded Linux devices.  Some examples include GTK+, Qt, wxWidgets, etc.  The Gimp is an example of a GTK+ application that runs on windows, plus there are many others.  The are C# bindings to GTK+ and Qt.  Another feature that seems to be coming on strong in the WinCE space is the Windows Communication Foundation (WCF).  But again, there are projects to bring WCF to Mono, depending what portions you need.  Embedded Linux support for scripting languages like Python is very good, and Python runs very well on 200MHz ARM processors.

There is often the perception that WinCE is realtime, and Linux is not.  Linux realtime support is decent in the stock kernels with the CONFIG_PREEMPT option, and real-time support is excellent with the addition of a relatively small real-time patch.  You can easily attain sub millisecond timing with Linux.  This is something that has changed in the past couple years with the merging of real-time functionality into the stock kernel.


In a productive environment, most advanced embedded applications are developed and debugged on a PC, not the target hardware.  Even in setups where remote debugging on a target system works well, debugging an application on a workstation works better.  So the fact that one solution has nice on-target debugging, where the other does not is not really relevant.  For data centric systems, it is common to have simulation modes where the application can be tested without connection to real I/O.  With both Linux and WinCE applications, application programing for an embedded device is similar to programming for a PC.  Embedded Linux takes this a step further.  Because embedded Linux technology is the same as desktop, and server Linux technology, almost everything developed for desktop/server (including system software) is available for embedded for free.  This means very complete driver support (see USB cell modem and printer examples above), robust file system support, memory management, etc.  The breadth of options for Linux is astounding, but some may consider this a negative point, and would prefer a more integrated solution like Windows CE where everything comes from one place.  There is a loss of flexibility, but in some cases, the tradeoff might be worth it.  For an example of the number of packages that can be build for Embedded Linux systems using Openembedded, see


It is important to consider trends for embedded devices with small displays being driven by Cell Phones (iPhone, Palm Pre, etc).  Standard GUI widgets that are common in desktop systems (dialog boxes, check boxes, pull down lists, etc) do not cut it for modern embedded systems.  So, it will be important to consider support for 3D effects, and widget libraries designed to be used by touch screen devices.  The Clutter library is an example of this.


Going back to the issue of debugging tools, most people stop at the scenario where the device is setting next to a workstation in the lab.  But what about when you need to troubleshoot a device that is being beta-tested half-way around the world?  That is where a command-line debugger like Gdb is an advantage, and not a disadvantage.  And how do you connect to the device if you don’t have support for cell modems in New Zealand, or an efficient connection mechanism like ssh for shell access and transferring files?


Selecting any advanced technology is not a simple task, and is fairly difficult to do even with experience.  So it is important to be asking the right questions, and looking at the decision from many angles.  Hopefully this article can help in that.  For additional assistance, please do not hesitate to contact BEC Systems — we’re here to help.


MontaVista Linux 6 is based on OpenEmbedded Technologies

Very interesting news from MontaVista yesterday as they announced MontaVista Linux 6.  It turns out they are using bitbake, which is the core of the OpenEmbedded build system.  Along with the adoption of OpenEmbedded in many leading Embedded Linux efforts (Beagleboard, Gumstix, Bug Labs, etc), this is a resounding affirmation that the OpenEmbedded way of building distributions is worthy of consideration.  It is a well known fact that OpenEmbedded has some deficiencies: a steep learning curve, the testing/quality efforts could be improved, and the organisational aspects of the project are somewhat lacking.  That said, OpenEmbedded is still the most effective tool for building Embedded Linux distributions that I’ve found yet, and is a very viable solution if you are willing to spend some time to learn it, or hire a consultant to get your project set up and work through the rough spots.  MontaVista is attempting to address these difficulties with their MVL6 commercial offering.

Below are few a notes from watching a few videos on the MV web site:

  • Discussion with Jim Ready
    • around 2006 things changed
    • Semiconductor vendors now porting Linux to all their CPUs, so MV has moved up the stack
    • MVL6 first embedded linux distro produced in post-victory phase (Linux has won)
    • very friendly with other open source pieces (I assume OpenEmbedded)
    • enable open source
    • The decision the use Linux is the right decision technically.  What is the right business decision to accompany the technical decision?
  • MVL6 Demonstration
    • toolchains pre-packaged
    • sources are all downloaded from MV servers
  • Interview with Joe Green
    • manager of developer tools team
    • MLV6 is new approach
    • previous editions were consistent for all targets
    • MLV6 more flexible model.  Market specific distros depending on market.
    • source driven product.  Previous products based on binary RPMs.  MLV6 is a lot easier to build the whole environment.  Very customized/fine-tuned distro.
    • key problems MLV6 solves
      • Complete starting point for target hardware
      • complete build system, so you have total control vs pre-canned distribution that sets all the rules
    • Integration platform based on Bitbake.  Working closely with community.  Enhanced in a number of ways.  Customers have the option of bringing in software developed by the OpenEmbedded community and bringing it into their project.
    • Gives you a system that you can configure at almost any level.
    • Enhancements to make it easy to make changes, as well as provide a pre-built starting point.
    • Features to control sources, and environment so the host environment does not contaminate your build.
    • Essentially hardening, and commercializing the bitbake and openembedded projects.
    • Why MVL6 vs roll-your-own.
      • Qualified starting point
      • Support.  Community or Semi-conductor vendor support is hit or miss.
      • Consistent environment accross multiple systems.
      • Includes devrocket graphical tools

Overall, this seems like a very good approach.  I’m glad they are not re-inventing the wheel, but instead have chosen to build on top of a proven open source solution, and leverage the work that is being done by a very active community of OpenEmbedded developers.  The OpenEmbedded project currently has 61 developers (not all are active), and weekly contribution rate is very good.  This type of effort would be very difficult to match by any but the largest commercial organizations.  This should be a win-win situation for all involved.  As a common base technology, the Bitbake and OpenEmbedded projects should improve due to MV’s involvement.  Through the MVL6 product, the OpenEmbedded project and way of doing things will get a lot more exposure and there will be a lot more developers who understand how it works.  Developers who are familiar with the OpenEmbedded build system will be able to apply their knowlege to a greater number of projects.   And developers building products will have more options for getting the support and techology they need.


How to set up a NFS root filesystem for embedded Linux development

Although ssh and friends work really well for embedded systems, occasionally you want to set up a NFS root for development.  One of the scenarios where nfsroot is useful is if you are making a lot of rootfs changes, and you don’t want to spend the time to reprogram the flash on the target system.  Fortunately, the Linux kernel includes complete support for NFS root, and does not require any userspace changes.  This setup assumes the following:

  1. the target system has Ethernet support built into the kernel
  2. the target system is on the same network as a Linux workstation
  3. said network includes a DHCP server

This configuration emphasises “simple” and does not require you to spend 4 hours trying to get a bootp server configured on a test network.

Kernel Support

Make sure you have the following options turned on in the kernel:


Then, add the following to your kernel CMDLINE:

  • ip=dhcp root=/dev/nfs nfsroot=<nfs server IP>:/path/to/nfsroot

Workstation Setup

The following setup is for Ubuntu.

apt-get install nfs-user-server

(edit /etc/exports to contain something like the following)


/etc/init.d/nfs-user-server restart

If you are using OpenEmbedded, then instruct OE to generate a rootfs tar image, and extract to your nfsroot directory:

cd /path/to/nfsroot
sudo tar -xvf <path to OE images dir>/Angstrom-image...rootfs.tar

And the result

eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
Sending DHCP requests ., OK
IP-Config: Got DHCP answer from, my address is
IP-Config: Complete:
     device=eth0, addr=, mask=, gw=,
     host=,, nis-domain=(none),
     bootserver=, rootserver=, rootpath=
Looking up port of RPC 100003/2 on
Looking up port of RPC 100005/1 on
VFS: Mounted root (nfs filesystem) readonly.
Freeing unused kernel memory: 220k freed
Write protecting the kernel text: 2792k
Write protecting the kernel read-only data: 788k
INIT: version 2.86 booting