Do you need “software update” functionality in your Embedded Linux system?

In this day and age, most embedded systems include a way for users to easily update software once the device has been deployed.  This article discusses the requirements for a field update mechanism along with pointers for how to implement.

Update Mechanism Requirements

Requirements for a field update mechanism might be:

  • easy for users to perform updates.
  • little chance of “bricking” a unit if reset occurs during update.
  • ability to update all of the software in the unit
  • in this case, update from files on a USB Storage device
  • ability to program NOR and NAND flash

Other systems may have requirements to update from a network, Compact Flash card, etc.

The USB wrinkle

System update is typically done by the bootloader.  In this case, we are using a Compulab cm-x270 module ( that has a proprietary boot loader and does not support updates from a USB storage device.  Therefore, we need to run the updates from the context of the Linux operating system which has drivers for USB and the NAND flash.  We decided that the best way in this case is to have a small Linux OS image that contains enough functionality to update the main filesystem.  This also provides us with the capability to recover if the main filesystem ever gets corrupted.  The flash layout for the system is:

Partition Flash Device Size File System Description
Kernel NOR 1536KiB NA Linux kernel
Update rootfs NOR 2304KiB Initramfs (RO) Small rootfs that contains just enough functionality to update the system
Main rootfs NAND 512MiB JFFS2 (RW) Main rootfs including application

There are other schemes that could work as well.  One way might be to have two identical application partitions and only update one at a time.  A reset while updating the kernel or the update rootfs does have the potential of “bricking” the system, but most of the updates will be for the Main rootfs, so we think this risk is low enough to be acceptable in this system.  The update rootfs is loaded into RAM before it is used, so there is little chance of it being corrupted during normal operation.  Running the update rootfs from ram is also convenient in that we can easily re-program the update rootfs in flash as we are not directly mounting the filesystem in flash.

Building the Update rootfs

The base update rootfs is built with OpenEmbedded as described in  Once we had the system booting into the update rootfs, we had to add the following functionality:

  • Boot the main rootfs from NAND flash
  • look for update files on a USB Storage device and flash the kernel, update rootfs, and main rootfs as needed.

Booting the NAND flash image is accomplished using the busybox switch_root application.  This utility allows you to easily switch to another rootfs from an initramfs.  There are a few gotchas to be aware of when working with switch_root:

  • switch_root must be “exec’d” from the init process with PID=”1″.
  • switch_root requires a “/init” file to be present

The above functionality was accomplished by writing a shell script called /init.  If an error occurs, the /init shell script then launches the standard init process in /sbin/init which provides a terminal and allows for easy debugging.  The mtd-utils package provides a handy utility called nandwrite which can be used to write jffs2 images to NAND flash (handles bad blocks, etc).  At some point, we may remove all the extra functionality from the update rootfs like the standard sysvinit, terminal login, etc, but for now it is handy to have for debugging.  To give you an idea how easy it is to write to flash from a shell script, consider the following snippets:

# write Image into update rootfs parition
cat /usb/nor_rootfs.img > /dev/mtd2
# write Image to NAND rootfs partition
flash_eraseall /dev/mtd3
nandwrite /dev/mtd3 /usb/nand_jffs2.img
# and to launch rootfs in nand flash
exec switch_root -c /dev/console /jffs2 /sbin/init 5


Having typically used bootloaders for system update in the past, I’m very pleased with how this mechanism worked out.  Thanks to the flexibility of Linux and OpenEmbedded, this only took several days to implement and debug.  Writing update programs with shell scripts at the application level gives you a lot of flexibility and allows you to use standard Linux drivers for USB and Flash access which are very robust.  At some point if there is interest, we may look at cleaning up our code and contributing functionality to OpenEmbedded to generate a more generic update initramfs.  Please contact us or leave a comment if you are interested in collaborating on something like this.


Building a tiny “safe” boot image using OpenEmbedded

We are currently working a project that contains cm-x270 module ( ) which contains 4MB of NOR flash and 128MB of NAND flash.  The Linux kernel and a small root file system (rootfs) will be stored in NOR flash, and the main rootfs in the NAND flash.  The rootfs in the NOR flash will be used to update the system from a USB Flash drive in the field.  This article describes how to build a small boot rootfs using OpenEmbedded.  We also look at various ways to store the image in flash.


In many embedded systems, field upgrades is a requirement.  There are many ways to organize this, but one of the most reliable is to have two rootfs partitions.  The first partition is a “safe” rootfs that is only used to update the second partition.  The second partition contains the main application and can be updated in the field.  If a programming error, or power loss occurs while updating the 2nd partition,  you can always boot into the first rootfs partition and restart the process.

The kernel and the bootloader in this system take up about 1.5MB (out of 4MB total) of the NOR flash.  This leaves us with about 2.5MB for the update rootfs.  Because we are building the rest of the system using OpenEmbedded (OE), it may make sense for us to also build this update rootfs using OE to keep everything in one build environment.

uclibc, glibc, or klibc?

There are several libc libraries that are commonly used to build Embedded Linux systems.  glibc is the standard that is used in desktop and sever systems and is what is being used in the main NAND rootfs partition.   uclibc is smaller than glibc is often used in space constrained systems.  klibc is a very minimal libc subset that supports operations typically used in an initramfs.  Because we are already using glibc, we decided to try this first as we could then use the same toolchain and build tree and just generate two images — one for NAND, and one for NOR.

OpenEmbedded support for small images

OE includes a task-base-minimal meta target that can be used as a basis for a small images, or you can create your own based on task-base.  You will then need to create a custom image recipe that will use this meta task.  An example is shown below:

PR = "r0"

export IMAGE_BASENAME = "svs_nor"
export PACKAGE_INSTALL = "task-min"
# keep extra language files from being installed
export IMAGE_LINGUAS = ""

RDEPENDS = "task-min"

IMAGE_FSTYPES = "tar jffs2 ext2 cramfs squashfs squashfs-lzma cpio.gz"

#EXTRA_IMAGECMD_jffs2="--pad=0x1000000 --eraseblock=0x40000"

inherit image

In this case, I had defined a custom recipe for task-min based on task-base, which installs the following packages:

root@compulab-pxa270:~$ ipkg list_installed
base-files - 3.0.14-r58 -
base-passwd - 3.5.9-r2 -
busybox - 1.2.1-r11 -
initscripts - 1.0-r86 -
ipkg - 0.99.163-r1 -
ipkg-collateral - 1.0-r5 -
libc6 - 2.5-r4 -
libgcc1 - 4.1.1-r10 -
libipkg0 - 0.99.163-r1 -
makedevs - 1.0.0-r2 -
sysvinit - 2.86-r32 -
sysvinit-inittab - 2.86-r32 -
sysvinit-pidof - 2.86-r32 -
task-min - 1.0-r1 -
tinylogin - 1.4-r3 -
update-rc.d - 0.7-r0 -

Storing the NOR filesystem in flash

OE can generate images for many different flash filesystems and initrd mechanisms.  These are specified in the IMAGE_FSTYPES variable in the above recipe.  In this case, I chose to generate many different filesystems so I could see evaluate how much compression I would get with each filesystem type.  The results:

Rootfs Type Size (bytes)
squashfs-lzma 1785856
tar.bz2 1918243
cpio.gz 2024558
squashfs 2109440
cramfs 2265088
jffs2 2455396
ext2 (uncompressed) 5481000

I ended up choosing the cpio.gz format as it can be loaded directly into an initramfs filesystem by the kernel.   There are several advantages to using an initramfs for this task:

  • You don’t have to mess around setting up tmpfs filesystems for directories that need to be writable.
  • You can update the NOR flash partition while you are running out of the initramfs filesystem.
  • It is fairly small with only squashfs-lzma being smaller.

Future Optimizations

For now, the glibc based solution is good enough because it fits in the flash space we have and it is easy to build with our existing build environment.  In the future, we may move to a uclibc or klibc based solution and try to reduce our flash size (OpenEmbedded supports both klibc and uclibc).  The above image sizes can also be reduced by removing ipkg and other components that are not needed.

This exercise illustrates many of the advantages of Linux and OpenEmbedded.  With very little work, I can produce exactly what I need.  The ability of the 2.6 Linux kernel to load a cpio archive into an initramfs is a very elegant solution for small boot images.


Remote access for embedded systems behind a firewall

When deploying embedded systems, wouldn’t it be nice if you could easily set up remote access for troubleshooting and updates?  With Embedded Linux systems, this is quite easy to do.  This article will explore how to use SSH to set up remote access to an embedded system behind a firewall without using VPN or special network setup and presents a few tips for making this setup work well.

The Problem

With today’s complex embedded systems, testing and deployment can be a challenge.   During the initial testing and roll-out, a new system may need to be carefully monitored to assure it is operating correctly.  Embedded Systems programming is different than application programming in that often you are interfacing with external I/O.  The behavior of things external to the embedded system are often very difficult to model during development; therefore time must be spent in the field testing the system in real-world situations.  Being able to remotely access the system during testing is very convenient.

As with most networked computer systems, networked embedded systems are typically deployed on a network behind a firewall. It is very easy to log in and administer any Linux system you have direct network access to — the challenge is getting to a system behind a firewall.

Options for remote Access

There are several general options for remote access to networks:

  • VPN
  • Port forwarding at the firewall
  • PC remote access technologies.  These are typically single session type connections that are initiated by the PC user.

The disadvantage of the above technologies is they require involvement from the IT department or users of the remote network.  BEC is currently involved in the deployment of a system that uses embedded computers in six different buildings.  Each building has its own firewalled network.  Getting remote access to all of these networks is a big task and everyone is busy.  An ideal solution would be a way to communicate directly with each embedded computer from a central location, which brings us to SSH …


SSH (Secure Shell) is a set of standards and an associated network protocol that is used to establish a secure channel between a local and remote computer.  SSH encrypts all traffic (including passwords) and is generally considered to be very secure if used properly.  Some uses for SSH are:

  • remote terminal access.
  • transfer files between computers.
  • tunnelling and port forwarding between computers

The fact that SSH can be used to forward ports over an encrypted tunnel is what makes remote access possible using what is called a Reverse SSH tunnel.  Because the embedded system is behind a firewall with no direct access from the Internet, the connection must be initiated from the embedded system.   A reverse SSH tunnel is set up as follows:

  • the embedded systems make a SSH connection to an external server that is internet accessible.
  • The SSH connection is set up so that a port on the external server (let’s pick 2900) is forwarded back over the SSH tunnel to port 22 (port used for SSH) on the embedded system.
  • The developer connects to 2900 on the external server using SSH.  Because port 2900 on the external server gets forwarded over the SSH tunnel to port 22 of the embedded system, the developer is actually logging into the embedded system.

This connection is illustrated in the figure below:


In the above figure, the external server is typically a system located in the developer’s office with only port 22 exposed to the Internet.  Port 2900 is only accessible to developers on the same private network as the server.


When talking about any type of remote access technology involving the internet, it is important to consider the security implications.  The reverse SSH mechanism is fairly secure, as everything is encrypted.  SSH is widely used for remote access to servers; therefore it receives a lot of testing and scrutiny.  With the reverse SSH remote access setup, the most likely attack is probably to compromise the external server and then try to access the remote network the embedded system is located on through the embedded system.    If the external server is properly secured, then this mechanism should be fairly safe.  One of the advantages of this scheme is that the embedded system is not directly exposed to the Internet, therefore  securing the embedded system itself is not all that important.

It is obviously very important that the people administering the remote network understand what you are doing — that is where this article can come in handy.  If there are concerns about external developers accessing a network, one mode of operation may be for the the SSH tunnel to be manually started and stopped so that administrators of the remote network know exactly when remote access is enabled.


A few ideas for making this work well:

  • It may be useful to put the reverse SSH connection in a cron job on the embedded system so that the connection is automatically set up.  If you do this, I advise killing the connection and restarting every so often.  Reverse SSH connections are somewhat fragile and occasionally need restarted if there is a network glitch.
  • Use the SSH TCPKeepAlive option.  Some firewalls/routers will drop a network connection if there is no activity.


Using SSH for remote access is another example of the “Big Win” with Embedded Linux — how industrial grade technologies developed for use in server and desktop systems can be readily used in embedded systems ( ).  Remote access during the deployment and testing of embedded systems can drastically reduce the amount of time and effort required to roll out an embedded system.


How to build GNU/Linux for an embedded x86 computer

Recently, I had a customer who needed an embedded Linux distribution running on an Advantech PCM-9371 single board computer (SBC).  The PCM-9371 contains a low voltage Celeron or Pentium III processor.  This article describes why the Openembedded build system was chosen and a few tips for running Openembedded on a x86 system.

The hard way!

There are many ways to put Linux on a x86 embedded system.  There are hundreds of distributions out there and the temptation is always to do something “quick” and just try to trim down one of the standard distributions.  Other shortcuts are often taken, such as compiling your application on a workstation that is running a different distribution and different versions of the libraries used in the embedded systems.  This approach can work, but there are many possible pitfalls:

  • Several years down the road, it will be difficult to compile applications for your target system as the libraries on the target system are very dated compared to tools and libraries used in standard desktop distributions.  What this means is you will need to keep a “golden” compile machine around for the life of your project — not fun!  I went through an experience like this on a previous version of this same project where it took us a week to set up a build machine to compile applications using a very old version of Debian.
  • It is difficult to gather GPL sources used to build the distribution you are using.  You are required to supply GPL source code upon request to customers who purchase your product.
  • To build applications correctly, you really need a build machine that is running the same distribution as the target, unless you set up a cross-compilation or scratchbox environment.
  • Most distributions are quite large and have difficulty fitting on CF card.
  • Most distributions are difficult to customize in a clean, systematic way.


Considering the disadvantages of using an standard x86 distribution for this project, I chose to use OpenEmbedded (OE).  The OpenEmbedded project ( ) is a full featured build system for building embedded Linux distributions.  The development is done on a Linux workstation of your choice and the entire distribution is cross-compiled to the architecture of your choice.  The entire build process is automated, including building the toolchain and an image that can be installed on the target embedded system.  Some of the features of OpenEmbedded are:

  • Over 3000 packages that can be built.  About anything you would ever need for an embedded system.
  • Support for a wide variety of target architectures.  ARM and x86 processors are well supported.
  • Build system is very self contained.  This means you can do the builds on a variety of different host machines.  Many of the tools OE needs are built by OE, so there is very little dependence on host tools besides a few basics such as Python, make, and other standard tools.  This also means it is much easier to pull the build system out of the archives in several years, load it on a current machine, and do a build.
  • All source code used in the build is downloaded, so GPL compliance is easy.
  • and the list goes on …

Build and Installation

Setting up an OE build is beyond the scope of this article, but it basically involves selecting a machine and distribution.  In this case, I chose the x86 machine and the generic OE distribution.  Several hours later, I had an image in the form of a tar.gz file that I decompressed to a Compact Flash (CF) card.  After installing grub on the CF card, I had a booting Linux distribution that included all the basics (like ssh, ipkg package manager, shell, standard file utilities, etc) in less than 4MB of disk space.

Adding GTK+ and Xwindows

This system required support for GTK+ and Xwindows.  To add this involved running the following commands: bitbake xserver-kdrive and bitbake gtk+.  The OE build system then automatically downloaded and compiled the need packages.  The image size is now around 20MB — still fits very easily on a CF card.

Getting X running

There are several options for running X on a x86 system.  In this case, I chose the quickest way so that my customer could get started with application integration as quickly as possible.  Linux and Xwindows support the standard VESA Framebuffer mode.  Then can be enabled by selecting the CONFIG_FB_VESA kernel config option.  The kdrive version of X can then be started by running Xvesa.  In the future, we may move to an accelerated X driver if needed.


We now have Linux running off of a CF card with full X/GTK+ support and the customer can proceed with integration, testing and application development.  Getting to this point took about 1 day of development as we did not encounter any major problems — much less time than it took to resurrect an old build environment used in previous versions of the project.  Future work will involve integrating applications into the OE build system, and customizing the distribution as needed.  Openembedded has proved to be a very valuable tool for getting embedded Linux systems running quickly and maintaining them in a controlled way.


Adding software to a Moxa UC7408 using OpenEmbedded

One of the big advantages of using Linux in embedded systems is the thousands of applications and drivers available for it ( ).  One of the challenges of Embedded Linux is building these applications.  Building applications for an embedded system often involves cross-compiling which is frustrating on a good day.  This article details how to use OpenEmbedded (OE) with the existing toolchain Moxa provides to build a number of applications for the Moxa UC7408.

Web Application Support

My needs for this project were to build a web application that ran on the UC7408 (See for a review of the Moxa UC7408).  The Moxa firmware is fairly full featured and provides php for web application development.  This is probably fine for many applications, but I prefer to build web applications as detailed in this article: .  Because the rest of the application is written in Python, there are advantages to also implementing the web application in Python.  I also wanted to use the sqlite database.  A quick look at the php supplied with the Moxa firmware suggested it was built without sqlite support.  In summary, I needed the following components:

  • Python
  • Clearsilver
  • sqlite
  • minicom
  • vim
  • strace
  • subversion
  • wget

The above is a fairly small list, but what you don’t see are all the dependencies that also need to be built and installed.  This ends up being many more components.   Some of the items are for development only, but this is Linux — why not have nice tools running on the target system?

OE to the rescue

Ideally, the Moxa system would have a complete root file system built with OE running on it.  But I did not have the time or budget to implement this, so I did the next best thing — use the rootfs and toolchain supplied by Moxa along with OE to compile just the applications I needed.  Typically, OE builds the toolchain and Glibc, but fortunately OE provides a very slick way to use an external toolchain and glibc with OE.  How to do this is detailed in the OE manual:  Below is the setup I used:

Environment variables:

export PRE_BUILT=/usr/local/mxscaleb/armv5b-linux

export CCACHE_DIR="${TOPDIR}/ccache"
export PYTHONPATH="${TOPDIR}/bitbake/lib"
export BBPATH="${TOPDIR}:${TOPDIR}/openembedded:${TOPDIR}/bitbake"
export PATH="${TOPDIR}/bitbake/bin:${TOOLCHAIN}:${PATH}"
export LANG=C

Local.conf file:

DL_DIR = "/build/sva_oe/downloads"
BBFILES := "/build/sva_oe/openembedded/packages/*/*.bb"
PREFERRED_PROVIDERS = "virtual/qte:qte virtual/libqpe:libqpe-opie"
PREFERRED_PROVIDERS += " virtual/libsdl:libsdl-qpe"
PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}gcc-initial:gcc-cross-initial"
PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}gcc:gcc-cross"
PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}g++:gcc-cross"
PREFERRED_PROVIDER_virtual/libx11 = "diet-x11"
ASSUME_PROVIDED += " virtual/libc "
MACHINE = "moxa"
IPKG_ARCHS = "all arm armv4 armv5te xscale ${MACHINE}"
TARGET_CC_ARCH = "-mcpu=xscale"
TARGET_OS = "linux"
TARGET_FPU = "soft"
DISTRO = "moxa-disro"
DISTRO_NAME = "moxa-distro"
DISTRO_TYPE = "release"
INHERIT += " package_ipk package_tar debian"
IMAGE_FSTYPES = "jffs2 tar"
export CC  = "mxscaleb-gcc-3.3.2 ${HOST_CC_ARCH}"
export CXX = "mxscaleb-g++ ${HOST_CC_ARCH}"
export CPP = "mxscaleb-gcc-3.3.2 -E"
export LD = "mxscaleb-ld"
export AR  = "mxscaleb-ar"
export AS  = "mxscaleb-as"
export RANLIB  = "mxscaleb-ranlib"
export STRIP  = "mxscaleb-strip"
TARGET_CPPFLAGS_append = " -I${PRE_BUILT}/include "
TARGET_LDFLAGS_prepend = " -L${PRE_BUILT}/lib -Wl,-rpath-link, ...
${PRE_BUILT}/lib -Wl,-rpath-link,${PRE_BUILT}/qt2/lib "

With the above setup, I was able to build the applications I needed with very little effort.  There were a few little problems I ran into, but I was able to quickly work around them.  The result is a number of packages that can be installed on the target system:

bigreqsproto-dev-X11R7.0-1.0.2-r0.tar.gz      python-crypt-2.4.3-ml0.tar.gz
busybox-1.2.1-r1.3.tar.gz                     python-curses-2.4.3-ml1.tar.gz
busybox-udhcpd-1.2.1-r1.3.tar.gz              python-datetime-2.4.3-ml0.tar.gz
clearsilver-0.10.3-r0.tar.gz                  python-db-2.4.3-ml0.tar.gz
clearsilver-dbg-0.10.3-r0.tar.gz              python-devel-2.4.3-ml0.tar.gz
clearsilver-dev-0.10.3-r0.tar.gz              python-distutils-2.4.3-ml0.tar.gz
clearsilver-doc-0.10.3-r0.tar.gz              python-email-2.4.3-ml0.tar.gz
inputproto-dev-X11R7.0-1.3.2-r0.tar.gz        python-fcntl-2.4.3-ml0.tar.gz
ipkg-0.99.163-r1.tar.gz                       python-gdbm-2.4.3-ml0.tar.gz
ipkg-dbg-0.99.163-r1.tar.gz                   python-hotshot-2.4.3-ml0.tar.gz
ipkg-dev-0.99.163-r1.tar.gz                   python-html-2.4.3-ml0.tar.gz
kbproto-dev-X11R7.1-1.0.3-r0.tar.gz           python-idle-2.4.3-ml0.tar.gz
libapr-0-0-0.9.12-r0.tar.gz                   python-image-2.4.3-ml0.tar.gz
libapr-0-bin-0.9.12-r0.tar.gz                 python-io-2.4.3-ml0.tar.gz
libapr-0-dev-0.9.12-r0.tar.gz                 python-lang-2.4.3-ml0.tar.gz
libaprutil-0-0-0.9.12-r0.tar.gz               python-lib-old-and-deprecated-2.4.3-ml0.tar.gz
libaprutil-0-bin-0.9.12-r0.tar.gz             python-logging-2.4.3-ml0.tar.gz
libaprutil-0-dev-0.9.12-r0.tar.gz             python-mailbox-2.4.3-ml0.tar.gz
libcrypto0.9.7-0.9.7g-r1.tar.gz               python-math-2.4.3-ml0.tar.gz
libexpat-bin-2.0.0-r1.tar.gz                  python-mime-2.4.3-ml0.tar.gz
libexpat-dev-2.0.0-r1.tar.gz                  python-mmap-2.4.3-ml0.tar.gz
libexpat-doc-2.0.0-r1.tar.gz                  python-netclient-2.4.3-ml1.tar.gz
libexpat1-2.0.0-r1.tar.gz                     python-netserver-2.4.3-ml0.tar.gz
libgcrypt-dbg-1.2.2-r0.tar.gz                 python-pickle-2.4.3-ml0.tar.gz
libgcrypt-dev-1.2.2-r0.tar.gz                 python-pprint-2.4.3-ml0.tar.gz
libgcrypt-doc-1.2.2-r0.tar.gz                 python-profile-2.4.3-ml0.tar.gz
libgcrypt11-1.2.2-r0.tar.gz                   python-pydoc-2.4.3-ml0.tar.gz
libgdbm-dbg-1.8.3-r2.tar.gz                   python-pyserial-2.2-r1.tar.gz
libgdbm-dev-1.8.3-r2.tar.gz                   python-pysqlite2-2.2.2-ml1.tar.gz
libgdbm-doc-1.8.3-r2.tar.gz                   python-pyxml-0.8.4-ml0.tar.gz
libgdbm3-1.8.3-r2.tar.gz                      python-re-2.4.3-ml0.tar.gz
libgnutls-bin-1.4.0-r1.tar.gz                 python-readline-2.4.3-ml0.tar.gz
libgnutls-dbg-1.4.0-r1.tar.gz                 python-resource-2.4.3-ml0.tar.gz
libgnutls-dev-1.4.0-r1.tar.gz                 python-shell-2.4.3-ml0.tar.gz
libgnutls-doc-1.4.0-r1.tar.gz                 python-stringold-2.4.3-ml0.tar.gz
libgnutls-extra13-1.4.0-r1.tar.gz             python-subprocess-2.4.3-ml0.tar.gz
libgnutls-locale-en+boldquot-1.4.0-r1.tar.gz  python-syslog-2.4.3-ml0.tar.gz
libgnutls-locale-en+quot-1.4.0-r1.tar.gz      python-terminal-2.4.3-ml0.tar.gz
libgnutls-locale-pl-1.4.0-r1.tar.gz           python-tests-2.4.3-ml0.tar.gz
libgnutls-openssl13-1.4.0-r1.tar.gz           python-textutils-2.4.3-ml0.tar.gz
libgnutls13-1.4.0-r1.tar.gz                   python-threading-2.4.3-ml0.tar.gz
libgpg-error-dbg-1.3-r1.tar.gz                python-tkinter-2.4.3-ml0.tar.gz
libgpg-error-dev-1.3-r1.tar.gz                python-unittest-2.4.3-ml0.tar.gz
libgpg-error-locale-de-1.3-r1.tar.gz          python-unixadmin-2.4.3-ml0.tar.gz
libgpg-error-locale-fr-1.3-r1.tar.gz          python-xml-2.4.3-ml0.tar.gz
libgpg-error-locale-pl-1.3-r1.tar.gz          python-xmlrpc-2.4.3-ml0.tar.gz
libgpg-error-locale-ro-1.3-r1.tar.gz          python-zlib-2.4.3-ml1.tar.gz
libgpg-error-locale-vi-1.3-r1.tar.gz          sqlite3-3.3.7-r2.tar.gz
libgpg-error0-1.3-r1.tar.gz                   sqlite3-dbg-3.3.7-r2.tar.gz
libice-dbg-X11R7.1-1.0.1-r0.tar.gz            strace-4.5.14-r3.tar.gz
libice-dev-X11R7.1-1.0.1-r0.tar.gz            strace-dbg-4.5.14-r3.tar.gz
libice6-X11R7.1-1.0.1-r0.tar.gz               strace-doc-4.5.14-r3.tar.gz
libipkg-dev-0.99.163-r1.tar.gz                subversion-1.3.1-r1.tar.gz
libipkg0-0.99.163-r1.tar.gz                   subversion-dbg-1.3.1-r1.tar.gz
liblzo-dbg-1.08-r14.tar.gz                    subversion-dev-1.3.1-r1.tar.gz
liblzo-dev-1.08-r14.tar.gz                    subversion-doc-1.3.1-r1.tar.gz
liblzo1-1.08-r14.tar.gz                       subversion-locale-de-1.3.1-r1.tar.gz
libneon-bin-0.25.5-r0.tar.gz                  subversion-locale-es-1.3.1-r1.tar.gz
libneon-dev-0.25.5-r0.tar.gz                  subversion-locale-fr-1.3.1-r1.tar.gz
libneon-doc-0.25.5-r0.tar.gz                  subversion-locale-it-1.3.1-r1.tar.gz
libneon25-0.25.5-r0.tar.gz                    subversion-locale-ja-1.3.1-r1.tar.gz
libpython2.4-1.0-2.4.3-ml5.tar.gz             subversion-locale-ko-1.3.1-r1.tar.gz
libreadline-dbg-4.3-r3.tar.gz                 subversion-locale-nb-1.3.1-r1.tar.gz
libreadline-dev-4.3-r3.tar.gz                 subversion-locale-pl-1.3.1-r1.tar.gz
libreadline-doc-4.3-r3.tar.gz                 subversion-locale-pt-br-1.3.1-r1.tar.gz
libreadline4-4.3-r3.tar.gz                    subversion-locale-sv-1.3.1-r1.tar.gz
libsm-dbg-X11R7.1-1.0.1-r0.tar.gz             subversion-locale-zh-cn-1.3.1-r1.tar.gz
libsm-dev-X11R7.1-1.0.1-r0.tar.gz             subversion-locale-zh-tw-1.3.1-r1.tar.gz
libsm6-X11R7.1-1.0.1-r0.tar.gz                tcl-8.4.11-r3.tar.gz
libsqlite-bin-2.8.17-r2.tar.gz                tcl-dbg-8.4.11-r3.tar.gz
libsqlite-dbg-2.8.17-r2.tar.gz                tcl-dev-8.4.11-r3.tar.gz
libsqlite-dev-2.8.17-r2.tar.gz                tcl-doc-8.4.11-r3.tar.gz
libsqlite0-2.8.17-r2.tar.gz                   time-1.7-r0.tar.gz
libsqlite3-0-3.3.7-r2.tar.gz                  time-dbg-1.7-r0.tar.gz
libsqlite3-dev-3.3.7-r2.tar.gz                time-doc-1.7-r0.tar.gz
libssl0.9.7-0.9.7g-r1.tar.gz                  tmp
libx11-6-X11R7.1-1.0.1-r1.tar.gz              update-rc.d-0.7-r0.tar.gz
libx11-dbg-X11R7.1-1.0.1-r1.tar.gz            util-macros-dev-X11R7.1-1.0.2-r0.tar.gz
libx11-dev-X11R7.1-1.0.1-r1.tar.gz            vim-7.0-r0.tar.gz
libx11-doc-X11R7.1-1.0.1-r1.tar.gz            vim-doc-7.0-r0.tar.gz
libx11-locale-X11R7.1-1.0.1-r1.tar.gz         vim-help-7.0-r0.tar.gz
libxau-dbg-X11R7.1-1.0.1-r0.tar.gz            vim-syntax-7.0-r0.tar.gz
libxau-dev-X11R7.1-1.0.1-r0.tar.gz            vim-tutor-7.0-r0.tar.gz
libxau-doc-X11R7.1-1.0.1-r0.tar.gz            wget-1.9.1-r6.tar.gz
libxau6-X11R7.1-1.0.1-r0.tar.gz               wget-dbg-1.9.1-r6.tar.gz
libxdmcp-dbg-X11R7.1-1.0.1-r0.tar.gz          wget-doc-1.9.1-r6.tar.gz
libxdmcp-dev-X11R7.1-1.0.1-r0.tar.gz          wget-locale-bg-1.9.1-r6.tar.gz
libxdmcp6-X11R7.1-1.0.1-r0.tar.gz             wget-locale-ca-1.9.1-r6.tar.gz
libxml2-2.6.22-r3.tar.gz                      wget-locale-cs-1.9.1-r6.tar.gz
libxml2-dbg-2.6.22-r3.tar.gz                  wget-locale-da-1.9.1-r6.tar.gz
libxml2-dev-2.6.22-r3.tar.gz                  wget-locale-de-1.9.1-r6.tar.gz
libxml2-doc-2.6.22-r3.tar.gz                  wget-locale-el-1.9.1-r6.tar.gz
libxml2-utils-2.6.22-r3.tar.gz                wget-locale-es-1.9.1-r6.tar.gz
libxt-dbg-X11R7.1-1.0.2-r0.tar.gz             wget-locale-et-1.9.1-r6.tar.gz
libxt-dev-X11R7.1-1.0.2-r0.tar.gz             wget-locale-fr-1.9.1-r6.tar.gz
libxt-doc-X11R7.1-1.0.2-r0.tar.gz             wget-locale-gl-1.9.1-r6.tar.gz
libxt6-X11R7.1-1.0.2-r0.tar.gz                wget-locale-he-1.9.1-r6.tar.gz
libz-dbg-1.2.3-r1.tar.gz                      wget-locale-hr-1.9.1-r6.tar.gz
libz-dev-1.2.3-r1.tar.gz                      wget-locale-hu-1.9.1-r6.tar.gz
libz1-1.2.3-r1.tar.gz                         wget-locale-it-1.9.1-r6.tar.gz
minicom-2.1-r0.tar.gz                         wget-locale-ja-1.9.1-r6.tar.gz
minicom-dbg-2.1-r0.tar.gz                     wget-locale-nl-1.9.1-r6.tar.gz
minicom-doc-2.1-r0.tar.gz                     wget-locale-no-1.9.1-r6.tar.gz
ncurses-5.4-r8.tar.gz                         wget-locale-pl-1.9.1-r6.tar.gz
ncurses-dbg-5.4-r8.tar.gz                     wget-locale-pt-br-1.9.1-r6.tar.gz
ncurses-dev-5.4-r8.tar.gz                     wget-locale-ro-1.9.1-r6.tar.gz
ncurses-doc-5.4-r8.tar.gz                     wget-locale-ru-1.9.1-r6.tar.gz
ncurses-terminfo-5.4-r8.tar.gz                wget-locale-sk-1.9.1-r6.tar.gz
ncurses-tools-5.4-r8.tar.gz                   wget-locale-sl-1.9.1-r6.tar.gz
openssl-0.9.7g-r1.tar.gz                      wget-locale-sv-1.9.1-r6.tar.gz
openssl-dbg-0.9.7g-r1.tar.gz                  wget-locale-tr-1.9.1-r6.tar.gz
openssl-dev-0.9.7g-r1.tar.gz                  wget-locale-uk-1.9.1-r6.tar.gz
openssl-doc-0.9.7g-r1.tar.gz                  wget-locale-zh-cn-1.9.1-r6.tar.gz
python-audio-2.4.3-ml0.tar.gz                 wget-locale-zh-tw-1.9.1-r6.tar.gz
python-bsddb-2.4.3-ml0.tar.gz                 xcmiscproto-dev-X11R7.0-1.1.2-r0.tar.gz
python-codecs-2.4.3-ml0.tar.gz                xextproto-dev-X11R7.0-7.0.2-r0.tar.gz
python-compile-2.4.3-ml0.tar.gz               xf86bigfontproto-dev-X11R7.0-1.1.2-r0.tar.gz
python-compiler-2.4.3-ml0.tar.gz              xproto-dev-X11R7.1-7.0.5-r0.tar.gz
python-compression-2.4.3-ml0.tar.gz           xtrans-dev-X11R7.0-1.0.0-r1.tar.gz

Installing the needed packages is just a matter of uncompressing a subset of the above files on the target system.


This exercise demonstrates how a developer can quickly (took me less than 4 hours) add a large number of applications (and dependencies) to about any Embedded Linux system.  Leveraging embedded Linux is being able to use the components you need.  Please contact us if you would like additional details or assistance.


Moxa UC7408 Review

I am currently evaluating a Moxa UC7408 for one of my customers.  The UC7408 is a small fanless industrial computer that runs Linux or Windows CE.  This article provides a basic overview of this unit and a review of some of the UC7408 features and the Linux distribution Moxa provides.


The basic specifications for the UC7408 are:

  • relatively small size and fanless
  • Intel Xscale IXP-422 266MHz Processor
  • 8 RS232/422/485 serial ports
  • 8 digital Input and 8 digital output ports
  • dual 10/100 Ethernet
  • PCMCIA, CompactFlash, Wireless LAN Expansion (supports 802.11b/802.11g)
  • Runs Linux or WinCE
  • Console serial port


The documentation Moxa provides is adequate and fairly detailed.  The moxa documents include a quick install guide, hardware user’s manual, and a 114 page user’s manual.  As good as this documentation is, a general knowledge of Linux is always helpful when working with embedded Linux systems.  Basics like using SSH, FTP, Telnet, and basic Linux system administration are essential when working with embedded Linux.

Linux Distribution

The Linux Distribution provided by Moxa is fairly full featured.  A few highlights with the v1.8 Linux firmware:

  • Provides a 26MB user JFFS2 flash partition.  This partition is read/write and is about 45% full leaving about 14.5MB for user files.  It is probably also possible to delete some of the files that are not required, freeing up more space.
  • Apache & PHP support
  • SSH
  • a number of other utilities are provided such as telnet, ftp, iptables, etc.
  • VI editor
  • Many command line utilities found in most Linux systems.
  • 2.4.18 Kernel

The Linux distribution seems to be based on MontaVista Linux and seems fairly solid.  I ran through some of the basics like mounting a CF card and everything seems to work.

Firmware Updates and Recovery Mode

Moxa provides a mechanism to update the firmware in the system.  This update is a global flash update that programs the entire flash and will erase all user changes.  The update works by enabling a RAM file system and copying the new flash image to the ram file system.  Running an update command that copies the firmware update file into flash.

The 7408 contains a stripped down version of Linux in a separate flash partition that can be booted in a recovery mode if the User flash partition becomes corrupted or unusable.  This mode can be enabled by pressing the “Reset to Default” button on the unit and powering it on.  In this mode, there is no ssh support and you must ftp an image from an ftp server to the device and then reflash it.  This requires you to set up an ftp server that the Moxa system can access.  While this is an entirely workable solution, it may be a challenge for inexperienced Linux users.


Moxa provides a Linux and Windows toolchain to use with the device.  I have built a number of packages with the Linux toolchain and it seems to work well.  It is based on GCC 3.3.2.  Cross compiling applications for embedded Linux is always a challenge and in a future article I will present a way to use OpenEmbedded to compile a number of packages using the Moxa toolchain.

Possible Improvements

There would be some benefits if the system was a little more open in the following areas:

  • access to the bootloader console and documentation.  This would allow developers to more easily flash their own software.
  • readily provide source for GPL components.  Moxa provides a form that you can fill out to receive GPL source code on a CE.  The cost is $100.

One of the big advantages of using Linux in an embedded system is the ability to use the many open source components available.  The easier an embedded solution provider makes this for developers, the more value their system provides.  See the white paper Tips for Planning Your Embedded Linux Project ( for more ideas on this subject.


Overall, the Moxa system seems very usable and the hardware is quite nice.  Stay tuned for future articles about how to get the most out of this system.





Autotools quick reference

At some point, most Linux developers need to master Autotools.  For most of us, this is a fairly painful process, but like any good tool, Autotools is extremely useful and well worth learning.  For example, if your program is set up correctly using Autotools, it will cross compile with almost zero effort in OpenEmbedded, or with slight effort manually.  Compare this to the tedious effort of manually referencing toolchain locations, cross toolchain library and header file locations, etc.  Despite first impressions, Autotools will make your life easier.  This article contains a collection of tips on how to do standard operations using Autotools.

Minimal Autotools files required for a simple C application

AC_INIT([fooprog], [1.0], [])
AM_INIT_AUTOMAKE([-Wall -Werror foreign])


bin_PROGRAMS = foo
foo_SOURCES = foo.c

The above files will create a binary name foo (compiled from foo.c) when you run the following steps:

  • touch NEWS AUTHORS ChangeLog
  • autoreconf -i
  • ./configure
  • make

How to add libraries to the link step

For example, if you want to link in the pthread library to your application, add the following line to

foo_LDADD = -lpthread

How to add compiler options

You can easily add compiler options using the foo_CFLAGS or foo_CPPFLAGS variables.


How to use pkg-config in Autotools

pkg-config is a helper tool that gives you library and include file information for various libraries installed.  For example, to link to alsa and glib in your project:



How to add a command line option to the configure script

    [  --enable-option-foo  help for option foo],

The above example sets a define on the compiler command line if configure is run with the –enable-option-foo command line option.

Disable shared libraries in your project

Add the following to


Pass a variable from to

Often there is a need to pass a variable in to a Makefile.  This is done with the AC_SUBST macro:


foo2 = @VAR_FOO@

Installing data files

By default, Autotools will install executables built when you run “make install”.  To install other non-executable files, you need to tell autotools explicitly about those files.  The standard place to install data files for an application is in $(datadir)/appname.

fooextradir = $(datadir)/@PACKAGE@
fooextra_DATA = \
	foo_data1.txt \

The above will install the foo_data1.txt and foo_data2.txt files in the $(datadir)/fooprog directory. $(datadir) is typically /usr/share/ or /usr/local/share, but it can be anything.

How does my program know where $(datadir) is located

Because $(datadir) can be anywhere in the system, we need some way to tell the program where it was specified to be at time of compilation so that it can locate foo_data1.txt and foo_data2.txt.  This can be done by passing a define at compile time:

AM_CFLAGS = -DDATADIR=\"$(datadir)\"

How to install data files during development without cluttering up my system directories

If you use the above method to install data files needed by your project, you may run into a case during development where you want to run the program from the build directory, but the program needs the data files at run time.  Since, the program is looking for them in the location specified by the DATADIR define, they need to be installed.  You could just run “make install”, but this would install them into /usr/local/share, require you to run as root, and clutter up your system directories.  A much more elegant solution is to specify the install prefix at configure time to be a temporary “install” directory in your source tree.  The –prefix option for configure requires an absolute directory, so the `pwd` trick is used to figure out what the current directory is.

./configure --prefix=`pwd`/install
make install

Your project will now be installed in a directory in your source tree named ./install.  This is also a nice way to test that the install for the project is working correctly as you can easily see everything that gets installed.  Now, the program has been compiled to reference extra files in ./install and will run just fine from the build directory.


The Linux kernel container_of macro

The Linux kernel contains a very useful macro named “container_of” that is used extensively in back-casting a data structure to its containing data structure.  This article includes a simple program that illustrates how this macro is used, and explains why it is so useful.

If you do a lot of C programming, this program is worth figuring out :-).

The program source can also be downloaded from the following location:

/* test code to illustrate use of Linux kernel container_of macro
 * Copyright (c) 2008 Cliff Brake, BEC Systems LLC
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License version 2 as
 * published by the Free Software Foundation.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

/* This program illustrates how the container_of macro works.
 * The container of macro is very useful in multi layered
 * software systems where you have progressivly more detailed
 * software layers.  Below is an example of a bus layer,
 * and then a device layer where a number of different
 * devices might register with the bus.
 * The device registers itself with the bus subsystem, and
 * then the bus subsystem makes a callback into the device.
 * Normally if there are multiple devices registered, the
 * bus subsystem must store and pass a device structure
 * when making callbacks.  With the container_of macro, this is
 * no longer necessary, and the bus subsystem only has to
 * know about one generic device structure, and does not need visibility
 * into lots of different device structures, or do tricks
 * by casting void pointers, etc.  With the container_of macro
 * we can backcast from the generic data structure, to the containing
 * datastructure.  This forces good separation of code in that
 * that bus layer cannot modifiy data structures that are specific
 * to the device layer.

 * (from Linux kernel source)
 * container_of - cast a member of a structure out to the containing structure
 * @ptr:	the pointer to the member.
 * @type:	the type of the container struct this is embedded in.
 * @member:	the name of the member within the struct.
#define container_of(ptr, type, member) ({			\
	const typeof( ((type *)0)->member ) *__mptr = (ptr);	\
	(type *)( (char *)__mptr - offsetof(type,member) );})

#define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER)

 * BUS layer code

/* generic bus device structure */
struct bus_device
	int general_device_param_a;
	int general_device_param_b;
	void (*device_callback)(struct bus_device * bd);

/* the following is a global list of
 * devices that have registered with the
 * bus subsystem.  Normally this would
 * be something like a dynamic linked list.
struct bus_device * bd_list[5];

/* function to register a device with the bus */
void register_with_bus(struct bus_device * bd)
	/* since this example only deals with one
	 * device, will put it in slot 0

	bd_list[0] = bd;

void start_bus()
	int i;
	struct bus_device * bd;

	/* make callbacks to all devices on bus */
	for (i=0;i<sizeof(bd_list)/sizeof(bd_list[0]);i++) {
		bd = bd_list[i];
		if (!bd) continue;
		/* call device callback with generic
		 * bus device structure

 * device X specific code
 * this would normally be in a different module

/* structure that holds device X specific stuff, as well as
 * generic bus_device structure
struct device_x
	int device_x_specific_param_a;
	int device_x_specific_param_b;
	struct bus_device bd;

void device_x_callback(struct bus_device * bd)
	/* if we know the structure type that contains the bus_device structure,
	 * we can extract a pointer to the containing structure using the container_of
	 * macro

	/*                                   ptr       type       member  */
	struct device_x * devx = container_of(bd, struct device_x, bd);

	/* the above statement expands to
	 * struct device_x * devx = (
	 * {
	 *   const typeof( ((struct device_x *)0)->bd ) *__mptr = (bd);
	 *   (struct device_x *)( (char *)__mptr - ((size_t) &((struct device_x *)0)->bd) );
	 * }
	 * );

	printf("device_x_callback called!, device_x_specific_param_a = %i\n",

void device_x_init()
	/* dynamically allocate structures */
	struct device_x * devx = malloc(sizeof(*devx));
	memset(devx, 0, sizeof(*devx));

	/* set a parameter in the device_x structure so
	 * we can test for this in the callback
	devx->device_x_specific_param_a = 1001;

	/* set up callback function */
	devx->bd.device_callback = device_x_callback;

	/* we register the generic bus device structure
	 * as the bus layer does not need to know
	 * about the device_x stucture.  Note, the
	 * devx structure is not stored anywhere, yet
	 * its location is being preserved without
	 * specifically passing it to the bus
	 * layer.

int main()

	/* test the above system */

	/* first, initialize device_x */

	/* now, start the bus.  This should make
	 * a callback into the device_x

/* when run, this program returns:
 * device_x_callback called!, device_x_specific_param_a = 1001

Gumstix Overo Connector Spreadsheet

One of the first things I do when designing a system based on a processor module is create a spreadsheet listing all the connector I/O.  This is the easiest way I’ve found to make sure nothing gets missed.  My first pass at a I/O spreadsheet for the Gumstix Overo is available at:


GESBC-9302E kernel update to 2.6.24, and reboot fix

I recently updated the GESBC-9302 machine support in OpenEmbedded to include the 2.6.24 released kernel.  Also, Glomation has kindly provided me with a patch for the software reboot problem which I have integrated into the OpenEmbedded build for the GESBC.   The patch files are part of the OpenEmbedded dev branch or can be obtained here:

Other related OpenEmbedded bits:

Future features to be integrated include RTC driver configuration, and support for loading Initramfs images from NOR flash.


Compulab EM-X270 Review

The EM-X270 is a full featured computer board from Compulab ( designed for handheld/mobile applications.  The board includes options for about anything you might need in a portable system including a PXA270 processor, GSM radio, GPS, Wifi, Bluetooth, Audio, SD/MMC, USB, battery circuitry, and the list goes on.  The EM-X270 is designed to give you a big head start in designing a handheld computing device, and can significantly reduce time to market and design costs.  This article provides a high level overview of this product, how it might be used, and lots of photos.

EM-X270 Overview

The EM-X270 is a computer board about the size of a display you might find in a typical PDA.  While the size is probably a little large to be used in consumer devices like phones and PDAs, it seems like a nice size for slightly larger products used in vertical industries.


As already mentioned, the EM-X270 includes about all the base features you might need in a mobile computing device.  An expansion connector is also included that allows you to easily add custom circuitry to the system.  The stack-up between the EM-X270 and an expansion board is shown below:


The EM-X270 differers from most other embedded computer modules in that it is a little more application specified, and quite a bit more circuitry is included in the design such as power supplies, battery charging circuitry, radios, etc.

Why EM-X270?

Why are we interested in the EM-X270?  If all you need is a generic handheld computer, there are plenty of options from PDAs to off-the-shelf industrial handhelds.  However, if you need a handheld that provides some additional features not typically found in existing devices, then you can use the EM-X270 plus a custom baseboard and packaging to add your high value features.  Depending what you need to add, the expansion board can be a simple 4 layer board which is very easy to design and manufacture.  All of the complex PCB design and assembly is already done for you on the EM-X270 module.  This allows low to moderate volume manufactures design products where they would not otherwise have the time and budget for designing a full custom processor board.   Some example applications might be handheld products for the medical, industrial, and test and measurement industry.

Compulab’s pricing model is also very interesting.  Compulab will custom build EM-X270 modules with just the options you need.  The base price without any extra features (such as radios) is $122 for 1000 pieces.

The EB-X270 Evaluation Kit

Compulab provides a EB-X270 Evaluation kit for getting started with EM-X270 development.  Like their other Evaluation klits, is is very full featured and very well done.   An extender board is provided that adds Ethernet, JTAG, and several other useful connectors for development.


The eval kit also includes all necessary cables, a display, battery, etc.


Software for the EM-X270

Compulab provides software support for both Windows CE and Linux.  This includes very up-to-date support for kernel version 2.6.23.  EM-X270 machine support has also been included in the OpenEmbedded project for building full featured Linux images.  This combination will drastically reduce development costs compared to solutions provided by many other vendors.  All too often, single board computer vendors provide an ancient kernel, a hacked up root file system and call it a “Linux BSP”.  Actually, any time the term “Linux BSP” is used, you should probably be a bit wary.  A much more attractive solution for those of us building products using Linux is for processor and module vendors to say “our products are supported in the mainstream kernel and OpenEmbedded sources”.  Compulab is doing it right.

OpenEmbedded provides many interesting options for devices like the EM-X270.  One of the most interesting might be the ability to run the OpenMoko software on industrial devices.

Compulab provides a Linux demo image built with OpenEmbedded that runs the GPE Palmtop Environment.  The method for loading this image is interesting, and similar to the method I have been using for some time with systems based on the cm-x270 module ( in that it used mtd-utils from a running Linux system to erase and write to the NAND flash.  The bootloader in the EM-X270 is able to boot a kernel and rootfs directly from a USB Flash disk.  This “LiveDisk” image, as Compulab calls it, contains a small utility that programs an image from the USB flash disk directly into NAND flash.

As with any modern Linux based embedded system, there is a lot going on in a very small package.  Managing this complexity effectively requires some amount of experience.  Stay tuned for future articles that discuss using OpenEmbedded to build software for the EM-X270.

Additional Photos





The Embedded WiFi module Quest

original article written in Sept, 2007

How does one implement WiFi functionality in vertical, low volume portable products?  This is a good question, and one I’ve been struggling with for the past 3 months.  I have a customer who is designing a portable data acquisition system based on a AT91SAM9260 processor, and needs WiFi functionality.  The fundamental problem is that no one has time to deal with low volume customers, and the task of implementing WiFi functionality is obviously complex.  At volumes of 5000 units a year or less, it makes a lot of sense to go with a WiFi module rather than trying to integrate the WiFi chipset on the board.  This article covers some of the options we have looked at and some of the possible solutions.

How is WiFi done on x86 systems?

With desktop Windows, most devices are well supported by device manufacturers.  With x86 Linux some of the devices are well supported by OSS drivers, but these tend to be PCI or USB devices.  Some of the devices that are not supported can still be used by running the Windows NDIS drivers inside of ndiswrapper, which allows you to run the Windows driver on a x86 Linux system, but this obviously does not work very well on non-x86 processors such as ARM.

WiFi solutions for portable, embedded systems

While there are many off-the-shelf solutions for PCs, Notebooks, etc, the current solutions are generally in the USB formfactor.  The ideal solution for portable systems running non-x86 processors are the small modules that connect to the host processor using the SDIO or SPI bus — like the modules you would find in newer cell phones and PDAs.  Older modules tend to use the Compact Flash interface, but these modules tend to be larger and are not packaged as nicely for deeply embedded applications.  So Ideally we want:

  • a modules with a SDIO or SPI interface
  • packaging options such as solder down for robust packaging

The scope of this article is limited to modules that implement a SPI or SDIO interface.  Why are we interested in modules?  Putting a WiFi chip down on a custom PCB is a lot of work.  Because the WiFi silicon includes a processor, if things do not work, it may be very difficult to debug.  There is also the issue of factory calibration which usually requires proprietary PC based tools that can communicate directly with the WiFi silicon.  If the WiFi silicon is buried in your product, getting all this to work can be a challenge.  The test equipment required for calibration is expensive.  Add to this the difficulty in getting support from anyone who makes WiFi silicon …


It should be noted that WiFi modules are fairly complex devices.  They are typically based on a highly integrated IC that includes an ARM processor, Flash memory, and radio circuitry.  The fact that these modules include a significant amount of firmware on the module contributes significantly to the complexity of getting WiFi solutions to work properly.  Hopefully no bugs are encountered in the module firmware, because I can only imagine how difficult it would be to get these fixed for a small customer.   The fact that the module firmware is involved in many of the WiFi functions like authentication further complicates the problem with the abundance of authentication and encryption options available for WiFi.   The worst possible problem is the case where you must design an embedded WiFi system that must operate at the enterprise level in every environment.  It is one thing to design a system that works most of the time in most environments where WiFi is more of a convenience (like a Cell phone or PDA).  It is another thing to design an industrial grade system where it must work all the time and in all environments.

The Fundamental Problems with low volume embedded WiFi

I have never worked with a technology with so many dead ends as embedded WiFi modules.  Many of the companies I email or call never even bother to return emails and phone calls.  This is certainly true of the chipset manufactures.  From what I can gather, the fundamental problems are:

  • WiFi implementation is difficult; therefore, a manufacturer’s scarce resources are dedicated to high volume customers.  Low volume customers are a significant distraction.
  • WiFi companies are very busy right now — they are not hungry for business from small customers.
  • WiFi solutions are very competitive and highly proprietary.  Many companies are very secretive and will not release driver source code.  This is certainly true at the module firmware level.

There are many WiFi module manufacturers and resellers out there, but very few of them offer any type of driver solution for Windows CE or Linux other than binary modules that are supposed to work on perhaps one reference platform.  Sorry, this is not going to cut it.  So, with plenty of hardware available, the gating item is the availability of software drivers and knowledge of how to use the software.  It would seem to me that this is the perfect opportunity for chipset vendors to open source some drivers and develop vibrant support communities so that their modules can be used without a lot of hand-holding by the chipset vendors.

So What are the options?

Atheros AR6001

The Atheros AR6001 seems like a nice solution.  The driver situation for the AR6001 is progressing.  For Windows CE, there is an opensource WinCE 6.0 driver available at:  Unfortunately, this project appears to be a snapshot of code and does not have any significant amount of community activity or development.  Perhaps once more AR6001 modules are available, this will change.

For Linux, there are several options:

  • Atheros has released a driver that is available:  There are several issues with this driver in that it is written for a SDIO stack that will likely never be part of the mainstream kernel.
  • A driver is being developed as part of the OpenMoko project.

Embedded Works supplies Atheros based modules and development boards:    AR6001 development boards are also available from Cardaccess:


The Marvell 88W8385 seems to be a very popular IC in WiFi modules available from a number of different companies.  OSS Linux drivers are in progress and are reportedly somewhat functional at this point, so it is probably just a matter of time before these devices are well supported in Linux.  This chipset/driver is often referred to as “libertas”.  The SDIO stack being developed for the Linux mainline ( is where support for this device is being developed.

For Windows CE, it seems the only option is to get the drivers from Marvell, which is a very difficult and time consuming process — at least for low volume customers.  First you have to find a module reseller that can get you the source code from Marvell, and then the process takes about 3 months.

Other Options

There are a number of other WiFi Silicon manufacturers and module vendors, but the driver options from them seem to be very limited in the form of binary only drivers that will only work with certain processors/operating systems.  As noted before there is almost no hope of getting anyone to even talk with you if your volumes are low.  So your only hope is finding a module vendor that can do any driver work for a NRE fee, obtain the source from the silicon manufacturer, or use OSS drivers.


The Embedded WiFi situation is changing fast, so by the time you read this, it is probably already out of date.  We are currently at the phase where it is very difficult to deeply embedded WiFi in low volume products.  I expect this will change during the next year.  Eventually, WiFi modules re-sellers will figure out that supplying hardware is just one side of the equation.  As things get more and more complex, the software availability and support is becoming the gating item.

Many thanks to James Nahra, Dave Anders, and Erik Strack for sharing information and their WiFi experiences with me over the past few months.


The Embedded Industry Transformation

I just ran across an interesting article written by Doug Gaff titled “Is the embedded industry dead?“. The article points out the transformation that is happening in the “embedded” industry. For many systems, we are no longer constrained by minimal resources ( Embedded Systems are no longer isolated systems, but parts of larger systems.

A few of my own observations:

There is more to modern embedded systems than control and logic. We now have to worry about data. In the past the task might be to get a microcontroller to run a state machine, control motors, etc. Today, we need to present a calendar to the user through a web interface, collect information about the location of the Sun, upload statistics to a remote server, support a rich GUI application on a VGA color display. The control aspect is suddenly the small part of the problem, and managing data is often the primary concern.

Embedded system development now encompasses many disciplines including OS development, web services, advanced application development in high level languages, and the traditional hardware debugging and low level coding tasks. It is becoming less about writing code and more about integration of the right pieces. As Embedded Systems become more complex, access to specialized knowledge is becoming more important. There is just too much for one person (or even one company) to know. Companies developing competitive products in the future will be the ones who have access to experts.
Is your company taking advantage of this transformation?


Compulab cm-x270 PXA270 module review

We are in the process of supporting a customer who is designing a product that uses the Compulab cm-x270 “Computer-on-module”.  The cm-x270 is a small computer module 66x44x7mm that contains an Intel PXA270 ARM processor similar to those found in many PDAs or smart phones.  This review provides an overview of the cm-x270, why it was selected, and how we are using it.

Project Needs

For this project, we needed a low cost embedded computer that could drive a VGA (640×480) color display.  The graphical user interface for the device was fairly advanced, so we were also looking at a system that could run a modern OS with an advanced graphical toolkit.  After discussing the display requirements with the customer, we concluded that we needed a 32-bit processor with a LCD display controller.  As there was some custom circuitry needed, a configuration where the customer could design a custom baseboard and use an off-the-shelf computer module seemed optimal.  The schedule was also very aggressive and we did not have the time or budget to develop a full custom processor board.


After researching a number of computer modules, we chose the cm-x270 from Compulab.  The cm-x270 is a module that includes the following components:

  • Intel PXA270 ARM processor (up to 520Mhz)
  • 802.11b Interface
  • up to 128MB SDRAM
  • 128-512MB NAND flash
  • up to 4MB NOR flash
  • PCI
  • Serial Ports
  • Touch panel controller (UCB1400)
  • Host and Slave USB
  • and other features …

For more details visit the compulab web site (http:// ).  The cm-x270 provides a very impressive number of features in a very small space.

Below is a picture of the cm-x270 mounted on the Compulab SBC-X270 baseboard with a VGA display.  The system is running the GPE Palmtop Environment built with OpenEmbedded.


Low Cost

Perhaps the most impressive feature of the cm-x270 is the cost.  When ordering in volume, Compulab will build the module with only the components you need.  For prototyping, you can order a standard offering with all components populated.  Starting a price of $50, it is the lowest cost module we found.  The configuration we will be using costs around $98 in 2K quantities (64MB DRAM, 4MB NOR Flash, 512MHz + 128MB NAND Flash + Audio/Touchscreen + RTC).  Eventually, we may be able to reduce some of the features and get the price down around $70 per module.

At this price, the cm-x270 provides a very cost effective way to develop a low volume product (a few thousand per year) with advanced features provided by the PXA270 processor.


The bootloader and hardware design of the cm-x270 is proprietary.  Compulab does not supply schematics for the design or source code to the bootloader.  The documentation provided is adequate and provided all the information we needed to complete the design.  Compulab also provides schematics for their development baseboard, which is helpful.


The cm-x270 bootloader provides typical features found in an embedded processor bootloader, including: a serial console, NAND and NOR flash programming, download images from a tftp server, downloading images over a USB connection to a PC, and a number of other configuration commands.  The tftp download works very well and is fast.  The USB download is much slower, but is convenient for systems that do not have a network connection.  The USB download requires a windows PC with a specific version of Microsoft ActiveSync installed, which is unfortunate.  A more general solution that uses something like libusb ( instead of ActiveSync would be preferred as it could then run on both Windows and Linux computers.

Linux Support

Compulab provides a fairly complete port of the 2.6.16 Linux kernel for the cm-x270 in the form of a kernel patch.  Most of the peripherals we tested seem to work with the following exceptions:

  • The SD card driver is not preempt safe.  In this application, we need kernel preemption turned on to meet certain real-time requirements.
  • The Touch driver contains a small bug that does not allow applications using it to exit properly

Overall, support is pretty good and enabled us to get started with development quickly.

NAND Flash Driver

As the Compulab module only provides 4MB of NOR flash, using the NAND flash is a requirement when implementing a full featured Linux system with an advanced graphical user interface.  Compulab provides a binary module that is linked into a kernel driver to support the NAND flash.  This driver provides a block interface on which standard filesystems like ext2 can be used.  The NAND driver has worked fairly well, although we have seen several cases where we get I/O errors if we remove power to the device without shutting down the system properly.  The way to recover is to do a low level format the NAND flash.  This should not be an issue with this product as we will have a backup battery that will be used to power the system during shutdown if power is removed.  We may eventually move to an open source JFFS2 flash driver as time permits.

The proprietary NAND driver presents an additional wrinkle in that the driver cannot be linked into the kernel due to GPL licensing requirements.  One solution is to boot a small root file system from NOR flash that includes the NAND driver module, and then proceed to boot from NAND flash.

GNU/Linux Root File System

Compulab provides an example root file system (I think based on Debian) and suggests using scratchbox to build applications.  We looked at it briefly, but then decided to build our own rootfs using OpenEmbedded for the following reasons:

  • We can easily add additional packages from the extensive OpenEmbedded catalog as needed.
  • Allows us to have an rootfs independent of the module — we can more easily move to other processor modules or architectures in the future.
  • OpenEmbedded provides us with mechanisms for package management and building complete images that include our applications and tweaks to the OS.

For more information on OpenEmbedded, read our “Introduction to OpenEmbedded “.  With OpenEmbedded, we were able to quickly build a toolchain and a root file system that included the kdrive version of Xwindows and GTK+ libraries.  This rootfs fits in less than 32MB of flash disk space, which is much smaller than Compulab’s OS image.

We have also added machine configuration files to the OpenEmbedded project to support the cm-x270:

Design Support

Compulab seems mainly interested in customers who are interested in purchasing modules in volume.  They do not sell low cost development boards such as LogicPD’s $500 board.  Compulab requires you to purchase a fairly high cost development system ($1900) in order to get started.  Part of this cost is to pay for Compulab’s support costs.  For companies who do not have experience designing systems with PXA270 class processors, we highly recommend getting some up-front help from someone who has experience with these systems.  Even though the module takes care of a lot of the design complexity, it helps to have a detailed understanding of the PXA270 and associated components, as you still need to interact with them.  Consulting companies like BEC Systems can provide assistance in the form of hardware design reviews and Linux OS support to accelerate projects and help avoid costly mistakes and delays.  As Compulab designs only the module, there is only so much they can do when you run into problems — as shown in the next section.

Issues Encountered

Overall, the project using the cm-x270 has gone well.  The only real challenge we have faced is a latch-up issue with the ITE PCI controller on the cm-x270 module.  It appears that this part is very sensitive to the characteristics of the 3.3V supply to the module.  After a considerable amount of debugging and testing, we were able to solve this problem with some filtering and slowing down the rise time of the 3.3V supply.  This is a good example of where a project can get difficult very quickly when things don’t work as expected.


So far, the cm-x270 has worked very well in this project.  With its low cost, it is a very attractive option for companies wanting to implement advanced functionality in their products, but don’t have the time, sales volumes, or experience to justify a full processor board design.  Feel free to post comments about your experiences using embedded computer modules.


How to Implement a Web Application Framework in an Embedded Linux System

As devices are increasingly more networked, an embedded web server is becoming a standard way for users to interact with and configure an embedded device using a standard web browser. As an example, most pieces of networking equipment (such as routers and wireless access points) are configured this way. There are many ways to implement a web server. One way is to just write a monolithic program that handles the requests and outputs HTML using print statements. Without a lot of planning, you usually end up with something where logic is intermixed with presentation. The disadvantage of such an approach is that it quickly becomes very difficult to maintain your web application as it grows and changes. The current best practice with web application frameworks (such as Ruby on Rails) splits a web application into 3 distinct components (Model, View, Controller — MVC) so that changes to one component can be made with minimal impact to others. This article details a solution used in a recent project to implement such an architecture using the following open source components: SQLite, Clearsilver, and Python. The challenge was to find a solution that performed acceptably on a resource constrained 130MHz ARM-Linux system.

Components Used

The requirements for the web application framework for this system are:

  • Footprint must be fairly small — less than 10MiB.
  • Must enable us to implement a clean MVC type architecture.
  • Must support a high level language like python for rapid development.
  • Includes a database.
  • Reasonable performance — less than 1 second to render a typical page.

After researching and testing several options, the following components were chosen to implement the framework:

  • Web Server: Cherokee (1MiB)
  • Model: SQLite  (290KiB)
  • View: Clearsilver  (170KiB)
  • Controller: Python (2-3MiB)


Total size for the above components is about 4MiB.

The following sequence occurs during a typical web transaction:

  • An HTTP request is received by Cherokee
  • If the request matches the URI for the web application, Cherokee forwards the request to the web application framework via CGI.
  • Clearsilver parses the HTTP headers (including CGI GET and POST parameters) and provides the data to python in a easy to access HDF format.
  • Python looks at the URI and dispatches the request to the appropriate function.  Data is extracted from the SQLite database and an HDF datastructure is created.
  • Clearsilver is called to render the appropriate template.  Data from the HDF datastructure is used in the template to provide the dynamic content in the page.
  • The rendered HTML is passed back to Cherokee, and is then returned to the user’s web browser.

Each component is discussed in more detail in the following sections.


One of the issues with modern web frameworks is that most require a lot of processing power.  Many frameworks are written in an interpreted language which tend to be not very efficient on embedded systems such as the 130MHz ARM-Linux system used in this project.  As most of the application development for this project is done in Python, I tried several other Python based solutions with the following results:

  • Django: takes 10 seconds to render a page
  • webpy: takes 5 seconds to render a page
  • Clearsilver/Python: takes about 1 second to render a page

Anything over 1 second is very slow for a web interface.  With fastcgi, webpy could probably have been sped up to be acceptably fast, but I don’t think there is much hope for frameworks like django on this type of system.  I have read of similar experiences trying to run Ruby-on-Rails on ARM-Linux systems.  As I’m currently just using a basic CGI interface, the Clearsilver solution could also be sped up significantly as well, if the Python portion could be kept running between requests with fastcgi or a similar mechanism.

Cherokee Web Server

A web application typically consists of a web server (like Apache) and a program that provides the dynamic web content.  Some of the functions of a web server are:

  • receives HTTP requests
  • handles the request by serving static files/images or routes the request to a program that outputs dynamic content
  • authentication
  • encryption (SSL, TLS)

There are many web servers that can be used in embedded devices.  Some of them are:

Cherokee was chosen for this application because it provides a good balance between the size and functionality I am looking for.  It is also included in the Openembedded ( ) build system I am using.  Other systems will have different requirements — use the one that fits your application best.


Clearsilver ( ) is the real gem that was discovered during this exercise.   Clearsilver is a language-neutral HTML template system written in C.  It is used as the templating system for many high volume sites such as Google Groups 2.  Clearsilver also provides CGI handling functions and bindings to several languages including Python.  The fact that it is written in C and is fast for high volume sites also makes it fast enough on slower embedded systems.  Because Clearsilver is written in C, the templating system is already fast.  Parts or all of the application can also be moved to C as needed to get the required performance.  Having a performance upgrade path is nice.  Clearsilver also forces a strict separation of application logic and presentation templates, which keeps things in line with the MVC architecture.


SQLite ( ) is a small C library that implements a self-contained, embeddable, zero-configuration SQL database engine.  SQLite works as expected and also has bindings to a number of languages including Python.  A SQLite database is just a single file that requires no configuration, making it very easy to use.  The database engine runs in the same process as the application using it, which increases performance because there is no context switch when running database operations.   Because SQLite does not have its own process, it does not handle concurrency as well as other databases such as MySQL, but in an embedded system you seldom have more than one process using the database.  SQLite provides some support for concurrency with reader/writer locks for the entire database.  If your application is not real busy, this is often adequate if you need two processes to share some data.


Python is the glue that ties things together in this web application framework.  It sits between Clearsilver and the database.  The advantages of using a language like Python are rapid development and an extensive library.  Python is considerably slower than C, but there is always the option to move parts of the application into C as needed.

Openembedded Build System

How does one put together a system with all these components?  The answer is the Openembedded Build System (OE) (  OE includes support for all of the components discussed in this article.


The combination of Python, Clearsilver, and SQLite provides a compelling solution for a web application framework in an Embedded Linux System.  The solution is reasonably small, performs well, and provides a path to improve performance if needed.  We also get a clean architecture to create clean, maintainable web applications.

About the author — Cliff Brake owns BEC Systems, a consulting operation that helps customers utilize modern computer technologies in their products.  BEC offers a range of services to make your embedded project a success including technology selection, development, troubleshooting, and training.  Please visit our website ( ) for more information and other free resources.