Compulab cm-x270 kernel update to 2.6.24

The kernel support in OpenEmbedded for the Compulab cm-x270 has been updated to version 2.6.24.  The 2.6.24 kernel opens up several possibilities including better real time and high resolution timer functionality, an improved SD Card driver, and a full SDIO stack with support for SDIO Wifi devices.  Also included is a patch to set the default framebuffer depth to 16 for several commonly used displays.  Xfbev really does not like when you give it a 8-bit framebuffer when it is expecting 16, and typically returns a very unhelpful error message like: “error: Invalid argument”.


Marvell Embedded SDIO Wifi Success

As detailed in the article I wrote back in September of 2007 (, getting  embedded wifi modules functioning is not a simple task.  However, due to recent advances in the Linux kernel, it looks like a viable solution for low-mid volume products is emerging.  This article provides a few details on how to get a Zcomax (Zcom) XG-180MU module working with a PXA270 processor.

System Setup

The setup used includes:

  • Zcomax XG-180MU SDIO Wifi module
  • Compulab cm-x270 + sb-x270 system (includes the PXA270 processor)
  • Linux kernel 2.6.24-rc5
  • OpenEmbedded

Support for the Marvell 8385 and 8686 based Wifi chipsets is now included in the mainstream Linux kernel 2.6.24 release candidate source code.  To use this driver, enable the CONFIG_LIBERTAS_SDIO option.  I also had to enable the CONFIG_WIRELESS_EXT option to prevent a compile error, but this is probably required anyway to build a functioning wireless system.


The firmware for the Marvell SDIO wifi solutions is downloaded at runtime, so the next challenge is to find firmware files.  I am working on a project that has access to a driver from Marvell, so I was able to extract this firmware from their source code header files.  I’m sure there are other places where this firmware is available.   The firmware files must be placed in the following location:

root@cm-x270:~$ ls /lib/firmware/sd*
/lib/firmware/sd8385.bin         /lib/firmware/sd8385_helper.bin

The version of udev used in OpenEmbedded determines the location of the firmware files.  The Libertas Linux driver requests these firmware files, and a userspace component of udev provides these files to the kernel.


The following kernel modules are required:

root@cm-x270:~$ lsmod
Module                  Size  Used by
pxamci                  7008  0
libertas_sdio           8488  0
mmc_core               46868  2 pxamci,libertas_sdio
libertas               85288  1 libertas_sdio
ieee80211              29956  1 libertas
ieee80211_crypt         4768  1 ieee80211

And the result:

root@cm-x270:~$ modprobe pxamci
root@cm-x270:~$ mmc0: new SDIO card at address 0001
libertas: eth1: Marvell WLAN 802.11 adapter
ifconfig -a
eth1      Link encap:Ethernet  HWaddr 00:60:B3:34:77:66
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Very nice!

So it looks like there is finally an embedded Wifi solution for products that sell less than 500,000 units per year.  I also briefly tried this on a AT91SAM9260 system, but it did not work.  So there is still some work to do on the Linux SDHC driver for the AT91SAM9260.


Linux 2.6.23 for the Compulab cm-x270

Updated 2.6.23 kernel patches for the Compulab cm-x270 are now available in OpenEmbedded:

The cm-x270 support in OpenEmbedded has undergone a lot of clean-up in recent weeks.  Now is a good time to give OpenEmbedded a try if you need a full featured Linux distribution for the cm-x270.


How to implement realtime periodic tasks in Linux applications

Have you ever wondered what is the best way to implement periodic tasks in Linux applications — something better than usleep()?  This article covers a number of issues related to this subject including real-time tasks, the different timers available, timer resolution, and how to implement periodic tasks accurately so that error is not accumulated.  The recent inclusion of the high-resolution timers in the mainstream kernel makes this a very interesting subject.

High Resolution Timers

Historically Linux has done all timing off the OS timer tick which is usually between 1ms and 10ms.  While this is adequate for many tasks, it is really nice to have a high resolution timer for timing tasks.  With the integration of the high resolution timers into the mainstream Linux source tree, this is now possible.  From a user space perspective, there are no API changes.  The only difference you will notice is that now you can sleep for less than OS timer tick period.  clock_getres() can be used to check the timer resolution and will tell you instantly if you have high resolution timer support.  If the clock resolution is 1ns, you have high res timer support.  Realistically, you can’t delay for 1ns in a Linux application, but delays in the range of 100us should be possible, and depending on the configuration, much better performance is possible.  The kernel config entry for high resolution timers is CONFIG_HIGH_RES_TIMERS.

The difference between CLOCK_REALTIME and CLOCK_MONOTONIC

Some of the Linux timer functions take a clockid_t argument that can be specified as CLOCK_REALTIME, or CLOCK_MONOTONIC.  The big difference between these two is that changing the system time will have an affect on CLOCK_REALTIME, thus affecting your timers.  Changing the system time will have no affect on CLOCK_MONOTONIC — it will always count upward.  For periodic tasks, CLOCK_MONOTONIC may be more applicable.  The best way to get burned using CLOCK_REALTIME is when your application takes a timestamp, does something, takes another time stamp and then compares them.  If you are using CLOCK_REALTIME, and the system time gets changed between the two timestamps, your comparison will not be valid.  For most timeouts and relative timekeeping in Linux, use CLOCK_MONOTONIC.  This issue becomes more important as many systems now have a process that periodically sets the time automatically from network time servers, and you have no idea when this might happen.


Kernel preemption ( ) makes a huge difference in the performance of real time applications by allowing the kernel to be preempted by higher priority application processes.  Historically, any kernel code that was runnable ran before the kernel returned control to applications.  This all changes with kernel preemption.  Kernel preemption has been available in mainline kernels for some time now (I think since 2.6.16).  The worst offender I’ve found for locking the kernel for long periods of time have been flash drivers — especially proprietary ones.  But, even jffs2 can cause problems in realtime applications without kernel preemption.

The PREEMPT_RT patch

Much of the realtime work being done for Linux is maintained in the PREEMPT_RT patch.  Bits of this patch have already been merged into the mainline kernel, but there is still a lot of very useful functionality in the patch and it should be considered if you are doing any type of realtime work.  More details will be presented in a future article.

Absolute vs Relative timekeeping

When implementing a periodic task, you really want to base your timekeeping off an absolute time, versus relative delays such as usleep().  It is not possible to achieve precise periodic activation with a relative sleep such as usleep().  The reason for this is you must first get the current time, make a calculation to determine how long to sleep, and then call the relative sleep function.  If your process gets preempted between the time you acquire the timestamp, and the sleep, your relative sleep time will probably be wrong.  This problem is solved by using the clock_nanosleep() function.  clock_nanosleep() can be called with the TIMER_ABSTIME value in the flags argument.  If  the TIMER_ABSTIME flag is set, then clock_nanosleep() will sleep until the absolute timer value is reached.  It does not matter if you get preempted between the time you take a timestamp, and the sleep function.


In summary, if you want to implement accurate, realtime, periodic tasks in a Linux application, consider the following:

  • Use high resolution timers.  Although Linux is still limited in its response time, at least the scheduling resolution is now quite high.
  • Enable kernel preemption.  This makes sure long kernel processes don’t get in the way of your real-time application process.
  • Use the CLOCK_MONOTONIC for relative timekeeping.  You don’t want your application to lock up due to the system time changing.
  • use the clock_nanosleep() function instead of relative delays like usleep().  This is the only way to accurately schedule periodic tasks.
  • if needed, apply the PREEMPT_RT patch.

The clock_nanosleep man page also includes a lot of useful information about timer functions.


Linux 2.6.22 for the Compulab cm-x270

I just completed porting the cm-x270 Linux kernel patches to the 2.6.22 kernel.  Kernel build is available in OpenEmbedded.  Patches are available in the OE tree:

Shortly I will be making 2.6.22 the default kernel for the cm-x270 in OpenEmbedded and posting instructions on how to use JFFS2 with the NAND flash on the cm-x270.


Is Linux ready for real-time applications?

Have you ever wondered about the real-time response of Linux and how well it performs?  I recently developed a system where I had the requirement to send a packet of data out a serial port every 30ms.  This article describes how stock Linux can be used to accomplish this and solutions to several problems I encountered.  There are several concepts that need to be understood such as how to set a Linux process to real-time,  and kernel preemption.

All of the work referenced in this article was done on a Compulab cm-x270 system which is based on the PXA270 ARM processor.  Before utilizing any real-time optimizations, the packets sent out the serial port where anything but consistent.  There was a significant amount of jitter in the packet timing and occasionally we would see delays of up to several seconds.  Obviously this was not good enough.

Real Time Process Priority

The first step was to partition the application into several threads, and give the serial communication thread a priority of real time.  Linux supports several scheduling policies:

  • SCHED_OTHER: default universal time-sharing scheduler policy used by most processes.
  • SCHED_BATCH: intended for “batch” style execution
  • SCHED_FIFO: First In-First Out scheduling. Can only be used with static priorities higher than 0.  A SCHED_FIFO process will always run before a SCHED_OTHER or SCHED_BATCH process.  A SCHED_FIFO process runs until it is blocked by an I/O request, or is preempted by a higher priority process.
  • SCHED_RR: Round Robin scheduling.  SCHED_RR is similar to SCHED_FIFO, except the process will only run for a maximum time quantum if there is a process of equal priority waiting to be run.

Practically, in a simple system there is really not much difference between SCHED_FIFO and SCHED_RR as you will typically only have at most a couple real time processes and you will probably assign them different priorities.

The sched_setscheduler() system call can be used to set a process or thread priority — see the Linux man pages for more details.

After setting the communication thread to real time, there was a noticeable improvement in the packet scheduling, but there was still way too much jitter and the occasional several seconds of silence with no packets.

Preemptible Kernel

After some experimentation, and reading, it was suspected that the NAND flash driver in the system was locking the kernel for long periods of time.  With older Linux kernels, the kernel was not preemptible.  Any process running in the kernel had to finish before something else could run.  Recent 2.6 Linux kernel include an option to enable Kernel Preemption (CONFIG_PREEMPT).   This allows a real-time process to preempt the system even if it is running in kernel space.   After enabling the CONFIG_PREEMPT option in the kernel, the scheduling of packets was very consistent most of the time with almost no perceptible jitter.

Be careful what you put in your real-time thread

Now the system was running fairly well, but we still noticed at system startup that there were still some long delays between packets.  It turns out the application was writing log messages to a file on NAND flash in the real-time thread using the standard glib logging functions.  At system startup, the NAND driver blocks for long periods of time (up to several seconds).  After the system has been running for a short time, this went away.  The solution in this case was to send any logging messages from the communication thread to a separate normal priority logging thread.  This fixed any remaining issues and the packet scheduling is now rock solid all the time.  The leads to an fundamental real-time concept which seems very obvious in hindsight:

Do not access any I/O (such as storage or network interfaces) that may block for long periods of time in a real time process.

Although we had designed all the primary I/O to be handled this way, we had not given any consideration to log file writes, which are an important part of any program.

The glib library provides several very nice mechanisms for sending data between threads.

Linux Real-time

With improvements in the 2.6 Linux kernel, Linux works very well for some real-time applications.  While I would still be very careful in using Linux for critical, hard-real time requirements due to its overall complexity, Linux works very well for multi-media, communication, and data acquisition systems.


Do you need “software update” functionality in your Embedded Linux system?

In this day and age, most embedded systems include a way for users to easily update software once the device has been deployed.  This article discusses the requirements for a field update mechanism along with pointers for how to implement.

Update Mechanism Requirements

Requirements for a field update mechanism might be:

  • easy for users to perform updates.
  • little chance of “bricking” a unit if reset occurs during update.
  • ability to update all of the software in the unit
  • in this case, update from files on a USB Storage device
  • ability to program NOR and NAND flash

Other systems may have requirements to update from a network, Compact Flash card, etc.

The USB wrinkle

System update is typically done by the bootloader.  In this case, we are using a Compulab cm-x270 module ( that has a proprietary boot loader and does not support updates from a USB storage device.  Therefore, we need to run the updates from the context of the Linux operating system which has drivers for USB and the NAND flash.  We decided that the best way in this case is to have a small Linux OS image that contains enough functionality to update the main filesystem.  This also provides us with the capability to recover if the main filesystem ever gets corrupted.  The flash layout for the system is:

Partition Flash Device Size File System Description
Kernel NOR 1536KiB NA Linux kernel
Update rootfs NOR 2304KiB Initramfs (RO) Small rootfs that contains just enough functionality to update the system
Main rootfs NAND 512MiB JFFS2 (RW) Main rootfs including application

There are other schemes that could work as well.  One way might be to have two identical application partitions and only update one at a time.  A reset while updating the kernel or the update rootfs does have the potential of “bricking” the system, but most of the updates will be for the Main rootfs, so we think this risk is low enough to be acceptable in this system.  The update rootfs is loaded into RAM before it is used, so there is little chance of it being corrupted during normal operation.  Running the update rootfs from ram is also convenient in that we can easily re-program the update rootfs in flash as we are not directly mounting the filesystem in flash.

Building the Update rootfs

The base update rootfs is built with OpenEmbedded as described in  Once we had the system booting into the update rootfs, we had to add the following functionality:

  • Boot the main rootfs from NAND flash
  • look for update files on a USB Storage device and flash the kernel, update rootfs, and main rootfs as needed.

Booting the NAND flash image is accomplished using the busybox switch_root application.  This utility allows you to easily switch to another rootfs from an initramfs.  There are a few gotchas to be aware of when working with switch_root:

  • switch_root must be “exec’d” from the init process with PID=”1″.
  • switch_root requires a “/init” file to be present

The above functionality was accomplished by writing a shell script called /init.  If an error occurs, the /init shell script then launches the standard init process in /sbin/init which provides a terminal and allows for easy debugging.  The mtd-utils package provides a handy utility called nandwrite which can be used to write jffs2 images to NAND flash (handles bad blocks, etc).  At some point, we may remove all the extra functionality from the update rootfs like the standard sysvinit, terminal login, etc, but for now it is handy to have for debugging.  To give you an idea how easy it is to write to flash from a shell script, consider the following snippets:

# write Image into update rootfs parition
cat /usb/nor_rootfs.img > /dev/mtd2
# write Image to NAND rootfs partition
flash_eraseall /dev/mtd3
nandwrite /dev/mtd3 /usb/nand_jffs2.img
# and to launch rootfs in nand flash
exec switch_root -c /dev/console /jffs2 /sbin/init 5


Having typically used bootloaders for system update in the past, I’m very pleased with how this mechanism worked out.  Thanks to the flexibility of Linux and OpenEmbedded, this only took several days to implement and debug.  Writing update programs with shell scripts at the application level gives you a lot of flexibility and allows you to use standard Linux drivers for USB and Flash access which are very robust.  At some point if there is interest, we may look at cleaning up our code and contributing functionality to OpenEmbedded to generate a more generic update initramfs.  Please contact us or leave a comment if you are interested in collaborating on something like this.


The Linux kernel container_of macro

The Linux kernel contains a very useful macro named “container_of” that is used extensively in back-casting a data structure to its containing data structure.  This article includes a simple program that illustrates how this macro is used, and explains why it is so useful.

If you do a lot of C programming, this program is worth figuring out :-).

The program source can also be downloaded from the following location:

/* test code to illustrate use of Linux kernel container_of macro
 * Copyright (c) 2008 Cliff Brake, BEC Systems LLC
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License version 2 as
 * published by the Free Software Foundation.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

/* This program illustrates how the container_of macro works.
 * The container of macro is very useful in multi layered
 * software systems where you have progressivly more detailed
 * software layers.  Below is an example of a bus layer,
 * and then a device layer where a number of different
 * devices might register with the bus.
 * The device registers itself with the bus subsystem, and
 * then the bus subsystem makes a callback into the device.
 * Normally if there are multiple devices registered, the
 * bus subsystem must store and pass a device structure
 * when making callbacks.  With the container_of macro, this is
 * no longer necessary, and the bus subsystem only has to
 * know about one generic device structure, and does not need visibility
 * into lots of different device structures, or do tricks
 * by casting void pointers, etc.  With the container_of macro
 * we can backcast from the generic data structure, to the containing
 * datastructure.  This forces good separation of code in that
 * that bus layer cannot modifiy data structures that are specific
 * to the device layer.

 * (from Linux kernel source)
 * container_of - cast a member of a structure out to the containing structure
 * @ptr:	the pointer to the member.
 * @type:	the type of the container struct this is embedded in.
 * @member:	the name of the member within the struct.
#define container_of(ptr, type, member) ({			\
	const typeof( ((type *)0)->member ) *__mptr = (ptr);	\
	(type *)( (char *)__mptr - offsetof(type,member) );})

#define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER)

 * BUS layer code

/* generic bus device structure */
struct bus_device
	int general_device_param_a;
	int general_device_param_b;
	void (*device_callback)(struct bus_device * bd);

/* the following is a global list of
 * devices that have registered with the
 * bus subsystem.  Normally this would
 * be something like a dynamic linked list.
struct bus_device * bd_list[5];

/* function to register a device with the bus */
void register_with_bus(struct bus_device * bd)
	/* since this example only deals with one
	 * device, will put it in slot 0

	bd_list[0] = bd;

void start_bus()
	int i;
	struct bus_device * bd;

	/* make callbacks to all devices on bus */
	for (i=0;i<sizeof(bd_list)/sizeof(bd_list[0]);i++) {
		bd = bd_list[i];
		if (!bd) continue;
		/* call device callback with generic
		 * bus device structure

 * device X specific code
 * this would normally be in a different module

/* structure that holds device X specific stuff, as well as
 * generic bus_device structure
struct device_x
	int device_x_specific_param_a;
	int device_x_specific_param_b;
	struct bus_device bd;

void device_x_callback(struct bus_device * bd)
	/* if we know the structure type that contains the bus_device structure,
	 * we can extract a pointer to the containing structure using the container_of
	 * macro

	/*                                   ptr       type       member  */
	struct device_x * devx = container_of(bd, struct device_x, bd);

	/* the above statement expands to
	 * struct device_x * devx = (
	 * {
	 *   const typeof( ((struct device_x *)0)->bd ) *__mptr = (bd);
	 *   (struct device_x *)( (char *)__mptr - ((size_t) &((struct device_x *)0)->bd) );
	 * }
	 * );

	printf("device_x_callback called!, device_x_specific_param_a = %i\n",

void device_x_init()
	/* dynamically allocate structures */
	struct device_x * devx = malloc(sizeof(*devx));
	memset(devx, 0, sizeof(*devx));

	/* set a parameter in the device_x structure so
	 * we can test for this in the callback
	devx->device_x_specific_param_a = 1001;

	/* set up callback function */
	devx->bd.device_callback = device_x_callback;

	/* we register the generic bus device structure
	 * as the bus layer does not need to know
	 * about the device_x stucture.  Note, the
	 * devx structure is not stored anywhere, yet
	 * its location is being preserved without
	 * specifically passing it to the bus
	 * layer.

int main()

	/* test the above system */

	/* first, initialize device_x */

	/* now, start the bus.  This should make
	 * a callback into the device_x

/* when run, this program returns:
 * device_x_callback called!, device_x_specific_param_a = 1001

GESBC-9302E kernel update to 2.6.24, and reboot fix

I recently updated the GESBC-9302 machine support in OpenEmbedded to include the 2.6.24 released kernel.  Also, Glomation has kindly provided me with a patch for the software reboot problem which I have integrated into the OpenEmbedded build for the GESBC.   The patch files are part of the OpenEmbedded dev branch or can be obtained here:

Other related OpenEmbedded bits:

Future features to be integrated include RTC driver configuration, and support for loading Initramfs images from NOR flash.