Categories
Uncategorized

Understanding the NXP i.MX6UL Pin Mux (Part 2)

In the previous post, it was noted that bit 30 needs to be set in the i.MX6UL pad config if you want to read the state of a GPIO output. Digging into this a bit more, we find the following text in the Documentation/devicetree/bindings/pinctrl/fsl,imx-pinctrl.txt file:

SION(1 << 30): Software Input On Field.
Force the selected mux mode input path no matter of MUX_MODE functionality. By default the input path is determined by functionality of the selected mux mode (regular).

Categories
Uncategorized

A Linux Kernel Tracing Tutorial

The Linux kernel has a fairly extensive tracing infrastructure that is quite useful for debugging.  There are a number of things you can do with tracing, but the focus of this article will be the traditional printk type debugging we often end up doing to trace initialization issues with a driver.  The following links provide additional information on the linux kernel tracing infrastructure:

In this example, I am working on a new audio driver.  The typical experience with a new driver is that you install it and nothing happens because something is not registered correctly with the Linux driver model.  So, the first thing I do is start with with the platform_device_add() function in my drivers init function.  To observe the kernel activity around the kernel platform code, I can do the following:

cd /sys/kernel/debug/tracing/
echo 0 > tracing_on (keep trace from filling up until we set filter)
echo function_graph > current_tracer
echo platform* > set_ftrace_filter
echo 1 > tracing_on
cat trace_pipe (leave running in a different shell)
<insmod my driver>

After executing the above, we see the following.  For this example, trace_pipe is preferred because the trace is then emptied and only new information is shown.

0) + 30.518 us   |  platform_device_alloc();
0)               |  platform_device_add() {
0)   0.000 us    |    platform_uevent();
0) + 30.518 us   |  platform_uevent();
0)   0.000 us    |  platform_uevent();
0) + 30.518 us   |    platform_match();
0) + 30.518 us   |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();

...

0) + 30.518 us   |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();
0)   0.000 us    |    platform_match();
0) ! 3936.767 us |  }
0) + 30.518 us   |  platform_uevent();
0) + 30.518 us   |  platform_device_alloc();

From the above, I can conclude that the platform_match() is not succeeding, because I would expect some more activity.  At this point I chose to add a printk:

diff --git a/drivers/base/platform.c b/drivers/base/platform.c
index 7a24895..f9ce0c7 100644
--- a/drivers/base/platform.c
+++ b/drivers/base/platform.c
@@ -662,6 +662,8 @@ static int platform_match(struct device *dev, struct device_driver *drv)
        struct platform_device *pdev = to_platform_device(dev);
        struct platform_driver *pdrv = to_platform_driver(drv);

+       trace_printk("pdev->name = %s, drv->name = %s", pdev->name, drv->name);
+
        /* Attempt an OF style match first */
        if (of_driver_match_device(dev, drv))
                return 1;

Now, if I re-run the trace, I see the following:

 0)               |      /* pdev->name = soc_audio, drv->name = davinci_emac */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc_audio, drv->name = snd-soc-dummy */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc_audio, drv->name = soc-audio */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc_audio, drv->name = omap-pcm-audio */
 0)   0.000 us    |    }
 0) ! 4241.943 us |  } /* platform_device_add */

From the above, it looks like we have a simple mismatch between “soc_audio” and “soc-audio.”  Fixing this problem, and re-installing the module, we now have:

 0)               |    platform_match() {
 0)               |      /* pdev->name = soc-audio, drv->name = snd-soc-dummy */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc-audio, drv->name = soc-audio */
 0)   0.000 us    |    }
 0) + 91.553 us   |    platform_drv_probe();
 0) ! 4241.943 us |  } /* platform_device_add */

Now we can see that the names match, and the probe function is now being called.  At this point, we may want to turn on tracing of some additional functions to try to determine what is happening next.

echo "platform* snd* mydriver*" > set_ftrace_filter

And the result:

 0)               |      /* pdev->name = soc-audio, drv->name = snd-soc-dummy */
 0)   0.000 us    |    }
 0)               |    platform_match() {
 0)               |      /* pdev->name = soc-audio, drv->name = soc-audio */
 0) + 30.517 us   |    }
 0)               |    platform_drv_probe() {
 0)               |      snd_soc_register_card() {
 0) + 30.518 us   |        snd_soc_instantiate_cards();
 0) ! 17852.78 us |      }
 0) ! 17883.30 us |    }
 0) ! 22125.24 us |  } /* platform_device_add */

With the above additional information, we can continue to learn more about the flow through the kernel.

While all of the above could have been done with printk’s, it would have been more time consuming.  The kernel function tracing capabilities allow us to quickly get a high level view of the flow through the kernel without manually adding a bunch of printk statements.  The kernel tracing features are completely contained in the kernel without requiring additional user space utilities which makes it very convenient to use in embedded systems.  The low overhead is also important in resource constrained embedded systems.

Categories
Uncategorized

OMAP3 Resume Timing

One of the most common power management modes for ARM processors is the suspend mode.  In this mode, peripherals are shut down when possible, the SDRAM is put into self-refresh, and the CPU is placed in a low power mode.  A useful bit of information is to know how soon the system can respond to a resume (wake) event.

It turns out the answer to this question depends on where you want to do the work.  In the first test, I created a simple application that continuously toggled a GPIO.  This is an easy way to tell determine with a scope when the application was running.  I then measured the time between the wake event, and when this application signal started toggling.  It was a consistent 180ms.

For the next test I simply toggled a gpio in the omap3_pm_suspend() first thing after resume.  This tells me roughly how fast I can execute code in the kernel after a wake event.  This turned out to be 250uS.

So the good news is we can run kernel code very quickly after a resume event (250uS), but it currently takes a long time until user space processes start running again (180mS).  The 180ms can likely be reduced with some work, but this at least gives us a baseline.  180ms is fairly quick from a human perspective (seems pretty much instant), but for industrial devices that are responding to real-world events, then 180mS can be a long time.

Categories
Uncategorized

Linux kernel stats, and long term advantages

I just read an interesting interview with Greg Kroah Hartman.  According to Greg, we add 11,000 lines, remove 5500 lines, and modify 2200 lines every single day.  This rate of change is something that few organizations have the resources to match.  It is interesting that Google chose to use the Linux kernel in the Android project even though they implemented most other parts of the system.  Another interesting thing to consider is the consequences of the Linux kernel license, which requires many drivers to be released as GPL.  While initially, this may be viewed as a disadvantage, it leads to some interesting long-term implications.  The obvious effect is that most Linux driver code is maintained in the mainline kernel source.  This leads to other effects.  One is that things are allowed to change, and get better.  The USB stack in both Linux and Windows has been re-written several times, but the difference is that in Linux the old software does not need to be maintained, as most drivers are maintained in the kernel, and can simply be modified along with core changes.  Another advantage is that drivers tend to be much simpler as common code is merged.  As a result, a Linux driver tends to be about 1/3 the size of a driver in other operating systems.  Yes, getting driver code accepted in the mainline kernel can be difficult at times, but it is clearly the best long-term option.

Categories
Uncategorized

Linux PM: OMAP3 Suspend Support

This article provides an overview of the Linux kernel support for the suspend state in the TI OMAP3.  Power management has always been one of the more difficult parts of a system to get right.  The OMAP3 power management is quite extensive.  There are many levels of very granular control over the entire system.  Initially, we are going to focus on a simple power state where we want to put the CPU in a low power mode (basically not doing anything), but resume very quickly once an event occurs and continue running applications where they left off.  This is typically called “Suspend”.

Power States

We must first match up terminology between the Linux kernel and the OMAP documentation.  The Linux kernel supports several suspend power states:

  • PM_SUSPEND_ON: system is running
  • PM_SUSPEND_STANDBY: system is in a standby state.  This is typically implemented where the CPU is in a somewhat static state.  After all the peripherals have been shut down, and the memory put into self refresh, the CPU executes an instruction that puts the CPU into a low power state.  After the next wake-event occurs, the CPU resumes operating at the next instruction.  The CPU state is static in this case.
  • PM_SUSPEND_MEM: This goes one step beyond STANDBY in that the CPU is completely powered down and loses its state.  Any CPU state that needs to be saved is stored to SDRAM (which is in self refresh), or some other static memory.  When the system resumes, it has to boot through the reset vector.  The bootloader determines that the system was sleeping, and re-initializes the system, enables the SDRAM, restores whatever state is needed, and then jumps to a vector in the kernel to finish the resume sequence.  This mode is fairly difficult to implement as it involves the bootloader, and a lot of complex initialization code that is difficult to debug.
  • PM_SUSPEND_MAX: From browsing the kernel source code, it appears this state is used for “Off” or perhaps suspend to disk.

The OMAP3 documentation uses the following terms:

  • SLM: Static Leakage Management.  Includes “suspend” functionality.
  • Standby:  OMAP3x retains internal memory and logic.  (uses 7mW)
  • Device Off: System state is saved to external memory(0.590mW).  This could theoretically map to the PM_SUSPEND_MEM state.
  • WFI: Wait for Interrupt.  This is an ARM instruction that puts the CPU in the standby state.

Based on the source code in the Linux kernel, there is currently only support for the Standby state:

static int omap3_pm_enter(suspend_state_t unused)
{
	int ret = 0;

	switch (suspend_state) {
	case PM_SUSPEND_STANDBY:
	case PM_SUSPEND_MEM:
		ret = omap3_pm_suspend();
		break;
	default:
		ret = -EINVAL;
	}

	return ret;
}

What kernel branches/versions support PM

Kevin Hilman has been maintaining an OMAP3 power management branch at http://git.kernel.org/?p=linux/kernel/git/khilman/linux-omap-pm.git.  Bits are being merged upstream.  Some of the things that don’t seem to be merged yet are SmartReflex support, and a number of other details.  But, it seems that basic suspend/resume support is available in the upcoming 2.6.32 kernel.

More Details on the Standby Mechanism

Due to the granularity and complexity of the the OMAP3 power management, relevant pieces of documentation are scattered throughout the OMAP35x Technical Reference Manual.  The following were helpful:

  • Table 3-17: details the MPU subsystem operation power modes
  • Table 3-18: Power Mode Allowable Transitions.  Standby
    mode can enter from active mode only by executing the wait for interruption (WFI) instruction.
  • SDRC, “Self Refresh Management”.  The OMAP3 can automatically put the SDRAM in self refresh mode when the CPU enters the idle state.
  • MPU Power Mode Transistions: The ARM core initiates entering into standby via software only (CP15 – WFI).

The omap34xx_cpu_suspend function is called to put the CPU into standby mode.  The actual instruction that stops the CPU is the ARM WFI (wait for interrupt) instruction.  This is a little different than previous generations of ARM CPUs that used coprocessor registers.

Testing

With the 2.6.32-rcX kernels, suspend/resume appears to work at least from the console:

root@overo:~# echo mem > /sys/power/state
PM: Syncing filesystems ... done.
Freezing user space processes ... (elapsed 0.00 seconds) done.
Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done.
Suspending console(s) (use no_console_suspend to debug)

<after key press on serial console ...>

omapfb omapfb: timeout waiting for FRAME DONE
CLIFF: going into suspend
Wake up daisy chain activation failed.
CLIFF: returning from suspend
Successfully put all powerdomains to target state
Restarting tasks ...
done.
root@overo:~#

The CLIFF messages are debug statements placed in the low-level suspend code to make sure we were getting to that point.  The console is disabled early in the suspend process, so we don’t get the debug messages until after resume.  The no_console_suspend kernel option can be used to leave the console enabled during suspend, but that feature does not seem to work.

Summary/Future work

This covers the basics of Linux Suspend/Resume support for the OMAP3 CPU.  There are many more options and details such as configuring wake sources, dynamic power management while running, etc.

Categories
Uncategorized

Integrated CAN solutions for Linux

I just received an email notification from EMS (http://ems-wuensche.com/) that support for their CAN controllers is now in mainline Linux kernels.  The EMS PCI products are supported in 2.6.31, and the CPC-USB product will be supported in 2.6.32.  I’ve used various Linux CAN stacks in the past, but none were as well integrated as the SocketCAN solution that has been merged into the mainline Linux kernel.  Though I’ve not used the EMS products yet, the fact that support is in the mainline kernel source provides a good indication that they know how to do things correctly, and that the software is of good quality.

The Microchip MCP2515 is another option for interfacing CAN with a Linux system, but uses the SPI bus so it typically requires a custom hardware design.   The availability of a USB adapter provides an easy way for CAN to be used with about any existing Linux system including small embedded systems.  From looking at the kernel source code, it appears that EMS solutions use the SJA1000 CAN controller, which has been an industry standard controller for many years.

Message buffering is an important consideration when selecting a Linux CAN solution.  Often in a system, the Linux computer is responsible for data processing, user interface, and other CPU intensive operations.  The real-time control is often relegated to a separate microcontroller.  The CAN bus is then just a convenient and robust way to get data from the control part of the system to the data processing part.  The amount of hardware buffering in the CAN controller determines how quickly the system needs to respond to process the CAN data, so the CAN controller has space to receive new data.  The SJA1000 provides a 64-byte receive fifo, or roughly space for 4 CAN messages.  This provides a little more buffering than the 2 messages in the MCP2515.  Dealing with full bandwidth CAN data in a Linux system can be a challenge  with small amounts of buffering as you typically have to implement the Linux RT patch, and spend some time tuning it.  Another upcoming CAN solution that looks nice is the TI High End CAN Controller (HECC) found in the new Sitara CPUs.  This controller provides 32 hardware mailboxes which should provide even more buffering, which makes it even better suited for operating systems like Linux.  As with any technology, there are tradeoffs, so its nice to have options.

One area the EMS solution may be very useful is for prototyping and development.  The ideal development flow when developing an Embedded Linux product is to development as much as possible on a PC.  The EMS product allows you to easily implement SocketCAN functionality on a PC so you can talk to the rest of your system directly from a PC.  The final embedded solution may be based on a more cost effective solution like the MCP2515 or the TI HECC, but being able to interface to the rest of the system directly from a PC has the potential to improve development efficiency.  This is part of the “Big Win” with Embedded Linux where the same technology scales from the embedded system, to the PC, to the server.

Categories
Uncategorized

Best practices for kernel development with OpenEmbedded

A common question is how do you do kernel development with OpenEmbedded?  Typically, OpenEmbedded builds a kernel by checking the source out of a git repository, or by applying patch files to a released version of the kernel.  See the many recipes for examples.  This works very well for a Linux distribution build system, but is less than ideal for ongoing kernel development where you will have many iterations over a period of time. The OE work directory is considered transient, and should not be used for any ongoing development.  Fortunately, the kernel build system is self contained, so it is a very simple matter to reference the toolchain built by OpenEmbedded and work outside the OpenEmbedded build system.

Use Git

Trust me on this one, just use it!  It will make your life easier.  There is a learning curve to any powerful tool.  Another way to state this is if a tool does not do much, then it would be easy to learn.  So expect to spend a little time learning to use git.  With modern Linux kernel development, much of your time is spent integrating work that others are doing.  For example, if you are using the OMAP3 CPU, you might want to start with linux-omap branch, integrate some patches from the OE beagleboard build, merge the OMAP PM tree, and then merge some bits from the Linux-wireless tree.  This is the way embedded Linux development is done today.  Git makes it possible to have many parallel developments going at their own pace, including your own.  Even in your own development, you will find significant advantages to being able to quickly create and merge branches, revise commits, etc.  There are several articles about git on this site.

Importing an OE kernel into git

If you are starting with an existing OE kernel tree, then you typically need to set up an equivalent tree in your local git workspace.  Typically you just look at the recipe to see how to manually reconstruct it.  For example, lets consider the SRC_URI from the palm-omap1 recipe:

SRC_URI = "${KERNELORG_MIRROR}/pub/linux/kernel/v2.6/linux-2.6.22.tar.bz2 \
           http://www.muru.com/linux/omap/patches/patch-2.6.22-omap1.bz2;patch=1 \
	   file://defconfig"

To set this kernel up in git, we might do the following:

  1. git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
  2. cd linux-2.6
  3. git checkout -b my_branch v2.6.22
  4. wget http://www.muru.com/linux/omap/patches/patch-2.6.22-omap1.bz2
  5. bzip2 -d patch-2.6.22-omap1.bz2
  6. git am patch-2.6.22-omap1

If the git am fails, then you might need to add a subject and author line to the patch in the following form and then retry git am.

From: Name <name@company.com>
Subject: omap1 patch

Now when you do a git log, you will see the above patch applied.  git apply can also be used to apply patches, but I prefer git-am as it automatically handles new files and fills in the commit message and author.

Setting up the toolchain

Now that you have a kernel, you must set up a toolchain so you can build it.  I typically set up my kernel directory next to my OE build directory in my project space, so I can then source the following script to set up the toolchain:

CROSS_COMPILER_PATH=`cd ../oe/build/angstrom-2008.1/tmp/cross/armv7a/bin; pwd`
BUILD_ARCH=`uname -m`
OE_STAGING_PATH=`cd ../oe/build/angstrom-2008.1/tmp/staging/${BUILD_ARCH}-linux/usr/bin; pwd`
STAGING_KERNEL_DIR=`cd linux-2.6; pwd`
export PATH=$PATH:$CROSS_COMPILER_PATH:$OE_STAGING_PATH
export ARCH=arm
export CROSS_COMPILE=arm-angstrom-linux-gnueabi-

Adjust to whatever you are building for.  Now you can simply type make.  The reason the OE_STAGING_PATH is added to the PATH in the above example is so that the uboot mkimage utility can be used directly from the OE build if your build target is make uImage.

Installing modules

Sometimes you need to re-install all modules on the target because you changed kernel version, etc.  Typically OE packages up all modules and adds the one you specify to the rootfs.  Because we are building the kernel outside the OE directory, OE can no longer do this.  However, its still very easy to install modules from your kernel development directory:

rm -rf modules_install; INSTALL_MOD_PATH=modules_install make modules_install
rsync -av modules_install/lib/modules/2.6.22 root@192.168.1.115:/lib/modules/

The first command installs the modules to a temporary directory, and the second command rsync’s the modules to your target system.

Creating a kernel recipe with your changes

Once you get to the point where you want to create a recipe with your changes, you can easily export a patch, or series of patches using git diff or git format-patch.  Simply add the patch files to your kernel recipe SRC_URI.  You can also teach bitbake to fetch your kernel source directly from a git repository, but I’ve generally found the patch mechanism to be adequate and easier in most cases.  As an example, you can create one large patch with all your changes that can be applied on the 2.6.22 released kernel:

git diff HEAD..v2.6.22 > my_kernel_changes.patch

Other ways …

This is just one flow and there are many variations.  There is also a new mechanism in OpenEmbedded called srctree that allows you to use OpenEmbedded in an external source tree.  srctree has the potential to further streamline this type of development.

As with any task, you want to use the tools that make sense for the task.  OpenEmbedded makes sense for building Linux distributions, but it is not a kernel development tool.  Git and the kernel build system make sense for kernel development.

Categories
Uncategorized

Embedded Linux versus Windows CE

Occasionally I am asked how Embedded Linux compares with Windows CE.  I have spent the past 5 years doing mostly embedded Linux development, and the previous 5 years doing mostly WinCE development with a few exceptions, so my thoughts are no doubt a little biased toward what I understand best.  So take this with a grain of salt 🙂  In my experience, the choice is often made largely on perception and culture, rather than concrete data.  And, making a choice based on concrete data is difficult when you consider the complexity of a modern OS, all the issues associated with porting it to custom hardware, and unknown future requirements.  Even from an application perspective, things change over the life of a project.  Requirements come and go.  You find yourself doing things you never thought you would, especially if they are possible.  The ubiquitous USB and network ports open a lot of possibilities — for example adding Cell modem support or printer support. Flash based storage makes in-field software updates the standard mode of operation.  And in the end, each solution has its strengths and weaknesses — there is no magic bullet that is the best in all cases.

When considering Embedded Linux development, I often use the iceberg analogy; what you see going into a project is the part above the water.  These are the pieces your application interacts with, drivers you need to customize, the part you understand.  The other 90% is under water, and herein lies a great deal of variability.  Quality issues with drivers or not being able to find a driver for something you may want to support in the future can easily swamp known parts of the project.  There are very few people who have a lot of experience with both WinCE and Linux solutions, hence the tendency to go with what is comfortable (or what managers are comfortable with), or what we have experience with.  Below are thoughts on a number of aspects to consider:

SYSTEM SOFTWARE DEVELOPMENT

Questions in this realm include CPU support, driver quality, in field software updates, filesystem support, driver availability, etc.  One of the changes that has happened in the past two years, is CPU vendors are now porting Linux to their new chips as the first OS.  Before, the OS porting was typically done by Linux software companies such as MontaVista, or community efforts.  As a result, the Linux kernel now supports most mainstream embedded cpus with few additional patches.  This is radically different than the situation 5 years ago.  Because many people are using the same source code, issues get fixed, and often are contributed back to the mainstream source.  With WinCE, the BSP/driver support tends to be more of a reference implementation, and then OEM/users take it, fix any issues, and that is where the fixes tend to stay.

From a system perspective, it is very important to consider flexibility for future needs.  Just because it is not a requirement now does not mean it will not be a requirement in the future.  Obtaining driver support for a peripheral may be nearly impossible, or be too large an effort to make it practical.

Most people give very little thought to the build system, or never look much beyond the thought that “if there is a nice gui wrapped around the tool, it must be easy”.  OpenEmbedded is very popular way to build embedded Linux products, and has recently been endorsed as the technology base of MontaVista’s Linux 6 product, and is generally considered “hard to use” by new users.  While WinCE build tools look simpler on the surface (the 10% above water), you still have the problem of what happens when I need to customize something, implement complex features such as software updates, etc.  To build a production system with production grade features, you still need someone on your team who understands the OS and can work at the detail level of both the operating system, and the build system.  With either WinCE or Embedded Linux, this generally means companies either need to have experienced developers in house, or hire experts to do portions of the system software development.  System software development is not the same as application development, and is generally not something you want to take on with no experience unless you have a lot of time.  It is quite common for companies to hire expert help for the first couple projects, and then do follow-on projects in-house.  Another feature to consider is parallel build support.  With quad core workstations becoming the standard, is it a big deal that a full build can be done in 1.2 hours versus 8?  How flexible is the build system at pulling and building source code from various sources such as diverse revision control systems, etc.

Embedded processors are becoming increasingly complex.  It is no longer good enough to just have the cpu running.  If you consider the OMAP3 cpu family from TI, then you have to ask the following questions: are there libraries available for the 3D acceleration engine, and can I even get them without committing to millions of units per year?  Is there support for the DSP bridge?  What is the cost of all this?  On a recent project I was involved in, a basic WinCE BSP for the Atmel AT91SAM9260 cost $7000.  In terms of developer time, this is not much, but you have to also consider the on-going costs of maintenance, upgrading to new versions of the operating system, etc.

APPLICATION DEVELOPMENT

Both Embedded Linux and WinCE support a range of application libraries and programming languages.  C and C++ are well supported.  Most business type applications are moving to C# in the WinCE world.  Linux has Mono, which provides extensive support for .NET technologies and runs very well in embedded Linux systems.  There are numerous Java development environments available for Embedded Linux.  One area where you do run into differences is graphics libraries.  Generally the Microsoft graphical APIs are not well supported on Linux, so if you have a large application team that are die-hard windows GUI programmers, then perhaps WinCE makes sense.  However, there are many options for GUI toolkits that run on both Windows PCs and Embedded Linux devices.  Some examples include GTK+, Qt, wxWidgets, etc.  The Gimp is an example of a GTK+ application that runs on windows, plus there are many others.  The are C# bindings to GTK+ and Qt.  Another feature that seems to be coming on strong in the WinCE space is the Windows Communication Foundation (WCF).  But again, there are projects to bring WCF to Mono, depending what portions you need.  Embedded Linux support for scripting languages like Python is very good, and Python runs very well on 200MHz ARM processors.

There is often the perception that WinCE is realtime, and Linux is not.  Linux realtime support is decent in the stock kernels with the CONFIG_PREEMPT option, and real-time support is excellent with the addition of a relatively small real-time patch.  You can easily attain sub millisecond timing with Linux.  This is something that has changed in the past couple years with the merging of real-time functionality into the stock kernel.

DEVELOPMENT FLOW

In a productive environment, most advanced embedded applications are developed and debugged on a PC, not the target hardware.  Even in setups where remote debugging on a target system works well, debugging an application on a workstation works better.  So the fact that one solution has nice on-target debugging, where the other does not is not really relevant.  For data centric systems, it is common to have simulation modes where the application can be tested without connection to real I/O.  With both Linux and WinCE applications, application programing for an embedded device is similar to programming for a PC.  Embedded Linux takes this a step further.  Because embedded Linux technology is the same as desktop, and server Linux technology, almost everything developed for desktop/server (including system software) is available for embedded for free.  This means very complete driver support (see USB cell modem and printer examples above), robust file system support, memory management, etc.  The breadth of options for Linux is astounding, but some may consider this a negative point, and would prefer a more integrated solution like Windows CE where everything comes from one place.  There is a loss of flexibility, but in some cases, the tradeoff might be worth it.  For an example of the number of packages that can be build for Embedded Linux systems using Openembedded, see http://cgit.openembedded.org/cgit.cgi/openembedded/tree/recipes.

GUI TRENDS

It is important to consider trends for embedded devices with small displays being driven by Cell Phones (iPhone, Palm Pre, etc).  Standard GUI widgets that are common in desktop systems (dialog boxes, check boxes, pull down lists, etc) do not cut it for modern embedded systems.  So, it will be important to consider support for 3D effects, and widget libraries designed to be used by touch screen devices.  The Clutter library is an example of this.

REMOTE SUPPORT

Going back to the issue of debugging tools, most people stop at the scenario where the device is setting next to a workstation in the lab.  But what about when you need to troubleshoot a device that is being beta-tested half-way around the world?  That is where a command-line debugger like Gdb is an advantage, and not a disadvantage.  And how do you connect to the device if you don’t have support for cell modems in New Zealand, or an efficient connection mechanism like ssh for shell access and transferring files?

SUMMARY

Selecting any advanced technology is not a simple task, and is fairly difficult to do even with experience.  So it is important to be asking the right questions, and looking at the decision from many angles.  Hopefully this article can help in that.  For additional assistance, please do not hesitate to contact BEC Systems — we’re here to help.

Categories
Uncategorized

Compulab cm-x270 kernel updated to 2.6.29 in OE

The cm-x270 kernel support in OpenEmbedded has just been updated to version 2.6.29.

Categories
Uncategorized

Wi2Wi W2CBW003 Wifi/Bluetooth module review

The Wi2Wi W2CBW003 is a highly integrated module that provides both Wifi and Bluetooth radios for embedded designs. This module is ideal for embedded designs, as it provides a lot of functionality in a small package and includes standard interfaces like SPI, SDIO and serial that connect with most embedded CPUs. With the availability of modules like the W2CBW003 and standard drivers in the Linux kernel, including radio functionality in an embedded device is very doable, even for low volume products. Wi2Wi provides an evaluation board for the W2CBW003 with a SDIO connector, UART connector, and BT Audio Connectors. For this review, the eval board was connected to a Marvel PXA270 ARM processor, and evaluated with current Linux and associated software.

W2CBW003 Overview

The W2CBW003 module integrates both WiFi and Bluetooth functionality in a 12mm x 12mm x 1.6mm package. The WiFi portion is based on the Marvell 88W8686, and the Bluetooth on the CSR BC04. Both of these components are well supported by Open Source software. Some other features include:

  • Separate Antennas for WiFi and BT.
  • ROHS compliant
  • Single Supply at 3.3V
  • 802.11g support (54Mbps)
  • Both SPI and SDIO interfaces for WiFi
  • UART interface for BT
  • PCM audio interface for BT

Pictures of the module and the demo board are shown below.

img_1798_small

img_1800_small

Evaluation System Configuration

The test software included with the Wi2Wi eval board is for a Windows PC, is provided in binary format only, and was not used in this review. For this review, I plugged the W2CBW003 demo board into a Compulab cm-x270 board (PXA270 cpu) running a 2.6.29-rc7 Linux kernel and a recent OpenEmbedded Angstrom distribution. With the exception of the Marvell Wifi Firmware, all software in this setup is Open Source and is available in the Linux kernel, and as packages in the OpenEmbedded Project.

When booting the kernel, you will see the following messages:

mmc0: new SDIO card at address 0001
libertas_sdio mmc0:0001:1: firmware: requesting sd8686_helper.bin
libertas_sdio mmc0:0001:1: firmware: requesting sd8686.bin
libertas: 00:19:88:06:0b:2e, fw 9.70.3p25, cap 0x00000303
eth2 (libertas_sdio): not using net_device_ops yet
libertas: PREP_CMD: command 0x00a3 failed: 2
libertas: PREP_CMD: command 0x00a3 failed: 2
libertas: eth2: Marvell WLAN 802.11 adapter

The “command 0x00a3 failed” messages are harmless, and have to do with features that are not supported. After the system boots, you will now see a new ethX network device:

root@cm-x270:~# ifconfig -a
...
eth2      Link encap:Ethernet  HWaddr 00:19:88:06:0B:2E
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:65902 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1758 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13550002 (12.9 MiB)  TX bytes:251627 (245.7 KiB)

The “iwlist eth2 scanning” command will list available access points.

Connecting to Open WiFi Networks

A connection to an open wifi network can be accomplished by placing the following in /etc/network/interfaces:

/etc/network/interfaces:
iface eth1 inet dhcp
    wireless_mode managed
    wireless_essid any

And now execute “ifup eth2”:

root@cm-x270:~# ifup eth2
udhcpc (v1.13.2) started
run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1
Sending discover...
Sending select for 192.168.1.115...
Lease of 192.168.1.115 obtained, lease time 86400
run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1
adding dns 208.67.222.222
adding dns 208.67.220.220

root@cm-x270:~# iwlist eth2
iwlist: unknown command `eth2' (check 'iwlist --help').
root@cm-x270:~# iwconfig eth2
eth2      IEEE 802.11b/g  ESSID:"bec3"
          Mode:Managed  Frequency:2.437 GHz  Access Point: 00:18:39:C1:AD:4A
          Bit Rate:1 Mb/s   Tx-Power=13 dBm
          Retry short limit:8   RTS thr=2347 B   Fragment thr=2346 B
          Encryption key:off
          Power Management:off
          Link Quality=84/100  Signal level=-37 dBm  Noise level=-87 dBm
          Rx invalid nwid:0  Rx invalid crypt:14707457  Rx invalid frag:0
          Tx excessive retries:58  Invalid misc:3   Missed beacon:0

WPA Secured WiFi Networks

The OpenEmbedded console image includes the WPA Supplicant packages which is used to manage wireless connections to secured networks. To set up the system for WPA encryption, modify the following files:

/etc/network/interfaces:
iface eth2 inet dhcp
   wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
   wpa-driver wext
/etc/wpa_supplicant/wpa_supplicant.conf:
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
eapol_version=1
ap_scan=1
fast_reauth=1

network={
      ssid="bec"
      proto=WPA2
      key_mgmt=WPA-PSK
      pairwise=CCMP TKIP
      group=CCMP TKIP
      scan_ssid=1
      psk="ascii passphrase"
      priority=10
}

Then, execute “ifup eth2”, and you should see something like:

root@cm-x270:~# ifup eth2
WPA: Configuring Interface
udhcpc (v1.13.2) started
run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1
Sending discover...
Sending select for 192.168.1.115...
Lease of 192.168.1.115 obtained, lease time 86400
run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1
adding dns 208.67.222.222
adding dns 208.67.220.220
root@cm-x270:~# iwconfig eth2
eth2      IEEE 802.11b/g  ESSID:"bec"
          Mode:Managed  Frequency:2.412 GHz  Access Point: 00:40:10:10:00:03
          Bit Rate:1 Mb/s   Tx-Power=13 dBm
          Retry short limit:8   RTS thr=2347 B   Fragment thr=2346 B
          Encryption key:<too big>   Security mode:open
          Power Management:off
          Link Quality=64/100  Signal level=-68 dBm  Noise level=-89 dBm
          Rx invalid nwid:0  Rx invalid crypt:-1463809279  Rx invalid frag:0
          Tx excessive retries:22524  Invalid misc:3   Missed beacon:0

Other Observations

With the above networks, the bec access point was much further away than the bec3 AP, so you will notice the difference in link quality. “iwlist eth2 rate” can be used to list the current connection rate. When the network is idle, it sits at 1Mb/s. When downloading a large file, it will climb to 36 or 54Mb/s, depending on link quality.

Production Issues

The review demonstrates that it is fairly simple to set up a demo quality Embedded Linux system with WiFi. Some of the issues that would likely need addressed for a production system include link management, test software for certification, and power management.

There are several ways to programatically control WPA Supplicant including linking to the wpa supplicant control interface, or using DBus. There are several WiFi management applications available including Gnome NetworkManager (used in desktop systems), and connman which seems a little better suited for embedded systems.

Unless you use a completely pre-certified module + antenna solution, you will likely need to do some level of agency certification. As many products use a custom antenna, this is an important issue to consider and plan for. While Marvell provides test software and firmware, it will likely require some work, as their software is designed to be used with their in-house drivers rather than the libertas driver which is available with modern kernels.

Also, if you want to minimize the power usage, some work will be required to figure out the power modes supported, and how to implement/control them. With this module, the Wifi and BT portions run off the same crystal, so if you only want the BT active, you will need to actively power manage the Wifi portion to a low power state instead of completely disabling it.

Conclusion

TheW2CBW003 module is an attractive solution for products that need WiFi functionality. With the availability of modules like this, and mainstream open source software, the technology is available to about anyone, including low volume manufacturers. Standard interfaces such as SDIO make it possible to interface this module with about any modern ARM processor that can run Linux. Software support in the Linux kernel, wpa supplicant, and the Linux wireless tools provide the needed software support to implement a very complex system with relatively little effort.

Categories
Uncategorized

Socketcan CAN-bus drivers added to OpenEmbedded

I just added a recipe to OpenEmbedded to build the Socketcan kernel modules from the socketcan SVN.  So if you are using the latest OpenEmbedded metadata, you can:

  • bitbake socketcan-modules
  • scp <oedir>/build/angstrom-2008.1/tmp/deploy/glibc/ipk/<machine>/socketcan-modules_0.0+svnr917-r0_cm-x270.ipk  root@<target IP address>:

and then on the target system:

  • opkg install socketcan-modules_0.0+svnr917-r0_cm-x270.ipk
  • opkg files socketcan-modules
Package socketcan-modules (0.0+svnr917-r0) is installed on root and has the following files:
/lib/modules/2.6.27/extra/net/can/can-bcm.ko
/lib/modules/2.6.27/extra/drivers/net/can/sja1000/ems_pci.ko
/lib/modules/2.6.27/extra/net/can/can-raw.ko
/lib/modules/2.6.27/extra/drivers/net/can/sja1000/sja1000.ko
/lib/modules/2.6.27/extra/drivers/net/can/softing/softing.ko
/lib/modules/2.6.27/extra/drivers/net/can/sja1000/ems_pcmcia.ko
/lib/modules/2.6.27/extra/drivers/net/can/sja1000/sja1000_platform.ko
/lib/modules/2.6.27/extra/drivers/net/can/can-dev.ko
/lib/modules/2.6.27/extra/drivers/net/can/softing/softing_cs.ko
/lib/modules/2.6.27/extra/drivers/net/can/sja1000/pipcan.ko
/lib/modules/2.6.27/extra/drivers/net/can/mcp251x.ko
/lib/modules/2.6.27/extra/net/can/can.ko
/lib/modules/2.6.27/extra/net/can/can-isotp.ko
/lib/modules/2.6.27/extra/drivers/net/can/vcan.ko

Then to use the modules:

  • depmod
  • modprobe mcp251x
  • modprobe can-bcm

What is CAN?  Can stands for Controller-area network and is popular in industrial and automotive applications.  A convenient way to add CAN to your Embedded Linux system is with the Microchip MCP2515.  This device connects to a SPI bus which means it can be interfaced with a number of popular SOC’s such as the PXA270, OMAP3, 91SAM9xxx, etc.

More on Socketcan later …

BTW, this recipe really illustrates how easy it is to compile kernel modules outside a kernel in OpenEmbedded:

DESCRIPTION = "Socketcan kernel modules"
HOMEPAGE = "http://developer.berlios.de/projects/socketcan/"
SECTION = "kernel/modules"
LICENSE = "GPL"
DEPENDS = "virtual/kernel"
PV = "0.0+svnr${SRCREV}"

SRC_URI = "svn://svn.berlios.de/socketcan/trunk;module=kernel;proto=svn"

S = "${WORKDIR}/kernel/2.6"

inherit module

This recipe tells OE to download the source, cross-compile it against your target kernel build dir, and then package it for easy install on the device.  This is the way things should work — no messing around figuring out make options, kernel source paths, compiler env variables, etc.

Categories
Uncategorized

Fix for PXA270 MMC/SD Controller Write Corruption

During system verification for a customer, we noticed occasional failures when writing to a SD card using the Marvell PXA270 MMC/SD controller.  The failure is a 4KB block of data is shifted on byte, where the first byte is duplicated and the last byte is dropped. The test app that found this problem is available here.  This test app simulates an application that writes data to a SD card at a certain rate.  We found that typically in about 2000 – 10000 hours of simulated testing, we would see a failure.  Its very interesting that the field failure rates matched this rate as well.

This failure had all the signs of a hardware (likely DMA) problem, and sure enough there is a Marvell Eratta for this issue.  A patch has been developed and tested by several people and is available in this mail thread.

Categories
Uncategorized

How to implement an interrupt driven GPIO input in Linux

With Linux, some of the things that seem like they should be easy are not — at least at first glance.  For example, how do you read an interrupt driven GPIO input in a Linux application?  With simpler microcontroller systems, this is straightforward, but with a system like Linux, you have to navigate through several layers of software (and for very good reasons).  You can’t handle interrupts directly in a Linux application, so this means you need a kernel component involved.  This operation of reading a GPIO resembles a key press, so the Linux input subsystem might be a good place to start looking.  Once we take that route, we discover the gpio_keys driver.

Configuring the gpio_keys driver

The gpio_keys driver is configured with a few lines of code in your Linux board configuration file.  An example is below:

static struct gpio_keys_button svs_button_table[] = {
  { .code = KEY_RECORD,
    .gpio = PP_GPIO_MIC_EN,
    .active_low = 1,
    .desc = "MIC_EN",
    .type = EV_KEY,
    .wakeup = 0,
  },
};

In this application, we are reading button presses on a cell phone style headset.  Once the gpio_keys driver is configured, a new entry will show up in /dev/input/eventX.  An application can then do a blocking read on this device.

Reading the GPIO in an application

To read the GPIO, we simply do a blocking read on the new /dev/input/eventX device.  The read will block until there is a change in GPIO state.  An example is show below:

#define MIC_INPUT_DEV  "/dev/input/event0"

static gboolean mic_button_callback(GIOChannel *source, GIOCondition condition, gpointer data)
{
  struct input_event ev;
  int bytes_read;

  g_io_channel_read_chars(source, (gchar *)&ev, sizeof(ev), &bytes_read, NULL);

  if (bytes_read > 0) {
    if (bytes_read != sizeof(ev)) {
      s_debug(1, "warning, only read %i bytes from mic input");
      return TRUE;
    }
  } else {
    return TRUE;
  }

  if (ev.type != EV_SYN && ev.value == 1) {
    /* button pressed, do something ... */
  }

  return TRUE;
}

void mic_button_init()
{
  GIOChannel * micbutton = g_io_channel_new_file(MIC_INPUT_DEV, "r", NULL);

  if (micbutton == NULL) {
    s_debug(TRUE, "Error initializing mic button");
    return;
  }

  g_io_channel_set_encoding(micbutton, NULL, NULL);

  guint id = g_io_add_watch(micbutton, G_IO_IN, mic_button_callback, NULL);
}

The above example also shows how to incorporate the GPIO read into a GLib mainloop so that you don’t need to create a separate thread. ( As a side, GLib mainloop programming is worth learning!)  Using this method, reading a GPIO interrupt is easy and requires very few lines of code.  This is typical of complex systems like Linux — if you know how to do something, it is relatively easy, but getting started down the right path is sometimes the challenge.

Categories
Uncategorized

Linux Input Testing and Debugging

The Linux input layer has made a lot of progress in recent years.  When writing a new input driver (such as keyboard, trackball, etc), it is useful to be able to monitor input events using a test application.  This article describes two ways to accomplish this using kernel input debugging, and the evtest utility.

kernel input debugging

The Linux kernel includes some support for printing input events as debug messages.  To use, set the following in the kernel config:

CONFIG_INPUT_EVBUG=y

Because the print messages are at the KERN_DEBUG level (7), we typically need to change the debug level:

echo 8 > /proc/sys/kernel/printk

At this point, you will see messages like the following on the console:

evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 108, Value: 1
evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0
evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 1
evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 106, Value: 1
evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0
evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 0
evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0
evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 1
evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0
evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 1
evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0
evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 0
evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0
evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 1
evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0
evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 0
evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0

evtest

The linux-input project (http://linuxconsole.sourceforge.net/input/input.html) includes a neat little tool named evtest that is very handy for testing input devices.  If you are using OpenEmbedded, simply:

bitbake linux-input

If you are not using OpenEmbedded, you can glean enough details about how to get and build the this package from the OE recipe:

http://gitweb.openembedded.net/?p=org.openembedded.dev.git;a=blob;f=packages/linux-input/linux-input_cvs.bb

Install the resulting package on your device.   To use, run evtest with a /dev/input/eventx argument:

root@machine1:~$ evtest /dev/input/event0
Input driver version is 1.0.0
Input device ID: bus 0x19 vendor 0x1 product 0x1 version 0x100
Input device name: "gpio-keys"
Supported events:
  Event type 0 (Sync)
  Event type 1 (Key)
    Event code 103 (Up)
    Event code 105 (Left)
    Event code 106 (Right)
    Event code 108 (Down)
    Event code 116 (Power)
Testing ... (interrupt to exit)
Event: time 1009.433605, type 1 (Key), code 108 (Down), value 0
Event: time 1009.433737, -------------- Report Sync ------------
Event: time 1010.735605, type 1 (Key), code 108 (Down), value 1
Event: time 1010.735740, -------------- Report Sync ------------
Event: time 1011.075586, type 1 (Key), code 106 (Right), value 1
Event: time 1011.075720, -------------- Report Sync ------------
Event: time 1011.103379, type 1 (Key), code 108 (Down), value 0
Event: time 1011.103550, -------------- Report Sync ------------
Event: time 1011.333924, type 1 (Key), code 108 (Down), value 1
Event: time 1011.334058, -------------- Report Sync ------------
Event: time 1011.339880, type 1 (Key), code 106 (Right), value 0

Notice that evtest lists the keymap when it is initially run which is very handy, and then lists key events as they occur.  The key descriptions are also included with events so you don’t have to manually decode the keycodes.

Categories
Uncategorized

Should you be using monotonic timers?

In a previous article (http://bec-systems.com/web/content/view/78/9/ ), I covered some of the basics of Linux timers.  Any time you are doing any type of fixed time delay in a program, you should really be using monotonic times, so the delay will not be affected by system time changes.  In an effort to save cost, some embedded systems today do not have a battery backed up RTC, and instead get the time via GPS, NTP servers, or other clever means.  What this means is your applications had better be able to handle the system time changing as the system time may not be set until well after the unit boots.  This article describes how you can quickly test your system for timer problems.

There are two cases where delays may fail if you are using non-monotonic timers.  The first is if the time advances forward by a large amount.  Delays will expire immediately in this case.  The other case is if the time advances backward by a large amount.  Delays will never expire in this case.  To test for these situations, write a simple test application (sample included below) that rapidly changes the time and then test your application while the time is changing.

#include <stdio.h>
#include <time.h>

int main()
{
  time_t system_time;
  static int count = 0;

  printf("Starting test application...\n");
  while(1)
  {
    // Sleep for 0.01 seconds
    usleep(10 * 1000);

    system_time = time(NULL);
    system_time += 30;
    stime(&system_time);
    printf("Cycle %d - system date/time set to %s", ++count, ctime(&system_time));
  }
}