Have you ever wondered about the real-time response of Linux and how well it performs? I recently developed a system where I had the requirement to send a packet of data out a serial port every 30ms. This article describes how stock Linux can be used to accomplish this and solutions to several problems I encountered. There are several concepts that need to be understood such as how to set a Linux process to real-time, and kernel preemption.
All of the work referenced in this article was done on a Compulab cm-x270 system which is based on the PXA270 ARM processor. Before utilizing any real-time optimizations, the packets sent out the serial port where anything but consistent. There was a significant amount of jitter in the packet timing and occasionally we would see delays of up to several seconds. Obviously this was not good enough.
Real Time Process Priority
The first step was to partition the application into several threads, and give the serial communication thread a priority of real time. Linux supports several scheduling policies:
- SCHED_OTHER: default universal time-sharing scheduler policy used by most processes.
- SCHED_BATCH: intended for “batch” style execution
- SCHED_FIFO: First In-First Out scheduling. Can only be used with static priorities higher than 0. A SCHED_FIFO process will always run before a SCHED_OTHER or SCHED_BATCH process. A SCHED_FIFO process runs until it is blocked by an I/O request, or is preempted by a higher priority process.
- SCHED_RR: Round Robin scheduling. SCHED_RR is similar to SCHED_FIFO, except the process will only run for a maximum time quantum if there is a process of equal priority waiting to be run.
Practically, in a simple system there is really not much difference between SCHED_FIFO and SCHED_RR as you will typically only have at most a couple real time processes and you will probably assign them different priorities.
The sched_setscheduler() system call can be used to set a process or thread priority — see the Linux man pages for more details.
After setting the communication thread to real time, there was a noticeable improvement in the packet scheduling, but there was still way too much jitter and the occasional several seconds of silence with no packets.
Preemptible Kernel
After some experimentation, and reading, it was suspected that the NAND flash driver in the system was locking the kernel for long periods of time. With older Linux kernels, the kernel was not preemptible. Any process running in the kernel had to finish before something else could run. Recent 2.6 Linux kernel include an option to enable Kernel Preemption (CONFIG_PREEMPT). This allows a real-time process to preempt the system even if it is running in kernel space. After enabling the CONFIG_PREEMPT option in the kernel, the scheduling of packets was very consistent most of the time with almost no perceptible jitter.
Be careful what you put in your real-time thread
Now the system was running fairly well, but we still noticed at system startup that there were still some long delays between packets. It turns out the application was writing log messages to a file on NAND flash in the real-time thread using the standard glib logging functions. At system startup, the NAND driver blocks for long periods of time (up to several seconds). After the system has been running for a short time, this went away. The solution in this case was to send any logging messages from the communication thread to a separate normal priority logging thread. This fixed any remaining issues and the packet scheduling is now rock solid all the time. The leads to an fundamental real-time concept which seems very obvious in hindsight:
Do not access any I/O (such as storage or network interfaces) that may block for long periods of time in a real time process.
Although we had designed all the primary I/O to be handled this way, we had not given any consideration to log file writes, which are an important part of any program.
The glib library provides several very nice mechanisms for sending data between threads.
Linux Real-time
With improvements in the 2.6 Linux kernel, Linux works very well for some real-time applications. While I would still be very careful in using Linux for critical, hard-real time requirements due to its overall complexity, Linux works very well for multi-media, communication, and data acquisition systems.