Categories
Uncategorized

A quick serial logging application

Recently when debugging a device connected to a rs485 bus, I needed a simple application to dump the raw data coming over the rs485 bus.  Minicom does all kinds of weird terminal stuff, plus it will not display binary data, so that was out.  While looking for serial analyzer programs for Linux, and pondering writing my own simple C application, it occurred to me to use pyserial.  With a few lines of code, I got exactly what I needed:

import serial
import sys
import struct

if len(sys.argv) != 3:
    print "Usage: serial-dump.py <serial port> <baud rate>"
    sys.exit(-1)

print "Dumping data on %s" % (sys.argv[1])

ser = serial.Serial(sys.argv[1], sys.argv[2], timeout=0.3)

while(1):
    buf = ser.read(500)
    if len(buf) > 0:
        print "Read %i bytes:" % (len(buf)),

        for i in buf:
            value = struct.unpack('B', i)[0]
            print "%02x" % (value),

        print ""

The full source is available here.  This could obviously be improved to do true blocking reads, automatically parse protocols, implement a gui, etc.  Ideas for efficiently reading data using pyserial is available in a previous post.  Another example of using a scripting language for quick tasks is http://bec-systems.com/site/226/a-really-nice-hex-calculator.

Categories
Uncategorized

How to Optimize Python reads for general data

The Python file object read() function acts a little differently than the standard read() found in C.  This article describes some of these differences and how to optimize reads for general continuous data streams such as reading data from a collection device through a pipe.

Python read()

The Python read function seems to be optimized for reading files and text oriented streams.  By default, a read() call will block until a EOF is encountered.  This is very handy for reading files from a disk.  You can just slurp them up with one read() statement.  If you pass a size parameter to read(), it will block until size bytes have been received.  This is less than ideal for reading a continuous data stream where some of thye data may be stuck until the size threshhold is reached.

Non-blocking read()

The way you get read to return with whatever data is available even if it is less than the read size parameter is to set the file object up in non-blocking mode.  This can be done using the fcntl module:

 		flags = fcntl.fcntl(fp, fcntl.F_GETFL)
            	fcntl.fcntl(fp, fcntl.F_SETFL, flags | os.O_NONBLOCK)

However, now the application does not block while waiting for data and spins using CPU resources.

select()

Enter the select call.  The Python select module does much the same thing as the C select() function.  In this case, it can be used to block waiting for data from a non-blocking file object with the added benefit of a timeout.  So the resulting code might look like:

 	fp = os.popen(<application that returns data to stdout>, 'r')

	flags = fcntl.fcntl(fp, fcntl.F_GETFL)
    	fcntl.fcntl(fp, fcntl.F_SETFL, flags | os.O_NONBLOCK)

	while 1:
		[i, o, e] = select.select([fp], [], [], 5)
		if i: s_ = fp.read(1000)
		else: s_ = ''

		if s_:
			logging.debug("received %i bytes of data, total = %i" % (len(s_), total))
			<do something with s_>

The above select statement blocks until data is available from the fp object or times out after 5 seconds.  Reading continuous data streams in Python is very possible, but usually requires the file object to be set in non-blocking mode and a select used to block while waiting for data.

Categories
Uncategorized

Tips for reading a serial data stream in Python

Interfacing with a RS232 serial device is a common task when using Python in embedded applications.  The easiest way to get python talking to serial ports is use the pyserial project found at  http://pyserial.sourceforge.net/.  This module works on most platforms and is straightforward to use (see examples on project web site). However, getting the read function in this module to operate in an optimal way takes a little study and thought.  This article investigates how the pyserial module works, possible issues you might encounter, and how to optimize serial reads.

We start out with several goals as to how we want the application to behave in relation to the serial port:

  • application must block while waiting for data.
  • for performance reasons, we want to read decent size chunks of data at a time if possible.  Python function calls are expensive, so performance will be best if we can read more than one byte at a time.
  • We want any data received returned in a timely fashion.

A key parameter in the pyserial Serial class is the timeout parameter.  This parameter is defined as:

 timeout=None,           #set a timeout value, None for waiting forever

The Serial class read function also accepts a size parameter that indicates how many characters should be read.  Below is the source for the read function on Posix systems (Linux, etc):

     def read(self, size=1):
        """Read size bytes from the serial port. If a timeout is set it may
           return less characters as requested. With no timeout it will block
           until the requested number of bytes is read."""
        if not self.fd: raise portNotOpenError
        read = ''
        inp = None
        if size > 0:
            while len(read) < size:
                #print "\tread(): size",size, "have", len(read)    #debug
                ready,_,_ = select.select([self.fd],[],[], self.timeout)
                if not ready:
                    break   #timeout
                buf = os.read(self.fd, size-len(read))
                read = read + buf
                if self.timeout >= 0 and not buf:
                    break  #early abort on timeout
        return read

The easy way to use this module is to simply set the timeout to None, and read size to 1.  This will return any data received immediately.  But, this setup is very inefficient when transferring large amounts of data due to the Python processing overhead.

To meet our goal of reading multi-byte blocks of data at a time, we need to pass the read function a size greater than 1.  However, if timeout is set to None, the read will block until size bytes have been read, which does not meet the goal of returning any data read in a timely fashion.  The solution then is to:

  • set the read size high enough to get good performance
  • set the timeout low enough so that any data received is returned in a reasonable timeframe, but yet the application spends most of its time blocked if there is no data.

As an example, a size of 1000 and a timeout of 1 second seems to perform well.  When used this way, the pyserial module performs well and returns all data read quickly.

Categories
Uncategorized

How to Implement a Web Application Framework in an Embedded Linux System

As devices are increasingly more networked, an embedded web server is becoming a standard way for users to interact with and configure an embedded device using a standard web browser. As an example, most pieces of networking equipment (such as routers and wireless access points) are configured this way. There are many ways to implement a web server. One way is to just write a monolithic program that handles the requests and outputs HTML using print statements. Without a lot of planning, you usually end up with something where logic is intermixed with presentation. The disadvantage of such an approach is that it quickly becomes very difficult to maintain your web application as it grows and changes. The current best practice with web application frameworks (such as Ruby on Rails) splits a web application into 3 distinct components (Model, View, Controller — MVC) so that changes to one component can be made with minimal impact to others. This article details a solution used in a recent project to implement such an architecture using the following open source components: SQLite, Clearsilver, and Python. The challenge was to find a solution that performed acceptably on a resource constrained 130MHz ARM-Linux system.

Components Used

The requirements for the web application framework for this system are:

  • Footprint must be fairly small — less than 10MiB.
  • Must enable us to implement a clean MVC type architecture.
  • Must support a high level language like python for rapid development.
  • Includes a database.
  • Reasonable performance — less than 1 second to render a typical page.

After researching and testing several options, the following components were chosen to implement the framework:

  • Web Server: Cherokee (1MiB)
  • Model: SQLite  (290KiB)
  • View: Clearsilver  (170KiB)
  • Controller: Python (2-3MiB)

web_app_framework1

Total size for the above components is about 4MiB.

The following sequence occurs during a typical web transaction:

  • An HTTP request is received by Cherokee
  • If the request matches the URI for the web application, Cherokee forwards the request to the web application framework via CGI.
  • Clearsilver parses the HTTP headers (including CGI GET and POST parameters) and provides the data to python in a easy to access HDF format.
  • Python looks at the URI and dispatches the request to the appropriate function.  Data is extracted from the SQLite database and an HDF datastructure is created.
  • Clearsilver is called to render the appropriate template.  Data from the HDF datastructure is used in the template to provide the dynamic content in the page.
  • The rendered HTML is passed back to Cherokee, and is then returned to the user’s web browser.

Each component is discussed in more detail in the following sections.

Performance

One of the issues with modern web frameworks is that most require a lot of processing power.  Many frameworks are written in an interpreted language which tend to be not very efficient on embedded systems such as the 130MHz ARM-Linux system used in this project.  As most of the application development for this project is done in Python, I tried several other Python based solutions with the following results:

  • Django: takes 10 seconds to render a page
  • webpy: takes 5 seconds to render a page
  • Clearsilver/Python: takes about 1 second to render a page

Anything over 1 second is very slow for a web interface.  With fastcgi, webpy could probably have been sped up to be acceptably fast, but I don’t think there is much hope for frameworks like django on this type of system.  I have read of similar experiences trying to run Ruby-on-Rails on ARM-Linux systems.  As I’m currently just using a basic CGI interface, the Clearsilver solution could also be sped up significantly as well, if the Python portion could be kept running between requests with fastcgi or a similar mechanism.

Cherokee Web Server

A web application typically consists of a web server (like Apache) and a program that provides the dynamic web content.  Some of the functions of a web server are:

  • receives HTTP requests
  • handles the request by serving static files/images or routes the request to a program that outputs dynamic content
  • authentication
  • encryption (SSL, TLS)

There are many web servers that can be used in embedded devices.  Some of them are:

Cherokee was chosen for this application because it provides a good balance between the size and functionality I am looking for.  It is also included in the Openembedded (http://openembedded.org ) build system I am using.  Other systems will have different requirements — use the one that fits your application best.

Clearsilver

Clearsilver (http://www.clearsilver.net/ ) is the real gem that was discovered during this exercise.   Clearsilver is a language-neutral HTML template system written in C.  It is used as the templating system for many high volume sites such as Google Groups 2.  Clearsilver also provides CGI handling functions and bindings to several languages including Python.  The fact that it is written in C and is fast for high volume sites also makes it fast enough on slower embedded systems.  Because Clearsilver is written in C, the templating system is already fast.  Parts or all of the application can also be moved to C as needed to get the required performance.  Having a performance upgrade path is nice.  Clearsilver also forces a strict separation of application logic and presentation templates, which keeps things in line with the MVC architecture.

SQLite

SQLite (http://www.sqlite.org/ ) is a small C library that implements a self-contained, embeddable, zero-configuration SQL database engine.  SQLite works as expected and also has bindings to a number of languages including Python.  A SQLite database is just a single file that requires no configuration, making it very easy to use.  The database engine runs in the same process as the application using it, which increases performance because there is no context switch when running database operations.   Because SQLite does not have its own process, it does not handle concurrency as well as other databases such as MySQL, but in an embedded system you seldom have more than one process using the database.  SQLite provides some support for concurrency with reader/writer locks for the entire database.  If your application is not real busy, this is often adequate if you need two processes to share some data.

Python

Python is the glue that ties things together in this web application framework.  It sits between Clearsilver and the database.  The advantages of using a language like Python are rapid development and an extensive library.  Python is considerably slower than C, but there is always the option to move parts of the application into C as needed.

Openembedded Build System

How does one put together a system with all these components?  The answer is the Openembedded Build System (OE) (http://openembedded.org).  OE includes support for all of the components discussed in this article.

Summary

The combination of Python, Clearsilver, and SQLite provides a compelling solution for a web application framework in an Embedded Linux System.  The solution is reasonably small, performs well, and provides a path to improve performance if needed.  We also get a clean architecture to create clean, maintainable web applications.


About the author — Cliff Brake owns BEC Systems, a consulting operation that helps customers utilize modern computer technologies in their products.  BEC offers a range of services to make your embedded project a success including technology selection, development, troubleshooting, and training.  Please visit our website (http://bec-systems.com ) for more information and other free resources.