Identifier Type</th>	Size</th>	Purpose/Usage</th>	Example</th></tr></thead>
Bluetooth Address (EUI-48)</td>	48 bits</td>	Device-level unique ID (public or random)</td>	12:34:56:78:9A:BC</td></tr>
UUID (16/32/128-bit)</td>	16/32/128b</td>	Service/characteristic identification</td>	0x180D, 12345678-...</td></tr>
Company Identifier</td>	16 bits</td>	Manufacturer/vendor identification</td>	0x004C (Apple)</td></tr>
System ID (Device Info)</td>	64 bits</td>	Encodes device's MAC address (optional)</td>	(varies)</td></tr>
Local Name</td>	Variable</td>	Human-readable device name</td>	"Charge HR"</td></tr>
Appearance/Class Codes</td>	16 bits</td>	Device type/classification</td>	0x0340 (Heart Rate Sensor)</td></tr></tbody></table></figure> Bluetooth Mesh</h2> Bluetooth Mesh networking introduces several unique identifiers (IDs) at various layers of the protocol to manage devices, messages, and network security. These IDs serve different purposes than traditional Bluetooth or BLE device addresses.</p> Unicast Address (Node Address/Element Address)</strong></p> Each node (device) in a Bluetooth Mesh network is assigned a unique unicast address</strong> during the provisioning process. This is a 16-bit value that uniquely identifies the node or its elements within the mesh network.</li> A node can have multiple elements, each with its own unicast address in a contiguous range.</li> </ul> </li> Group Address</strong></p> 16-bit values used to address multiple nodes simultaneously (e.g., all lights in a room). Nodes can subscribe to one or more group addresses, enabling efficient publish/subscribe communication.</li> </ul> </li> Virtual Address</strong></p> 128-bit values derived from a UUID. They allow addressing of a logical group of nodes, regardless of their physical location or provisioning sequence. Virtual addresses are mapped to a 16-bit address space for use in messages.</li> </ul> </li> Device Key (DevKey)</strong></p> A unique 128-bit cryptographic key assigned to each node during provisioning. It is used for secure communication between the provisioner and the node, especially for configuration and management messages.</li> </ul> </li> Network Key (NetKey) and Application Key (AppKey)</strong></p> Network Keys</strong> are shared among all nodes in a mesh network and are used to secure network layer messages.</li> Application Keys</strong> are used to secure application-specific messages, allowing for separation of different functionalities (e.g., lighting vs. sensor data).</li> </ul> </li> Model Identifier</strong></p> Each mesh model (defining device behavior, such as a light switch or sensor) is assigned a unique identifier:</li> SIG-adopted models</strong>: Identified by a 16-bit model identifier.</li> Vendor-specific models</strong>: Identified by a 16-bit vendor ID and a 16-bit model ID.</li> </ul> </li> Message Sequence Number</strong></p> Every message sent by a node includes a sequence number</strong> to prevent replay attacks and ensure message uniqueness.</li> </ul> </li> Device Property ID</strong></p> For sensor and property models, each property (e.g., temperature, humidity) is assigned a unique Device Property ID</strong> as per the Bluetooth Mesh device properties specification.</li> </ul> </li> Node ID</strong></p> Some documentation refers to a Node ID</strong> (or nodeId), typically a 2-byte value, as a unique identifier for a mesh device within the network, used for addressing and control.</li> </ul> </li> </ol> IPv6/EUI-64</h2> IPv6/EUI-64 addresses are a 64-bit number commonly used in IPv6 and typically displayed as 8 groups of four hexadecimal digits: 2001:0db8:85a3:0000:0000:8a2e:0370:7334</code>. Consecutive groups of zeros can be replaced with a double colon. For example, 2001:0db8:0000:0000:0000:0000:0000:0001</code> can be shorted to 2001:0db8::1</code>. Note, leading zeros in the group are also omitted. IPv6 address blocks are assigned by the Internet Assigned Numbers Authority (IANA), which allocates large blocks of addresses to Regional Internet Registries (RIRs). These RIRs then distribute smaller blocks to Internet Service Providers (ISPs) and other organizations. The ARIN (American Registry for Internet Numbers) provides blocks of IPv6 addresses. The annual fee for a 3X-Small block (288 addresses) is $262.50 at the time of writing.</p> `There are also EEPROM and other ICs that have EUI-64 numbers preprogrammed, such as the Microchip 24AA02E64</a>.</p>` QR Codes</h2> QR codes provide the opportunity to easily read a UUIDs or some other long ID on every device using a phone camera. This relieves some of the complexity of humans having to deal with long IDs.</p> Entering a known device in the field into an application is easy with QR codes. However, finding a device (one among many) in the field can be difficult unless the application has a feature designed just for this.</p> </figure> There is no central registry for QR codes. Instead, the process of QR code generation and use is decentralized and relies on the vast number of possible QR code combinations to avoid conflicts.</p> SNMP/Private Enterprise Numbers (PEN)</h2> A SNMP Private Enterprise Number (PEN)</strong> is a unique identifier assigned by the Internet Assigned Numbers Authority (IANA) to organizations, individuals, or entities. It is primarily used in network management protocols like Simple Network Management Protocol (SNMP) to identify objects defined by an enterprise within the Management Information Base (MIB) structure.</p> PENs are part of the Object Identifier (OID) hierarchy under the prefix 1.3.6.1.4.1</code>, which corresponds to .iso.org.dod.internet.private.enterprise</code>. This prefix ensures that each PEN is globally unique.</p> Any entity can request a PEN assignment from IANA at no cost</a>.</p> Automobile VIN</h2> The modern 17-character Vehicle Identification Number (VIN) format was first conceptualized and standardized by the United States National Highway Traffic Safety Administration (NHTSA) in 1981. This standardization aimed to unify the previously inconsistent vehicle identification practices across manufacturers and facilitate more effective tracking of recalls, registrations, and thefts. The VIN system introduced by NHTSA was later adopted globally and is now governed by the International Organization for Standardization (ISO), specifically under ISO 3779 (content and structure) and ISO 4030 (location and attachment)</p> An example of an automobile VIN (Vehicle Identification Number) is:</p> 1P8ZA1279SZ215470</strong></p> VINs are always 17 characters long and use a combination of capital letters and digits. Each section of the VIN provides specific information about the vehicle:</p> The first three characters (World Manufacturer Identifier) indicate the manufacturer and country of origin.</li> Characters 4–8 describe features such as model, body type, and engine.</li> The 9th character is a check digit for security.</li> The 10th character indicates the model year (e.g., “S” for 1995, “R” for 1994 or 2024).</li> The 11th character shows the assembly plant.</li> The last six characters are a unique serial number for that vehicle.</li> </ul> KNX Serial Number</h2> KNX is a popular building automation standard. The KNX serial numbers</a> follow the following format:</p> Length:</strong> The KNX serial number is always 6 bytes (octets) long.</p> Manufacturer Code:</strong></p> Octets 6 and 5</strong> (the first two bytes, when read from left to right) contain the official KNX Manufacturer Code assigned to each manufacturer by the KNX Association.</li> </ul> </li> Device-Specific Portion:</strong></p> Octets 4 to 1</strong> (the remaining four bytes) are managed by the manufacturer.</li> This part may encode information such as production site, date, series, batch, or simply a sequential unique number.</li> </ul> </li> </ul> BACnet</h2> BACnet Device IDs, also known as Device Object Identifiers or Device Instance Numbers, are unique numerical values assigned to each BACnet device on a network to identify them distinctly.</p> The Device ID is a 22-bit unsigned integer ranging from 0 to 4,194,302 (decimal). The value 4,194,303 (0x3FFFFF) is reserved and cannot be used as a valid Device ID.</p> This Device ID corresponds to the object instance number portion of the BACnet Device Object’s Object_Identifier property.</p> The full Object_Identifier is a 32-bit value composed of:</p> Object Type (10 bits): For a Device object, this is fixed as 8.</li> Object Instance (22 bits): The Device ID itself.</li> </ul> BACnet IDs are not assigned during manufacturing, but rather during setup/installation.</p> Matter</h2> Matter devices include a Device Attestation Certificate (DACs) generated through a hierarchical Public Key Infrastructure (PKI) process, ensuring each device is uniquely authenticated. Each device is cryptographically verified, making spoofing difficult.</p> The DAC is part of a chain of trust:</p> Product Attestation Authority (PAA):</strong></p> Root certificate issued by the Connectivity Standards Alliance (CSA) or an approved third party.</li> Stored in the Distributed Compliance Ledger (DCL), a global registry of trusted certificates.</li> </ul> </li> Product Attestation Intermediate (PAI):</strong></p> Intermediate certificate signed by the PAA.</li> Typically includes the manufacturer’s Vendor ID (VID).</li> </ul> </li> Device Attestation Certificate (DAC):</strong></p> Unique per-device certificate signed by the PAI.</li> Contains the device’s VID, Product ID (PID), public key, and serial number.</li> </ul> </li> </ol> Additionally, devices use rotating keys, which makes tracking difficult and preserves privacy. End users scan a QR code to onboard devices.</p> Matter attempts to be:</p> Authentic (not counterfeit)</li> Secure (resistant to hacking)</li> Private (resistant to tracking)</li> Interoperable (works with any ecosystem)</li> </ul> Serial Numbers</h2> Other custom serial numbers take many different forms. Some examples are shown below:</p> Format Type</th> Example</th> Description</th> Typical Use Cases</th></tr></thead> Sequential</td> D000001</td> Incremented by one, often padded with zeros</td> Manufacturing, inventory</td></tr> Alphanumeric</td> TBL-101A234</td> Letters and numbers, often with category or model encoded</td> Electronics, equipment rental</td></tr> Random</td> X9F4-7T32-ZY6</td> Unpredictable, generated by algorithm</td> Security-sensitive items</td></tr> Date/Batch Embedded</td> 20230715-001-123</td> Encodes date and batch with sequence number</td> Quality control, traceability</td></tr></tbody></table></figure> Systemd network interfaces</h2> Systemd’s predictable network interface name encodes the physical path of the device. Example of how enp0s13f0u2c2</strong> is constructed:</p> en</strong>: Ethernet device (prefix for Ethernet interfaces).</li> p0</strong>: PCI bus 0</li> s13</strong>: PCI slot 13 (device 13 on the bus)</li> f0</strong>: PCI function 0</li> u2</strong>: USB port 2 (the device is connected via USB, possibly a USB Ethernet adapter)</li> c2</strong>: USB configuration or child port 2 (sometimes used to indicate a specific port or configuration on a USB hub)</li> </ul> Summary</h2> I’m sure there are many other examples – please leave a comment if you’d like to share other interesting formats. In a future post, we’ll explore a new standard for device IDs that is optimized for human interaction in the field. Subscribe below to stay informed of the latest developments.</p> How to implement Zephyr web applications 2025-04-08T00:00:00+00:00 Are you considering implementing a web UI in your Zephyr RTOS</a> application and perplexed by the endless number of web technologies available today? This article traces my journey through trying several options and presents some of the tradeoffs.</p> </figure> Modern microcontrollers (MCUs) have many connectivity options (Ethernet, WiFi, Bluetooth, Cellular, etc.), and when coupled with the Zephyr OS, implementing a web application on these devices often makes sense. Once you start doing this, you soon realize that this is not Ruby-on-Rails. Rather, Zephyr is still a very constrained environment (see my article on the differences between MPUs and MCUs</a>). While we can still leverage modern web technologies, we need to be selective because of the constraints of MCU systems. An example of a web UI in the Zephyr SimpleIoT project</a> is shown below:</p> </figure> Primary concerns in an MCU system</h2> Size</h2> The memory resources on MCU and MPU systems are vastly different</a> – a difference of over 1,000x.</p> </td> MCU (Internal)</strong></td> MCU (External)</strong></td> MPU (Server)</strong></td></tr> Flash</strong></td> 0.5-2MB</td> 16MB</td> 500GB</td></tr> RAM</strong></td> 0.25 - 1MB</td> 16MB</td> 16GB</td></tr></tbody></table></figure> While it is possible to use larger memory devices with MCUs, like an SD card or SDRAM, the above table is for the more common scenarios.</p> If we want to serve the web application entirely from a Zephyr device, then the web assets must fit in flash. The RAM size also limits the number of connections we can handle and the size of the data payloads we can transmit and receive.</p> Stability (includes maintainability and security)</strong></h2> The constraints of an MCU drive us toward doing more in the browser (frontend) because resources there are free and plentiful. This typically ends up being a Single-Page Application (SPA) architecture, where the web application is a free-running application that fetches and sends data to the MCU (backend). All rendering happens in the browser instead of the MCU. Because of this, the web application needs to be stable. Extra care must be taken so that the application does not crash and freeze or pop up annoying messages telling the user to reload the page, which can easily happen with JavaScript.</p> We also need to consider how the application is hosted. A typical web application is hosted in the cloud in one place. If there is a problem, we have complete control over updating and fixing it as needed. Embedded applications are a completely different matter – there may be thousands of devices. If there are problems, it is not as simple as updating a single server. We may have a firmware update mechanism in the device, but it is often up to the user to decide when to update. Users expect embedded devices to be stable. Many of them are used in critical applications. An instability in the web UI of the device does not give a positive impression, even though it may only be a web frontend crash, and the device is still functioning fine.</p> </figure> It seems simple to write an initial prototype, but maintaining a web application over time is a different matter. Security can also be a concern as many web frameworks pull in mountains of JavaScript dependencies via NPM, which are difficult to audit for security problems.</p> State Management</strong></h2> One of the hardest parts of a web application is maintaining state. The reason for this is that user interfaces can be deeply nested and complex. The naive approach is to distribute the state into each UI component. The difficulty with this is keeping everything in sync. There has been a continual stream of new frontend state management libraries (Redux, MobX, VueX, React Context, XState, react-easy-state, etc.). This is not an easy problem.</p> Options</h2> Serve site/assets from external server</h2> One way to work around the asset size constraint in MCUs is to serve large assets (like a Bootstrap CSS file) from an external server.</p> </figure> There are several potential issues with this:</p> ❌ The external asset must always be available, and if it goes away or changes, then the web application on the device must be updated to point to the new location.</p> ❌ If the entire site is served from an external server, then you must host all versions of the web application or be very careful to maintain backward compatibility to all previous versions of firmware running on the Zephyr device.</p> ❌ You may need the Zephyr UI to function if the Internet is down or unavailable (example: air-gapped secure setups or during device setup). In many systems, device state is mirrored to an upstream server, so the only time you would use the local UI is during setup or when the Internet is down.</p> ✅ The most reliable way to host a Zephyr web application is on the device itself.</strong> You certainly have many more options if you host it externally, but there are tradeoffs that must be considered.</p> No Build, just HTML, JS, CSS</h2> DHH and others are advocating No-Build</a> web solutions. However, they have Rails for a backend, and we have Zephyr. I investigated this approach, but manually mutating HTML elements in JavaScript in a complex app is messy and scattered – you’ll eventually create a framework anyway, so you may as well start with something close to meeting your needs.</p> </figure> HTML/JS/CSS development is powerful and flexible, but is tailored to design, not programming. In its raw state, it is like developing a program where everything is a global variable. If you have a dynamic web application responding to real-world events, this quickly becomes a programming problem that is difficult to manage with 200 global variables. You need an abstraction on top of this. Again, a powerful templating engine on the backend helps a lot, which we do not have in Zephyr applications.</p> React, Vue, Angular (non-compiled frameworks)</h2> Solutions like React are too large for many MCU applications as they typically require a megabyte or more of web assets to be transferred from the host. Additionally, you have the issue of JavaScript crashes and the continual bloat and churn of the NPM package ecosystem. Updating to new versions of packages can be a lot of work as APIs sometimes change. While this solution can work for large-scale cloud-hosted apps, it is not ideal for embedded web UIs where you often don’t have control over the device and can’t easily patch the system if there is a problem.</p> </figure> I’ve been writing/maintaining a 50K SLOC Meteor/React app for the past 10 years – it works and has been very successful. However, we’ve had a handful of frontend crashes. Being a cloud app, we were able to deploy fixes for this quickly, but it is still an inconvenience for users. In another project developed by a very experienced JavaScript developer, we also had a number of JavaScript crashes in the frontend. There are likely ways to reduce or avoid these stability problems, but it is not inherent in the system/language. It requires a high degree of carefulness and discipline. It is not easy to refactor JavaScript and know that you have fixed all the issues caused by your code change.</p> </figure> There are slimmer solutions like Preact which may solve the size issue, but you likely lose some of the integration with all the libraries written for React.</p> htmx</h2> htmx is the cool kid on the block right now – supposedly simple and easy to use. I tried this for the first iteration of a Zephyr web application and got a prototype running, but concluded it works best if you have a powerful backend that can do most of your rendering, and the frontend just handles a little bit of reactivity. Additionally, with htmx, you typically open up a network connection for every widget to get updates, and this requires a lot of connection resources on the backend, which are expensive in a Zephyr system.</p> </figure> Svelte</h2> Svelte is a compiled language that generates small assets, so it works well from a size perspective. It also appears to be a well-maintained project with good tooling. However, there are two downsides: 1) it is still JavaScript, so you have potential stability issues when maintaining applications long-term. 2) It puts the “JS” in the “HTML” instead of the “HTML” in the “JS”</a> (like React or Elm). This is largely a matter of preference, but as a programmer, I prefer the latter. The success of React indicates that many others do, too.</p> Functional Languages</h2> Functional languages are becoming more common in frontend development because they solve many reliability and refactoring challenges. Some options include ClojureScript, Elm (discussed below), PureScript, ReasonML, F#, Scala.js. Some of these options likely have smaller asset sizes than non-compiled options.</p> Elm</h2> Elm</a> is a compiled language that produces small assets. It is like React JSX in that it puts the “HTML” in the “Elm</a>“. Elm is a functional language that solves many of the challenges with web programming, including stability and managing state. Runtime errors in an Elm application are very rare. I wrote the Simple IoT UI in Elm, which is now 18,610 lines of code, and have maintained it since 2018. It has never crashed, and it is easy to add new features. Code density/reuse in Elm is very high. If it compiles, it generally works, so very little time is spent in the browser console debugging stuff. The elm-ui</a> package offers a powerful style and layout system that eliminates most of the pain with web programming and allows someone (like me) who is not a dedicated web developer to maintain a pretty nice web application without learning the intricacies of CSS.</p> </figure> As you maintain an Elm project, you’ll find that the language pushes you to a cleaner design. It nudges you in the right direction. The opposite generally happens in JavaScript projects, where a lot more discipline is required to keep things from becoming a mess.</p> Elm allows for creating powerful functions like the following for a text input form:</p> textInput Point.typeBitRate "Bit rate" "250000" </code></pre> This line of code renders a form element and automatically posts a new point to the backend of type BitRate</code> when the content is changed, nothing else in the system needs to change for this to happen. There are no “handlers” that need to be attached to this element. Elm also has some of the best tooling in this industry – elm-review, elm-land, etc. This tooling is largely made possible by the language.</p> Elm is a small project and community, but it is a stable/friendly community that has existed for many years, and many companies are successfully using Elm</a>. The Elm package system is fairly comprehensive and has most of what you might need to implement web apps. Since Elm packages are pure Elm, the entire ecosystem is stable and relatively secure</a>. There is some concern about the future of Elm as the compiler does not get frequent updates. Some have produced new versions of the compiler to meet special needs. The packages and tooling around Elm are where most innovation is happening right now, and it is phenomenal. Elm got it mostly right, so there is not much that needs immediate improvement in the core.</p> Summary</h2> There is no silver bullet in technology. With any choice, there are tradeoffs. Most people stick with mainstream JavaScript for frontend programming as the “safe” approach. However, in Zephyr/MCU-based systems, there are constraints that we don’t face in server environments.</p> There are significant challenges in the HTML/JS/CSS world. While it is nice to have options, the continual churn of new frameworks (jQuery -> Backbone -> Ember -> Angular -> Meteor -> React -> Angular2 -> Vue -> Svelte -> Next.js) (and many others) indicates there is some fundamental problem with the Javacript approach that has not been solved in a general way. The number of NPM package dependencies pulled into a typical project is staggering and nearly impossible for small teams to audit for security and longevity. NPM installs routinely break for various reasons. NPM suffers from some of the same problems as Python, as many packages compile C code in non-standard ways during the package install process, which can be fragile.</p> There is also the issue of what might be viewed as an impedance mismatch – what the technology mega-companies produce and use is not always the best fit for small companies/teams with limited resources. Facebook faces different problems on a different scale than a small embedded team. While React is right for them, we can’t assume it is right for everyone. Most of us are not facing Google-scale problems</a> that require a system as complex as Kubernetes. Ansible and a few servers work just fine. Facebook has near-infinite resources. They have ways to deal with the inherent stability problems of JavaScript. A small company/team does not have these resources, so different technologies like Elm may be a better fit.</p> When faced with the need for a Zephyr web application, I tried htmx first, then looked at “no build”. Neither of these worked very well for me. Having worked on decent-sized React and Elm projects, Elm looked like a better fit for the next try due to asset size, reliability, and maintainability. Additionally, I had already solved the problem of how to effectively deal with data in a distributed IoT system</a>, and had frontend code already written</a> to interface with this dataflow in Simple IoT. It was easy to port the Simple IoT frontend to a smaller version tailored for MCUs, and thus far it seems to be working well. Check out the Simple IoT Zephyr Networking Example</a> to learn more.</p> Struggling to scale your IoT system? Simplify your data. 2025-03-29T00:00:00+00:00 Have you struggled with adapting your IoT system to new applications and requirements? Is handling configuration updates at both the cloud and edge a challenge? Is synchronizing required data between cloud and edge instances a challenge? This article describes a simple way to model data that eliminates many of these difficulties.</p> We can scale an IoT system both horizontally</strong> (deploy more units) and vertically</strong> (add features and address new applications). While the ideas presented in this essay are focused on vertical scale, any simplification will likely help with horizontal scale as well.</p> Previously, we explored data-centric architectures in IoT systems</a>. This post expands on this by describing how to represent data using Nodes and Points in a way that drastically simplifies IoT data storage, exchange, and synchronization. This is in contrast to encodings and mechanisms used in traditional web and cloud systems.</p> How are IoT systems different?</h2> IoT systems 🌐 have the following properties, which differentiate them from traditional computer/web/cloud applications:</p> 🌐 IoT systems are inherently distributed (equipment is located in many physically different locations).</p> 🌐 Data must exist concurrently in multiple places (cloud and many edge instances).</p> 🌐 Data is not fully replicated on every instance (a cloud server instance might contain the entire dataset, whereas an edge instance only has a tiny fraction of the dataset that is relevant at that site).</p> 🌐 Instances are not symmetrical (edge instances are very different than cloud instances. It is not a cluster).</p> 🌐 There can be many instances (thousands of edge devices).</p> 🌐 The network is even less reliable (edge systems are often in uncontrolled environments).</p> 🌐 The system must continue to operate during network interruptions (it is not a cluster).</p> We can attempt to address these challenges by adding constraints such as: 1) data only flows in one direction, 2) device configuration is only changed in the cloud, and 3) the data payload format will not change over time. However, if we think realistically about these constraints, they are impractical for many systems. In distributed computer systems, change is the constant. Because a system can be networked, its capabilities are endless. The ability to connect to new and different systems means new systems and new value can always be created, and things will change</strong></em>. If you don’t, somebody else will.</p> The “struct”</em></h2> Cloud/Web APIs have traditionally used various encoding schemes and patterns such as Protobuf, JSON, REST, gRPC, etc. Tools exist to manage schemas, such as GraphQL and JSON Schema. Synchronization mechanisms can be used to manage conflicts (Raft, CRDT, etc.). Most of these schemes assume that the complex struct</em> with multiple and sometimes nested fields is the fundamental element of data representation, storage, and transmission. An example is shown below:</p> type EmployeeData struct { ID int Name string Age int Email string Phone string Street string City string Country string PostalCode string EmploymentHistory []struct { Company string Position string Years int } Skills map[string]struct { Proficiency string YearsExperience int } LastUpdated time.Time } </code></pre> Structs are great – they are a powerful tool for organizing data in a computer program. They are so useful that we have also assumed structs are the best way to store and transfer data outside a program. Thus, we have tables in SQL databases, REST APIs, Protobuf/JSON encoding, etc. This works fairly well and is likely the right solution in many domains. However, we have the following problems with using the struct</code> outside a computer program:</p> ❌ Updating the struct</code> tends to be an all-or-nothing proposition. Fetching the data from a REST API gives you the entire struct</code>. Likewise, when updating, you send the entire struct</code>. There are tools, such as GraphQL, that attempt to solve this problem.</p> ❌ Schema changes are painful. Many very elaborate tools are dedicated to solving the problem of database migrations.</p> ❌ API schema changes are even more painful.</p> ❌ Distributed systems are hard, especially if the struct</code> may be updated by multiple systems concurrently and then requires merging. CRDTs and various other distributed system algorithms attempt to solve this problem, but they are complex.</p> ❌ Nested data is problematic and compounds issues with schema changes and synchronization.</p> Data in IoT Systems</h2> What are desirable features for handling data in a distributed IoT system?</p> ✔️ We can generate/change data anywhere (in the cloud, at the edge, etc.).</p> ✔️ Synchronization between instances is granular, simple, and reliable.</p> ✔️ Data can flow in any direction.</p> ✔️ One mechanism/transport can be used for all types of data (config, sensor samples, etc.).</p> ✔️ All data changes in the system are recorded in history.</p> On the surface, this seems like a hard problem. If you have a struct</em> that exists in multiple places, and one field gets changed in one location, and another field gets changed in a second location, how do you reconcile the differences and record what has changed?</p> As mentioned before, there are solutions such as CRDTs or sending data changes over the wire. Some systems only store changes (mutations) and then compute the current state from all the changes. However, all these schemes are mostly just adding layers of complexity on top of an existing paradigm and are rarely a good fit for IoT systems where you accumulate many thousands of data samples (changes) over time. Maybe it’s time to change the paradigm …</p> Points and Nodes</h2> What if we rethink our concept of data representation? Instead of trying to make a generic struct</code> the fundamental unit of data, we take a more granular approach and focus on each piece of data in a struct. Can we create a simple unit of data that is easy to store and synchronize, and then build on top of this? This same unit of data is used for everything, including configuration, sample data, notifications, etc. The data unit we have arrived at in the Simple IoT project</a> (after many iterations) is the “Point.”</p> </figure> The data type is typically a string</code>, int</code>, or float</code>, but it can be anything you want.</p> Points are organized in a hierarchical tree of nodes, where each node contains an array of points.</p> </figure> Because a point</code> is defined by both a type</code> and a key</code>, it is possible to construct arrays and maps by populating the key</code> field with either an array index or a map key. This allows us to easily represent one level of structure (arrays and maps) in a node as shown in the following example:</p> </figure> If a deeper structure is required, then nest additional nodes.</p> Benefits</h2> There are multiple benefits to this data architecture.</p> ✔️</strong> Data is granular.</strong> An edge instance can update a node’s sensor reading, and the user can update a node’s config setting in the cloud simultaneously. The points are easily merged as they are received into the node at both locations, without conflicts. This works very well because different users/instances tend to update different points. The exception to this may occur when a user at the edge and a cloud user update a config item at the same time. In this case, we simply use the point with the latest timestamp.</p> ✔️</strong> The storage and communication infrastructure does not change.</strong> One of the biggest problems in traditional storage mechanisms and network APIs is that schema and encoding changes can be very painful. If you add a field to a data structure, every piece of code that touches the data as it travels throughout the system needs to be updated. With nodes and points, only the ends of your system need to be updated. Everything in between (transmission, storage, sync, etc) can stay the same.</p> </figure> ✔️</strong> Introspection and monitoring are easy.</strong> Since our data format is simple and fixed, it is easy to watch and decode traffic anywhere to see what is going on.</p> ✔️</strong> Integration is easy.</strong> Because we have standardized the data to be very simple, it is easy for one service to provide data that another can use.</p> ✔️</strong> Everything is data that can be acted upon.</strong> In the past, we might have had a rules engine that could respond to sensor data changes. Since config changes are now the same kind of data, the rules engine can now respond to and make config changes as well.</p> ✔️</strong> History is easily logged for all changes.</strong> Likewise, since config and sensor sample data are now the same type of data, everything that happens in the system can be easily recorded in history. The system is automatically auditable.</p> ✔️</strong> It is efficient.</strong> Because the data is granular, only what changes is transmitted and processed.</p> ✔️</strong> It works both internally and externally.</strong> While this data scheme was primarily developed to efficiently communicate between systems connected by networks, it also works very well for communicating between threads in an application. When an application is architected this way, it is very easy to extend, it as the communication scheme internally and externally is the same. This architecture scales.</p> Objections</h2> 🚩Objection #1: But isn’t this a lot of overhead to store a type/key/timestamp/etc with every bit of data?</strong></p> We need to focus on where the bottlenecks/hard/expensive parts of the system are. Synchronizing data from many different instances over networks is the hard part of IoT systems. Local storage and CPU processing are relatively cheap in comparison. Networks are also relatively slow compared to the additional processing time required to process this data. CPU and storage are now abundant, even on microcontrollers. Development, operations, and deployment are not cheap.</p> 🚩 Objection #2: Nodes and Points are hard to deal with.</strong></p> True, dealing with raw nodes</code> and points</code> is not as convenient as a struct</code>, but we can create tools</a> that easily convert structs</code> to and from nodes/points</code>. We do it all the time – JSON, Protobuf, etc. Computers are very good at encoding/decoding data – let them do it.</p> 🚩 Objection #3: How do you protect your data if anyone can update anything?</strong></p> AuthZ/AuthN is a valid concern and has not been fully solved yet. One idea is to authorize access to data at the node level – each user or instance would have access to a set of nodes. However, this says that an edge device could change its own config parameters. Perhaps this is OK because, without physical security, you don’t have true security. It may also be useful to differentiate betweenread and write access</a> to node data. Ideas are welcome!</p> 🚩 Objection #4: If points for a single struct are flowing in at different times, how do I know when a change is complete and when to process it?</strong></p> This is also a valid concern. One option is to set a timer that gets reset any time the config changes, and the changes are only processed once the timer expires. With a little creativity, we can probably come up with other schemes to signal the change is complete – perhaps a special point that is sent at the end of a sequence of points.</p> 🚩 Objection #5: What if the data is more complex than what can be represented with a simple data type like a number or string?</strong></p> You can store anything you want in the point data field (JSON, Protobuf, etc.) This should provide enough flexibility for things like messages, where it probably does not make sense to separate the data into separate points.</p> 🚩 Objection #6: How do you deal with schema changes?</strong></p> There may be instances where there are point type name changes, node topography changes, etc. In this case, each client, which provides the backend code for a node type, should be responsible for handling any schema changes. This localizes the migration logic to each client (including external clients that plug into the system at runtime) rather than requiring a central system to understand all migrations. A node may contain a version point if necessary.</p> 🚩 Objection #7: Do I have to represent all my data as points?</strong></p> If data needs to be distributed between instances on a network, then it is very beneficial to use points. However, for local processing of data, there is nothing preventing you from using more conventional techniques, storage, etc.</p> 🚩 Objection #8: How do you store data as nodes and points? SQL databases are not designed for this.</strong></p> That is an excellent question and part of our research in the Simple IoT project. We are currently using SQLite, but we are moving to NATS Jetstream</a>. Stay tuned!</p> 🚩 Objection #9: Will this scale to a bazillion devices?</strong></p> Probably not. If you need to connect a bazillion devices, then perhaps AWS IoT or something else is a better fit. However, what we are presenting here is a flexible architecture that will scale over time to many different applications and configurations as your needs evolve and change. How do you want to scale? Think carefully about this, as scaling in one dimension sometimes limits how far you can scale in another.</p> 🚩 Objection #10: This will not work for data sampled at 100 bazillion samples per second.</strong></p> For moderate data rates, transmitting blocks of data with every point by putting an array of data in a point data field may work. For high-rate data that flows, a custom data encoding and transport will likely be required.</p> 🚩 Objection #11: You can’t depend on time in a computer system, so last-write-wins (LWW) is not reliable.</strong></p> True, time, like networks, is not always reliable. However, in IoT systems, most of the time, you do not have the same data being edited in multiple locations, so LWW is only used in the rare case where two human users make changes at different locations. A little clock drift is not a huge deal. You can also do things like check if two systems have similar times before you synchronize data between them, and verify a system has a valid time before writing any data. You can also require that every instance in the system has its own RTC clock and that some engineering effort is made to ensure it is reliable and reasonably accurate.</p> 🚩 Objection #12: Why not send commands, events, changes, etc.?</strong></p> Sending “commands” is a common paradigm in IoT systems, but it is the siren song into complexity. How do you deal with cases where the command does not make it due to a network problem? There are valid cases for events such as orders or notifications, but in most cases, commands/events are an unnecessary abstraction and only complicate things. A sensor is at value X. The config setting is Y. Keep things declarative. Just send the data itself, keep some amount of history, and if you want to know when something changed, look at history. Utilizing new technologies like NATs Jetstream</a> is a critical piece of this puzzle.</p> 🚩 Objection #13: But this is different !?!?</strong></p> Yup, I’ve never seen anything like it before either. But it works.</p> Summary</h2> When we want to advance something, we often need to simplify it first.</strong> Piling on layers of complexity will only get us so far – at times, we need to go back to ground zero and re-think things.</p> There are many ways to specify contracts between systems, such as APIs and schemas. A more fundamental contract focuses on a more granular unit of data. Points in an IoT system are like atoms in matter. A Node tree in an IoT system is like the DNA in a living organism. The composition of points and structure of nodes (like atoms and DNA) is infinitely flexible. It is this simplicity and flexibility that allows atoms and DNA to define so many different types of matter and species of organisms. It is time to throw out the custom struct as the defined unit of data between systems and focus on something more granular, flexible, and scalable.</p> Want to learn more? Check out and follow the Simple IoT project</a>.</p> The Ability to Improve 2024-03-05T00:00:00+00:00 </figure> (while this article contains many general ideas, it is written from the perspective of product development.)</em></p> If you don’t continually improve, you soon lose the ability to do so …</strong></p> This thought came after an associate described a company that is struggling to manufacture one of their products:</p> They have been making a product since way back in the last century. EOL (end-of-life) is catching up with them. The circuit is really hairy analog stuff with ridiculously high precision. Pots everywhere to get it dialed in. But their voltage regulator is drifting in value significantly, whereas the old parts are rock steady.</p> </blockquote> How do we get into situations like this, and how can they be avoided?</p> A few more examples</h2> Below are a few more examples of situations that are very difficult to deal with in the lifecycle of a product:</p> A part becomes obsolete in a printed circuit design, and no one knows how to use the CAD system it was designed in. Rather than updating to an open/modern CAD system</a> years ago at a relaxed pace, exorbitant efforts are now spent trying to track down a few more parts or find a replacement. When this fails, it is now a frantic crash-and-burn project to revise the design with no time for testing and verification.</li> A serious security problem is found in a product, but the software build system has fallen into disrepair and no one knows how to set it up, and it will not run on current operating systems.</li> A new Linux kernel driver is needed to support a new USB peripheral, but the kernel we are shipping is 7 years old, and the SOC/SOM vendor has not provided any updates. Updating to the latest kernel would take months. Backporting the new driver is nearly impossible due to the many changes in the kernel and its subsystems.</li> Manufacturing was outsourced to another country years ago. Then COVID happened and the manufacturer shut down for an extended time. No one knows where the design files are or how to build the product to move it to another manufacturer.</li> The manufacturing process for a complex product has been the same for years. No one knows why things are done the way they are, but are afraid to change anything for fear that problems may crop up.</li> A code base has not been refactored as it grew and no tests were written. Adding new functionality is very difficult and usually causes bugs in unrelated areas, requiring an exorbitant amount of manual testing with each release.</li> An application does not have a cohesive architecture. Global variables and unrestrained function calls between various subsystems are common. As time goes on, the application becomes less and less stable. Fixing one bug often causes another. All developer resources are consumed in a circle of support, continuous bug fixing, and damage control.</li> A new customer is interested in purchasing a product, but it is missing a critical feature. The original development was contracted to an outside development firm, and there is no knowledge in-house on how to modify the software. The managers at the company are scrambling to find someone who can figure out how to make the changes needed.</li> </ol> Technical debt</em> is another term to describe systems that are not adequately maintained – eventually, it takes so much effort just to keep things going that no time is left for improvements.</p> Maintaining Systems</h2> How do you maintain the ability to react to changes, security problems, part shortages, and feature requests? How can you add value to your products over time? Maintaining systems (both hardware and software) is what keeps them malleable and flexible. If you don’t maintain stuff, then it soon becomes brittle and harder and harder to change. Modern systems are not ship-it-and-you-’re-done, but rather more similar to a living organism. If it is not maintained, it is dying.</p> The only way to make sure you can</em> maintain something is to maintain it.</strong> Make it better. Replace a part with a cheaper/better one. Improve something. Add a feature. Make it faster. Streamline and simplify manufacturing. If we don’t continually improve, then it becomes difficult to do anything without starting over. It is much easier to absorb a series of small costs over time than a large expense all at once. Small changes over time allow us to improve our processes. If we are not under pressure and something goes wrong, then we simply revert to the previous version or write tests to keep it from happening again.</p> This does not have to be hard, but it does require an investment in methodology and workflow. It requires a commitment to open tools, automation, and testing. It requires process – not the type that results in stifling bureaucracy but rather improves efficiency through automation and transparency.</p> There are several concerns related to change including return on investment (ROI), quality, and efficiency.</p> The ROI Objection</h2> There are two financial mindsets:</p> View R&D as a one-time capital expense where you design something, lock it down, and then try to extract as much value as possible – the “minimizing costs” mindset.</li> View R&D as an ongoing operating expense, improve things over time, and continually increase value – the “maximizing value” mindset.</li> </ol> The first is easier to measure and forecast. Financial people like and understand this model. The second requires intuition and vision. It involves some risk. As your processes become more efficient, the value you get from an improvement begins to outweigh the costs of making the change. Over time, these improvements have a compounding effect.</p> </figure> It depends if you are playing the short or long-term game …</p> The Quality Objection</h2> Stability and Quality is always a concern with any change in a system. With changes, mistakes will be made – especially early on. When this happens, there is an opportunity to learn and proactively improve the testing and verification process for next time, or reactively place blame, lock things down even harder, and remove responsibility. When someone introduces a bug, the question should be: “Did you create a test so it does not happen again?” If no mistakes are being made, not much is likely being done. The upside of growth and improvement should far exceed the costs associated with the occasional mistakes made in the course of changes. Done right, quality should follow the same value curve as the value shown in the graph above. The key is to not get discouraged by an initial drop in quality and focus on improving the processes, not minimizing or restricting change and responsibility.</p> The Efficiency Objection</h2> Efficiency is always a concern, and again there are two schools of thought:</p> It is more efficient if we just do things the way we’ve always done them, not risk change, and get what we can out of our investment. Change brings risk.</li> Efficiency can always improve, bringing lower costs and more value.</li> </ol> Again, the key is to be patient with the initial declines and persevere until you get to the point where efficiency is increasing and the cost and value lines cross.</p> Case Studies</h2> Tesla is an interesting case study on continuous improvement</a>. From a design and manufacturing perspective, their accomplishments are notable. There is always a better way to design or manufacture something.</p> Google’s blameless postmortem</a> practice enables proactive improvements for the future instead of reactively placing blame for the past.</p> Amazon’s “You build it, you run it”</a></em> philosophy allows people who know the most about the system to improve how it is run instead of going through endless rounds with gatekeepers who don’t understand the system.</p> Toyota’s implementation of “kaizen</a>” has produced excellent results</a>. Kaizen is a concept referring to business activities that continuously improve all functions and involve all employees from the CEO to the assembly line workers.1</a></sup></p> These practices are used at large successful companies, but in concept, they are simple and can be applied at any scale.</p> The Irony of Kaizen</h2> We can develop products with the thought – we are going to get this perfect and then go into production. Logically it would seem that if we expect to make improvements after it is in production, then we must be starting with an inferior product with flaws. However, Toyota has proved the opposite. Even when a model is first introduced, their quality is likely better than the average in the automobile industry. Kaizen does not mean we slop something together of inferior quality, but rather we do the best we can but have the humility to realize it can always be better. Kaizen is not limited to the assembly line, but all aspects including design, prototyping, quality control, testing, etc. If all of these processes are continually getting better, then the first item produced will also be better.</p> Requirements for Improvement</h2> Every person and every organization is different, so there is no one-size-fits-all recipe that will always work. But there are several ingredients that are generally beneficial including:</p> Metrics</li> Testing</li> Openness</li> Simplicity</li> Leveraging outside improvements</li> Iterate faster</li> Owning your platform</li> The right people</li> </ol> Metrics</h2> Not all change is an improvement – change for the sake of change is just churn and generally negative. You have to be able to measure something to know if things are improving or not. Improvement is very difficult without some metrics, because we are then just guessing and going off our intuition, and human intuition is often distorted by emotion, social pressures, and a host of other distractions.</p> Testing</h2> Testing is trendy today and many talk about it, but fewer individuals and teams have the discipline to implement testing consistently. If you are not confident enough to deploy new software to production after your tests pass</a> without doing some manual checking, then you possibly need better tests. Automated end-to-end and performance testing is critical. Consider the following statement about SQLite:</p> The reliability and robustness of SQLite is achieved in part by thorough and careful testing.</p> As of version 3.42.0 (2023-05-16), the SQLite library consists of approximately 155.8 KSLOC of C code. (KSLOC means thousands of “Source Lines Of Code” or, in other words, lines of code excluding blank lines and comments.) By comparison, the project has 590 times as much test code and test scripts - 92053.1 KSLOC.</p> https://www.sqlite.org/testing.html</a></p> </blockquote> Most of SQLite’s tests are not open-source. This is the real IP of SQLite.</p> Openness</h2> Openness is another necessary ingredient for improvement. This takes many forms:</p> Open access to tools, design files, and source code. Open-source software and tools are preferred when available.</li> Good documentation is available to all involved.</li> Work being done is in the open and can be reviewed by anyone at any time. Pull requests are created when work starts to provide visibility into work being done rather than when the work is finished. Transparency is a powerful ingredient for quality.</li> Metrics, including financial information</a> and customer feedback, are accessible to the team.</li> Anyone on the team to allowed and encouraged to fix or improve anything. There are no “turf battles” where only certain people are allowed to do certain things. Within practical limits, anyone can do any work, but the process (transparency and testing) ensures that substandard work will not be automatically used.</li> </ul> The above does not mean everything is open-source to the world, but rather those involved with the product – a team, organization, company, customers, partners, etc. We can learn a lot from the success of open-source projects and their associated culture and methodologies.</p> Simplicity</h2> As the complexity of systems continues to increase, we need to also be aware that humans interacting with these systems are finite – we can only handle so much complexity. Thus, efforts need to be made to simplify where possible. We often hear the phrase: “keep it simple!” But, from whose perspective? Design, Manufacturing, Users, Service, Ops, Support, Sales, Financial, CEO, etc. If we look at how much time these various groups interact with the product, we can arrange them in an inverted pyramid:</p> </figure> The CEO</em> is only one person and hopefully, there are many Users</em>, so if we can simplify things for the Users</em>, that will probably provide more benefits to the product than simplifying things for the CEO</em>. Likewise, if Design</em> can spend a little more time making Manufacturing</em> and Support</em> simpler, that is probably a good investment.</p> Simplifying a system’s architecture often provides the biggest wins. This can take the form of common data structures, how data is structured, how data is communicated and synchronized, lifecycle management, etc. A simplification at this level has the potential to improve the entire system.</p> Faster Iterations</h2> The most efficient way for humans to design complex systems is through iteration</a>. Some call this failing fast</a>. The faster you can design/develop, test, get feedback, and repeat, the more likely you will be successful.</p> </figure> There is certainly a place for some up-front planning, but you don’t want to get stuck there</a>. Build a minimal slice of your product</a> as soon as possible, even if it only has a few features. This can even be done with hardware – put down only the bare minimum parts on the PCB and build something. The rest can be added later. If it takes too long to build a PCB prototype, find ways to speed it up, or switch to another vendor. Automation and process is very important for fast iterations. CI (continuous integration) tells you quickly if your code fails tests. CD (continuous delivery) continually builds and deploys your work so that it can be tested in the complete system and evaluated by others. CAD and PLM automation</a> allows you to quickly package, track, and build new versions of PCBs.</p> A focus on iteration compensates for our many human weaknesses. It reveals our blind spots. It improves teamwork and project transparency. It helps avoid the scenario where the lone genius works for three months behind closed doors while the rest of the team is largely stalled. It ensures we are building what the customer actually needs. It encourages automation to avoid the tedium of manual testing and releases. It accelerates learning. Above all, it improves the efficiency of the entire process. It ensures we can quickly make changes and release them when needed, with confidence.</p> Owning Your Platform</h2> Owning your platform means you “own” and take responsibility for key parts of whatever it takes to build and maintain your products. It might be a printed circuit design, a build system, an application architecture, automation, infrastructure, process, etc. Owning your platform likewise is an investment for the future. It gives you more control and the ability to improve things. You are not locked into what another company provides at one snapshot in time. Certainly, there are limits to this and we cannot create an entire system from scratch so we all must use software and components that other companies create. However, we can build a system that has flexibility so that we can easily swap out one component for another. We can understand it well enough to test it. We can manage the build systems so that if something needs to be fixed or updated, we can do that. We build our device software when possible instead of relying on binary images that we can’t control. We are not afraid of developing our own automation and process software instead of relying entirely on outside vendors. Perhaps the most important thing is to own the integration points. This might include:</p> Top-level PCB (printed circuit board) designs (we can change electronic components and submodules as needed)</li> Software build systems (we can add, remove, and update software components)</li> PLM (project lifecycle management) (we write our own or use a system that can be easily extended through programming to meet our needs)</li> Core application architecture (can be adapted to specifically meet our needs)</li> Hosting (we can run our own servers or move applications between hosting providers as needs change)</li> CI/CD (can be adapted to any task/technology)</li> </ul> Integration is the most difficult (but also the most valuable) part of product development, so it is tempting to outsource integration. But this often locks you into one vendor and does not give you the flexibility to scale to higher volumes or additional products, as this running commentary</a> illustrates.</p> Ideally, you can reuse parts of your platform across many products, thus compounding your efforts over time. Even if your design files and source code are open-source, owning your platform (which includes so much more than design files) is what gives you the ability to efficiently design, test, maintain, and produce something. This is the hard part. If you are moving fast, a static snapshot of your product design at any point in time is of marginal value to competitors.</p> Leveraging Outside Improvements</h2> Leveraging outside improvements is another important activity. This can include upgrading to a new open-source software component, designing in a new integrated circuit, etc. It is easy to get stuck in the mindset that the “old” was “better.” In some aspects of life, this may be true, but with technology, things are generally getting better. Open-source software is generally more stable as time goes on due to better testing. Newer integrated circuits are more capable and lower cost. Many products today are largely an integration of many outside technologies, many of which are now open-source</a>. To improve, we must continually track and integrate these improvements. Developing using the mainline</a> (upstream) branch of open-source projects is a good strategy that companies are increasingly adopting</a>. Knowing when to adopt</a> new technology is a key skill.</p> If you use an outside development firm to develop parts of your product, it is critical that you receive a deliverable that is maintainable. Several suggestions for ensuring this include:</p> run the project like an open-source project. All activity is transparent and is committed to Git often.</li> own the integration points and testing.</li> use small iterations when possible. This ensures you are engaged and what is being built is what you need.</li> use open tools</a>/technology that everyone has easy access to where possible. Ensure that you can build or generate usable outputs from the design/source files. Even if you don’t have the expertise to use the tools, there is a better chance you can find someone else who can.</li> </ol> The Right People</h2> There is a saying often used by hikers: “Leave it better than you found it.” If we are working on a section of code that does not have tests, do we write them? If a manufacturing problem is found, do we improve the process to try to prevent the problem in the future, or just write it off? Do we give more responsibility and opportunity to everyone involved to improve things, or do we let a few mistakes restrict responsibility to only a few people?</p> For this to all work, you have to have people with integrity – who are genuinely interested in improvement and not just gaming the metrics. You have to have people with the discipline to write tests, which goes above and beyond just getting the job done and the ticket closed.</p> This all applies personally too … do you want to improve how you use your time? Then measure how you spend your time and share these measurements with someone. Use one of the many tools available to help track your time. Create your own tooling and process, and then improve it. Look for better ways to do all this.</p> Fundamentally, people don’t usually change much, so if someone is not interested in improving themselves, they likely will not be interested in improving the organizations they are a part of and be a drag on the process. So perhaps the most fundamental thing to learn in an interview is not only past accomplishments, but also if the candidate still has that burning desire to learn and improve themselves. It is a frustrating situation when a person who is wired for improvement is part of an organization that is not, or vice-versa. Both types of people and organizations exist.</p> Small Changes vs Large Changes</h2> We can make small or large changes. This might be the difference between an improvement vs. a pivot, slow and steady vs drama. The drama captures attention in the news – the new charismatic CEO brought in from the outside to shake things up. You typically don’t see great companies in the news, but you do see them in investment portfolios. In his book “Good to Great,”</a> Jim Collins studied several companies that produced exceptional results and concluded the following:</p> We keep looking for change in the wrong places, asking the wrong questions, and making the wrong assumptions. There’s even a tendency to blame Wall Street for the “instant results” approach to change. But the companies that made the jump from good to great did so using Wall Street’s own tough metric of success: a sustained leap in their stock-market performance. Wall Street turns out to be just another myth—an excuse for not doing what really works. The data doesn’t lie.</p> Now picture a huge, heavy flywheel. It’s a massive, metal disk mounted horizontally on an axle. It’s about 100 feet in diameter, 10 feet thick, and it weighs about 25 tons. That flywheel is your company. Your job is to get that flywheel to move as fast as possible, because momentum—mass times velocity—is what will generate superior economic results over time.</p> Right now, the flywheel is at a standstill. To get it moving, you make a tremendous effort. You push with all your might, and finally you get the flywheel to inch forward. After two or three days of sustained effort, you get the flywheel to complete one entire turn. You keep pushing, and the flywheel begins to move a bit faster. It takes a lot of work, but at last the flywheel makes a second rotation. You keep pushing steadily. It makes three turns, four turns, five, six. With each turn, it moves faster, and then—at some point, you can’’t say exactly when—you break through. The momentum of the heavy wheel kicks in your favor. It spins faster and faster, with its own weight propelling it. You aren’t pushing any harder, but the flywheel is accelerating, its momentum building, its speed increasing.</p> This is the Flywheel Effect</a>. It’s what it feels like when you’re inside a company that makes the transition from good to great. Take Kroger, for example. How do you get a company with more than 50,000 people to embrace a new strategy that will eventually change every aspect of every grocery store? You don’t. At least not with one big change program.</p> Instead, you put your shoulder to the flywheel. That’s what Jim Herring, the leader who initiated the transformation of Kroger, told us. He stayed away from change programs and motivational stunts. He and his team began turning the flywheel gradually, consistently—building tangible evidence that their plans made sense and would deliver results.</p> “We presented what we were doing in such a way that people saw our accomplishments,”Herring says. “We tried to bring our plans to successful conclusions step by step, so that the mass of people would gain confidence from the successes, not just the words.”</p> Think about it for one minute. Why do most overhyped change programs ultimately fail? Because they lack accountability, they fail to achieve credibility, and they have no authenticity. It’s the opposite of the Flywheel Effect</a>; it’s the Doom Loop.</p> Companies that fall into the Doom Loop genuinely want to effect change—but they lack the quiet discipline that produces the Flywheel Effect</a>. Instead, they launch change programs with huge fanfare, hoping to “enlist the troops.” They start down one path, only to change direction. After years of lurching back and forth, these companies discover that they’ve failed to build any sustained momentum. Instead of turning the flywheel, they’ve fallen into a Doom Loop: Disappointing results lead to reaction without understanding, which leads to a new direction—a new leader, a new program—which leads to no momentum, which leads to disappointing results. It’s a steady, downward spiral. Those who have experienced a Doom Loop know how it drains the spirit right out of a company.</p> https://www.jimcollins.com/article_topics/articles/good-to-great.html</a></p> </blockquote> Change is easiest to implement and absorb if it is incremental and not a drastic overhaul. Incremental change allows you to measure, adjust, and backtrack if needed. Incremental improvements can be implemented by anyone without permission and can then be evaluated and improved by others.</p> It is also important to understand the difference between foundational truths/values (things you should not quickly change) and inefficiencies that and ripe for improvement. The foundational principles of your business provide stability and need to be understood and valued.</p> Summary</h2> Do you want to be able to respond quickly to unexpected opportunities or issues that come up? The solution is simple – practice doing so now. If you regularly practice on the little things that by themselves may not really be worth the time to do, several things will happen:</p> you will be ready and practiced to make critical changes that will make a big difference such as fixing a critical security problem, responding to supply chain shortages, or making a change necessary to land a big sale.</li> the compounding effect of small improvements will eventually provide a sizable return in the future.</li> each small improvement is a small push on the flywheel of momentum. This adds positive energy to your organization and this momentum helps carry you through difficult times.</li> </ol> A culture of improvement is a long-term game. There will be some up-front pain as you develop your processes, but this is much easier than the long-term pain of accumulated technical debt which cripples your ability to do anything. You can compound the value from improvements or the cost of technical debt – take your choice.</p> ^1</sup>https://en.wikipedia.org/wiki/Kaizen</a></p> </div> Think Horses, not Zebras (Part 2) 2024-03-01T00:00:00+00:00 There is a popular quote</a> in medical circles:</p> When you hear hoofbeats, think of horses, not zebras. — Dr. Theodore Woodward</p> </blockquote> I recently posted several debugging experiences</a> where it was beneficial to examine simple scenarios before complex ones. Below are a few more …</p> NVMe slot not working</h2> An NVMe slot is fairly simple – it has a clock and one to four PCIe lanes. However, in a recent new hardware design, the slot was not working. We reviewed the kernel boot logs, scoped the clock signals, etc. We did notice the clock amplitude was lower than other designs (see below). The initial suspect was signal integrity as the PCIe data lines run at very high speeds, so we reviewed the routing, board stackup, impedance calculations, etc. Eventually, I compared the PCIe connector pinout to another design and it appeared that the RX and TX data signals were swapped. Sometimes RX vs TX (receive vs transmit) can be a little confusing based on which perspective the signals are referenced to (the host system or the target device). In the case of PCIe, it appears the signals are all named from the perspective of the host. After modifying to swap the RX and TX signals, the NVMe slot worked fine, even though the modification wires drastically violated signal integrity and impedance requirements. PCIe must be a fairly robust transport!</p> </figure> </figure> PCIe clock amplitude</h2> While verifying the hardware of a design, we noticed that the PCIe clock was at a lower amplitude than the specification required:</p> </figure> What we were measuring:</p> </figure> The Vcross was about 170mV, and the spec required it to be between 250 and 500mV. To verify our measurement technique, we measured the PCIe clock on another system and it it looked correct. The default output of the clock chip should have given us the correct amplitude. The clock chip was connected via an I2C bus, so we read back the registers to make sure they were set to the defaults. We then set the amplitude to the maximum the chip would output, and it still was not within range. Finally, we noticed the following in the datasheet:</p> </figure> Sure enough, we had terminations resistors for these signals on the PCB. After the resistors were removed, the clock amplitude was correct.</p> Cellular connection problems</h2> Cellular IoT systems can be tricky to troubleshoot. They are often installed in harsh environments (rain, cold, hot, damp, caustic sewer gasses, etc.). Occasionally a modem goes bad and connections fail. In one recent case, we replaced a modem and it still would not connect. The antenna, cable, and connections were the next suspect. After spending considerable time swapping components and not making any progress, I reviewed the configuration and noticed the APN was not set. After setting the cellular APN, it worked.</p> Zephyr Nanopb build error</h2> We recently updated the version of Zephyr we were using in a project. In the process, a protobuf generation process broke and another developer found a solution – nanopb was now an optional dependency that needed to be activated. However, the build still failed in my workspace. We discussed diffing our build workspace, but in the back of my mind, I was thinking – there has to be a simple reason. While reading the documentation, it occurred to me to check the west</code> version – sure enough, I was running 1.0 and the latest was 1.2. After updating, it worked.</p> What can we learn?</h2> Again, in the above cases, the initial assumptions were more complex than the actual problem. Suggestions from the previous article</a> are relevant here:</p> Double-check connector and component pinouts.</li> Compare to working designs.</li> Re-read the entire datasheet carefully.</li> </ul> Additional suggestions learned from these examples include:</p> Verify the configuration of the system is correct, never assume it is.</li> Make sure you have the correct version of the tools installed (ideally your build system would check this for you).</li> Re-read documentation for tools and software components. This will often spark ideas on where to look for problems.</li> </ul> Why Open Source for Product Development 2023-12-01T00:00:00+00:00 (this topic is also discussed in a podcast episode</a>)</p> One of the paradoxes in product development is Open Source. How can you personally or as a company benefit from participating in Open Source projects? Why should you share your great ideas and code? How can you build a business or a career by giving things away? How can an open-source project be a reliable supplier without contractual guarantees? Most successful technology companies today participate in open-source projects. Why do they do this?</p> Open source is difficult to fully understand and much has been said about it. There are many approaches to licensing and funding. Many question the equity and sustainability. Many get bogged down in ethical, moral, and political arguments. There are messy problems that don’t seem to have good answers. But we need to go deeper and examine the core issues of why companies are turning to open-source.</p> We will consider several product development perspectives:</p> Our motivation is to produce something of value</strong> that society can use and benefit from.</li> Software technology is advancing at a rapid rate of change</strong>.</li> Technology in modern systems is increasingly complex</strong>.</li> How can we maintain some level of control</strong> over the resources we use to ensure they will meet our needs?</li> And how does organizational culture</strong> impact our ability to deal with these issues and build modern systems?</li> </ul> </figure> Value</h2> The value of software is hard to quantify, and breakthroughs are difficult to predict. New software algorithms, methods, languages, and structures are sometimes much more effective than previous ones. It is these non-linear breakthroughs that we must leverage. Software is an unbounded sphere of opportunity, free from the constraints of physical locality, physical devices, duplication costs, and, in many cases, computing resources. Modern computers, even those we hold in our hands, are capable of running more software than we can write in multiple lifetimes and storing a lot of information. Systems that improve human efficiency and capability potentially add value and wealth to society. The best software can do more with simpler expressions and less effort. The best software can scale to larger problems. The best software is easier to maintain and thus the rate at which value is added over the long term increases. The best software is more reliable and can be easily deployed without worrying about it crashing or malfunctioning.</p> Complexity</h2> We should clarify our use of the term complexity – we are talking about inherent and necessary complexity, not artificial, human-created complexity (some call this accidental complexity</a>). The human body is incredibly complex, but this is necessary for it to do all the things it does. Modern microprocessors are complex, but they need to be in order to do the amount of processing they do. The Linux kernel is complex, but only because this is required. In modern systems, software is potentially orders of magnitude more complex than hardware. We might spend X hours designing the hardware and XY hours writing and maintaining the software over the lifecycle of the product. This is hard to comprehend because the hardware can be seen, but the software is largely invisible. Software is increasingly the driver of value in modern systems, whether the systems are products or processes.</p> Change</h2> The development of hardware versus software requires a different mindset and process. With hardware, the effort is mostly in the initial development as we design, specify, and integrate the components needed. Once designed or qualified, a hardware component can be used for 5 or 10 years with very few changes. Its value is driven by minimizing variance (change), which is how we define hardware quality. Software systems are completely different – the inherent value in software is that it can be changed</strong> over a product lifecycle and its value subsequently increases, so the value in software comes from maximizing (instead of minimizing) changes. Software in a product may be updated hundreds or even thousands of times over its life. This is why it is so difficult for a company that is really good at hardware to also be good at software, or vice versa – a different culture and mindset are required.</p> Control</h2> How can we secure and have some level of control over the critical software components we need to build products? First, we should clarify the scope of the problem. Modern systems are increasingly assembled from hardware and software components we did not create ourselves. In a Linux-based system, there are many millions of lines of source code. The few thousands of lines of custom application code we might write for a product, which may represent many man-years of effort, are only a drop in the bucket compared to the rest of the software running in the system. However, it is easy to ignore the “rest” of the software, as it is largely invisible. How you manage the “rest” is critically important and perhaps the hardest part. And this “rest” is increasingly being supplied by open-source projects. Many engineers are good at creating things from scratch, but fewer are good at understanding, modifying, debugging, and managing software technology that others create. This skill is increasingly valuable.</p> Ideally, we would create everything we need and not be dependent on anyone else. However, we won’t get very far on our own if we need to create our own microprocessors, operating systems, programming languages, databases, and the myriad of other components needed to build systems. Likewise, these components by themselves are of little value unless they are combined into a larger system to do something useful. No one person, company, or even nation can build and maintain all the technology building blocks required to build modern systems. Human success is largely dependent on the degree to which we are able to work with others. The scope of the Linux kernel is beyond even the largest companies or governments. Even if an organization had the manpower, it would likely not have the culture required to pull off a project like the Linux kernel, the PostgreSQL database, or many other useful OSS projects. For smaller organizations, the capability to build even the most basic building blocks is often beyond our reach.</p> Culture</h2> Thus far we have established that to create value</strong> with modern systems, we must be able to build complex</strong> systems using software technology, much of which we did not create, and be able to change</strong> this software over time to deliver more value. In all this, we must have some level of control</strong> over this software. If there are problems, can we fix them? Can improvements be implemented and new features added? Secondly, can we influence development over time if we are not doing it ourselves? When we source software technology, there are many options, but let’s reduce it to two for the purpose of discussion:</p> Find a vendor with a proprietary software product. Initially, it fits our requirements – at least during the sales cycle. As changes/fixes are needed in the future, this vendor is hired to provide these updates or we work around them.</li> Find an open-source software project. We may purchase support from a company supporting this project or we may make needed changes ourselves.</li> </ol> There are no doubt times when #1 is the best solution, but increasingly #2 is being chosen. Why is this?</p> Let’s again review the challenges with software development:</p> Obtaining the software we need (value</strong>), as we only have the resources to write a tiny fraction of it.</li> Making changes</strong> over the product lifecycle, as this is where value is added.</li> Having some level of control</strong> and influence over the software we use.</li> Managing the extreme complexity</strong> of modern systems.</li> Providing an environment and finding the people where software innovation can happen (culture</strong>).</li> </ol> As we review these challenges, we start to realize that the primary benefits of open-source software are cultural. Software innovation is not created with a manufacturing-type mentality but is more art than science – a process of discovery, experimentation, collaboration, inspiration, and simplification. Developing great algorithms and code is a creative process that cannot be easily planned or scheduled, other than allocating the resources and time to work on it. The development of software innovations is not always predictable and cannot be forced. It requires the rare combination of the right people, the right environment, the right level of collaboration, and the freedom to experiment and innovate. All these conditions rarely exist inside corporations. Companies are realizing they cannot always hire the people needed or provide the environment for the level of innovation required. So the next best thing is to support open source projects where this innovation is happening. There are two ways to do this: time and funding.</p> Many authors of innovative software projects are driven by the craft itself, not wealth. Funding is necessary, but they are not working on a project to become rich – there are much easier ways to make money. They are driven by the intrinsic challenge, the joy of discovery, and the reward of building something useful. The first step in making sure an open-source project will be maintained and available in the future is to ensure that the authors have the funding they need. There will never be perfect equity in that authors will receive exact compensation proportional to the value they create, but this matters little to most OSS authors</a>, and in reality, this is also the case with most developers employed by companies. Many developers are more than satisfied with earning a good living and having the freedom to work on whatever they choose. Most are generous and glad to see their work being used. However, there are cases where OSS authors do not receive funding comparable to what they could earn working elsewhere, so it would bewise for companies to audit all the OSS components they are dependent on and ensure that these projects are adequately funded</strong>. Contributing financially also makes it more likely the authors of the project will respond to the company’s needs, but we need to be careful that this process does not become restrictive to the point where the environment for innovation is destroyed.</p> Contributing time (code, documentation, support, feedback, etc.) to open-source projects is another way to help ensure a project will be useful and available to companies needing it. Contributing to the community around an open-source project is critical for the project’s health and is part of the flywheel of innovation. If someone is a valuable contributor, then they will also have some influence on the direction and priorities of the project. They will become skilled at understanding and using the project. They will also become better programmers, as they work with some of the best in the world, and are exposed to well-written software. Open source projects provide an environment for innovation that is often not possible inside a company, so having your engineers do some work in OSS projects provides them with an alternative environment to do software development.</p> Why Open-Source Works</h2> It is somewhat of a puzzle why the open-source environment works so well for software innovation. Transparency is a powerful tool for great work. When you do things in public, you do them better. In OSS projects, there are no barriers to working with the best people in the world. Freedom is likely another ingredient. When there is the latitude to experiment, it is more likely that new and better methods will be discovered. Formal software development methodologies are good at managing more predictable efforts to solve business problems, but they seem to be of less value in OSS projects at driving innovation. Each successful OSS project is different than the previous one and is a unique combination of various factors that cannot be easily replicated.</p> Many people still stumble at the fact that if my company contributes to an open-source project, we are helping our competitors. This is true. However, if we don’t participate in open-source projects, we likely won’t get to where we need to go. It may no longer be optional. Open source is not like walking into a discount store and picking something off the shelf at a tremendously good deal. It is not a one-time decision where we select something and are done. It is an ongoing process of continual change and improvement. And the better we are integrated into this process, the more we can leverage and use open source. Economic opportunity is not a fixed-sized pie, where we are all competing for the largest slice. With software, the duplication costs are near zero, so this is a perfect opportunity for economic growth or an overall increase in society’s wealth. We can work to grow the size of the pie, so everyone can have a larger slice.</p> Often in writing, the author learns and understands things he did not know before, and this essay is no exception. The benefits of open source are not primarily in getting stuff for free or pooling resources in a socialist sense for the greater common good, but rather in providing an environment and connecting the right people for innovation to happen.This dynamic has happened occasionally in the past inside companies such as Bell Labs</a> and Lockheed Martin’s Skunk Works</a>. But this environment is obviously difficult to replicate or more companies would do so. Even the largest and most successful technology companies, such as Google and Facebook, aggressively support and promote open source. At the time of this writing, Google employs over 27,000 engineers – why would they need open-source projects? With that number of developers, why can’t they create anything they would ever need? Google’s support for OSS projects is not an altruistic gesture, but rather a calculated business decision – they are supporting the processes and people that will produce the software technology they need. This is a long-term game that requires experience, skill, understanding, and patience. It is an example of humility, which is the source of all true greatness – the realization that there are some things I cannot do, even if I have huge financial resources. It is the realization that there are people out there who are much better at certain tasks than anyone in my company or anyone I can hire. It is the realization that my company organization is not the perfect environment for the development of needed software breakthroughs and highly refined building blocks, so we need to support projects that are. Because it is counterintuitive, many companies will not make this leap, especially those who have been successful in the past without it, or those building products that in the past did not require advanced software systems. Ironically, it is often the smaller companies that need OSS components the most that are resistant to adopting and contributing to OSS projects. However, the opportunity is available for anyone to participate, support, influence, and use the most advanced software innovations available today. And these innovations are largely happening in OSS projects. Your engineers can work with some of the most gifted developers in the world and learn skills that will greatly enhance their careers and benefit your organization. You will be plugged into an engine of innovation that produces value. You will have more control over the needed building blocks. You can be part of the process that produces the valuable software you need, rather than just a consumer.</p> Open-source projects are far from perfect. They are messy at times, as human efforts tend to be. Their future is not guaranteed, but nothing in this world is guaranteed. The lack of formal business contracts/methods is perplexing to most business people. However, open source appears to be the best model we’ve discovered thus far for software innovation, and the results are amazing. Open source is the only supply chain that can deliver the needed value</strong>, innovate and change</strong>, manage the increasing complexity</strong>, give you some amount of control</strong>, and provide the culture</strong> necessary for innovation. Will you be a part of it?</p> Think Horses, not Zebras 2023-09-22T00:00:00+00:00 There is a popular quote</a> in medical circles:</p> When you hear hoofbeats, think of horses, not zebras. – Dr. Theodore Woodward</p> </blockquote> “Zebra” is the American medical slang for arriving at a surprising, often exotic, medical diagnosis when a more commonplace explanation is more likely.1</a></sup> What does this have to do with product development? Like the medical profession, we often diagnose problems – we call it debugging</a>. Below are three recent cases where I would have been helped by applying this approach a little more rigorously.</p> Simple IoT Startup</h2> I recently encountered a bug</a> where the compiled version of Simple IoT would not work on one of my computers. Interestingly, go run</code> worked fine. What was the difference?</p> At first, it seemed that a race condition was at play – the compiled version was faster, did not have the race detector enabled, and would likely have different characteristics. SIOT is a highly concurrent application with many clients running in parallel. However, after spending some time debugging, I could not find any issues with startup, rather the NATS client in SIOT could not get data from the embedded NATS server. Again, race conditions were assumed, so I tried an external vs. embedded NATS server – no difference.</p> I finally observed the network traffic with Wireshark and could not observe any NATS traffic in the failing version. The NATS client was not even sending any requests. I then looked at /etc/hosts</code> and there was no localhost entry. After adding this, everything worked properly. See this discussion</a> for more analysis of this issue.</p> In this case, a race condition was the zebra and the missing entry in /etc/hosts</code> was the horse.</p> i.MX8 Stability</h2> We recently ran into a problem with a new product that uses a Variscite VAR-SOM-MX8</a> in a custom baseboard. The software was stable on the Variscite development baseboard, but the same SOM (System on Module) would crash in the custom hardware. The system would never crash until after the Yoe updater initramfs ran and the main rootfs was loaded. Hardware was the immediate suspect – the power supply is probably not able to handle the dynamic load when multiple cores spin up, etc. However, the power looked stable and we could not find any hardware issues. In the end, we found a missing device tree line ("fsl,imx8qm")</code> in the Linux kernel DTS file for the new design. This missing construct caused some critical kernel initialization code not to run.</p> In this instance, a subtle power problem was the zebra and a missing line in the DTS file was the horse.</p> STM32H7 USB Instability</h2> In a recent STM32H7 design, we wanted to use a High-speed USB Phy with the STM32H7 and Zephyr. This is not supported by Zephyr yet, but with a few hacks</a>, we got it working. We are trying to push a fair amount of data over this link (10’s Mb/sec) so some optimization and performance work was required on both the sending and receiving end. In the mix of all this, we got a new revision of the target hardware, and at some point started seeing an increase in error rates and the target not enumerating and connecting properly to the host. With a lot of moving parts and a lot of data flowing through the system, it is sometimes hard to pinpoint the problem. In this situation, we tried to narrow the scope by testing older/newer versions of Zephyr and comparing STM32H7 reference designs to the customer hardware. We found the reference designs performed much better. We also learned that a previous version of the custom hardware also performed like the reference designs, so it appeared there was a hardware problem with the latest custom hardware.</p> When something worked before, and now is experiencing high levels of communication errors, the first thought was signal integrity. On the PCB design, there is a fairly long trace from the USB Phy to the connector, so this was an obvious suspect. Although we were careful to design the PCB such that the impedance of these traces was controlled, there is always the possibility that the PCB vendor did not implement the board stack-up correctly or some other strange issue. One of the engineers analyzed the trace impedance on the old and new boards with a network analyzer. This is a fairly involved test procedure that requires cutting traces, soldering coax cable to the PCB, and a fair amount of expertise. The USB traces between the two revisions looked equivalent and the impedance measured was correct. The next suspect was the high-speed signals between the MCU and Phy – we had changed some of the length matching between the two versions, so perhaps we messed something up. In the process of looking at these signals, the following was observed when scoping the data line:</p> </figure> The data line was running at two different voltage levels. The problem ended up being the power to the VDDIO pin on the phy was floating, so the signals from the phy to the MCU were not being driven at the right voltage levels. Simply scoping all the power pins on the Phy would have found this problem in a few minutes.</p> In this instance, signal integrity was the zebra, and a floating power pin was the horse.</p> What can we learn?</h2> In all of the above cases, the initial assumptions about the cause of the problem were considerably more complex than the actual problem turned out to be.</p> Problem</strong></td> Zebra (complex)</strong></td> Horse (simple)</strong></td></tr> SIOT Startup</td> Race conditions</td> Host networking config</td></tr> i.MX8 Stability</td> Dynamic power supply problem</td> Missing line in device tree file</td></tr> SMT32H7 USB errors</td> Signal integrity</td> Floating Phy IO power</td></tr></tbody></table></figure> Hindsight is always 20/20, so it’s easy to say we should have solved all these problems more quickly than we did. But there is always something to be learned, and I think as humans, our minds generally go to the more exotic scenario. Part of this may be overconfidence in our ability to get the simple things right, like copying a DTS file from a reference design, or supplying power to a chip. How can we possibly mess up something so simple? So we tend to overlook the simple and obvious and jump to the obscure and exotic, which exercises our most advanced abilities and knowledge. The more experienced we are, the more likely we may be to think this way. To overcome these human tendencies, it is helpful to acknowledge first that they exist (humility), and then put in place a process for future endeavors to help guide us in our debugging efforts. Consider the following maxim:</p> When debugging a problem, always verify and eliminate the simple/obvious possibilities first. Work from simple to complex.</strong></p> This is a pragmatic approach because verifying the simple possibilities usually does not take much time, so little is lost in starting there. It may be boring or tedious, which is why we subconsciously tend to avoid these, but we can save a lot of time if we just follow this simple rule. A general process might be:</p> isolate the problem as much as possible (remove unused functionality, etc.)</li> write down everything you know, possibilities (all of them), and ideas of what you might try.</li> sort your ideas from simple to complex and work on the simple ones first.</li> document all tests and results.</li> meet as a team regularly to discuss results and brainstorm the next steps.</li> </ol> Some specific ideas:</p> verify basic network connectivity any time a network is involved, even if on the same machine.</li> don’t ignore any problem in the system. Often problems are related, and solving one problem sometimes solves another.</li> check that all components involved have good power.</li> compare to a known working reference design (both operation and design).</li> ask for help in relevant communities.</li> review and verify component pin-outs.</li> read the datasheets of the components involved.</li> read the errata for components.</li> work backward from a known working setup.</li> with web apps, open the console, and look for errors.</li> carefully review boot and build logs and diff with logs from working systems.</li> narrow the scope of the problem wherever possible.</li> try multiple systems.</li> carefully review any changes between working and non-working versions. Verify the operation of any new circuits or code.</li> don’t keep doing the same thing and expect different results.</li> </ul> Debugging is an art and a part of the development of any complex system. We must realize that our intuition often fails us in these situations and, with any human weakness, we must have a process to compensate. This debugging process can turn what is often a frustrating and scattered experience into something much more enjoyable and productive. Think horses first, and then zebras!</p> ^{1</sup> https://en.wikipedia.org/wiki/Zebra_(medicine)</a></p> </div> The Technology Trajectory 2023-01-03T00:00:00+00:00 See also a podcast episode</a> where we discuss this topic.</em></p> Modern systems require the integration of technology, often from many sources. This can come in the form of processors, integrated circuits, operating systems, open-source software components, databases, messaging systems, cloud infrastructure, programming languages, build systems, frameworks, etc. All technology follows a trajectory – an example is shown below. The timeline may be short or long. Your usability threshold may be low or high, sloped right or left, etc. But the fact remains that most technology will eventually be replaced by something newer. We can debate whether or not newer is better, but the fact remains that time marches on.</p> </figure> Why can’t we use a particular technology forever? With hardware, components may not be available. With software, we potentially have a little more control, but if we want to use a software component beyond the above cycle, then we have to commit to taking over maintenance. This means:</p> if there are bugs, we will fix them</li> if there are security problems, we will fix them</li> if there are build problems with new versions of compilers and other libs, we will fix them</li> </ul> For complex, networked projects, this can be a significant undertaking.</p> But a far bigger issue is that of value. Open source software and corresponding ecosystems are continually improved over their life and value is added. If you use projects in your development that are being improved, then value is being added to your product. The rate at which value is being added correlates to the slope of the curve above.</p> To extract the most value from technology, it is generally best to adopt technology when the trajectory is rising and there is project momentum, not when everyone else has moved on. There are several reasons for this:</p> the rate at which value is being added is greatest during this period. This makes what we are building better.</li> you have an opportunity to participate in and influence the development</li> you have more time to benefit from the technology</li> more support is available as there are more people participating in the project</li> in programming languages, there are more maintained 3rd party packages available</li> </ul> We also need to select technologies that have a reasonably long life cycle as switching out a complex component can be a lot of work.</p> There are many other perspectives by which to evaluate technology – this is just one of them. Technology follows a trajectory from inception, growth, maturity, decline, and finally obsolescence. How do you know where a particular technology is on the curve? It is probably partly gut feeling, part experience, part asking others, and perhaps there are analytical ways to approach this as well (search engine ranking, etc). But the most important thing is to use this concept as a framework for objective thinking and evaluation. Effective technology selection is a long-term effort. We need to have a process. We may have to watch a project for a year or more to get a good feel for it. Most of us can’t significantly change the trajectory of most technologies, but we can choose where on the curve we will participate. Inefficiencies occur early and late on the curve. If we are always too early, we likely won’t get much done. If we are always too late, then our products will be a few steps behind everyone else’s.</p> Go for IIoT Systems 2022-12-09T00:00:00+00:00 As developers, we have many options for programming languages. On one hand, it is great to have choices. On the other hand, it can be a little overwhelming. Many times in life, we need to make decisions before we have the experience to know the best answer. There are many opinions as to what is best, and this article is just another one. There is no one-size-fits-all – a lot depends on your team and the task. In this article, we will examine the challenges we face in developing industrial IoT (IIoT) systems, review desirable attributes in a programming language, and discuss how Go meets these needs.</p> Peter Bourgon defines the industrial programming context</a> as:</p> in a startup or corporate environment;</li> within a team where engineers come and go;</li> on code that outlives any single engineer; and</li> serving highly mutable business requirements.</li> </ul> Additionally, an industrial IoT system can be thought of as a system that gives users access and control of remote systems. There are many variations of this, but let’s focus on a simple case that is typically composed of the following pieces: Edge device (remote equipment, sensors, etc), Cloud, and Browser.</p> IoT Components</figcaption></figure> All of these pieces must be dealt with at some level and will require some programming. The fundamental requirement of this system is networking – data must be moved between the above three instances. IIoT systems are inherently distributed systems. Distributed systems are hard</a>, thus the distributed problem needs to be solved well if you want to have a scalable, reliable IIoT system. It is natural to focus on the end application at the edge and add IIoT as an afterthought. However, this rarely works very well.</p> The context of this article is that of an average programmer working in small teams where engineers do a lot of different things. Many of us are not full-time programmers doing deep theoretical work, but find ourselves building these complex systems. Some of the challenges/requirements we face include:</p> resources are constrained at the edge (CPU, Disk space, etc)</li> networks can be unreliable</li> config/state needs to exist in multiple places (edge & cloud)</li> software updates need to be deployed to remote places (edge & cloud)</li> end-to-end (edge <-> browser) real-time response is often desired</li> many parallel operations are happening (collecting data, running rules, managing the system, etc)</li> physical access to edge systems so debugging problems can be difficult</li> edge systems typically use Embedded Linux, which can be a challenge to develop and maintain</li> IoT systems are inherently concurrent and distributed</li> many of us are doing this part-time, are not computer science gurus, and need pragmatic solutions.</li> </ul> Attributes of Go</h2> Go has several attributes that make it a good fit for this problem:</p> excellent reliability. I’ve written about this before</a>. (our systems are not crashing in the field)</em></li> it is a compiled language (reasonably fast and small)</em></li> runtime is small, embedded, and efficient (this means there is very little overhead to distributing the entire runtime with each binary. This makes deployment easy.)</em></li> statically linked (this makes it much easier to deploy updates as everything you need is embedded in one binary file)</em></li> easy to embed assets (again, this makes it easy to deploy in one file without containers)</em></li> is garbage collected, so we don’t have to worry about managing memory (we don’t have memory leaks, invalid pointers, and other problems that can cause field failures)</em></li> typed language (compiler catches more problems early instead of having crashes in the field)</em></li> popular and well-supported (will continue to improve and be around for a long time)</em></li> excellent concurrency support (a lot is going on in these systems, so this helps)</em></li> popular in cloud and distributed systems (there are plenty of stdlib features and 3rd party packages – metrics, monitoring, logging, communication, etc to help us build an IoT system)</em></li> language is simple (easy to learn, read, and maintain)</em></li> tooling is excellent, compile is blazing fast, no Makefiles, etc, built-in support for profiling, and easy to cross-compile for other systems (we spend less time messing around and more time deploying value to our customers)</em></li> </ul> Go is not the best solution in every case, but if your situation matches what is described above, it might be a good fit.</p> Go removes a lot of complexity</h2> Cloud and edge systems typically run Linux. You may outsource parts of the cloud portion and perhaps even parts of the edge device, but you are generally responsible for the edge device. It must run 24/7. You likely need to deploy updates, maintain security, and add functionality over the product lifecycle to meet customer needs. You will run into issues and need to be able to debug them. You can’t walk over to an edge or cloud system and press the reset button or hook up your JTAG debugger. Most embedded software in the past has been developed for MCUs (microcontrollers) and written in C/C++. This is a fine solution for disconnected control-focused products, but a Linux MPU (microprocessor) solution provides you with more flexibility and functionality for data-focused connected devices that process/store a lot of data and have complex user interfaces. Linux provides unparalleled support for a large number of devices, USB and networking functionality is reliable and mature, and it scales to processing and storing large amounts of data. In an MCU platform, you may write most of the software in the system. In some cases, you may use an RTOS (Real-time operating system), but even these are relatively small as resources are limited. In a Linux system, you are running millions of lines of code you did not write. We can use the iceberg analogy – the part underwater includes Linux and system libraries which you don’t see most of the time. The part you see above water is your application, which is a tiny portion of the total software in the system.</p> </figure> Part of the value of modern systems is the huge number of software components you can reuse in your system as there are software libraries/packages for many tasks. Building modern systems is partly the creative combination of existing/reusable technology and the creation of new technology (the code you write).</p> </figure> So the question arises: How am I going to get up to speed on all this?</strong> One solution is to minimize the amount you need to interact with the stuff underwater. If you build applications with C++, the compiler does not give you a lot of help in linking to libraries and cross-compiling to edge system architectures. (Cross compiling is compiling native code on one architecture for another – typically developing on an x86 workstation and then running on an ARM Linux target system). Thus, you need to rely on complex build systems (CMake, Yocto, Buildroot, etc). Every time you add/update a library/package, you are faced with the following questions:</p> How do I integrate this library into my app build?</li> How do I cross-compile to my edge system?</li> </ul> It’s not easy, thus projects such as Yocto and Buildroot exist. All developers need to have a cross tool-chain (compiler and libraries) installed on their development computer that matches the libraries in the edge system. Deploying an application update often means deploying additional libraries, or library updates. So we add container technology to try to manage all this, which is another layer of complexity.</p> Some solutions (like Raspbian for the Raspberry PI) advocate developing directly on the target device to avoid the complexity of cross-compiling. This works fine for small one-off maker projects, but for the development of large industrial programs, it is a poor solution as it is like developing on a PC from the 90s. Sure, cross-compiling is easier, but everything else is much harder.</p> Python or Node.js may look attractive in that they are easier to program than C++ and the program is interpreted on the target device, but many of the underlying packages for these languages depend on C++ code, as interpreted languages are slow, as a result the build problem is worse because it is even more difficult to cross-compile this code in the Python and Node.js build systems. Additionally, a separate run-time is required on the edge system which is relatively large and must be kept in sync with the application.</p> Go sidesteps most of this because it is a mostly pure ecosystem. There are no host library dependencies in a pure Go application – everything it needs is embedded in a single binary file (typically 2-10MB compressed). The Go standard library is rich and has many needed functions. Additionally, there is a rich selection of pure Go 3rd party packages</a> that cover most other needs. Adding packages to a Go application is as simple as go get some/package/name</code>. There are no development tool requirements other than a standard Go installation and an editor. No complex build tools are required to add 3rd party packages or to cross-compile to different target systems (macOS, Windows, Linux). It has built-in support for profiling. If it crashes, you always get a stack trace. It’s so easy, and the learning curve is small.</p> Many software tools today add layer upon layer to solve the problems with the underlying technology. While this helps in some cases, it often adds additional complexity. You can’t gloss over complexity – when things go wrong, you still need to understand the entire stack anyway. Abstraction has its place but can only take you so far. If there are problems in one technology layer, adding additional layers sometimes just makes it worse. Go threw everything out and started over</a>. Simplicity is the only sustainable approach to technology.</p> Go is a relatively new language (released in 2009), so many people have not heard of it. Learning a new language does take some time. Go has a small learning curve, but that is nothing compared to the learning curve of building, deploying, and maintaining C++ code in edge systems. Go allows you to quickly become productive and spend very little time fighting tools over the product lifecycle. Moving your Go application forward over time often does not require any changes to the base Linux system because they are decoupled.</p> Go is familiar and simple</h2> Google originally developed Go</a> as a solution to the pain they were experiencing with C++, and they are still investing in it</a>. Go syntax is similar to C and was developed by some of the same people</a> who were involved in the development of the C language and Unix back in the 1970s. Thus, it feels very familiar to those of us who have spent time developing in C, Java, C#, etc. Go eschews the complexity that has overtaken C++. Language features are added slowly and carefully. Compile errors are generally easy to understand and debug. Backward compatibility has been excellent. Since 2014, I’ve implemented several systems in Go and have not had any significant issues after updating the Go or 3rd party package versions. Go developers take API compatibility seriously, so this helps.</p> Go is a simple language but does not force you to do things correctly (as a language like Rust does). You can still make a concurrency mess with channels. Go has nil</code> values, so you can still crash your program if you try to reference a nil</code>. If you don’t check returned error values, you’ll probably run into problems. But, if you follow a few simple best practices and idioms, it works well and is a nice experience. Simple programming languages are much easier to read and maintain.</p> Go Concurrency Support</h2> IIoT systems have a lot going on, so you need good support for concurrency. Interpreted languages like Python and Node.js are weak in these areas. Go has excellent support for concurrency by providing channels and the go</code> keyword as language features. This allows for running concurrent tasks and communication between them with very little overhead. Go channels are easy to misuse, so it requires a little effort and practice to learn how to use them correctly, but once understood, it is a very simple and reliable way to manage concurrency in a system.</p> Go is an Internet language</h2> While older languages like C/C++ and Python have been excellent tools for solving programming problems in the past (and are still the best in some domains like MCU development, machine learning, etc), the problems we are solving in IIoT systems are different today in several ways:</p> we are dependent on large ecosystems of 3rd party packages to implement all the functionality we need</li> we need better tools for concurrency and distributed systems</li> we need simpler models for network programming</li> we need software that is easier to build and deploy</li> </ul> Communication between systems is the fundamental problem IoT systems solve and Go is good at this. Go is the best-in-class for a new breed of software: an Internet programming language</a>. As evidence of this, most cloud and distributed software infrastructure is written in Go these days (Kubernetes, Docker, NATS, etc). Cloud companies that scale to thousands of systems are very sensitive to cost, thus they need efficient platforms. This efficiency in Go also benefits us at the edge where resources are constrained. An added benefit is that you can write your edge and cloud applications in the same programming language.</p> But Go has no GUI support …</h2> Although there have been many excellent efforts to develop a native GUI library for Go, none of them have emerged as the standard. There is a good reason for this … the browser is the standard UI platform these days</strong>. Nothing else can come close to touching its capabilities. It is cross-platform, reasonably performant, and very well-supported. The browser has so many options for programming, packages, etc. The browser has built-in support for video and other forms of media and is highly optimized. Developing applications in the browser is more difficult to get started than in a C++ GUI toolkit. But there is a much richer ecosystem of packages and functionality available in the browser platform. The browser platform is stable and you are ensured of backward compatibility – it will be around and well-supported for a long time. The web platform is built on solid engineering principles that have proven to scale. Many UI technologies designed to replace the browser programming model have come and gone: Java, Macromedia Flash, and Microsoft .net – yet the standard browser programming model of HTML5 + Javascript has displaced all of these and is growing at an astounding rate. Google Flutter</a> is the latest attempt to displace the web platform, but it remains to be seen yet if it will be successful. Most C++ GUI toolkits are highly dependent on a single company that may be here today and gone tomorrow (look at the history of Qt</a> for an example), or licensing requirements may change</a> at any time – there are a lot of unknowns. Plus, with the web platform, you can view the UI remotely just as well as locally. You can also use the same UI building blocks in the cloud as in the edge device. If designed correctly, web applications also function very well on mobile devices, eliminating the need to develop separate mobile apps. With the web platform you can address all of the following:</p> on-device LCD UI (runs a browser full screen)</li> remote access to the device using a browser</li> cloud application UI</li> mobile device access (browser, PWA, Cordova, etc)</li> </ul> The experience of a native phone app might be slightly better than a web application on mobile devices, but how many of us can afford to develop 4 different UI applications for a product (native, web, Android, iOS) when one will do in most cases? The question is not what is possible or technically best, but what is practical with the limited resources we have.</p> Languages like Elm</a> simplify front-end (web) development in many of the same ways that Go has simplified back-end development.</p> But Go is not the best at …</h2> Very true – Go is not as easy as Python, as widespread as C++, as popular as Javascript, as safe as Rust, or as good at concurrency as Elixir. But, Go is almost as good as the best at everything and its tooling is much better than any other language ecosystem – especially for distributed applications that need to be deployed to remote and different architectures. It does not have any of the significant drawbacks other languages have. And especially for networked IoT systems, it is an excellent choice. The below decision matrix likely reflects my biases, but illustrates this point (green is best, red is worst):</p> </figure> But what if I need to use C++ …</h2> There are certainly cases where you may need to use C++ or some other language. You may choose to implement a native UI using Qt or LVGL. Machine learning code is often written in C++. There is no reason your system needs to be written in one language – Go can be used in combination with other languages. There are several ways this can be implemented:</p> call the C++ code directly from Go (CGO)</li> run the C++ and Go code as separate applications and communicate between them using a message bus</li> </ol> Method #2 is preferred in many cases as it keeps your Go code clean, and the C++ code can be built using standard C++ build systems instead of trying to force the Go build system to build C++. Simple IoT</a> is one framework that is designed for exactly this scenario:</p> </figure> This architecture keeps your Go code easy to build/develop/deploy. Your core logic, which manages config/state and communicates with other systems, can be written in Go. This core logic must be reliable and easy to maintain. Specialty code like machine learning models can be dedicated C++ applications. NATS is a modern message bus written in Go that is easy to embed in any Go program and is a great way to connect different applications.</p> </figure> Optimize for where you will spend most of your time</h2> Linux-based IoT edge systems have plenty of memory, storage, and processing power. Thus, features are continually added over the product lifecycle, which can be 5-10 years in an industrial system. The time spent maintaining and improving a product often swamps the initial development. Therefore, it makes sense to optimize for the long term – adding features and deploying updates over the next ten years. Go makes it easy to build and deploy updates with the latest Go and package versions.</p> Friction</h2> Developer productivity is difficult to fully understand. We often think of the features we need in our languages/tools. Commercial software vendors focus on features. Development methodologies are designed to increase productivity by better organizing what needs to be done. But another aspect that is easy to overlook is the amount of time we spend messing around working on stuff that does not directly add value to our product. I call this “friction.” This friction can manifest itself in unforeseen problems that are more common in complex systems: learning curves, build issues, issues with libraries/packages, lack of flexibility, security problems, deployment problems, bugs, race conditions, stability problems, technical debt, lack of control over parts of the system, etc. Go does an excellent job of reducing friction in the development process. You focus on what needs to be done, and the rest is mostly invisible.</p> Summary</h2> In this article, we explored many of the attributes of Go that make it ideal for implementing IoT systems. Go provides a modern programming ecosystem where the build tools are first class, the language is simple, and there is little friction to get things done. Go also provides features such as garbage collection and concurrency support in the language which are very helpful in IoT systems where data is distributed and a lot is going on. Go is an Internet language that is increasingly used to develop distributed systems, which includes IoT systems. It is a stable (almost boring in many respects) technology and is a great option for developers who are familiar with C/C++ and want to expand into Embedded Linux/IoT – the skills you have are very applicable in Go.</p> What is the main constraint in your projects?</p> Application performance?</li> Getting things done and shipped?</li> </ol> If #1, then perhaps C++ or Rust is a good choice. If #2, then take a look at Go.</p> Go is not the best solution for every team/project, but for the scenario described in this article, it has worked incredibly well. With Go, we can develop and deploy remote systems with confidence.</p> Layered, Event, and Data-Centric Architectures in Distributed IoT Systems 2022-06-14T00:00:00+00:00 With the advent of the browser, cloud, embedded Linux systems, and networked microcontrollers, distributed systems are everywhere. There are many models for communication in distributed systems – we will look at the tradeoffs between three of them. There are many perspectives to consider – initial implementation, client libraries, maintenance, adding features now and in the future, client compute and storage requirements, network bandwidth, data structure, etc. This article will discuss a number of these concerns and present several options.</p> The motivation for this discussion is synchronizing data in IoT systems where we have the following requirements:</p> Data (state or configuration) can be changed anywhere, at edge devices or in the cloud, and this data needs to be synchronized seamlessly between instances. Sensors, users, rules, etc. can all change data. Some edge systems have a local display where users can modify the configuration locally as well as in the cloud. Rules can also run in the cloud or on edge devices and modify state.</li> Data bandwidth and allowance is limited in some IoT systems (example Cat-M modems (< 100kb/s) and cellular IoT plans (< 50MB/month)). Additionally, connectivity is not always reliable, and systems need to continue operating if not connected.</li> </ol> The first is the Layered</strong></em> model. In this model, various instances talk directly to other instances. If any information needs to be shared, it is sent directly through an API (application programming interface) call using REST, CoAP, and other protocols – sometimes traveling through multiple layers.</p> </p> The layered, or API model’s advantage is simplicity – almost anything can connect to a REST endpoint. Most networked devices have built-in support for HTTP. Many companies provide services through REST APIs. Polling is typically used in the layered model to determine when things change, which limits the response time to the polling frequency. There are mechanisms that can be used in a layered architecture to obtain real-time response such as HTTP long polling, WebSockets, Server SIde Events (SSE), callbacks/webhooks, CoAP subscribe/notify, etc. However, an event bus is generally a much cleaner implementation if real-time response is needed. Firewall and security concerns can also make callbacks impractical on edge devices. If a device at the end of a layered architecture needs new information, often multiple layers need to be modified to handle this new information, which can make changes expensive.</p> A second model is the Event</em> or Message Bus</em></strong> model. Implementations include NATS, MQTT, Kafka, and many others. In this model, every instance in the system is connected to a message bus. All data communication is routed through the bus. Clients can post or subscribe to topics of interest so that updates happen in real-time.</p> </p> Some advantages of the event bus include real-time response and streaming workflows. All communication and information is on the bus, so if an instance needs to send or receive some information, it simply subscribes or publishes it. Event buses often leverage one connection to the message server so that each message does not need to establish a new connection, which can bring some efficiencies and reduce needed resources</a> (such as number of TCP connections). However, the event bus protocol is considerably more complicated than REST, so a message bus library is typically required to implement the client application.</p> The third model is the Data-Centric</strong></em> model. In this model, each client instance contains a set of data it is interested in. In the below example, the cloud system has an aggregate of all the data in a system, and the browser and edge instances are subscribed to a subset of the data they are interested in or have access to, represented by the different colored blocks.</p> </p> The data models the entire system, so if an action (setting an output, sending a notification, etc) is required, a bit of data is changed, and then the system takes action based on this data. The needed data is replicated to each instance in the system and is automatically synchronized in any direction when it changes.</p> To all you Go programmers out there, this may immediately raise a big red flag. What? Communicate by sharing common data?</p> Don’t communicate by sharing memory; share memory by communicating.</em></p> R. Pike</em></p> </blockquote> Pike’s comment is typically used in the context of an application where multiple threads (goroutines) share data in memory, and locks are used to keep multiple threads from writing to it at the same time. This model can be hard to reason about due to race conditions. An analogous model in distributed systems may be a common database that all instances use and all config/state is stored in this remote database. Locks may be analogous to database transactions. A common database is a good solution for many cloud based distributed systems where everything is always connected and, and this model allows data to be modified from multiple locations. However, with IoT systems, we have the following constraints that make it impractical to use a single remote database as our primary configuration/state data store for everything in the system:</p> We often don’t have the network bandwidth or data allowance (ex: LoRaWAN or Cat-M modems and low cost cellular IoT plans)</li> Network connections are unreliable, and edge instances need to continue working (collect data, run rules, etc.), even if the network is down.</li> Network latency can be long – especially with low bandwidth connections.</li> </ol> This leads us to a “local first”</strong> data model, where all configuration and state needs to be stored local to an edge device. However, this data is often viewed or edited in the cloud, so it also needs to be present there. We could craft new data structures and messages for every new bit of data and send custom messages over a REST API or event bus every time anything on either end changes. However, a simpler approach is to design a generic data model that can hold any kind of data, and then a mechanism to synchronize this generic data model. Then, adding functionality is much simpler as we can write/read data wherever needed, and synchronization between instances happens automatically with no extra effort. However, we have to learn to work within the constraints of the data model. A new feature might involve tweaking the UI and writing some code on the edge instance. Often, nothing in the middle needs to change:</p> </figure> Each instance has its own local copy of whatever data it needs. But instead of the developer needing to manually manage the synchronization of data between systems using custom messages, changes to the data are automatically synchronized in any direction.</p> The data-centric architecture requires more computing resources and storage to run it – typically an embedded Linux system. The below chart illustrates the trade-offs between client complexity and programming simplicity.</p> </p> For less capable networked devices such as microcontrollers (ex: Arduino), you may be limited to a layered approach, as HTTP may be the only supported protocol. However, as microcontrollers advance, many of them are now supporting event buses such as MQTT. The cost of a data-centric architecture is an up-front cost when building the base system. As time goes on, the incremental costs of adding features is much lower. Since adding features is typically a continuous, long term exercise, the initial cost is paid back many times over. A data-centric architecture also introduces some discipline in how data is structured, stored, and synchronized. This helps avoid the technical debt that rapidly accrues in less structured approaches.</p> All architectures have their place. If the application is fixed in functionality, and you only need to send a temperature reading to the cloud once per hour and this is never going to change, then a microcontroller with a layered architecture is fine. If you need to send data in both directions and real-time response is needed, then moving up to an event bus makes sense. If you have devices that have a decent amount of configuration that can be modified at the device and in the cloud, runs rules locally, and features will continually be added to the system, then a data-centric architecture is beneficial. The data-centric architecture requires a more complex system to start with, but the incremental effort to add features down the road is drastically reduced because the developer does not need to worry about the data storage/synchronization problem. Additionally, the data model is available at each instance, so programming is simpler.</p> It is possible to support multiple architectures in one system. For example, in Simple IoT</a> all data modifications are sent over an event bus which is the primary programming API</a>. A client can connect to the event bus of a local or remote Simple IoT instance. If the local and remote instances are connected using an upstream sync connection</a>, then writes to the common data-set of either instance are automatically synchronized to the other. If a client does not want to implement a local data store, they can connect to a remote instance’s event bus. NATS JetStream</a> is another example of a store built on top of an event bus. Some event bus implementations like NATS allow for easy request/response transactions</a>, which is similar to a layered REST/RPC type interface. Thus clients in a system like Simple IoT can range from simple HTTP clients, to mid-range event-bus clients, to advanced Linux based, data-centric clients with a local data store.</p> In computer science, focusing on data is typically the right approach.</p> I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships.</em></p> Linus Torvalds</em></p> </blockquote> Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.</em></p> Fred Brooks</em></p> </blockquote> Distributed systems are no different – the focus needs to be on the data. Synchronizing data efficiently and reliably between distributed instances is a hard problem to solve, and the focus of much research and development effort today (example: the Riffle project</a>). This requires some up-front thought, planning, and constraints, but the payoff is simpler data models that result in systems that are more reliable and cost effective over the product life-cycle. Data-centric architectures free developers from the tedium of synchronizing data and allows focus on the application problem at hand. Data-centric architectures also provide discipline and structure around the data synchronization problem that helps avoid technical dept build-up which is a great impediment in the flow of value to end users.</p> Each distributed system architecture has its place, and with a correct implementation, we can support multiple architectures in one system. This gives us the option to support simple as well as advanced clients, depending on the need. However, as the cost of advanced MPUs systems running Linux such as the Raspberry PI continue to fall, deploying advanced IoT clients to the edge makes a lot of sense.</p> In future articles, we’ll examine in more depth the techniques used to store and synchronize data in data-centric architectures.</p> Thanks to Khem Raj</em></em>, Bruce Stansell, Gina Brake, and Collin Brake for reading drafts of this.</em></p> Business Contracts 2022-03-21T00:00:00+00:00 Business contracts are often executed when two companies work together. At the most basic level, a contract should describe a common understanding between the two parties. The contract should assist and aid the relationship. However, many contracts are written by lawyers with the primary goal of covering every legal loophole and taking every legal advantage possible for their clients. This often results in contracts that are very long, filled with legal jargon, benefiting only one party, and in general, a burden to process.</p> For a business relationship to be successful and sustainable, there has to be value</strong> for both parties and there has to be trust</strong>. If these two key ingredients are lacking, there is little motivation to continue the relationship.</p> We can look at business relationships as a positive-sum game, or a zero-sum game. A positive-sum game is where the relationship is structured as a win-win – both parties find good value in the relationship and are motivated to continue it. There is a belief that the “pie” of opportunity will grow through their combined efforts. The engagement is voluntary and not constrained by force or compulsion. Zero-sum thinking is where one party is always trying to gain the maximum advantage over the other party. This is the “fixed pie” mentality where we think the resources in business are fixed, and if anyone else gains something, it is my loss. This attitude is often found in purchasing departments where they are trying to obtain the lowest possible price for a good or service, net 60 payment terms, etc. I recall in one case a manager of a company exclaiming, as we went into a meeting with a vendor, “Let’s go beat ’em up!”</p> Business relationships are more complex than getting the lowest price or the greatest legal advantage. The people/companies you want to do business with are likely already busy. They likely have plenty of opportunities. They will seek engagements with those who play the positive-sum game. If a company plays the zero-sum game, they will end up doing business with second rate companies who don’t have any other choice. They won’t get good service, and trust in the relationship will be lacking. They will be forced to hop from one vendor or customer to other. There will be few long term relationships. As Bob Metcalfe once wrote</a>, “First, don’t “hire” anyone. B people hire C people – they collect resumes and choose the person they want to honor with a job. A people recruit A people.“</em> The same applies to business relationships.</p> With this background, let’s examine the topic of contracts a company may present to an independent contractor or consultant providing services to the company.</p> Again, the best developers (the ones you want) are successful and busy. They will not mess around in business relationships where people are playing the zero-sum game. They are successful because they are trustworthy and expect the same in return. They are generous with their time and knowledge because they understand the universal law that the more you give, the more comes back to you.</strong> But they also expect the same in return. They are confident in their abilities and do not take measures that lock their customers into their services, but on the contrary, make themselves “easy to fire</a>.”</p> A fundamental problem with trust is that people generally don’t tell you when they don’t trust you. Actions that damage trust are likely unintentional (otherwise you would not do them), but you never get feedback on this. So, I’m giving you a hint now – an onerous contract, with one-sided indemnification clauses, damages trust. It tells the other party that “I might take advantage of you.” It implies that if bad things happen due to a product flaw, “you could be on the hook.” It sows seeds of doubt. Most often, this is likely not the thought of the company presenting the contract, and the contract language is just the result of lawyers trying to do an exceptional job of protecting their client from any possible liability. However, a contract is still a contract. If disaster strikes, the contract will be used in court by various parties trying to recoup damages. Products are often sold over a long time (5-10 years or more). Companies often change owners. Therefore, it is very important that a contract be fair and just. When a developer is presented with an onerous contract, immediately the question arises – can I trust this company? Is it worth it? Even if the project goes forward, it will be with some reserve and guard. It will not be with the wholehearted attitude that is an ingredient for success.</p> As we consider indemnification phrases in a contract, we need to establish who should assume liability for a product: 1) a consultant/contractor who is billing by the hour, or 2) the company selling the product over a span of many years? Risk should be proportional to reward. A hourly contract developer has no reward for a product’s success, other than the baseline costs of doing the development. The company selling the product is the one who will potentially reap the multiplying rewards of a successful product. Therefore, they should be the ones who bear the risk as well. It is not just to expect otherwise. In the situation where a developer is providing consulting or contracting services to a larger company developing a product, it is important that the contract clearly state that:</p> The Company assumes sole responsibility for all testing of the Work, including but not limited to software and hardware, and also assumes sole liability for any use or misuse of the Work developed by Contractor for the Company.</strong></p> An argument may be made to the developer – you are not a lawsuit target because you are not a large company with deep pockets – so just sign the contract. This may be true, but what if the consultant does grow the business to the point where there are more assets? Products live a long time – a lot can change. As mentioned before, a company can change hands. If there is a lawsuit, then anything is possible.</p> An individual developer has limited control of the development of a complex project and the resulting product. (We are not talking about fixed-bid turnkey product design where a larger company designs the entire product, but rather hourly development services). Typically, there is extensive involvement from other developers. The contract developer does not control how the product is manufactured, marketed, sold, supported, or even how other parts of the system are developed. In complex systems there will be bugs, and often there are time to market pressures that prevent extensive testing and verification. These are all factors that are out of the developer’s control and are choices made by the product owner. Again, this further illustrates that those who are ultimately in control of a product (the owner) should be responsible for any liability the product might bring.</p> This brings us to a common concern of the company – they want to be sure the product is developed to professional standards. A contract phrase might be: “Contractor hereby warrants that its work shall be of professional quality and performed consistent with generally accepted industry standards”</em> Who defines these standards? How will they be defined in court? The reality is that few developers have the time and resources to really do the job they want to do. Schedules are always pressing, forcing compromises. Ultimately the product owner makes the decisions concerning these trade-offs – not the developer. Time to market is critical. Therefore it is difficult for a developer to warrant any work will be at any particular industry standard – as the product owner is ultimately in control. The universal laws of reality apply: “Good-Fast-Cheap” – pick two – all three do not exist. In any product development effort, there are compromises.</p> There is a very practical way to handle the concern that a company’s product is being developed to their standards, and that is to encourage collaboration and openness between all developers on the project. Make sure all source code and design files are shared in Git repositories that are easily accessible to all developers – both employees and contractors. Encourage all developers to work on any part of the system needed to get their job done. Google does</a> it</a> – maybe you should, too. Discourage engineering “turf” where developers feel they “own” something in the sense that no-one else should mess with it. This is essentially the process used in Open Source Software, and successful companies have adopted these practices – the results speak for themselves. If this works in a large company, is there any excuse for not doing this in a small company? Open source projects don’t have contracts that specify all contributors have to do a good job, yet open source projects contain some of the highest quality code available. Peer review and doing your work in public (even if only within a company) is a powerful motivator. Because the development is in an open environment, it is quickly evident who is going a good job and who is not. The same dynamic can happen in a company. If there is open collaboration, frequent integration, and early testing, you will soon get a good feel if developers are efficient and are generating work that is of “professional quality.” Share weekly reports with the entire team of what each developer did and how much time they spent on the project. This will produce a level of accountability among peers that will far exceed what any manager can force. If someone is spending excessive amounts of time and not getting anything done, other developers will start to question it. And you will know of problems early in the project at a time when you can do something about it, rather than after a product has been sold in quantity and some horrible defect arises that results in lawsuits. Which is the better option? To summarize:</p> If a company wants to ensure quality work is being done by their developers, then they must foster a collaborative, transparent development environment where:</strong></p> All development is done in the “open” in Git repositories that the entire team has access to.</strong></li> Any developer can contribute to any part of the system required to do their job.</strong></li> Integration is done often.</strong></li> Product testing, by the owner, is started early in the development cycle.</strong></li> </ol> No amount of contract legalese can come close to being as effective as the above.</p> Whether you intend it or not, your legal contracts are both a reflection and driver of your company culture, and its resulting success. They impact the people you will attract – both employees and contractors. Do your contracts reflect a fair and generous mindset where you are willing to take responsibility for the things you should take responsibility for? Do they turn away or create reserve in people you need the most? This is a serious issue because the best people often come through referrals. Will a developer recommend their talented and busy friends if they are required to sign uncomfortable contracts that must be negotiated to even approach an acceptable level? Any friction in this process is slowing you down. Yes, a one-sided legal contract that takes everything you can get may save you some liability in the unlikely event of a product lawsuit. But it also has the effect of introducing friction in relationships that are key to your product being completed in the first place. Which is more important? Which should have priority? In summary, a contract should:</p> clearly represent the concerns of both parties</li> document what is expected</li> not be one-sided</li> be fair</li> avoid complex legal jargon as much as possible</li> aid, not hinder the business relationship</li> include provisions for how the relationship will be terminated, if needed</li> </ul> I’ve been working as a consultant for 20 years. My most successful and longest relationships have been with companies where I have never executed a single contract. Is this a coincidence? Is it cause or effect? I don’t claim to know, but my experiences resonate with the words of Seth Godin:</p> A handshake deal, on the other hand, is about the future. Either side can claim loopholes or wriggle out of a commitment, but the consequence is clear—if you disappoint us, we won’t be back for more. The participant in a handshake deal is investing in the future, doing more now in exchange for the benefits that trust and delight and consistency bring going forward.</em></p> from: https://seths.blog/2014/10/handshakes-and-contracts-the-future-and-the-past/</a></p> </blockquote> Somehow an essay about contracts has morphed into a discourse on company culture, development practices, and ultimately success. This is the power of writing – it helps you connect concepts that you did not previously realize were connected. In the end, everything is connected. Your contracts, your company culture, the type of people you attract, etc. It all matters.</p> References:</p> https://www.technologyreview.com/s/400489/invention-is-a-flower-innovation-is-a-weed/</a></li> http://www.unixwiz.net/techtips/be-consultant.html</a></li> http://sethgodin.typepad.com/seths_blog/2014/10/handshakes-and-contracts-the-future-and-the-past.html</a></li> http://badsoftware.com/contract1.htm</a></li> https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext</a></li> https://medium.com/@Jakeherringbone/you-too-can-love-the-monorepo-d95d1d6fcebe</a></li> https://www.clearviewlegal.co.uk/ohlawdy/contracts-are-communication</a></li> </ul> Technology 2020-11-27T00:00:00+00:00 The fire of information technology – will you use it:</p> to create</strong> or consume?</li> to comfort</strong> or torment?</li> for education</strong> or entertainment?</li> as a tool</strong> or weapon?</li> to temper</strong> or incite?</li> for community</strong> or faction?</li> for harmony</strong> or rebellion?</li> to encourage</strong> or discourage?</li> to spread light</strong> or darkness?</li> to bless</strong> or curse?</li> to compliment</strong> or ridicule?</li> to save time</strong> or waste time?</li> to be generous</strong> or selfish?</li> to heal</strong> or hurt?</li> to share ideas</strong> or focus on self?</li> to be open and direct</strong> or talk behind backs?</li> to focus on things we can do</strong> or complain about things we have no control over?</li> to spread truth</strong> or falsehoods?</li> to express thankfulness</strong> or discontent?</li> to preserve our autonomy</strong> or be manipulated and exploited by others?</li> for productive</strong> pursuits or indulgence in vanity?</li> to engage in real life</strong> or escape into some virtual universe?</li> </ul> Do we have integrity such that our digital communications match what we would do in person? We live on a great sea of information, and the default path is to be tossed about on the waves of unverifiable communication that wash over our society, and in the end have little to show but discontent and unhappiness. Technology is an excellent tool, but a poor friend, a useful servant, but an oppressive master. Technology is no substitute for the things that bring true joy and satisfaction in life – family, friends, giving, companionship, creativity, craftsmanship, mastery, accomplishment, nature, and faith. Like fire, information technology is only useful when used intentionally and constrained to some useful purpose. Otherwise, it will rob us of our time, relationships, humanity, skills, potential, and ultimately our soul.</p> Adventures with a ATSAMA5D27 board 2020-01-29T00:00:00+00:00 Recently, we brought up a new PCB (printed circuit board) a customer had designed based on the Microchip ATSAMA5D27C-D1G</a> processor. We encountered several interesting challenges.</p> </figure> The first issue was that the design only had serial ports (and possibly a USB Host port that could be used in device mode) for bootstrapping the system. Microchip provides a tool named SAM-BA to do this. There are two implementations of SAM-BA. The older implementation does not support this CPU. The newer implementation</a> does support this CPU, but the documentation and examples are sparse (at least when we were using it). The standard way to use SAM-BA is to directly program the bootstrap and bootloader into SPI flash. However, this design only has eMMC flash. So we came up with a way to bootstrap at91bootstrap and u-boot over the serial port, and then load a Linux kernel with an embedded initramfs from a USB flash disk. This initramfs contains an updater</a> that is used to program the eMMC flash. After the updater runs the first time, the system will then boot from eMMC. We developed and verified this process on a development board, and it is documented in a Github issue</a> (The SAM-BA team has since disabled Github issues on this repository)</em>, and as part of the Yoe Distro sama5d27 support</a>. Microchip has also since provided us with a procedure to program eMMC using SAM-BA, but the procedure is complex and we never had a need for it once we figured out the above method.</p> After receiving 10 prototypes of the new board, we learned pretty quickly that VCCBU must have voltage for the CPU to start. After that, the first board loaded u-boot and everything looked good, but things went south from there. U-boot would only load perhaps one out of twenty times. Basic checks of power supply voltage levels all looked good. The custom board used a 12MHz crystal for the main clock, where Microchip’s SOM1 module uses a 24MHz crystal. We decided to try a 24MHz crystal, and after that change, u-boot would reliably load over SAM-BA. We’re not sure why (12MHz is slower than 24MHz, so things should run better), but initializing a clock tree is complex and there is likely something that was not configured correctly in software. Lesson #1: Unless you want to spend a lot of time writing chip initialization software and debugging strange problems, follow a reference design as closely as possible.</strong> Working at the level of chip initialization often requires intimate knowledge of the MPU and likely the support of those who designed the chip. This is much more effort than most teams have time for and is best left to the MPU vendor.</p> Now that u-boot was loading reliably, we could turn our attention to the kernel. At this point, a couple boards would start the kernel, but most failed with no kernel messages after the “Starting kernel …” message on the serial console. The boards that would start the kernel would crash partway through the boot process. We started looking at power supplies and observed a 100mV sawtooth on the core power supply.</p> </figure> The SAMA5D27 datasheet specifies a maximum of 15mV of ripple for the core supply.</p> </figure> The ADP5023 power supply used for the core voltage had a mode pin that can be used to force it into PWM-only mode, instead of PWS/PWM mode. After forcing the supply into PWM-only mode, some boards would start Linux. However, there were still stability problems later in the boot cycle.</p> After trying many other things, we finally raised the voltage of the core supply slightly by changing one of the feedback resistors. Presto! Lesson #2: If a CPU has stability problems and everything else looks OK, trying raising the core voltage.</strong> The system was now stable. After communicating with Microchip, we learned that the CPU actually required a higher voltage than the documentation specified. Microchip provides a power supply that is designed to be used with the SAMA5D27 – if this supply would have been used, then we would have gotten the right voltage.</p> This bringup experience was harder and longer than most – especially for a board of this complexity. But, it illustrates an important point – full custom MPU design is hard – even with a part like the SAMA5D27-D1G which integrates SDRAM in the CPU package. It can require a significant amount of time. This design was done by a competent designer and was extensively reviewed, but it still contained serious problems that took weeks to resolve. In the end, we were still able to make the first PCB revision work with minor modifications. However, it is very possible that the first revision of a PCB like this might need to be scrapped and chalked up to a learning experience leading to months of project delays.</p> Lesson #3: Use a SOM</a> (System on Module) unless you have a good reason to do a full custom design, and you have the time and resources to pull it off.</strong> You are likely not Google or Apple</a>. Yes, you will pay slightly more for a SOM than the raw MPU system components, but the SOM cost may be offset by a lower cost baseboard. A full custom MPU design may require 8 layers and very small routing features where a baseboard for a SOM may only require 4 layers and use larger features, resulting in a less costly PCB. Additionally, a baseboard for a SOM may have a lot less parts on it, and requires less effort to set up for assembly – simpler ones can even be hand built to bypass weeks of assembly lead-time.</p> There are dozens of SOM vendors offering products today that fit about any price/quality/certification price point. Some SOMs contain pre-certified WiFi and Bluetooth radios. Others meet various mil-spec requirements. This situation today is much different than 10 years ago when there were only a few expensive SOM models available. SOM modules allow us to bring advanced technology to a problem, and implement a solution with a small team in a reasonable amount of time. Implementing a SOM based MPU design still requires a fair amount of knowledge, but the unknowns are drastically reduced, and chances of success are far better. Much of the design complexity is already handled, freeing you up to work on the value a product provides. Any time we can simplify our development effort, our chances of success are much greater.</p> Working with Microchip on this project was a good experience. Without their support, we would not likely have resolved the issues we faced in a timely fashion. This brings us to Lesson #4: When developing complex system, you are likely going to need some level of support.</strong> This may come from IC vendors, open source communities, consultants, etc. In this case, the MPU documentation was not up to date, and the chip designers were able to point out the problem. We could be critical of Microchip for this oversight, but with a 3000+ pages datasheet, this is a lot of detail, and there will be mistakes and oversights. The fact that most of the datasheet is available and accurate is a commendable accomplishment. Getting this level of documentation from some vendors is like pulling teeth. Before any of us cast the first stone, we must ask the question – is everything we do completely and accurately documented? Overall, the SAMA5D27 documentation is pretty good.</p> The support provided on the SAM-BA Github issues was also essential for us to figure out a way to program the device. The procedure we found for programming this system using only a serial port is relatively simple and easy to implement. Though it may violate some SDRAM refresh timing specs, in practice it works very well – and that is what matters – getting things done. It is unfortunate that Microchip has since closed the Github issues for the SAM-BA project, indicating they are not really interested in building a community around the project, but rather Github is only a way to distribute the tool that is developed in-house. As systems get more complex, Open Source communities are proving to be the only viable mechanism to develop and support all the various needs users might have. Large companies readily acknowledge this as many core technologies are now Open Source, and much effort is put into building communities around them. But, lack of Open Source vision aside, Microchip is a reliable partner – especially for smaller companies wanting to use advanced technology and needing support.</p> Drone for Continuous Integration 2019-11-26T00:00:00+00:00 Recently, we have been evaluating Continuous Integration</a> (CI) systems for a variety of projects (both OSS and customer). There are many OSS options to chose from</a>. Because we already use Docker containers for Yocto/OE builds</a>, Concourse</a> and Drone</a> made the short list. Both communities seem responsive</a> and helpful</a>.</p> https://github.com/YoeDistro/yoe-distro</a></figcaption></figure> Building OE in a container</a> is important, so naturally Drone and Concourse seem like good options as they are focused on containers. Containers make sense for most CI tasks as they allow you to easily control the environment without getting bogged down in setting up the host. Because Concourse does not seem to easily support bind mounting volumes</a> on the host system, we ended up implementing a prototype in Drone</a>. This was improved and merged</a> into the yoedistro master. Ironically, we are not currently building in a container, as the yoedistro CI machine is a 64-core ARM system and we had some problems with the container support. Hopefully we can get containers working again at some point. Since the initial evaluation, Drone has added an exec runner</a>, which allows us to execute a pipeline directly on a machine without a container.</p> Drone is an excellent fit for most of our projects. There are more capable systems (like Concourse, Jenkins, etc), but the simplicity of Drone is a big advantage in that it will more likely be used by developers. A tool is only useful if it gets used. With Drone, a developer can simply cd to a Git workspace that has a .drone.yml</code> file, and then run drone exec</code> and it will run the entire pipeline locally. Because drone typically does everything in containers, this works fairly well – it does not really matter what the host system is. This is powerful because developers can test the entire pipeline on their local machine before enabling it in the drone server/agents for automated testing/deployment. If there are problems, they can be debugged locally rather than the tedious process of commit, run the pipeline on the server, view the logs, repeat. Solutions that are simple to use make a big difference in productivity.</p> Below is a quote from a Drone user:</p> Today we have over 60 engineers interacting with Drone daily, and they really like it, due to its simplicity. We’ve always had a small DevOps team, so we tend to heavily favor solutions that don’t require a lot of maintenance overhead, and Drone fit the bill. Since most developers have at least a basic understanding of Docker, they hardly need any help maintaining their pipelines, and there have been a couple of instances of developers creating their own plugins! This way our team can focus on exciting upgrades to our infrastructure, even as teams regularly update their deployment pipelines and new projects are created every month.</p> https://blog.drone.io/why-quinto-andar-chose-drone/</a></p> </blockquote> Drone makes some interesting trade-offs in that it trades some flexibility for simplicity and developer ease of use. Initially, Concourse was my first choice as it is a more powerful tool, but because it does not support bind mounting host volumes or exec runners, we ended up using Drone. This tool makes sense for many teams because it is more likely to be used by the average developer without having a dedicated CI/Operations team.</p> This is a trend we are seeing at successful companies where developers are more responsible for everything related to their project, including testing, continuous integration, deployment, and operations. The quote below gives us some insight into how things are done at Amazon:</p> There is another lesson here: Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.</p> — Werner Vogels (CTO @ Amazon)</strong>, https://queue.acm.org/detail.cfm?id=1142065</a></p> </blockquote> Drone fits into this model well because it enables developers to become engaged in CI and deployment.</p> One potential problem with drone is the lack of support</a> for some Git systems like Assembla. One workaround would be to store pipelines in Github/Gitlab/Gitea, etc, and then fetch code from other repos. But it would quickly make sense to add auth/webhook support for whatever Git system is used if it does not already exist.</p> Reflections on KiCad and EDA Tools 2019-11-12T00:00:00+00:00 A recent interview</a> with a KiCad developer prompted some reflection on KiCad and EDA (electronic design automation) tools in general. Below are samples of several PCB (printed circuit board) designs, created with KiCAD, and implemented as part of the SimpleIoT project</a> in the last couple months.</p> </figure> </figure> The experience has been excellent. Above all, the tool is very fast, efficient to use, and stable. Schematic and PCB integration works well enough, and routing and copper pours are easy. Switching between inches and millimeters can be done on the fly. The KiCad library has many parts in it, and other organizations, such as DigiKey</a>, Seeed</a>, SnapEDA</a>, and Ultra Librarian</a> are also providing libraries. If a KiCad symbol/footprint for a part is not already available, it is relatively easy to create new symbols and footprints as needed. There is a good KiCad support forum</a>. KiCad is a pleasure to use and production-ready for standard PCB designs.</p> Choosing an EDA tool is an investment. There are several aspects to consider:</p> purchase/licensing cost of the tool</li> time invested in learning the tool</li> time spent implementing designs</li> time spent maintaining designs</li> </ol> #2, #3, and #4 swamp #1 for many projects – at least when low end EDA tools are used. Thus we want to be confident that the tool we choose will be around for a while and will be well maintained. With all the churn in the EDA industry in the last 10 years, it is hard to have a lot of confidence in the long term prospects of many tools. For KiCad, however, the development statistics indicate that momentum around the project is increasing and that this tool will likely be around for awhile.</p> https://github.com/KiCad/kicad-source-mirror/graphs/contributors</a></figcaption></figure> With 183 contributors, the KiCad project appears to be healthy – it is doing something right.</p> Some may fear that open source projects (like KiCad) are taking away the opportunity for companies to make money and are not sustainable. Has the advent of Linux decreased the amount of money being spent on operating systems? On the contrary, Linux has greatly increased the operating system ecosystem – much more so than if proprietary Unix were the only option. Redhat, Ubuntu, and many other companies seem to be doing well. Linux is used in everything from watches to supercomputers. Unix would never have enabled this breadth of products – it is simply too large a task for any one company or team. Collaboration is the only way to accomplish a project of this magnitude. When the commodity layers are taken care of, then people are free to work on higher level problems. If a commercial EDA tool does about what KiCad does, that product is likely not going to fare well long term. But there is plenty of space to add value higher up the stack. Examples might include tools that work with KiCad such as auto routers, advanced placement tools, library management tools, RF design, SI/EMC simulation, high voltage design rules, etc. There is endless opportunity. The “pie” of opportunity is not fixed – it can grow. Contribute code to KiCad that will enable proprietary tools to run in tandem. If companies build on top of KiCad, then they will not have to duplicate the task of building a basic schematic and PCB layout editor and can instead focus on other features, and as a result, there will overall be more value for society. Companies will already have a large market of people who are more likely to spend incremental amounts of money to add functionality as needed. The alternative is for every company to try to provide the entire EDA stack, which can be a tough sell, as it is expensive and customers will need to change everything they are currently doing. There is no reason EDA software cannot become modular like the software development ecosystem.</p> KiCad may displace a few commercial EDA products, but the value and opportunity KiCad will help create will far outweigh this loss. More companies will build more stuff, with less friction and effort. There will be more innovation. More problems will be solved (and likely a few created). There will be more collaboration. Even now, it is practical for developers, who only design PCBs occasionally, to use production grade tools.</p> EDA tools should not be evaluated merely on technical features and price, but also on the hassle factor, the capability to facilitate collaboration, and the ease of long term product maintenance. Who will maintain your product in two years when a part goes obsolete, the design needs tweaked, and the designer and the obscure tool used on the original design is long gone? If you contract out a PCB design, why not require that the design be implemented using a tool that anyone can use in the future? Part of the value of a product is its maintainability. Companies who can quickly adapt as things change will have an edge. The costs of not being able to quickly respond to some product need, or of having to learn new tools and re-implement a design are significant. Over the life of some products, maintenance costs may be much larger than the initial development costs.</p> In years past, I’ve used a number of different PCB design tools (Mentor Board Station, OrCad, P-Cad, gEDA, Eagle, etc). Some have been good experiences, others not so much. While there are no doubt EDA tools that have more features and are better than KiCad in some ways, KiCad is the low friction solution for your standard PCB design. It can be used easily by any one at any time on any computer. This flexibility is a big part of KiCad’s value. EDA tools are the “build system” for your hardware. They work best if they are open and easy to use by anyone on the team. There are many “free” PCB design tools available, but few of these are open. Some are cloud based, are very convenient to use, and technically a marvel that we can run a PCB design tool in a browser.. But the design is still locked up in a proprietary format. It is not easy to customize or extend these tools. And the sheer number of these commercial options available today suggests that long term they probably won’t all be around. “Open” is more important than “Free” in my opinion – especially for the long term.</p> KiCad is certainly a noteworthy project in the technology landscape, and deserves consideration. The availability of such a tool will change the world. The open source hardware movement will grow. Companies who learn to leverage this movement will benefit in the same way that most companies now benefit from the use of open source software. More wealth will be created for society. The traditional barriers between hardware engineers, software developers, and PCB designers will be reduced. Just as hardware engineers have had access to about any software development tool they want to use, now software developers have access to tools to develop and tweak hardware. The small amount of time required to learn KiCad is a worthwhile investment.</p> The Humble Programmer 2019-10-31T00:00:00+00:00 In 1972, Edsger W. Dijkstra</a> published a paper titled The Humble Programmer</em></a>. Dijkstra was trained in math and physics and was a university professor for much of his life. This paper is an interesting reflection on the history of computers and contains thoughts for the future. A few quotes are included below:</p> Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with. In other words: both goals point to the same change.</em></li> To put it in another way: as long as machines were the largest item on the budget, the programming profession could get away with its clumsy techniques, but that umbrella will fold rapidly.</em></li> The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.</strong></em></li> Another lesson we should have learned from the recent past is that the development of “richer” or “more powerful” programming languages was a mistake in the sense that these baroque monstrosities, these conglomerations of idiosyncrasies, are really unmanageable, both mechanically and mentally. I see a great future for very systematic and very modest programming languages.</em></li> The best way to learn to live with our limitations is to know them. By the time that we are sufficiently modest to try factored solutions only, because the other efforts escape our intellectual grip, we shall do our utmost best to avoid all those interfaces impairing our ability to factor the system in a helpful way.</em></li> We shall do a much better programming job, provided that we approach the task with a full appreciation of its tremendous difficulty, provided that we stick to modest and elegant programming languages, provided that we respect the intrinsic limitations of the human mind and approach the task as Very Humble Programmers.</em></li> </ul> Dijkstra was 42 years old when we wrote The Humble Programmer</em>. At a similar age of 47 years, I too have come to appreciate the value of simple programming languages and a careful approach to writing software. Two modern languages that embody some of these principles are Go</a> and Elm</a>. While on the surface these languages may seem very different (one is procedural and one is functional), the values of the languages are similar. Both are designed with a great respect for the finite capability of the human brain. Both strive for simplicity first, and features are only added to the language after careful consideration. Both have excellent tooling that helps ensure consistency and correctness. Both are not just a syntax, but rather a culture and mindset. Experience on recent projects indicates that Go and Elm are effective languages, especially when I compare to previous experiences on C++/Javascript projects. There are rarely run-time exceptions. Compiler errors are relatively easy to understand and fix. Tooling is simple and almost invisible. Program maintenance and refactoring is enjoyable. The combined cognitive load of the language and the tooling is small freeing up resources for solving the problem at hand.</p> Today, there is a huge amount of churn in the software development world. Especially in front-end development, new frameworks are being released at an astounding rate. Many have their place, and perhaps each addresses some need. But it seems there is something fundamentally wrong with so many solutions for the same problems. Thus, it is encouraging to see new approaches where the authors have carefully studied the past, tried to learn from everything that has been done, and have created solutions where simplicity is valued. This is key to developer effectiveness, and really to life in general.</p> Why are Go applications so reliable? 2019-10-10T00:00:00+00:00 Go does a lot of things well (good performance, easy to learn, very productive, extensive stdlib, excellent tooling, etc), but after programming with Go for three years (both embedded Linux and cloud applications), stability is the characteristic that really stands out.</p> </figure> In the past, I’ve observed the development of several fairly large C++ applications. Stability problems were more common than they should have been. Memory leaks, null pointers, crashes, etc. On one project (over 500,000 SLOC</a>) we built elaborate mechanisms to capture core dumps when things crashed, at times needed to run Gdb on the target system, and spent a fair amount of time in Valgrind tracking down memory leaks. The worst problems were the rare, intermittent ones which only happened in the field and were very difficult to instrument. It is possible to write reliable C/C++ code (one example is the Linux kernel), but it takes focused effort (code review, testing, coding standards, etc). My experience with Javascript has been similar. What if there was a programming language where basic program stability did not have to be a primary concern?</p> I’ve now participated in four decent-sized Go projects and stability problems are a thing of the past. I only recall one memory leak. Crashes are rare. Several attributes of Go contribute to this:</p> The pattern for returning multiple variables (including error) and aligning the happy path to the left</a> encourages error checking before doing anything with the data a function returns. Therefore you don’t end up trying to do things with nil variables. It is a little verbose, but it works and is a small price to pay for the problems it solves.</li> Garbage collection takes care of memory leaks.</li> The language encourages simplicity, and simple things are generally more reliable.</li> The type system catches many errors during compile time.</li> </ol> I’m sure there are others, but these are the ones that stand out to me. Go strikes a nice balance between safety and simplicity. There are safer languages (Rust, Haskell, etc), but they require more effort to learn, and probably require more coding effort (I don’t have a lot of experience, so I could be wrong about this). Every tool has its use and place. If I was developing an application on a space craft going to Mars, I’d probably use Rust. If I am writing a quick script to process some test data, I typically use Python. When writing MCU code, I still use C/C++, but someday may use Rust. But when I’m writing a reasonably complex cloud or edge (embedded Linux) application, I use Go. In Go, you can get things done quickly, and be reasonably confident that the application you deploy will not crash or leak memory.</p> Go’s type system and simplicity require a little more typing than Python, but likely less effort than more complex languages like Rust or Haskell. Go is a pragmatic language that gets you 98% there without a lot of overhead. In business, you generally don’t want to add a lot of process overhead on the entire company for the 2% of people who may be causing problems, as it imposes drag on the other 98% trying to get things done. Rather, it makes sense to address the 2% at a personal level. Likewise, with Go, the 2% of the time you have to fix a nil bug, or a unhandled case the type system did not catch, is acceptable as 98% of the time you are working in a language that is very productive with very little overhead. Go is ideal for many applications.</p> Auto-formatting/linting Go code 2019-04-22T00:00:00+00:00 Some things in life you just have to experience to truly appreciate the value. One of these is auto-formatting/linting source code. When I started programming in Go in Vim, I naturally looked for editor support, and found the excellent vim-go</a> project. Through this, I learned about gofmt</a>and then goimports</a>. These tools can be configured in your editor to automatically format your code when you save. goimports goes a step beyond and adds missing imports and removes unused ones.</p> Auto-formatting is quickly</a> becoming</a> the</a> norm</a>. The Javascript world also has an excellent formatter available named Prettier</a>. There are formatters for many other languages as well including C/C++</a>, shell</a>, Elm</a>, Rust</a>, etc. The neoformat</a> and ALE</a> plugins add auto-formatting functionality to Vim/Neovim. An example of how to enable these plugins in Neovim is included in my dotfiles</a>.</p> Below are instructions on how to set up Visual Studio Code</a> to auto-format code. Even though I use Neovim</a> for most of my code editing, I still try out VS code and IntelliJ IDEA</a> from time to time to learn what’s new in cutting-edge editors. I also recommend VS Code for most new Go developers – especially those coming from embedded systems development where an IDE is traditionally used. Give it a try – it will only take you a few minutes.</p> The instructions below should work with most systems (Windows, Linux, Mac). First, make sure Go and VS code are installed. Then install the Go VS code extension.</p> </figure> Next, type Ctrl-Shift-P to bring up the command palette and run Go: Install/Update Tools.</p> </figure> This will display a list of tools – select them all.</p> </figure> Next, configure VS Code to format on save. Open the settings by selecting File->Preferences->Settings from the menu. Select the “Format on Save” option and set the timeout to something larger than the default, like 5000ms.</p> </figure> Last, select goimports as the code formatting tool – this will automatically manage the imports in your file, as well as formatting the code.</p> </figure> Now, open a Go source file. Mess up the formatting of some code and then save – the code should be automatically formatted. Similarly, if you remove one of the import statements that is being used, and save, the imports will be corrected. Unused imports will be removed.</p> These tools allow you to focus on writing software and let the tools worry about formatting and imports. Prettier code with less effort – why not?</p> Microcontroller (MCU) or Microprocessor (MPU)? 2019-04-09T00:00:00+00:00 As technology advances, there are two basic processing platforms for implementing embedded systems</a>. The first is the Microcontroller Unit</a> (MCU). These devices have varying amounts of integrated Flash (<= 2MB) and RAM (<= 1MB), and are designed to run bare-metal code or a real-time operating system</a> (RTOS), like FreeRTOS. The second is the Linux-capable Microprocessor Unit</a> (MPU). An example of an MCU based system is most Arduinos, and an example of an MPU based system is the Raspberry PI. An MPU typically does not have embedded Flash and RAM – at least on the same die. The fundamental difference between MCU/RTOS and MPU/Linux systems is the memory architecture and the amount of memory in the system.</p> </figure> There are other differences as well, summarized in the table below.</p> Feature</strong></td> MCU</strong></td> MPU</strong></td></tr> # power supplies</td> 3.3V</td> VCORE, VIO, VDDR, etc.</td></tr> USB Host</td> limited</td> Full support many drivers</td></tr> Networking</td> limited</td> Options for GB and multiple interfaces </td></tr> Cost</td> lower $1-$20</td> higher, starts at $10 (cpu + power/mem) </td></tr> Programming Languages</td> C/C++ Micropython Rust</td> C/C++, Go, Python, Nodejs, Java, Erlang, Rust, about anything</td></tr> Realtime</td> hard</td> soft</td></tr> Data processing</td> limited</td> excellent</td></tr> Processing power</td> less</td> more</td></tr> Display size</td> small</td> any size </td></tr> Startup time</td> fast</td> slow</td></tr> Expandable over time</td> less so</td> more so</td></tr> On-chip Peripherals</td> more </td> less </td></tr> PCB real-estate</td> less</td> more (external flash/RAM)</td></tr></tbody></table> Performance is related to memory architecture. In most MCUs, the memory architecture is fairly simple. Code is executed directly from flash, and on-chip SRAM is accessed directly. Programs do not need to be loaded from flash into RAM before running them. This architecture is very simple, leading to predictable real-time response and code execution. It takes a consistent amount of time to load code from flash into the processor for execution, so the timing for every instruction is fairly predictable. However, with this architecture, clock rate is typically limited to around 190MHz (example STM32F4). Some of the newer STM32 parts, such as the STM32F7 and STM32H7, add a L1 cache, which allows the CPU to run at 216MHz and 400MHz respectively.</p> MPUs have a more complex memory architecture in that they page code from Flash into SDRAM (both external to the MPU), and then from SDRAM into two or more levels of cache memory located on the MPU. The local cache memory is very fast, allowing these processors to run at high clock rates (1GHz or more in some cases). A memory management unit</a> (MMU) implements a virtual memory system in which physical pages from RAM are mapped into a virtual address space. This is very efficient as physical pages are only mapped in as needed and can be discarded if RAM is needed elsewhere. But this memory management introduces delays that are relatively long for some hard real-time systems. The first access to a block of code that has not been run recently takes time, as code needs to be paged from a file in flash into SDRAM and then loaded into the respective caches. But subsequent accesses from cache are blazing fast. An MMU also provides protection against one user space process corrupting memory of another process, or corrupting kernel memory. For complex systems (server/desktop) running multiple processes, this protection increases the system reliability. For many Embedded systems, there is only one main application, and protecting processes from each other is less of a concern. However, this memory protection is still useful in that a bug in the application does not crash the system, and recovery mechanisms can be built in to restart the app, or debug it during development.</p> In an MCU, program flow is controlled through simple loops and state machines (no operating system) or a real-time operating system</a> (RTOS). In an MPU system, a fairly complex operating system like Linux is typically required to manage the multiple levels of memory and storage, schedule the multiple processes that are running, and provide drivers for the complex hardware systems found in an MPU (USB, large displays, networking, etc). There is a great gulf between these two systems. Even though a RTOS may run on an MPU, there are much longer delays in execution due to the memory architecture. You still have to load code from flash to SDRAM and then through several layers of cache memory before the CPU can execute it. Once it is in cache, the code executes much faster and can do more work overall, but the occasional delays are still there. You might be able to play tricks by locking lines of code in cache or by using the small amount of on chip SRAM that may be present, but by the time you do this, you are now back to the smaller memory sizes on an MCU, and may as well just use an MCU.</p> After understanding the differences in memory architectures, we can understand that MCUs and MPUs are optimized for different things. An MCU is optimized for simplicity, cost, and predictable (real-time) response times. An MPU is optimized for getting the maximum amount of work done over time. Cost and real-time response is a secondary concern. There is not really a lot of middle ground between an MCU and MPU, evidenced by the large gap in memory sizes. You can’t fit standard Linux in an MCU, as there is only so much NOR Flash and SRAM that will fit on an MCU die. Once you move to more dense memory technologies such as NAND Flash and SDRAM, these must be implemented as separate dies and typically separate integrated circuit packages. These technologies are so efficient at storing large amounts of data, that you see a huge jump in memory capacity once you switch to the off-chip memories.</p> MCUs and MPUs also have different origins. Today’s 32-bit MCUs have descended from simpler 8-bit MCUs and are scaled-up embedded controller technologies. MPUs have descended from desktop and server computer systems and are scaled-down technologies. The differing priorities of these two paradigms reflect their different uses.</p> Keeping these differences in mind, the most fundamental question to ask in selecting an MCU or an MPU is whether hard real-time performance and reliability is most critical (control-centric application</strong>), or whether data processing performance and connectivity is most important (data-centric application</strong>). If both of these are important in your application, you should consider having both an MCU and MPU in the system, saving yourself untold pain.</p> Reliability is an interesting topic. I have seen MPU/Linux systems operate very reliably, and have also experienced issues that were very difficult to solve. MPU systems are many times more complex and have more unknowns, which translates into more risk. There are many more physical components and solder connections required to implement the system, all of which can fail over time with environmental stress. There are many millions of lines of code in an MPU system that you did not write, but you are still responsible for all of it – it all has to work for the system to function. Although an MPU system can be made fairly reliable, simple statistics tell us that MCUs will generally be more reliable than an MPU because there is less hardware that can fail and less lines of code running that may contain bugs.</p> One example of a problem we experienced when implementing control in an MPU system is a product where we were collecting data, and the data was being transferred back to a cloud server over a cellular modem. Unfortunately, the cellular modem in the system was not 100% reliable, and the only way to recover in some instances was to reboot the system when we detected a network failure, which cycled power to the modem (this modem module itself ran Linux). At one point, we added some basic control functionality to the system to control plant blowers based on a schedule. The control worked fine until the system had to restart due to modem connectivity issues, then the blower control was inactive during the reboot cycle (perhaps 30s or so). This was not a fatal problem in this application but also not ideal. The obvious solution is to get a more reliable modem or develop a better recovery method, but with units in the field and other development priorities, it is not always so simple. This is a classic example of the tension between complex data/connectivity systems and reliable control.</p> In an MCU system, the hardware system is relatively simple, and you are using a relatively small RTOS, or none at all. You write a greater percentage of the code in the system yourself, so if you have a simple task to do, and write reliable code, there is a potential to have a more reliable system than a comparable MPU system. However, if you are trying to do complex data processing and connectivity tasks on an MCU (such as writing your own database, network, or USB stack), chances are there will be bugs in your code and you will have a less reliable system than if you would have chosen an MPU using proven technologies. If your application is large and complex, then writing it in a safer language like Go or Rust on an MPU may provide a more reliable and maintainable product than trying to implement the same functionality in C++. Again, if you need both reliable real-time control and advanced data processing, then put both an MPU and MCU in the system.</p> Another advantage of an MPU system is they tend to be more general purpose and more functionality can be added over time. There is little danger of running out of code space, which often happens in MCU environments. With interfaces like USB, additional peripherals can be added in the future as requirements change, and with Linux, drivers for a vast array of hardware are available.</p> In some cases these lines between MCUs and MPUs are blurred by technologies like Linux RT extensions, or uCLinux, but you still must keep the above principles in mind when selecting the building blocks for your product. MCUs and MPUs are very different devices designed to handle different tasks. Before choosing, you should understand what you are trying to do, and what is important for your product.</p> Reference:</p> Atmel MCU vs MPU paper</a></li> ST page referencing both MCU and MPU</a></li> </ul> Accepting Constraints in Build Systems 2019-04-01T00:00:00+00:00 As Embedded Systems become more complex, the complexity of the process to build the software for these systems also increases. As humans, our ability to deal with complexity is limited, so we develop tools and processes</strong> to manage the complexity. In the end, these tools and processes are about constraints and patterns</strong>. A well-designed tool or process encourages you to do things in a way that is consistent and maintainable</strong>, which leads to reliable and predictable</strong> results.</p> </figure> As an example, consider using CMake vs standard Makefiles for a C++ application build system. This is a trade-off I often encounter, as my customers are using Linux to build products, and they need to develop a relatively complex software application to implement the product functionality. There are usually a handful of library dependencies as well (such as Qt, OpenCV, etc) – after all, the reason you use Linux is it provides a large amount of advanced functionality that you cannot afford to write yourself.</p> For a simple application, CMake is somewhat constraining and annoying. Its syntax is a little clumsy, and it is definitely less flexible than a Makefile. Thus some developers conclude that a Makefile is a better choice for Linux application builds. As the application grows, the Makefile also grows, and the number of dependencies increases. Then comes the time to cross compile the application so it will run on the target system. With a little work, we can convince the Makefile to accomplish this. Tool-chain file names and paths are often hard-coded in the Makefile. As the variations of systems the application needs to build on increase, all these variations are coded in the Makefile.</p> Five years down the road, something major changes. The application developers need some new features in a new version of OpenCV, or the System on Module (SOM) the target system is using just went obsolete. The developer who wrote the original Makefiles is gone. The Linux host distributions have changed. At this point the application build system collapses. So much has changed, and so much is hard coded in the Makefiles that getting them working again is a painful and tedious process. With some effort, they are finally building on one developer’s machine. But on the next developer’s machine, they fail with a different error.</p> The same dynamic happens when developers hack together tools to build an operating system image for an embedded device. This typically involves things like building the application or Linux kernel manually outside the build system, then writing a script to unpack an image file, insert the new binary, then pack the image file back up. The application build process is heavily dependent on the configuration of the developer’s workstation.</p> There is a better way. Use tools like Yocto/OpenEmbedded and CMake as they are were meant to be used. These tools do have a learning curve, and require some investment to learn, but in the end they are well worth it. The reason tools like OpenEmbedded and CMake work is that they are consistent, and use the same patterns over and over. We can illustrate the basic architecture of OpenEmbedded in a simple sketch, as shown below. OpenEmbedded is made up of a relatively small number of primitives:layers, recipes, packages, classes,</em> conf files, and tasks</em>.</p> </figure> Dependencies can be specified at the layer, recipe, package, and task level. Once these basic concepts are understood, then it is easy to find things in the build system and make necessary modifications. The boundaries are precise and well-defined. These patterns are not as flexible as a Makefile or Bash script, but they abstract huge amounts of complexity. This allows our mind to see the big picture and not get bogged down in thousands of details and to reliably execute extremely complex tasks. The same patterns are used over and over and are easy to recognize. Dependencies are well-controlled. The cost is to understand and use the established patterns</strong> and to accept the constraints</strong>. These constraints may be a little inconvenient at times. But when you compare this to the work these tools do for you, these inconveniences are relatively minor.</p> If you are going to build complex products using complex technologies like Linux, then you need powerful tools to manage this complexity. There is a learning curve. You may need to invest in training or hire someone to help you get started. Most other industries recognize this. Car mechanics undergo continuous training to learn how to work on new vehicles/systems. Nurses and other professionals receive training on new techniques and equipment. Build systems also require a significant investment in learning. We routinely hire experts (doctors, dentists, plumbers, carpenters, etc) in our personal lives to do things we don’t know how or have time to do. But somehow many developers think the build system and associated infrastructure should not need attention, can be ignored, or is trivial and can be implemented from scratch. Never mind the fact that the underlying software in the system (bootloader, kernel, and libraries) are many times more complex than the custom application software written for the system. You must be able to maintain</strong> it all. You can’t ignore the stuff you did not write – it all has to work. It has to be reliable and predictable</strong>. A build system is largely invisible until it starts causing problems.</p> Getting something working initially is relatively easy; keeping it working for years is another matter. It is similar to product development – building the first prototype that somewhat works often goes fairly quick. Getting something into production is a long, arduous slog. The build system is the foundation of your software project. It must be reliable and predictable</strong> because you can’t do much without it. A build system must do its part well without a lot of fuss, so we can focus on the problem being solved. A well implemented build system produces an image ready for installation with a single command and no manual steps. You check source code into git, run the build command, and out pops an image ready to deploy. There is no copying around binaries, etc. The efficiency gained by the effort to implement such a system pays itself back many times over the life of a project – especially those that are produced for 5-10 years.</p> Yocto and CMake are only used as examples in this article and may not be the best solution for every project. If you have something that is working well, keep doing what you are doing. But if what you are doing is not working, then ask yourself some questions:</p> Is my current system the best way to build complex software?</li> Is the process</strong> to build your project’s software simple and easy to reproduce by any developer on any machine?</li> Is it well documented?</li> Is it maintainable</strong>?</li> Is it consistent</strong> across all your products?</li> Is it reliable and predictable</strong>?</li> </ul> SD Card Write Speed Tests 2019-03-13T00:00:00+00:00 </figure> During Embedded Linux development, we often need to write large operating system images to SD cards for testing. Recently, I purchased a USB 3.0 SD card reader</a> from Plugable. Before that, I used a USB 2.0 SD card reader from IOGear.</p> </figure> How much better does a USB 3.0 card reader perform compared to a 2.0 card reader? To test, I wrote a large (2.2GB) image to a Transcend v30 card using both readers.</p> IO Gear USB 2.0 Reader</strong></td> Plugable USB 3.0 Reader</strong></td></tr> 15.3 MB/s</td> 33.5 MB/s</td></tr></tbody></table> The USB 3.0 reader is over twice as fast with this same card!</p> I then used the USB 3.0 reader to compare a number of cards.</p> SD Card Brand</strong></td> Size</strong></td> Grade</strong></td> Tested Speed</strong></td></tr> Transcend</td> 8GB</td> C10 HC</td> 8.8 MB/s</td></tr> Sandisk</td> 32GB</td> C10 HC I</td> 16.8 MB/s</td></tr> Transcend</td> 16GB</td> C10 HC I V30</td> 33.5 MB/s</td></tr> Sandisk</td> 64GB</td> XC I V30 A1</td> 44.5 MB/s</td></tr></tbody></table> The markings on the cards mean various things. The Wikipedia SD Card page</a> provides a nice overview. A few details are summarized below:</p> Bus interface</strong></p> I: 50MB/s Half/Full duplex, 104MB/s Half Duplex</li> </ul> </li> Capacity</strong></p> HC: Secure Digital High Capacity (SDHC) format – supports cards 2GB to 32GB in size</li> XC: Secure Digital eXtended Capacity (SDXC) format – supports cards 32GB to 2TB in size</li> </ul> </li> Speed</strong></p> 10 with C around it: Minimal sequential writing speed is 10 MB/s</li> 3 with U around it: UHS Speed Class: 30MB/s</li> V30: Video Speed Class: 30MB/s</li> A1: Application Performance Class 1: Read 1500 IOPS, Write 500 IOPS</li> </ul> </li> </ul> If you have not upgraded to a USB 3.0 card reader, it is probably worth your time to do so.</p> One note of caution – there are counterfeit cards for sale that look like SanDisk or Transcend cards. They are not manufactured by these companies, and use lower quality components. If the price seems too good to be true, it probably is. Always buy from a reputable supplier.</p> Getting started with Embedded Linux 2018-10-04T00:00:00+00:00 </figure> Recently I was asked by a developer, who has done windows development for 10 years, how to get started with Embedded Linux. Embedded Linux covers a lot of ground and includes a broad range of components/skills to put together an entire system. Below are a few suggestions.</p> Do I have the right Mindset?</h3> There are many different types of people; there are those who are who are a good fit for Linux, and those who are not. There are those who think Open Source Software (OSS) and Linux are the craziest thing they ever heard of, and those who think it’s a great idea. There are those who thrive on openness and collaboration, and those who feel threatened by it. A few specific aspects of this mindset are described below.</p> Code Gardening</h4> Much of your time as an embedded Linux developer is spent figuring stuff out (vs writing new code). Figuring out stuff is harder than writing new code (at least initially). It is similar to weeding the garden – often not something you are raring to do, but after you are finished, it feels really good. Much of the truly valuable work in life is like this. It initially appears kind of boring, but once you get started there are many interesting and rewarding aspects.</p> You have to like build systems and automation. Linux systems are complex. You can’t throw millions of lines of kernel and user space code together with a simple Makefile or a fancy IDE. Proficiency with command line tools is required. Builds must be clean and completely automated if you want to keep your sanity. Once you allow one manual step in the build, the floodgate has been opened, and these manual steps will start to accumulate at a frightening pace until you have an unmaintainable mess.</p> Collaboration</h4> Because of the complexity, you have to be able to collaborate with others. This requires a generous mindset. How can I make this easy for someone else to pick up once I am gone? Most likely this someone will be you in a couple months or years when you have forgotten many of the project details. The same really applies to coding. Do I write my code to be clever, do I hack it together, or do I focus on making it maintainable and easily understand by others? Automation and simplicity reduce cognitive load, freeing up resources to think about the problems that actually need solved, and allow multiple people to work on a project without a lot of friction.</p> The flip side of this is humility. With a complex system like Linux, there will be a lot of things you don’t know, and some difficult problems to solve. There is no one “manual” for Linux where you can go look up an answer. So we rely on open source communities for help. There are many people willing to help, so the key is to have the wisdom to know where and when to ask for help, and then the humility to ask. In Linux communities (and OSS in general), you will be surrounded by developers who are much smarter and more knowledgeable than you. At the very least, there will always be someone who knows something you don’t. If you don’t have the humility to acknowledge this and be willing to learn, then you will likely never be comfortable or effective working in an OSS environment.</p> NiH</h4> This can all be a challenge for developers who are endowed with a healthy dose of NiH (Not Invented Here Syndrome). These are people who imagine themselves happiest writing and maintaining their own code, and who are not inclined to share it with others. Some imagine themselves as great software architects and coders (very few of us really are) who can make something better. They don’t want to spend long, arduous hours learning and debugging a complex system written by someone else. If this describes you, then perhaps Embedded Linux is not for you.</p> Learning</h4> Are you a learner? Do you enjoy learning about new technology, tools, techniques, etc? Everything is always changing in the Linux world. New and compelling solutions are coming out every year. The real power of Linux and Open Source is knowing when and how to leverage these new technologies to solve real problems. If you are more comfortable doing things the way they have always been done, then perhaps you should just keep doing what you are doing.</p> The Problem of Abundance</h4> Can you deal with a lot of choices? With Embedded Linux, there are many options for about every aspect of the system. There are multiple init systems. There are many programming languages. There are many libraries available for various tasks. There is no one, true way to implement Embedded Linux – you will have to make a lot of choices.</p> Ideas for Learning Embedded Linux</h3> Books/Presentations/Documentation/Blogs</h4> There are many books published on Linux, Embedded Linux, Yocto, etc. The Yocto documentation</a> is pretty good. Videos from the Embedded Linux Conference</a> are freely available. There are many blogs that cover Embedded Linux topics (such as this one). Subscribing to blog RSS feeds in feedly</a> is a good way to keep up with recent developments.</p> The list of learning resources is endless, but perhaps the key thing to remember is there is no one place you can go to learn everything you need to know, like you might for a commercial RTOS. The good news is there is information available in many different formats, so you can choose what best fits your learning style.</p> Run Linux on your computers</h4> One of the best ways to learn a lot about Linux is to simply use it for your personal and work computers. Ubuntu is very easy to get started with. If you go with something like Arch Linux, the install process will teach you a lot. I am very pleased with the combination of Arch Linux plus the XFCE desktop environment. This is a no-nonsense setup that does its job well, is very customizable, and stays out of my way. Any distribution will work – just pick something.</p> Embedded Linux aside, there is little to lose, and a lot to gain from running Linux. Linux systems are naturally much more immune to viruses, malware, and various other attacks than Windows. Linux systems are much more flexible than Apple computers. Some things might not look as pretty, or might take a little more work to set up. But the stability, flexibility, and consistency of a Linux system will save you time in the long run. As a software developer, most of our time is spent in an editor and browser – these work very well in Linux. Additionally, the terminal is a first class citizen in Linux rather than an afterthought.</p> Learn Git</h4> Git is used in almost all open source projects these days, and enables new levels of collaboration. You need to know Git</a>.</p> Do small projects with Raspberry PI, etc.</h4> There are a number of embedded computer boards out there that are very well supported and quite cheap such as the Raspberry PI. Find something in your home that you want to automate – there’s nothing like a real project to motivate learning. With Linux, you learn best by doing. This will also give you an opportunity to learn application programming under Linux.</p> Build and install OpenEmbedded/Yocto or Buildroot</h4> Take the previously discussed home automation project and create a custom build for the board using Yocto/OE</a> or Buildroot</a>. This will give you practice learning how to put together a custom embedded Linux distribution. Create recipes to package, and include your custom applications in the image at build time (instead of installing them afterwards).</p> These tools have a learning curve and may seem intimidating initially, but Linux is a complex system, and you need powerful tools.</p> We think the BEC OE template</a> (now yoe-distro</a>) is a good way to set up Yocto/OE for a project.</p> Modify the Linux Kernel</h4> Most embedded Linux projects require a small amount of kernel modification. This can be something as simple as changing the GPIO configuration on an ARM SOC (System on chip), or writing a driver for some custom hardware. There are many resources available for learning Linux kernel programming. Perhaps try creating a custom driver for your PC or embedded Linux board. This driver could do something trivial, like monitor several GPIOs, etc.</p> Modify and install U-boot</h4> U-boot is a common bootloader for Embedded Linux systems (but not the only one). Locate the source code for your embedded board and learn how to build, modify, and install a custom version of U-boot.</p> Explore the many Application Programming Options</h4> There are so many interesting application programming options with Linux. Of course you can stick with traditional C/C++. But today, there are many other languages and environments that may be better for various applications (Python, NodeJS, Golang, Qt/QML, etc).</p> Find something that is broken and fix it</h4> There are many opportunities to improve open source code. Find something that does not work quite right, fix it, and get the changes accepted in the upstream project. This will provide valuable experience working with OSS communities. When working with complex systems like Linux, you will run into problems that are very hard to solve. Being comfortable with asking, and getting help from various OSS communities will be key to your success. Again, your mindset is key here. People who don’t have the right mindset will create fake names and email aliases to hide their true identity when asking questions. People with the right mindset are willing to be up front and admit there is something they don’t know.</p> Helping Others</h4> One of the best ways to really learn something is to teach it or help others. Once you reach a certain level of proficiency, there are many opportunities in open source communities (stack overflow, mail lists, IRC, etc) to help others. In doing so, we usually receive back much more than we give.</p> Maintain Yocto or Buildroot support for an embedded board</h4> There are many embedded Linux platforms out there that are not actively maintained or tested in Yocto or Buildroot. Get involved and help keep the platform current and working in the build system.</p> Set up a server in the cloud</h4> It is very cheap (less than $10/mo) to set up a server in Amazon AWS, Digital Ocean, Linode, Vultr, or one of the many other cloud providers. Although this is technically not embedded Linux, the skills required are very similar. It will also open up interesting possibilities of creating systems that are a combination of Embedded Linux devices and cloud servers.</p> Publish an Open Source project on Github</h4> Publishing an open source project is a good experience as it helps you learn to create something that is generally useful. For example, I wrote a little app</a> to test serial ports and published it on Github. Over the past few years, other people have provided nice contributions to the application, making it much more useful, with little effort on my part. However, I had to spend a little more time up front making the app generally useful and documenting how to use it.</p> Summary</h3> Embedded Linux brings a level of functionality to systems that is unmatched by anything else out there, thus its popularity. With this broad scope of functionality comes complexity. Linux has a learning curve, and using it successfully requires some amount of discipline and interaction with various OSS communities. The resources are endless – time and motivation are the only limitations. In life, the rewards are proportional to the effort put forth. If you are committed, working with Embedded Linux is very rewarding, and brings a new level of functionality to products.</p> Git, Versioning, and Branching for Embedded Linux Development 2018-02-14T00:00:00+00:00 </p> When building a product using Linux, versioning and branching of your software is an important consideration. Everyone’s needs are different depending on the size of the team, culture, and testing requirements, so there is no one size that fits all. However, after working on a number of different projects for a dozen or so different companies, there are several practices that are often used.</p> The most fundamental is to use Git to manage your software development. It is surprising the number of small companies developing software who do not use Git, or any version control system yet. The many benefits of Git stand alone</a>, but the minute you enter into the Linux world (or the Open Source Software (OSS) world in general), there are thousands of projects available, and 99.9% of them are developed in Git. Having a good understanding of Git is essential in using OSS. This is not really specific to Linux development, but rather any modern software development (cloud, web, mobile, etc). Most foundational software technologies today are open source. There is no other development methodology that scales to the complexity of modern software systems, and no one company contains the resources needed to create and maintain these systems. Git does have a steep learning curve and usually requires 2-3 weeks of committed use for new users to become comfortable with it. However, it is well worth it.</p> An extension of using Git is to consider hosting your Git repositories in the cloud. There are many companies who provide this service at a reasonable cost (some like Gitlab</a> are free), or it is relatively simple to host your own git server (can be done for $2.50/mo at Vultr</a> using Gitea</a>). Some may balk at putting a company’s valuable IP (intellectual property) assets in the cloud (where hackers might have an easier time getting to them) vs behind the secure company firewall. However, which of the following is the main impediment to your business?</p> The danger of competitors hacking into your cloud server, stealing your IP, and then implementing a product with it.</li> Shipping products in a timely fashion, and then maintaining them.</li> </ol> In my observation, #2 seems to be the challenge for most – especially as systems become more complex. Unless you are a top secret government agency, or a cutting edge chip/algorithm company, your software IP is probably of marginal use to anyone else. The reason for this is that integration and execution is most often the challenge, not coding. There is plenty of free/OSS software out there to do about anything. Integrating this into something that will make a product is the hard part. It is difficult getting software and hardware to work together, debugging problems, communicating with cloud infrastructure, software updates, maintaining a product, adding features, etc. Modern products are typically not static entities where you design it, toss it over the wall to production, produce millions, make lots of money, and never see it again. Rather modern products are complex, dynamic beasts that have continually changing requirements, bugs to fix, features needed for new customers, manufacturing issues to solve, obsolete parts to replace, and are part of a larger complex system. Execution is the hard</a> and valuable</a> part. Increasing developer efficiency and collaboration is important, especially with a distributed team of developers. You can require your developers to use company issued Windows computers and fussy VPNs in an attempt to protect your IP, or you can simply host your Git repositories in the cloud secured by industry proven methods such as https and ssh. As systems become more complex, it is critical to be able to involve domain experts in the project (consultants, contractors, employees at a different office, manufacturing, etc). A Git server in the cloud is the foundation of distributed development collaboration. Emailing zip files may be temping as quick and easy, but it is not a sustainable way to develop products that have so many moving parts. As an extension of this, use a web based Git repository manager (like Gitea, Gitlab, or a hosted solution) to implement your Git server. These repository managers greatly simplify the management of Git repos (teams, users, permissions, creating repositories, issue tracking, etc).</p> Software should be released with a version number. Semantic Versioning</a> is a good place to start. Your needs may vary, but you may as well start with the good ideas of others. Additionally, a changelog</a> should be maintained that lists all changes for each software release. Being able to quickly understand what changed with each release is critical as multiple versions are being used. Keeping a changelog requires a little discipline, but is fairly easy to do if you keep an unreleased section at the top of your changelog file to track changes for the upcoming release, as described in the above link.</p> A discussion on branching only really makes sense if you are using Git. Source code branches greatly improve your development process flow. The ease with which Git allows you to create and merge source code branches enables development flows which were impractical before Git. With an Embedded Linux project you are typically building images that get loaded on some type of hardware device. In this case, it’s useful to have development</em> and production</em> releases. Production releases are sent to customers, and development releases are used internally to test new features under development. Every software component that is under active development (OE build system, custom applications, kernel, etc) should likewise have production and development branches that feed into their respective builds. For a more extensive discussion of this topic, consider this post</a>. Again, start with the good ideas of others, and then use what makes sense. An OE build can be configured to pull from the latest HEAD of your custom components, so that for each new build you get the latest from the respective branches for various components. To do this, you typically have something like the following in a bitbake recipe:</p> SRC_URI = "git://git.mycompany.com/myproject.git;branch=develop;protocol=ssh;user=git PV = "4.1.15+git${SRCPV}" SRCREV = "${AUTOREV}" </code></pre> In some cases, for production builds we’ll manually specify the Git hash or tag in the bitbake recipe so that versions of all components are explicitly locked down. To specify a git version, replace ${AUTOREV}</em> with a Git hash. Either way, if you include the Git version (SRCPV</em>) in the package version (PV</em>) variable, you can always figure out what Git version was used to build a package for a particular release. Below is an example of a package file name:</p> my-app_4.1.15+git0+a05d9b23b9-r0.10_var_som_mx6.ipk </code></pre> In this case, a05d9b23b9</em> tells you what Git version was used to build my-app</em>. This is often adequate if you don’t spend a lot of time building or patching old software releases. With each release of software, it is beneficial to tag the software components used in the build with a version. This allows a clear view of what changed between releases, and allows us to easily check out old versions of software.</p> With development and production branches, we should consider how to version the different branches. One simple way is to use a sequence like:</p> 1.0.0 (production)</li> 1.0.900 (develop)</li> 1.0.901 (develop)</li> 1.0.1 (production bug fix)</li> 1.0.902 (develop)</li> merge develop to production and do a new release</li> 1.1.0 (production)</li> 1.1.900 (continue development</li> 1.1.901 (develop)</li> </ul> In this case, the development version is always the last production release MAJOR.MINOR</em></a> and the PATCH</em> version starts at 900 and is incremented with each development release. If a version number ends in 9xx, you know it’s a development build. You can use more complex version schemes like 1.1.0-alpha1 (instead of 1.0.901), but then you might run into sorting problems where 1.1.0-alpha1 is listed after 1.1.0, even though 1.1.0 is a later release than 1.1.0-alpha1. Recent versions</a> of the BEC software updater</a> look at all update files on a USB disk and pick the latest one using a natural sort of version numbers within the text. In the example below, you can see how versions are sorted:</p> [cbrake@mars ver]$ ls -lv total 0 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.0 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.1 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.900 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.901 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.1.0 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.1.0-alpha1 </code></pre> In this case, 1.0.9xx works much better for development builds if you want update systems to automatically pick the latest release from a list of options. You can also use versions like 1.0.0+1.1.0-alpha1 to get around the sorting problem if you want something a little more explicit.</p> Versioning and branching does not have to be complex, and in many cases the above is adequate for small teams. Your needs may vary, but the important thing is to have a source control, versioning, and branching process in place. These practices require some discipline to implement, and may seem unnecessary to those who have not used them, but with a little experience we soon realize that efforts to introduce appropriate amounts of organization and process into our development efforts clearly communicates what we did, what we are doing, and what gets released to production. This then frees us up to focus on what really adds value.</p> Separation of Source and Build Directories 2017-11-16T00:00:00+00:00 </p> As we work with larger and more complex systems (i.e. Linux), more and more of our time is spent on integration and pulling different pieces together. We often need to debug or understand code we did not write – especially in build systems. To work effectively in this scenario you must be able to quickly search through a lot of source code. Therefore, we are always looking for ways to make this more efficient.</p> One pattern that is very useful is separating source and build files into two separate directories at the top level of the project. There are several reasons for this:</p> It is easy to grep the source (or today we ripgrep</a>) if it is not littered with transient build data.</li> Its easy to reset the build by simply deleting the build directory.</li> </ul> This is especially important for large builds like OpenEmbedded, where a build directory can be on the order of 40GiB. If your grep tool has to process 40GiB of data every time it runs, it will be very slow.</p> This separation should occur at the top level of the project. Ideally you want something like this for an OpenEmbedded build:</p> ├── build/ │ ├── sstate-cache/ │ └── tmp/ ├── downloads/ ├── sources/ │ ├── bitbake/ │ ├── meta-bec/ │ ├── meta-browser/ │ ├── meta-freescale/ │ ├── meta-freescale-3rdparty/ │ ├── meta-freescale-distro/ │ ├── meta-openembedded/ │ ├── meta-qt5/ │ ├── meta-variscite-fslc/ │ ├── meta-myproject/ │ ├── meta-mycompany/ │ └── openembedded-core/ ├── conf/ └── etc ... </code></pre> The sources</em> directory contains OpenEmbedded metadata (core, 3rd party, and your own custom meta-myproject layers). The downloads</em> directory might contain 3rd party sources or tar balls that are downloaded by the build tool. And the build</em> directory contains transient build data that can be deleted and re-created at any time. This configuration is used in the Yoe distribution</a>. It should be noted that in this example, the desire is to cleanly separate OE metadata (source) from build data. Source code for individual packages is considered build data from an OE perspective. Most of the development work in this case is creating/customizing OE recipes, images, etc, thus the need is to quickly grep the various OE meta layers.</p> This idea can be contrasted with the Poky reference distribution directory layout:</p> ├── bitbake ├── build │ ├── cache │ ├── conf │ ├── downloads │ ├── sstate-cache │ └── tmp ├── documentation ├── meta ├── meta-poky ├── meta-selftest ├── meta-skeleton ├── meta-yocto-bsp └── scripts </code></pre> In this case, there are multiple meta directories in the top level. If you want to search all of the OE meta data, you need to instruct your search tool to exclude the build</em> directory. Additionally, the build directory is dynamically generated, which includes the conf directory. If you want to completely reset your build, you need to clean several directories in the build directory. For a production build, you need some of the conf files to be in revision control; therefore, they are more difficult to manage when they are dynamically generated, or copied from another location. Why not move conf to the top and use it there?</p> Most decisions are tradeoffs, so there may be good reasons for the Poky directory structure. For a reference (demo) distro, simplicity and ease of setup may be valued over reproducibility. For production builds (builds used in products), rock solid reproducability is important. To get that, you need:</p> all source and configuration is locked down in version control</li> a build should be a minimal number of steps: checkout source, a very simple setup/configuration step, and then a build command. There should be no complex scripts that copy files around that may be modified later, or manual steps.</li> the output is ready to program into the target. There are no manual post processing steps.</li> keep things as simple as possible.</li> </ul> The separation of source and build can also be used/observed with many other build systems. With CMake, if you have a project directory named myproject</em>, you can do something like:</p> mkdir myproject-build cd myproject-build cmake ../myproject make </code></pre> Qt Creator also sets up a separate build directory by default.</p> With the linux kernel, you can specify a separate build directory with the ‘O’ option:</p> cd linux mkdir ../linux-build make O=../linux-build </code></pre> Think about directory structure – it can make a difference.</p> Understanding the NXP i.MX6UL Pin Mux (Part 2) 2017-11-06T00:00:00+00:00 In the previous post</a>, it was noted that bit 30 needs to be set in the i.MX6UL pad config if you want to read the state of a GPIO output. Digging into this a bit more, we find the following text in the Documentation/devicetree/bindings/pinctrl/fsl,imx-pinctrl.txt</em> file:</p> SION(1 << 30): Software Input On Field.</em> Force the selected mux mode input path no matter of MUX_MODE functionality. By default the input path is determined by functionality of the selected mux mode (regular).</em></p> In the i.MX6UL developers manual, we also find the following:</p> </p> When we look at the SW_MUX_CTL</em> register, we notice the SION</em> bit is located at the bit 4 position.</p> </p> So the question is: How does bit 30 in the device tree file get mapped to bit 4 in the SW_MUX_CTL</em> register? In the Linux kernel source we find the following:</p> drivers/pinctrl/freescale/pinctrl-imx.c:</em></p> /* The bits in CONFIG cell defined in binding doc/ #define IMX_NO_PAD_CTL 0x80000000 / no pin config need / #define IMX_PAD_SION 0x40000000 / set SION / ... / SION bit is in mux register / config = be32_to_cpu(list++); if (config & IMX_PAD_SION) pin->mux_mode \|= IOMUXC_CONFIG_SION; pin->config = config & ~IMX_PAD_SION; </code></pre> drivers/pinctrl/freescale/pinctrl-imx.h</em></p> #define IOMUXC_CONFIG_SION (0x1 << 4) </code></pre> From this code, we can see that bit 30 is a virtual bit that gets translated into bit 4 in the mux mode register.</p> Hope this helps in understanding how to configure the i.MX6UL gpio. Once you understand the details; the process is very simple.</p> Understanding the NXP i.MX6UL Pin Mux 2017-09-14T00:00:00+00:00 (note, the article is also applicable to the i.MX6ULL as these processors are very similar)</em></p> The NXP i.MX6UL application processor has a very flexible pin multiplexer, that is somewhat difficult to understand at first glance. Most times when we’re configuring the pin mux in Linux, we modify Device Tree files, so perhaps that is the place to start. The pin mux options for the i.MX6UL are defined in the arch/arm/boot/dts/imx6ul-pinfunc.h</em></a> file. The arguments to the macros in this file are defined as:</p> /* * The pin function ID is a tuple of * <mux_reg conf_reg input_reg mux_mode input_val> / </code></pre> The first 5 are defined in the pinfunc.h file, where the config value is defined in a user’s DTS file. As an example, consider the following section from imx6ul-pinfunc.h.</em> Column headings are added for clarity.</p> // headings: mux_ conf_ input_ mux_ input // reg reg reg val _val #define MX6UL_PAD_GPIO1_IO09__PWM2_OUT 0x0080 0x030C 0x0000 0 0 #define MX6UL_PAD_GPIO1_IO09__WDOG1_WDOG_ANY 0x0080 0x030C 0x0000 1 0 #define MX6UL_PAD_GPIO1_IO09__SPDIF_IN 0x0080 0x030C 0x0618 2 0 #define MX6UL_PAD_GPIO1_IO09__CSI_HSYNC 0x0080 0x030C 0x0524 3 1 #define MX6UL_PAD_GPIO1_IO09__USDHC2_RESET_B 0x0080 0x030C 0x0000 4 0 #define MX6UL_PAD_GPIO1_IO09__GPIO1_IO09 0x0080 0x030C 0x0000 5 0 #define MX6UL_PAD_GPIO1_IO09__USDHC1_RESET_B 0x0080 0x030C 0x0000 6 0 #define MX6UL_PAD_GPIO1_IO09__UART5_DCE_CTS 0x0080 0x030C 0x0000 8 0 #define MX6UL_PAD_GPIO1_IO09__UART5_DTE_RTS 0x0080 0x030C 0x0640 8 2 </code></pre> From the i.MX6UL processor reference manual (IOMUX Controller chapter), we can determine the mux_reg is indeed located at offset 0x80. We can also see that the mux_val</em> values match MUX_MODE in the datasheet.</p> </a></p> The mux_val</em> in the imx6ul-pinfunc.h</em> file also match the MUX_MODE field values in the above register.</p> The config register can also be located in the datasheet. Again, we notice that the offset of 0x30C matches config_reg</em> setting.</p> </a></p> Lets consider the last option listed in the pinfunc file: UART5_DTE_RTS. The input register is used to mux a pin to an input function inside the SOC – in this case, the UART5_RTS_B input. We find the input register at offset 0x640 and can see that the input value of 0x2 matches what we would expect in the reference manual.</p> </a></p> So, how do we know if the pinmux is configured correctly for a pin? One way is to use devmem2 and dump the register contents directly from user space:</p> root@imx6ul-var-dart:/sys/class/gpio# devmem2 0x20e0080 /dev/mem opened. Memory mapped at address 0x76f02000. Read at address 0x020E0080 (0x76f02080): 0x00000005 </code></pre> The mux value is 0x5, which configures the pin for the default GPIO1_9 function. Next we can look at the config register:</p> root@imx6ul-var-dart:/sys/class/gpio# devmem2 0x20e030C /dev/mem opened. Memory mapped at address 0x76f0b000. Read at address 0x020E030C (0x76f0b30c): 0x000010B0 </code></pre> If we further decode the bits in this register we find:</p> Bit index: 12─╮ 8─╮ 4─╮ 0─╮ Bits: 1 0000 1011 0000 </code></pre> and can conclude that the following options are set:</p> bits 3-5 = 6: DSE_43ohm</li> bits 6-7 = 2: Medium speed (100MHz)</li> bit 12: PKE: Pull/Keeper Enabled</li> </ul> There are additional options that can be configured such as hysteresis, pull up/down, open drain, and slew rate. I/O is very configurable on this device!</p> If we want to use this pin as a GPIO, it appears that everything is configured correctly, so let’s give it a try. GPIO in the i.MX6UL is banked in 32 bit registers, so we need to map this to a sequential GPIO in Linux. The formula for this mapping is:</p> linux gpio number = (gpio_bank - 1) 32 + gpio_bit</em></p> For GPIO1_9, this would be GPIO9.</p> linux gpio number = (1 - 1) * 32 + 9 = 9</em></p> To read the GPIO, execute the following commands:</p> root@imx6ul-var-dart:~# cd /sys/class/gpio/ root@imx6ul-var-dart:~# echo 9 > export root@imx6ul-var-dart:~# cat gpio9/value 0 (current gpio value) </code></pre> Another nuance of configuring the i.MX6UL pin mux in Linux is that for GPIO outputs, you need to set a magical bit 30 in the Device Tree configuration as show below. If you don’t set bit 30, it will always read back as 0.</p> MX6UL_PAD_JTAG_TDO__GPIO1_IO12 0x4001b0b0 /* 2_nSLEEP / </code></pre> The reason for this will be explored in a future post.</p> Why Git? 2016-08-18T00:00:00+00:00 Some time back, I gave a presentation</a> that included an overview of the Git version control system. I still occasionally get asked why Git should be used instead of Subversion, as it seems harder at first. Most developers don’t really understand Git until they have used it for awhile, and then they will have an “aha moment.” There are 3 features of Git that are especially interesting to me:</p> many repositories (vs. one large repository)</li> distributed development</li> cheap branches</li> </ol> The fundamental driver for better tools is increasing system complexity. Developers are required to manage and integrate more third party software, work in distributed teams, and more effectively re-use the software we do have. Git is a tool that helps you accomplish these goals.</p> Git is a fairly new project, but it is already nearly universal among open source software projects. Its uptake and acceptance speaks for itself.</p> Many Repositories</h2> In the past when most companies used Subversion, there was one huge company repository (I’ve seen them in the 10’s of GB in size) that held a hierarchical tree of all the company source code. This worked fairly well, and is comfortable in that it is very similar to a file system directory structure. However, this model is not very flexible in that it does not have a consistent way to re-use components between different projects. Some people simple copy source code. Some have a “common” project that is included in all their other projects. Subversion externals can be used. But with Git, typically a separate repository is created for each software component. There are perhaps several reasons for this, but one reason is that Git simply does not scale to huge multi-GByte repositories. However, this turns out to be a blessing in disguise as I think it is a better model in many cases. What we end up with is more of a catalog of software components rather than a rigid hierarchy.</p> </p> There is much emphasis these days made on modular, re-usable software components (Object Oriented Programming, plugins, etc.). Why not keep things modular at the repository level? Another example of this type of organization is the Internet itself. It is not a hierarchy of information, but rather a flat system that is organized by hyperlinks.</p> One of the benefits of organizing your source code this way is that it encourages clean boundaries between software components. Each software component needs to stand on its own fairly well without being propped up by header files located in an unrelated source tree 3 levels up in the directory hierarchy. This type of organization forces us to make better use of standard build system practices.</p> Distributed Development</h2> The “many repository” paradigm has been partly driven by the distributed development paradigm. Git solves the problem of multiple developers working in multiple repositories very well. Because we want to use and customize projects like the Linux kernel, U-boot, and OpenEmbedded in our projects, then we naturally find ourself in the situation where we need to manage multiple repositories. Yes, you can check the Linux kernel into your company Subversion repository, but you are much better off long term if you bite the bullet and implement your own Git infrastructure.</p> As we consider the product development process, we need to consider the life cycle of a product. Most products live for at least several years, and will go through several software iterations. If we can update the software components we use, then we can add value to the product in the form of new or updated drivers to support new peripherals, new libraries, performance improvements, etc. But we are dealing with millions of lines of source code, so we must have an efficient way to deal with software projects of this size. The below Figure 2, below, illustrates how you might organize a typical project. Notice we can pull updates from the U-boot and Kernel source trees at any time in the development process. We might have an outside team working an application, and we then easily synchronize the repositories when it makes sense.</p> </p> There are many other design flows possible. Once you have the ability to support multiple branches and repositories easily, it becomes trivial to implement a staging/testing repository for QA processes, maintenance repositories for supporting old releases, etc.</p> Cheap Branches</h2> The last feature discussed in this article is that of cheap branches. Just as Git excels in allowing us easily integrate changes from multiple repositories at a team level, Git also allows us to easily work with multiple branches at the personal developer level. This is a revolutionary feature, and perhaps has to be experienced to be fully appreciated. With SVN, branching and merging is painful; therefore branches are not used a lot. In Git, branches are easy, therefore branching is naturally encouraged. Below is a typical flow of development.</p> </a></p> In this case, a developer is working on a performance improvement that is largely experimental – depending how it works out, it may or may not be used. So he starts a branch and experiments with various changes. As he goes along, he makes nice granular commits that are easy to follow. He may reach a dead-end, delete the branch and start a new one. In this middle of this development, his boss asks him to make a quick bug fix. He checks out master, makes the change, switches back to the branch and keeps working. The switching between branches takes seconds and is very easy. Finally, he has some performance changes he is happy with. He merges master to his perf-improvement branch and tests his changes with the latest master changes. Everything looks good, so he then merges his branch to master. Git makes this type of development flow natural and easy. Because branching is painful in SVN, changes tend to be made in large commits that are difficult to understand.</p> The ability to keep separate tasks isolated is analogous to having a clean, neat desk with only one thing on it, or having a messy desk with a half a dozen different projects going at once. Git is a tool to help maintain different working contexts, and it allows work to be staged in a natural flow. In the end, Git will change how you work.</p> Using Go in place of a Spreadsheet 2016-06-28T00:00:00+00:00 Recently I needed to calculate NAND partition tables for a project where we will be supporting a number of different flash parts from 500MB to 2GB. I first tried this in a spreadsheet, but found it difficult to work easily with hex numbers and do the calculations I needed. I then looked into options for formatting text in columns from a program and found the nice text/tabwriter</a> Go library. With a few lines of code</a>, I was then able to get the below output, which is quite easy to read. The only tricky part was figuring out that for right justified data, you need to:</p> not use tabs for the padding character</li> add a trailing \t in the input data</li> </ol> The Go stdlib is quite extensive, and often has everything you need. This makes “go run” a nice solution for scripting tasks.</p> Old Partitions Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x1c0000 1.75 14 mtd2 uboot env 0x240000 0x40000 0.25 2 mtd3 linux 0x280000 0xa00000 10 80 mtd4 rootfs 0xc80000 0x186a0000 390.625 3125 mtd5 data 0x19320000 0x26ce0000 620.875 4967 total size = 0x40000000, 1024MiB New Partitions 500MB Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x500000 5 40 mtd2 uboot env 0x580000 0x40000 0.25 2 mtd3 linux1 0x5c0000 0x1400000 20 160 mtd4 linux2 0x19c0000 0x1400000 20 160 mtd5 rootfs1 0x2dc0000 0xa000000 160 1280 mtd6 rootfs2 0xcdc0000 0xa000000 160 1280 mtd7 log 0x16dc0000 0xa00000 10 80 mtd8 data 0x177c0000 0x7c40000 124.25 994 total size = 0x1f400000, 500MiB size of boot partitions: 45.75 MiB New Partitions 1G Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x500000 5 40 mtd2 uboot env 0x580000 0x40000 0.25 2 mtd3 linux1 0x5c0000 0x1400000 20 160 mtd4 linux2 0x19c0000 0x1400000 20 160 mtd5 rootfs1 0x2dc0000 0xfa00000 250 2000 mtd6 rootfs2 0x127c0000 0xfa00000 250 2000 mtd7 log 0x221c0000 0x3200000 50 400 mtd8 data 0x253c0000 0x1ac40000 428.25 3426 total size = 0x40000000, 1024MiB size of boot partitions: 45.75 MiB New Partitions 2G Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x500000 5 40 mtd2 uboot env 0x580000 0x40000 0.25 2 mtd3 linux1 0x5c0000 0x1400000 20 160 mtd4 linux2 0x19c0000 0x1400000 20 160 mtd5 rootfs1 0x2dc0000 0x1f400000 500 4000 mtd6 rootfs2 0x221c0000 0x1f400000 500 4000 mtd7 log 0x415c0000 0x6400000 100 800 mtd8 data 0x479c0000 0x38640000 902.25 7218 total size = 0x80000000, 2048MiB size of boot partitions: 45.75 MiB </code></pre> </code></p> Using Docker for OE/Yocto builds 2016-04-21T00:00:00+00:00 </a></p> Why Docker? When using OE to build software for products, we often run into the scenario where software needs to be built using the same version of OpenEmbedded over the course of several years. Production builds need to be predictable. We’ve also observed that old versions of OE often break as new Linux distros come out. This is just the result of the complexity of building tool chains. Additionally, for predictable builds you really don’t want to be changing the build OS. This requirement automatically rules out Arch Linux, Debian Unstable, Gentoo, etc as production build machines. Additionally, having developers debug OE build issues on varying workstation distributions is frustrating and time consuming.</p> There are several options:</p> keep a golden build machine around with a fixed OS</li> do builds in cloud on a machine with a fixed OS</li> build in a systemd-nspawn, or chroot</li> build in a virtual machine (Virtualbox, etc)</li> use Docker</li> </ul> I have used all of the above and recently started experimenting with Docker. As an example, I set up a container named cbrake/oe-build</a> and provided an example of how to use a Debian based container for OE builds.</p> Docker requires a bit of a mind-shift from virtual machines. The 1st aha moment with Docker comes when you realize data and build directories should be mapped from the host into the container such that the container does not really have any state. There are no ssh keys, etc in the container. This allows the container to be immutable and easily replicated across any number of machines. Stuff that needs to change goes in your build directory and is managed using Git. Users specific information lives on your host machine.</p> The 2nd aha moment with Docker is running commands with Docker instead of working in the context of Docker. For years, I have run OE builds in a systemd-nspawn. I’d always have two terminal windows open – one for the container context, and one for the host. With Docker, you typically just work in your host context, and spin up a Docker container every time you need to run a command in the container (bitbake, etc). If you define a few shell functions to wrap the docker commands, you hardly even know that commands are executing in the container. This might seem impractical, but it actually works quite well. Of course you can still run a container shell and work in the context of the container if you want, but its not really necessary or desired once you get the container set up.</p> Docker is a neat tool and allows you to have a consistent environment for developer and production build machines. It also encourages a natural separation of immutable containers and transient data that makes it much simpler to manage and distribute containers.</p> Setting up a Go development environment 2016-04-15T00:00:00+00:00 Go has a pretty neat development environment, and its helpful to set up a standard GOPATH on your workstation up front. This is what I do:</p> mkdir ~/go</p> </li> add the following to .bashrc (or some file that configures your env on login)</p> export GOPATH=~/go</li> export PATH=$GOPATH/bin:$PATH</li> </ul> </li> </ul> Now, after you log in, you can do things like:</p> go get -u github.com/monochromegane/the_platinum_searcher/... </code></pre> </code></pre> go get -u github.com/schachmat/wego </code></pre> (set up wego with forcast developer key) </code></pre> </code></pre> There are lots of</a> Go apps</a> out there and if you just set up a global GOPATH, they are trivial to install and use anywhere on your system! Generally your ~/go directory is also used for Go development projects, but that is a topic for another day.</p> </code></pre> Go Language for Embedded ARM Linux Systems 2016-03-08T00:00:00+00:00 In the quest for technologies that work well for embedded Linux systems, I recently gave Go another try. The last time</a>I tried this was very early on and there were some floating point issues on ARM that appear to be fixed now. Having spent a few days porting an existing application to Go, there is a lot to like about Go.</p> Compiles very fast</li> Nice set of default libraries that are nice to use</li> Binaries are reasonable size</li> Memory usage is reasonable</li> Very simple to deploy (run time and libs are bundled with a single executable)</li> many 3rd party libraries available</li> Tooling is good (build, unit testing, etc)</li> </ul> As an example of running a Go app on an embedded Linux system, enter the following in hello.go:</p> package main</p> import (</p> “fmt”</p> )</p> func main() {</p> fmt.Println(“Hello”);</p> }</p> To run on x86:</p> go run Hello.go </code></pre> To build for ARM Linux:</p> $ GOARCH=arm GOARM=5 go build hello.go $ file hello hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped </code></pre> As you can see, cross compiling is very easy with Go.</p> Now, to try this exe on a Raspberry PI 3 running Arch Linux:</p> [root@alarmpi ~]# ./hello Hello </code></pre> I did notice the time to build for ARM is a bit longer than building for x86:</p> $ time go build hello.go real 0m0.262s user 0m0.337s sys 0m0.030s </code></pre> $ time GOARCH=arm GOARM=5 go build hello.go </code></pre> real 0m2.533s user 0m4.930s sys 0m0.183s </code></pre> But, still pretty reasonable considering what all it is doing. The simplicity of all this is very attractive. There is no runtime to cross compile and deploy. Just one executable and you’re good to Go.</p> IOT Protocols: MQTT vs CoAP vs HTTP 2015-12-11T00:00:00+00:00 With the explosion of IOT (Internet of things), there are now more technologies we can use to build systems. In reality, we’ve been doing IOT for years. We’ve been networking devices for a long time. We’ve been collecting data from remote nodes. This is nothing new, but what the IOT movement brings to the table is technologies that are much lower cost, and more standardized. Two of these technologies are MQTT and CoAP. Both very interesting, and very useful. Recently, I helped a system manufacturer think through the architecture of a system with the following requirements:</p> They plan to sell a relatively small number (200-1000) of fairly expensive machines ($5,000-$10,000). The point here is we are not dealing with millions of $10 nodes, which is really the vision of IOT.</li> There might be as many as a dozen machines installed in each location.</li> Users of these machines need to monitor them remotely. The manufacturer would also like to monitor them for diagnostics and customer support.</li> The machines can be connected to the Internet at the install sites.</li> These machines are installed at various locations where the manufacturer doesn’t have much control over the network (firewall, filtering, which ports are blocked, etc).</li> The machines don’t need to talk to each other.</li> It should be relatively difficult to hack into a machine and mess up what it is doing.</li> Data will be mostly going from machines to the cloud.</li> </ol> There are several ways to address these requirements, but one of the simplest today is for each machine to talk to a server in the cloud, and then users monitor and control the machines through a web application running in the cloud. The web application can be designed to be viewed on both personal computers and mobile devices.</p> </a></p> So the question is which protocol do we use to communicate from the installed machines to the cloud? In this case, we really do not need a lot of efficiency, as the number of nodes is relatively small. Bandwidth requirements are not really a huge concern because a) the number of nodes is relatively small, and b) we can use the existing broadband infrastructure at the install sites. While it might be interesting to use one of the new protocols like MQTT or CoAP, we need to carefully consider them in the context of requirement #5 – we don’t have control over what ports are blocked or what type of filtering/proxying happens at the install site. Support costs can run very high debugging networking problems at remote locations. Because MQTT and CoAP typically use non-standard ports, they are less attractive for this application. Even if we did run MQTT or CoAP over port 80, the install site might have internet filters or proxies that would reject non HTTP data. So this puts us back to HTTP, perhaps with a REST implementation. As it turns out, this is not a bad way to go. Even though HTTP is old, it is still a perfectly valid protocol for this application. Some of the advantages include:</p> HTTP is the most compatible with existing network infrastructure, and you are pretty much guaranteed HTTP will work. If a web browser works, then machines in this scenario should be able to send data to the cloud.</li> Stateless HTTP/REST will traverse firewalls and NAT systems easily. Again, web browsers work.</li> Because the Machines initiate the connection to the server (users are not connecting directly to a browser on the machine), the machines are much easier to secure. Thy don’t even need any open ports. Most of the security effort in this system only needs to be focused in one place (cloud server).</li> HTTP provides well understood authentication and encryption mechanisms.</li> </ul> There are other applications where MQTT or CoAP might make more sense.</p> In a system connected via cellular modems where data is very expensive, you might want to consider CoAP, because it can use UDP which eliminates the overhead of TCP/IP, which is significant (both MQTT and HTTP require TCP/IP). Especially if you have 1000’s of nodes, and are trying to keep the monthly usage of each one under 5MB.</li> If the various machines need to talk to each other, then MQTT would be a good choice.</li> </ul> While I’d like to use MQTT or CoAP, in this application I’m going to recommend they stick with HTTP(S). Even though MQTT might be more convenient to work with, the support costs of debugging network issues at install sites will probably swamp any advantages during development. This project can benefit from the host of</a> Wifi enabled</a> microcontroller systems</a>, low cost embedded Linux</a> technologies, or cloud technologies</a> aimed at IOT systems. All of these support HTTP just fine.</p> OpenEmbedded / Yocto Usage Survey 2015-11-18T00:00:00+00:00 Recently, I sent a survey to the OpenEmbedded and Yocto mail lists asking a series of questions about how they use OE. 38 people responded. The average number of years using OE is 4.8 years.</p> Below are the results of the questions. (update 2015-11-23 – the below numbers are absolute numbers, not percentage)</p> </a></p> </a></a></a></a></a></a></a></a></a></p> On the question about tooling, 4 people responded they use Toaster (two specified toaster analysis mode). Two use devtool. One uses ADT.</p> I plan to follow up with additional articles discussing some of these results.</p> Thanks to all who responded to this survey.</p> Rebasing a set of changes with Git 2015-11-11T00:00:00+00:00 One of the common things we do during Linux kernel development is move a series of patches from one kernel version to a similar version (say Linux 4.1 to 4.1.12). This is required as new stable versions of particular kernel version are released. One approach is to merge, but then your changes are mixed in with upstream commits and are more difficult to manage. Git rebase offers a convenient way to move a set of patches. In the following example we have a series of changes we made (or patches we applied) on top of the 4.1 kernel.</p> </a></p> Now we want to move the same patchset to the 4.1.12 stable kernel. To do this, we can use the following commands:</p> git checkout -b 4.1.12-armv7-x0-dugen2 4.1-armv7-x0-dugen2 (create a new branch that will be rebased)</li> git rebase --onto v4.1.12 v4.1 4.1.12-armv7-x0-dugen2</li> </ul> A few details on the git rebase arguments:</p> --onto v4.1.12 (this tells git what version we want to move our changes to)</li> v4.1 (upstream – tells git where our patchset starts)</li> 4.1.12-armv7-x0-dugen2 (branch – is the branch to do the operation on)</li> </ul> After the rebase, our patchset looks very similar, only on top of the 4.1.12 kernel version.</p> </a></p> It this example (which included many more patches than shown above), I only encountered one conflict which was easy to resolve. This process can also be used to move a patchset to a new kernel version (say 4.2, or 4.3), but you would obviously expect more conflicts.</p> Modifying the BusyBox config in OpenEmbedded 2015-09-30T00:00:00+00:00 Recently, I needed to enable the eject</em> command in BusyBox for an OpenEmbedded (Yocto) based project. Below is a way to do this in a fairly painless way:</p> bitbake -c menuconfig busybox (enable the eject command in the config and save)</li> bitbake -c diffconfig busybox (this generates a config fragment, note the fragment file location)</li> recipetool appendsrcfile -w [path to layer] busybox [path to fragment generated in step #2]</li> </ol> Now, in my project layer, I have the following files:</p> recipes-core/busybox/busybox_%.bbappend:</em></p> SRC_URI += "file://fragment.cfg" FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:" </code></pre> recipes-core/busybox/busybox/fragment.cfg:</em></p> # Wed Sep 30 15:23:49 2015 CONFIG_EJECT=y CONFIG_FEATURE_EJECT_SCSI=y </code></pre> Wow, that is pretty neat! The same procedure should work on kernel recipes as well.</p> Notes:</p> If you have trouble running bitbake -c menuconfig</em> in a remote ssh shell, trying installing screen on the build machine.</li> recipetool is located in openembedded-core/scripts/recipetool</em></li> Chris L.</a> noted the -w option in recipetool would have simplified the append slightly (would avoid the subdir= on the SRC_URI, since it’d install it to workdir instead of the source tree) Great idea – example above modified to include -w.</li> </ul> Best practices for using CMake in OpenEmbedded/Yocto 2015-03-10T00:00:00+00:00 I’ve already written about using autotools</a> and qmake</a> in OE. With recent projects, we’re using CMake to build most C/C++ components. Like any good tool, CMake has a learning curve, but it seems to do its job quite well. Below are examples of a CMake build file, and corresponding OE recipe.</p> CMakeLists.txt</em></p> cmake_minimum_required(VERSION 2.8) project(MyApp) file(GLOB_RECURSE src/.cpp) add_executable(MyAPP ${src_files}) target_link_libraries(MyApp pthread) install(TARGETS MyApp DESTINATION bin) </code></pre> myapp.bb (OE recipe)</em></p> DESCRIPTION = “My xyz app” LICENSE = “CLOSED” inherit cmake SRCREV = “${AUTOREV}” PV = “1.1+gitr${SRCPV}” SRC_URI = “git://my-git-server/myapp.git;branch=master;protocol=ssh;user=git” S = “${WORKDIR}/git” BBCLASSEXTEND = “native”</p> Notice the OE recipe above has no build/install logic other than “inherit cmake”.</p> Below are some tips on using CMake.</p> put your source files in a src directory. This allows you to easily build a list by using GLOB patterns. If you put the source files in the root directory of your application source, then the GLOB pattern will also find temporary C files that CMake generates, and you’ll get multiple “main” errors. Some recommend against using GLOB patterns for finding source files because it may miss additions, but in practice I find the benefits of this pattern outweigh the negatives.</li> do all install operations in CMake if possible (vs do_install in the OE recipe). The reason for this is its much</strong> easier to debug install in CMake then OE. You can build the app for x86 on your workstation, run “make install” and easily verify the files get installed correctly. Typically, things get installed in /usr/local when building on your workstation, and /usr when built in the context of OE.</li> do most of your development on your workstation. Most of the time, you can test 99% of an application’s functionality on your workstation. Do what you can there with unit tests where development is much faster and easier.</li> change ownership of /usr/local by running “sudo chown -R $USER /usr/local</a></em>”. This allows you to run “make install” of the CMake project during development without sudo. It also helps ensure you are not accidentally installing stuff outside /usr/local.</li> you don’t need to add header files to the target sources. CMake automatically figures out include dependencies.</li> you can run “make VERBOSE=1”</em> to see the commands make is running during a compile.</li> cmake -DCMAKE_BUILD_TYPE=Debug can be used to add debugging information (essentially -g).</li> When developing on your workstation, do the following to build: mkdir build; cd build; cmake ../; make</em> The advantage of running cmake inside a build</em> directory is you can simply wipe the entire build dir to completely reset the build. Otherwise, the cmake files are mixed up with your source files.</li> The following is the CMake version of printf – very useful for debugging: message(some_var: ${some_var})</em></li> ccmake</em> and cmake-gui</em> are useful tools for examining the state of various CMake variables.</li> </ol> Most of these tips are general and have nothing to do with OE, but that is the entire point – using CMake with OE rarely requires you to do anything beyond what you normally do with CMake. So, if you use CMake correctly, there is usually not a lot more required to cross compile and package your apps in OE.</p> PCB Schematics Style Guide 2014-08-25T00:00:00+00:00 As I work at the intersection of hardware and software, I spend a fair amount of time reading schematics for printed circuit boards (PCBs). I’m astounded at how sloppy most schematics are, and have come up with a few suggestions to consider.</p> Schematics are not only design entry – schematics are documentation</strong>. Unless you are writing all the software, handling all the manufacturing, and debugging all field issues yourself, there will be others who will be using them. Be considerate.</li> Not everyone has a D size plotter (even B size printers are fairly rare). I have a rather large (27“, 2560x1140) monitor, and have seen schematics that I have to zoom in to read. Schematics should be created such that they can be printed on letter size paper</strong> (everyone has a letter size printer), and easily read – even by folks in their 60’s. This is fairly simple – just spend some time breaking large symbols into smaller sections, and use a few more pages. Even for viewing on a screen, putting less stuff on each page</strong> makes it easier to scroll through pages. Most people print out schematics for review or bench work – so having letter size pages makes this possible.</li> Schematics should be organized to have one function per page</strong>, or group similar functions on a page. Don’t intermix power supplies and an audio codec unless they are related. We realize the importance of organizing software in modules, and a large schematic is really no different.</li> Keep all pages the same size</strong>. When every page is a different size, you can’t easily scroll through pages, or print them out.</li> Avoid hierarchical schematics</strong>. For most PCB designs, hierarchical schematics offer no advantage over a flat schematic logically organized on pages. PCB’s can only be so big, therefore even if you have 20 pages for a schematic, 20 pages is really not all that much, and adding hierarchy to 20 pages just adds complexity and makes them harder to read. In a sense, your schematic page becomes a “hierarchical block.”</li> Off page references</strong> are the key to making smaller schematic pages work.</li> Spend a little more time putting a little art</strong> into your schematic symbols. If you have a DB-9 connector, and you draw a connector in the shape of a DB-9 (only takes a few more minutes), then you can instantly recognize what it is in the schematic, and it is much quicker to find the circuit you are looking for during debugging sessions. The same can be said for USB connectors, etc. In some cases, it may make sense to draw a connector symbol to match how it looks physically, so you can quickly find a pin on the bench without digging out a datasheet.</li> Schematic text must be easily searchable</strong>. I don’t want to install a clunky schematic viewer that only runs on X.XX version of windows just to search for text. I want to use the PDF reader I already have installed. Make sure schematics are cleanly exported to PDF, and are easy to search.</li> </ol> I’m sure there are many other ideas – in a brief search, I turned up the following:</p> http://blog.upverter.com/schematic-style-guide</a></li> http://www.k-state.edu/ksuedl/publications/Technote%208%20-%20Guidelines%20for%20Drawing%20Schematics.pdf</a></li> http://electronics.stackexchange.com/questions/28251/rules-and-guidelines-for-drawing-good-schematics</a></li> </ul> Library sizes for C vs C++ in an embedded Linux system 2013-12-18T00:00:00+00:00 Is the size of the libraries required for C++ (vs C) a concern in Embedded Linux systems? Most Embedded Linux systems likely include some C++ code, so this is probably not even a decision we need to make in many cases. However, often there is a need for a small initramfs that is used as part of the boot process (perhaps for software updates) before switching to the main root file system. In this case, it makes sense to keep the initramfs as small as possible, and we might be concerned here with the size of C++ libraries.</p> To find out what additional libraries are required when compiling with a C++ compiler vs C, a simple hello-world app was created. It was then compiled with gcc and g++. The outputs were then parsed with readelf to determine the required dynamic libraries. The results are shown below:</p> C:</p> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]</p> C++:</p> 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]</p> 0x0000000000000001 (NEEDED) Shared library: [libm.so.6]</p> 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]</p> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]</p> The sizes of packages for these libraries in an OpenEmbedded build are:</p> libc: 1.4KiB (libm is inlcuded with libc)</li> libstdc++6: 256KiB</li> libgcc1: 204KiB</li> </ul> (The above sizes are compressed, so the actually will be some larger, but are useful for comparison.)</p> The installed sizes of libstdc++6 and libgcc1 on a ext3 file system are 1.4MiB and 750KiB.</p> It may also take slightly longer to load a C++ application as there is time required to dynamically load and link additional libraries. Is an extra 2MiB of libraries a big deal? It depends on your situation, but should be considered any time you are trying to create a small image.</p> Setting up an OpenEmbedded Package Feed Server 2013-12-11T00:00:00+00:00 With the BEC OE build template</a>, you can easily set up an opkg feed server that serves up packages from your build directory. This allows you to easily install new packages during development, without generating and reflashing an entire image. To use:</p> On workstation:</p> edit local.sh, and define MACHINE_IP to point to your target machine, and re-run “source envsetup.sh”. Alternatively, you can export the MACHINE_IP variable in your environment.</li> run: oe_setup_feed_server</li> run: oe_feed_server</li> </ul> On target system:</p> opkg update</li> opkg install </li> </ul> Opkg will install dependencies, so if you have a complex package, this is so much easier than copying over an opkg file manually, and figuring out you need 6 other packages as dependencies.</p> Why systemd in Embedded Linux Systems? 2013-12-11T00:00:00+00:00 Recently I was asked why use systemd vs sysvinit in embedded systems? There are many discussions on this, and really most of the reasons people use it for servers and desktops are also valid for embedded systems. Lennart Poettering’s articles</a> explain very well why you might want to consider systemd. A few things that rank high on my list:</p> systemd systems generally boot must faster than sysvinit systems.</li> systemd monitors every process it starts, so you know conclusively if your app is still running.</li> systemd has flexible mechanisms for restarting apps that crash, and rebooting if you get so many restarts in X amount of time. This is very important in embedded systems where you need to keep things running.</li> systemd logging (journald) is very flexible. The stdout of every process that is started by systemd is captured in the journal. You can also capture kernel messages in the journal. The end result is you have one place to look for system logs.</li> systemd has support for watchdog chaining</a> – again, important for many embedded systems.</li> The systemd dependency mechanism is very flexible.</li> systemd has tools to debug the init process: systemd-analyze, systemd-cgls, systemd-cgtop, bootchart, pybootchargui, etc.</li> on demand launch of services can improve boot time and conserve resources.</li> </ol> In summary, with systemd I have more control over the system startup and shutdown, and have better mechanisms to monitor things to make sure they stay running.</p> OpenEmbedded Build Template 2013-06-28T00:00:00+00:00 How does one set an OpenEmbedded/Yocto/Poky/Angstrom build? There are many options. Some include:</p> Angstrom setup scripts</a></li> Poky</a></li> Freescale Community BSP</a></li> OpenEmbedded core standalone</a></li> </ul> (I’m sure there are many others, feel free to add in comments …)</p> Over the past years, we’ve supported a number of customers using OpenEmbedded to develop products using various SOC’s. We also try to keep builds going for a number of different systems so that we can continuously evaluate the state of OpenEmbedded and related Embedded Linux technologies. We also needed a standard way to set up builds for customers (some don’t have a lot of experience with OE and Git) that is simple and consistent. What we ended up with is the BEC OpenEmbedded build template</a>.</p> The goal is to have a quick entry point into OpenEmbedded that includes the necessary layers for a number of different machines, and automates a number of routine tasks such as installing images to a SD card, setting up a development feed server, etc. The build template is only updated when the build is stable and tested on a number of machines, so it provides a series of stable snapshots of OpenEmbedded and associated layers.</p> This build template currently tracks the master branches for all the layers used. This gives us a platform to track the latest OE changes. With most projects, that ability to use the features in the latest versions of software outweighs the stability benefits of OpenEmbedded release branches. There are times when the OpenEmbedded project goes through invasive changes (such as the systemd integration), and using the master branches is not practical, so in this case we simply use the last stable snapshot that builds and works. In most cases if there are issues, simply report or fix the issue and wait a week.</p> Perhaps the most controversial decision is the use of Git submodules for including OpenEmbedded layers. The Internet is full</a> of rants</a> against Git submodules. For heavy developer use, submodules may not be optimal. However, from a user perspective, Git submodules provide a simple mechanism for including external repositories in a project. If most of the submodules (OE layers) won’t be touched (typical OE user scenario), submodules work very well. The fact that Git locks down submodules to a specific commit ensures you are getting exactly what you think you are getting (vs a branch that may have been rebased, modified etc). If the git hash matches, you can be pretty sure it is the same as the last time you built it. This is an important factor in production build systems where you want to be sure of what you are building. Google repo is another option under consideration, but there are still some trade-offs to work through.</p> In the end, build systems are very personal, and must be customized for your product and development team. The number one requirement for a an Embedded Linux build system is that you can get repeatable builds with a single command. There must be no manual steps where human error can be introduced. This is just one way to accomplish this goal.</p> (2023-09-05 update: the BEC OE Build template has been replaced by the Yoe Distribution</a>)</p> OS Containers for Build Systems 2013-05-08T00:00:00+00:00 Since I’ve been running archlinux on some of my systems, one thing I’ve found useful is systemd-nspawn</a>. systemd-nspawn containers (or chroots on non-systemd systems) give you a quick way to install a Linux distribution, that can run inside an existing Linux system.</p> Some cases where systemd-nspawn containers (referred to as containers in this document) are useful:</p> At one point, OpenEmbedded would not build with GCC 4.8 (this is no longer the case with recent versions of OE). So a Debian or Ubuntu OS container was a quick way to get builds going again.</li> For product build systems (may live for many years), typically OE will eventually break as you upgrade the workstation distribution. For projects that need a long-lived OpenEmbedded build system, setting it up in a chroot makes a lot of sense.</li> Someone might be having a compile or build problem with a distribution you don’t currently have installed. With containers, you can quickly set up a test distribution to reproduce problems.</li> I’ve had cases where I need an older version of Qt for a project, but my workstation includes a newer version. Again, setting up a OS container is sometimes simpler than getting two versions of Qt to dwell together peaceably in the same distribution.</li> Backing up or replicating your entire build system is very easy – simply rsync the OS container directory to another machine.</li> </ol> So the solution is to select a relatively stable, long-lived distribution to host your product builds. Debian is good choice. Because the container is simply a directory in the host workstation filesystem, you can use host workstation tools (editors, git, etc) directly in the container filesystem. The only thing you need to use the chroot for is the actual building. If you make sure the user ID is the same between your workstation and nspawn container, then permissions are seamless – you can easily access files in the container from the context of your host workstation.</p> To set up a nspawn-container:</p> Install debootstrap. On arch systems, this needs to be obtained for the AUR.</li> host: sudo debootstrap --arch=amd64 wheezy ~/debian-wheezy/</em></li> host: sudo systemd-nspawn -D ~/debian-wheezy/</em></li> container: apt-get update && apt-get install ssh</em></li> container: edit /etc/ssh/sshd_config, and set port to something other than 22 (23 in this example)</em></li> container: /etc/init.d/ssh start</em></li> </ol> (This systemd-nspawn</a> man page gives examples for setting up other distributions.)</p> To set up a user in your container:</p> host: id</li> will return something like: uid=1000(cbrake) gid=100(users) …</li> container: adduser --uid 1000 -gid 100 cbrake</li> host: ssh-copy-id -p 23 localhost (will copy public key to container)</li> </ol> Now, on the host system, you can simply “ssh -p23 localhost”</em> any time you want to log into the container. Soft links between the project workspace on the host system, and the container can also make shifting between the two easier.</p> An alternative way to start the container once its set up is:</p> sudo systemd-nspawn -D ~/debian-wheezy /sbin/init</em></p> Its also handy to make the shell prompt in the container slightly different than the host OS so that you can easily tell the difference. To accomplish this, add the following to ~/.profile in the container OS:</p> export PS1=“[\u@wheezy \w]\$ “</em></p> To create a service that starts your container, put something like the following in /lib/systemd/system/debian-wheezy.service</p> [Unit] Description=Debian Wheezy [Service] ExecStart=/usr/bin/systemd-nspawn -D /scratch/debian-wheezy/ /sbin/init 3 KillMode=process [Install] WantedBy=multi-user.target </code></pre> Hopefully this gives you a quick overview of how OS containers can be set up, and used in your OpenEmbedded build systems.</p> Git submodules can now track branches 2013-04-22T00:00:00+00:00 As of version 1.8.2, Git submodules can now track branches instead of specific commits. This is good news as in many cases, this is exactly the behavior we want. However, Git submodules are still not as flexible as Google repo</a>, but since submodules are built into Git, the submodule command is a good solution in many cases.</p> The “git submodule update --remote</em>” command is the key to tracking branches with submodules. The following is from the Git man pages:</p> --remote This option is only valid for the update command. Instead of using the superproject’s recorded SHA-1 to update the submodule, use the status of the submodule’s remote tracking branch. The remote used is branch’s remote (branch..remote), defaulting to origin. The remote branch used defaults to master, but the branch name may be overridden by setting the submodule..branch option in either .gitmodules</strong> or .git/config (with .git/config taking precedence).</em></p> This works for any of the supported update procedures (--checkout, --rebase, etc.). The only change is the source of the target SHA-1. For example, submodule update --remote --merge will merge upstream submodule changes into the submodules, while submodule update --merge will merge superproject gitlink changes into the submodules.</em></p> In order to ensure a current tracking branch state, update --remote fetches the submodule’s remote repository before calculating the SHA-1. If you don’t want to fetch, you should use submodule update --remote --no-fetch.</em></p> So, if you already have a Git submodule set up, its a simple matter to run git submodule update --remote</em> to update the submodule to the latest master branch. If you want a different branch, simple edit .gitconfig</em>.</p> [submodule "meta-bec"] path = meta-bec url = git@github.com:cbrake/meta-bec.git branch = test </code></pre> Now, if you run git submodule update --remote,Git will update the meta-bec</em> submodule to the latest on the test</em> branch.</p> This functionality is purely a convenience feature in the submodule update</em> command. In the actual repository, Git still stores submodules pointed to a particular commit. The same thing could be accomplished with something like git foreach “git fetch && git checkout test”</em>. The branch option in .gitmodules functions more as documentation and convenience. It is very handy to be able to look at .gitmodules and quickly determine that submodule X is tracking branch Y. Normally, this would have to be documented elsewhere, or figured out in some other way. Also, for build systems where you want the build to always track the production branches of various projects, update --remote</em> gives you a convenient way to update the build tree.</p> A quick way to share files from any directory 2013-01-22T00:00:00+00:00 Did you ever need a quick way to share files from a directory on your computer? Or perhaps transfer a large file to another person? With nodejs</a> and express</a>, you can easily set up a temporary web server that allows users to browse and access a list of files in a directory. For convenience, I created a simple github project</a> that can be cloned into any directory, and then a server started in a matter of seconds. Yes, you could upload files to a server, or share them with a file sharing service, but if you can expose a random port on your computer to the person who needs the files, then this is faster, and does not require any intermediate steps. Check out https://github.com/cbrake/http-file-server</a> for more information.</p> OpenEmbedded Source Mirrors 2013-01-07T00:00:00+00:00 When using OpenEmbedded for product development, there are several reasons you may want to consider setting up a source mirror for your OpenEmbedded build:</p> over time, sources disappear from download locations</li> various servers for source packages may be off-line at the time a build is run</li> some servers may be very slow, which slows down your build</li> occasionally the checksums of a source package will change</li> </ul> For a production build system, you want the build to be reliable and consistent, so this means not depending on 3rd party web sites/servers for a clean build to complete. Fortunately, OpenEmbedded makes it easy to set up a source mirror with the PREMIRRORS</a> variable. When bitbake tries to fetch source code, it tries PREMIRRORS, the upstream source, and then MIRRORS. There are several advantages to using a PREMIRROR variable over a MIRROR for your source mirror:</p> your source mirror will be used first, thus slow web sites are not an issue</li> if the checksums of the package change, the build will not fail because its still using the original source package from the mirror. You are guaranteed to be always using the same source package.</li> </ul> Setting up a source mirror is as simple as copying the contents of your downloads directory to a web server, and then populating the following variable in local.conf:</p> PREMIRRORS_prepend = "\ git://./. http://my-server/sources/ \n \ ftp://./. http://my-server/sources/ \n \ http://./. http://my-server/sources/ \n \ https://./. http://my-server/sources/ \n" </code></pre> The Poky reference manual has more details</a>.</p> Perisistent device names for USB serial ports 2012-12-19T00:00:00+00:00 Currently, my workstation has two 8-port USB<->RS232 devices</a>, one dual port USB<->RS422/RS485 adapter</a>, and several single port adapters such as the very useful BUBII</a>. So with around 20 USB->serial devices, figuring out which /dev/ttyUSBx entry corresponds to which port is not really practical. However, with udev in Linux, you can easily give static names to each device. This is especially convenient to do with FTDI devices because each FTDI device has a serial number. In devices such as the 8-port RS232 adapter, there are 4, 2-port FTDI chips. I could not find a serial number available in the adapter with a Prolific IC, so I would avoid those until that is sorted.</p> udevadm can be used to discover the serial number for a FTDI device:</p> udevadm info --attribute-walk -n /dev/ttyUSB0\|grep serial </code></pre> After the serial number is known, rules can be created in the /etc/udev/rules.d/99-usb-serial.rules</em> as shown in this example</a>. Now, serial ports can be accessed using convenient names such as /dev/ttyUSB_beagle</em>.</p> A quick way to set up an OpenEmbedded feed server 2012-12-08T00:00:00+00:00 During development with OpenEmbedded (oe-core, meta-oe, meta-angstrom), I often find it useful to set up a feed server so that packages can quickly be installed on the target system without manually copying them over or building a new image. One way to do this is copy your deploy/ipk directory to an existing web server (perhaps Apache running on your workstation), or configure Apache to point at your OE build directory, etc. But, it might be more convenient if your build system could directly create an opkg feed with no extra configuration. In the below example, we use node.js</a> + express.js</a> to create a feed server (basically just a web server that serves up the ipk files).</p> tools/feed-server/app.js:</em></p> // nodejs script to start a http server with the feed from this directory // default is port 4000 var express = require('express') var app = express() app.use('/', express.static(__dirname + '/../../build/tmp-angstrom_next-eglibc/deploy/ipk/')) app.use('/', express.directory(__dirname + '/../../build/tmp-angstrom_next-eglibc/deploy/ipk/')) console.log("feed server started on port 4000") app.listen(4000) </code></pre> The express.directory</em> function is used to create a directory listing that can be browsed. With most other web servers, at least this much code is required just for configuration. With node.js, this is the code to create an entire server from scratch! This node.js app can be started with a bash function in the environment:</p> function oe_feed_server() { cd $OE_BASE bitbake package-index node tools/feed-server/app.js cd - } </code></pre> The package-index</em> target is used to rebuild the index files that list the available packages.</p> On the target system, the /etc/opkg configuration files must be modified to point to the feed server, and then you can run:</p> opkg update; opkg install <package> </code></pre> An exmaple build environement with this integrated is located:</p> https://github.com/cbrake/oe-build-core</a></p> https://github.com/cbrake/oe-build-core/commit/23352b9a43c60d67070abe5ac001aba9a9ac5cc4</a></p> https://github.com/cbrake/oe-build-core/commit/6baceb8b1e4477ccfd03aa553a5fcf501b398196</a></p> It might be argued that it is just as easy or easier to set up a more conventional web server. However, the benefit of node.js is that it is a full blown programming environment. You can quickly extend it to provide a web interface, perhaps add functionality to automatically push updated opkg configuration files to the target system that point to your feed server, etc. It is much more than just a web server.</p> Setting the root password in an OpenEmbedded image 2012-12-05T00:00:00+00:00 During development, often a blank root password is used for the embedded Linux target system. However, when deploying an embedded Linux system, often there is a requirement to set the root password to something non-obvious. One way to do this is boot the system, and change the password using the passwd command. Then copy the password hash from the /etc/shadow file into the below line in your image recipe.</p> ROOTFS_POSTPROCESS_COMMAND += "\ sed 's%^root:[^:]:%root:password_hash_from_etc_shadow:%' \ < ${IMAGE_ROOTFS}/etc/shadow \ > ${IMAGE_ROOTFS}/etc/shadow.new;\ mv ${IMAGE_ROOTFS}/etc/shadow.new ${IMAGE_ROOTFS}/etc/shadow ;" </code></pre> The ROOTFS_POSTPROCESS_COMMAND is useful for simple modifications like this to the rootfs image.</p> OpenEmbedded: configuring openssh to allow a blank password 2012-12-05T00:00:00+00:00 Noticed the following when browsing around in the OpenEmbedded sources the other day:</p> ROOTFS_POSTPROCESS_COMMAND += "openssh_allow_empty_password ;" </code></pre> This allows a blank password for development, which is conveient for running ssh/scp commands to the device. The above can be placed in an image recipe.</p> The command modifies the PermitEmptyPasswords config option in /etc/ssh/sshd_config or /etc/default/dropbear.</p> Running a reboot cycle test shell script with systemd 2012-08-29T00:00:00+00:00 One of the easiest ways to stress test an embedded Linux system is to continuously reboot the system. Booting is a difficult activity for a Linux system (similar to waking up in the morning). The CPU is maxed out. There are a lot of things happening in parallel. Software is initializing. There is a lot of filesystem activity. Often, if there is an instability in the system (and especially the filesystem), continuously rebooting will expose it.</p> Now that we use systemd for most new systems, we need to run a simple script X seconds after the system boots to increment the boot count, and reboot. This is the classic domain of a shell script. However, for a simple task like this, it’s a little cumbersome to create a seperate shell script and then call this script from a systemd unit. The following is a way to embed the script directly in a systemd unit.</p> Put the following in: /lib/systemd/system/cycletest.service</em></p> [Unit] Description=Reboots unit after 30s [Service] StandardOutput=syslog+console ExecStart=/bin/sh -c "\ test -f /cycle-count \|\| echo 0 > /cycle-count;\ echo 'starting cycletest';\ sleep 30;\ expr `cat /cycle-count` + 1 > /cycle-count;\ systemctl reboot;\ " [Install] WantedBy=multi-user.target </code></pre> To install and start the script:</p> systemctl daemon-reload</li> systemctl enable cycletest.service (enable the service to start on reboot)</li> systemctl start cycletest.service (start the service, should reboot in 30s)</li> </ul> Bitbake has a new way of displaying build status 2012-08-29T00:00:00+00:00 Now instead of displaying a scrolling log, bitbake will display a simple output that lists which tasks it is working on at the moment:</p> Currently 4 running tasks (185 of 3093): 0: gmp-native-5.0.5-r0 do_configure (pid 22919) 1: lzo-native-2.06-r1 do_configure (pid 27103) 2: expat-native-2.1.0-r0 do_compile (pid 7463) 3: ncurses-native-5.9-r10.1 do_compile (pid 9820) </code></pre> This really allows for a clear view of how the parallel threads option (BB_NUMBER_THEADS) in bitbake works.</p> Mounting a UBIFS partition using systemd 2012-08-16T00:00:00+00:00 Systemd</a> is becoming the defacto system and service manager for Linux, replacing the SysV init scripts. The Angstrom distribution</a> has supported systemd for some time now. Recently, I needed to mount a UBIFS filesystem in one of my projects. The main application is being started with systemd, so it seemed like a good fit to also use systemd to mount a data partition needed by the application. Systemd can use entries from /etc/fstab, but one additional wrinkle in this system is that I also wanted to run the UBI attach in a controlled way. This can be done with a kernel command line argument, but there are times in this system where we will want to format the data partition, so this requires a detach/attach operation.</p> The resulting systemd units are:</p> data.mount</strong></em></p> [Unit] Description=Mount data partition Requires=data-attach.service After=data-attach.service </code></pre> [Mount] What=ubi1:data Where=/data Type=ubifs </code></pre> data-attach.service</strong></em></p> [Unit] Description=Attach data ubi partition [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/sbin/ubiattach /dev/ubi_ctrl -m 6 ExecStop=/usr/sbin/ubidetach /dev/ubi_ctrl -m 6 </code></pre> add the following to my-application.service</strong></em></p> Wants=data.mount </code></pre> The only real problem I ran into is the “After” statement in the data.mount</em> unit. It turns out that the mount will run before the attach operation is finished unless this is included.</p> The RemainAfterExit</em> seems to be required so that ExecStop</em> can be run when the unit is stopped.</p> (There may be a better ways to do all this, so comments are welcome!)</p> One of the benefits of systemd is that everything is very controlled. If the dependencies are specified properly, there are no race conditions. Additionally, if you need to manage units (services, mounts, etc) from a program, it is much easier to check the states as everything is very consistent. For example, if you want to query the state of a unit, the systemctl show command will return easily parsed output (as shown below):</p> ... Before=umount.target After=data-attach.service systemd-journald.socket -.mount Description=Mount data partition LoadState=loaded ActiveState=active SubState=mounted FragmentPath=/lib/systemd/system/data.mount UnitFileState=static InactiveExitTimestamp=Thu, 01 Jan 1970 00:00:17 +0000 ... </code></pre> If the data-attach service is stopped, systemd automatically unmounts the data partition first – very nice!</p> Hopefully this example illustrates how to do simple tasks in systemd. It appears that instead of having a complex script or program to initialize a system, the systemd “way” is to create a number of small units that get connected with dependencies. This seems like a much more controlled and flexible approach.</p> A Review of Graphical Application Solutions for Embedded Linux Systems 2012-08-07T00:00:00+00:00 One of the decisions we face when building Embedded Linux systems is what components to use. With Open Source software, there is often more than one good option. Graphical libraries are no exception. In this article, we’ll examine GTK+, Qt, EFL, Android, and HTML/Javascript.</p> There are many factors that go into a choice like this, but some of them are:</p> Does the application need to run on Windows or MacOS?</li> Does the GUI need to be viewed remotely over a network?</li> Are dynamic effects (think iPhone) desired?</li> Does the application need to run on low end CPU’s (ones without a GPU)?</li> </ul> Putting some effort into selecting the right GUI technology is important for the following reasons:</p> Most of the development effort for the product will likely be in the graphical application, so it makes sense to maximize the productivity of the application developers. Typically there might be 3-10 application developers on a project for every system software developer.</li> You want a technology that will scale with new revisions and improvements of the product.</li> You want a technology that will be supported and improved long term.</li> </ul> With Open Source software, things are always changing. Leading technologies change. This can be illustrated by following Intel’s support for their open source software platforms. While some of this may be driven by politics, we can also see clear technical and licensing reasons why these shifts were made.</p> Moblin (GTK+), became public around 2007, merged into MeeGo in 2010</li> MeeGo (Qt), announced in 2010-02, cancelled 2011-09</li> Tizen (HTML/Javascript, EFL), announced in 2011-09</li> </ul> We can see similar shifts in other companies:</p> Nokia started with GTK+ as the GUI technology for their tablet computers, and then shifted to Qt</li> Samsung has been using GTK+ on top of DirectFB, and now is moving toward EFL/HTML/Javascript for some phones</li> Many phone manufactures are producing Android products</li> Palm moved from a proprietary GUI to HTML in webOS</li> </ul> GTK+</h2> GTK+ is part of the GNOME desktop project and is perhaps the most used graphical library for desktop Linux applications, and in the past has been very popular in Embedded systems. Nokia has invested heavily in GTK+ in the past with its early tablet products (N770, N800, N900, etc). However, with the advent of the iPhone and faster processors with GPUs, everything has changed. The standard is now dynamic GUI’s with sliding effects, etc. The Clutter project is a library that can be used to build dynamic GUIs and fits in with the GNOME stack. GTK+ supports Windows and MacOS, but probably not as well as Qt.</p> Qt</h2> Qt is a very mature project that is also extensively used for desktop projects, and recently is being used on some of Nokia’s phones. Qt was originally developed by the Norwegian company Trolltech. Originally, Qt was only offered with either a proprietary license or the GPL license. This meant if you wanted to write a proprietary application using Qt, you had to purchase Trolltech’s commercial license. This factor alone probably made GTK+ a much more popular solution for many years for Embedded Linux GUI’s, including most cell phone stacks. In 2008, Trolltech was acquired by Nokia, and shortly after that, Qt was offered under the LPGL license, giving it the same license as GTK+.</p> One of Qt’s compelling features introduced in Qt 4.7 is its QML (or Qt Quick) technology. This allows you to write declarative GUI’s in a Javascript like syntax with many automatic bindings. There is good support for dynamic operations like sliding effects, and the performance is reasonable, even on low-end systems without a GPU.</p> In the future, Qt 5.0 will require OpenGL, and hence be relegated to high end ARM CPU’s with a GPU, or desktop systems.</p> Qt’s cross platform support is excellent, and provides good native support for Linux, MacOS, and Windows.</p> Recently, Nokia has made efforts to set up Qt as more of a community project, instead of retaining exclusive control over it.</p> EFL</h2> EFL (Enlightenment Foundation Libraries) is a project that started out as the Enlightenment window manager, and grew into a set of general purpose libraries. It claims to be to be more efficient than GTK+ and Qt, and work with or without hardware acceleration. Recently, EFL seems to have garnered the commercial interest of Samsung, Intel, and others involved in the Tizen project. According to a presentation by Carsten Haitzler (one of EFL’s founders and developers), Samsung was using GTK+ and DirectFB, but switched to EFL after seeing the performance. Perhaps the most compelling story for EFL is the high performance over a range of hardware capabilities (from simple phones with low end processors, to high end smart-phones running OpenGL). Parts of EFL are also used in the commercial GUI FancyPants.</p> Android</h2> Android is an interesting GUI solution, especially as many developers have experience working on Android applications. Android now seems to be used in many applications where Windows CE was used in the past, probably due to its polished application development tool-set.</p> HTML/Javascript</h2> Application development with HTML and Javascript is one of the more interesting developments because many embedded systems are headless (don’t have a local display). Couple this with the fact that many users now have smartphones or tablets readily available, and it may make sense to simply use an external device for displaying the UI. There is often a requirement for accessing the UI of a device remotely, and in this case, HTML/Javascript works very well. If the UI needs to be displayed locally, then a fairly powerful CPU is required (an ARM Cortex-A8, etc) to run a modern web browser. If there is no physical display on the device, and the UI is accessed remotely on a computer or mobile device, then a less powerful CPU is required because the embedded device does not actually have to do any of the rendering. HTML/Javascript also has the benefit that it is a very popular technology, thus there are many experienced developers.</p> Summary</h2> Each of the above technologies has benefits and drawbacks. Understanding your project’s requirements, and what each solution offers is key to making the best decision.</p> A Linux Kernel Tracing Tutorial 2012-01-31T00:00:00+00:00 The Linux kernel has a fairly extensive tracing infrastructure that is quite useful for debugging. There are a number of things you can do with tracing, but the focus of this article will be the traditional printk type debugging we often end up doing to trace initialization issues with a driver. The following links provide additional information on the linux kernel tracing infrastructure:</p> </a>http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=tree;f=Documentation/trace</a></li> Debugging the kernel using Ftrace - part 1: http://lwn.net/Articles/365835/</a></li> Debugging the kernel using Ftrace - part 2: http://lwn.net/Articles/366796/</a></li> Secrets of the Ftrace function tracer: http://lwn.net/Articles/370423/</a></li> http://elinux.org/Kernel_Trace_Systems</a></li> </ul> In this example, I am working on a new audio driver. The typical experience with a new driver is that you install it and nothing happens because something is not registered correctly with the Linux driver model. So, the first thing I do is start with with the platform_device_add() function in my drivers init function. To observe the kernel activity around the kernel platform code, I can do the following:</p> cd /sys/kernel/debug/tracing/ echo 0 > tracing_on (keep trace from filling up until we set filter) echo function_graph > current_tracer echo platform > set_ftrace_filter echo 1 > tracing_on cat trace_pipe (leave running in a different shell) </p> After executing the above, we see the following. For this example, trace_pipe is preferred because the trace is then emptied and only new information is shown.</p> 0) + 30.518 us \| platform_device_alloc(); 0) \| platform_device_add() { 0) 0.000 us \| platform_uevent(); 0) + 30.518 us \| platform_uevent(); 0) 0.000 us \| platform_uevent(); 0) + 30.518 us \| platform_match(); 0) + 30.518 us \| platform_match(); 0) 0.000 us \| platform_match(); 0) 0.000 us \| platform_match(); ... 0) + 30.518 us \| platform_match(); 0) 0.000 us \| platform_match(); 0) 0.000 us \| platform_match(); 0) 0.000 us \| platform_match(); 0) 0.000 us \| platform_match(); 0) ! 3936.767 us \| } 0) + 30.518 us \| platform_uevent(); 0) + 30.518 us \| platform_device_alloc(); </code></pre> From the above, I can conclude that the platform_match() is not succeeding, because I would expect some more activity. At this point I chose to add a printk:</p> diff --git a/drivers/base/platform.c b/drivers/base/platform.c index 7a24895..f9ce0c7 100644 --- a/drivers/base/platform.c +++ b/drivers/base/platform.c @@ -662,6 +662,8 @@ static int platform_match(struct device dev, struct device_driver drv) struct platform_device pdev = to_platform_device(dev); struct platform_driver pdrv = to_platform_driver(drv); + trace_printk("pdev->name = %s, drv->name = %s", pdev->name, drv->name); + /* Attempt an OF style match first / if (of_driver_match_device(dev, drv)) return 1; </code></pre> Now, if I re-run the trace, I see the following:</p> 0) \| / pdev->name = soc_audio, drv->name = davinci_emac / 0) 0.000 us \| } 0) \| platform_match() { 0) \| / pdev->name = soc_audio, drv->name = snd-soc-dummy / 0) 0.000 us \| } 0) \| platform_match() { 0) \| / pdev->name = soc_audio, drv->name = soc-audio / 0) 0.000 us \| } 0) \| platform_match() { 0) \| / pdev->name = soc_audio, drv->name = omap-pcm-audio / 0) 0.000 us \| } 0) ! 4241.943 us \| } / platform_device_add / </code></pre> From the above, it looks like we have a simple mismatch between “soc_audio” and “soc-audio.” Fixing this problem, and re-installing the module, we now have:</p> 0) \| platform_match() { 0) \| / pdev->name = soc-audio, drv->name = snd-soc-dummy / 0) 0.000 us \| } 0) \| platform_match() { 0) \| / pdev->name = soc-audio, drv->name = soc-audio / 0) 0.000 us \| } 0) + 91.553 us \| platform_drv_probe(); 0) ! 4241.943 us \| } / platform_device_add / </code></pre> Now we can see that the names match, and the probe function is now being called. At this point, we may want to turn on tracing of some additional functions to try to determine what is happening next.</p> echo "platform snd* mydriver" > set_ftrace_filter </code></pre> And the result:</p> 0) \| / pdev->name = soc-audio, drv->name = snd-soc-dummy / 0) 0.000 us \| } 0) \| platform_match() { 0) \| / pdev->name = soc-audio, drv->name = soc-audio / 0) + 30.517 us \| } 0) \| platform_drv_probe() { 0) \| snd_soc_register_card() { 0) + 30.518 us \| snd_soc_instantiate_cards(); 0) ! 17852.78 us \| } 0) ! 17883.30 us \| } 0) ! 22125.24 us \| } / platform_device_add / </code></pre> With the above additional information, we can continue to learn more about the flow through the kernel.</p> While all of the above could have been done with printk’s, it would have been more time consuming. The kernel function tracing capabilities allow us to quickly get a high level view of the flow through the kernel without manually adding a bunch of printk statements. The kernel tracing features are completely contained in the kernel without requiring additional user space utilities which makes it very convenient to use in embedded systems. The low overhead is also important in resource constrained embedded systems.</p> The easy way to get serial terminal in Linux 2012-01-26T00:00:00+00:00 When doing embedded Linux development, most of us spend out time tethered to a target system with a serial cable, which is used for a serial console. Minicom is the defacto serial terminal software for Linux. However, Minicom is a little fussy in that you typically have to set it up for each port you want to use. This is no big deal, but is generally difficult for new users to understand, and yet another hurdle. And with 8-port USB->serial adapters, I have a lot of ports to set up.</p> Just recently, I discovered that screen can be used as a serial terminal program:</p> screen /dev/ttyUSB0 115200 </code></pre> A few notes on using:</p> to exit screen: Ctrl-a k</li> to write a hardcopy of the screen image: Ctrl-a h</li> to get help: Ctrl-?</li> </ul> All the neat features of screen are two numerous to list here, but one more that I’ll point out is the scrollback/copy feature (activated by Ctrl-a [ ). This allows you to scroll back and the navigation works much like VI – what could be nicer?</p> Verizon UML290 and Sprint U600 USB Modems in Embedded Systems 2011-03-31T00:00:00+00:00 Recently I tested support for the Verizon UML290 and Sprint U600 USB Cellular modems in an embedded Linux system. Both modems support 3G and 4G networks, but only the 3G modes were tested due to lack of 4G coverage at the testing location.</p> </a></p> Fortunately, both modems function very similar to previous modems, so with the drivers available in the Linux kernel, and standard pppd support in OpenEmbedded</a>, they worked fine.</p> The Verizon UML290 modem provides a challenge in that it must be manually switched between 4G and 3G modes. Typically this is done automatically by the vzaccess program Verizon supplies with the modem that runs on Windows. The solution for this system was to manually set the modem to 3G mode as detailed on the following page:</p> http://www.evdoinfo.com/content/view/3492/64/</a></p> It appears that some embedded systems such as the Cradlepoint routers have implemented automatic 3G/4G switching support for the UML290, so this is no doubt possible with a little effort.</p> The Sprint U600 modem appears to default to 3G, or automatically switch inside the modem.</p> The same pppd scripts can be used with both modems:</p> # /etc/ppp/peers/verizon_um290 user a password v connect "/usr/sbin/chat -v -f /etc/ppp/peers/verizon_um290_chat" defaultroute usepeerdns ttyACM0 921600 local usepeerdns debug -detach </code></pre> # /etc/ppp/peers/verizon_um290_chat '' 'ATZ' 'OK' 'ATDT#777' 'CONNECT' '' </code></pre> To initiate a connection:</p> pppd call verizon_um290 </code></pre> With Verizon cellular modems, it appears that port 22 is often blocked, so if you need to access a remote device via ssh, you may need to run ssh on a higher port number. With 4G networks, it appears that the networking setup may be different in that a public IP address may not be assigned. From the above evdoinfo.com page, we find the following text:</p> This fix will also work for users looking to use their device for remote based applications because it assigns a public facing IP address (3G ONLY). With eHRPD you’re assigned a private IP in either 3G or 4G mode, which has prevents UML290 users from accessing remote applications.</em></p> Perhaps the Rev A HDR Service mode will also work in 4G mode, but its seems as cellular networks become more complicated, there will be more issues to deal with in using USB Cellular modems for remote access in embedded systems.</p> Past articles on USB Cellular modems:</p> /353/sprint-598u-usb-broadband-modem-in-embedded-systems/</a></li> /203/using-a-verizon-usb720-modem-in-an-embedded-linux-system/</a></li> </ul> Git and Distributed Development 2011-03-29T00:00:00+00:00 This is part of an ongoing series of articles</a> on the Git version control system.</p> The “many repository</a>” paradigm has been partly driven by the distributed development paradigm. Git solves the problem of multiple developers working in multiple repositories very well. Because we want to use and customize projects like the Linux kernel, U-boot, and OpenEmbedded in our projects, then we naturally find ourself in the situation where we need to manage multiple repositories. Yes, you can check the Linux kernel into your company Subversion repository, but you are much better off long term if you bite the bullet and implement your own Git infrastructure.</p> As we consider the product development process, we need to consider the life cycle of a product. Most products live for at least several years, and will go through several software iterations. If we can update the software components we use, then we can add value to the product in the form of new or updated drivers to support new peripherals, new libraries, performance improvements, etc. But, we are dealing with millions of lines of source code, so we must have an efficient way to deal with software projects of this size. The below Figure 2 illustrates how you might organize a typical project. Notice we can pull updates from the U-boot and Kernel source trees at any time in the development process, and merge the changes with our own modifications. We might have an outside team working an application, and we then easily synchronize the repositories when it makes sense.</p> </a></p> There are many other design flows possible. Once you have the ability to support multiple branches and repositories easily, it becomes trivial to implement a staging/testing repository for a QA processes, maintenance repositories for supporting old releases, etc.</p> Even at a personal developer level, Git’s distributed capabilities offers many advantages. Each Git workspace is actually a full Git repository. This means you can check changes in locally, re-organize your changes, be able to track changes when off-line, etc. For this reason, many developers are now using git-svn</a> when they need to work with Subversion repositories. With git-svn, you have all the benefits of using git locally, and yet you can easily synchronize with Subversion repositories. And this leads us to our next topic: cheap branches (coming soon).</p> How do modern USB chargers work 2011-03-19T00:00:00+00:00 As we help customers design products, we often try to leverage the latest cell phone practices and technologies. One of these is USB charging. There has been a push in recent years to standardize on USB chargers for cell-phones. There are a number of organizations involved including the USB-IF</a> and OMTP</a>. Two of the specification available include:</p> [http://internal.omtp.org/Lists/ReqPublications/Attachments/59/OMTP CCLDC V1.1 Final.pdf](http://internal.omtp.org/Lists/ReqPublications/Attachments/59/OMTP CCLDC V1.1 Final.pdf)</li> http://www.usb.org/developers/devclass_docs/Battery_Charging_V1_2.zip</a></li> </ul> There is also a Chinese standard titled the “Telecommunications Industry Standard of the PRC“ that has been instrumental in influencing these standards.</p> The fundamental problem is a battery powered device needs to know if it is plugged into a computer USB port, or a dedicated charger. A standard USB port on a computer is only specified to provide 500mA. Typically a dedicated charger that plugs into a wall outlet provides more current than 500mA so that the battery can be charged quicker. To differentiate between a computer USB port, and a dedicated charger, the dedicated charger shorts the D+ and D- USB signals together with a resistance of less than 200 ohms (specified by USB-IF).</p> One of the goals of these standards is that any charger can be used with any battery powered device. The next question is how do we handle the case where every USB charger has a different rated current? How can a device that charges its battery at 1.7A be compatible with a charger that only outputs 0.7A? One theoretical solution would be for the device to query the current capacity of the charger and then only use that much current. It turns out a much simpler approach is used. The USB-IF provides the following chart in the above specification:</p> </figcaption></a></figure> As long as the charger outputs power in the Required Operating Range, the battery powered device must be able to use whatever power is available to charge its battery. If the device uses more current than the charger can supply, the charger simply goes into a constant current mode and the battery charges slower than it would with a higher capacity charger. Thus we have a very simple scheme where we can theoretically use any charger with any device.</p> As a simple test, I charged a Nokia N900 phone (which came with a 1.2A charger) with a smaller LG charger that is rated for 0.7A. I monitored the LG charger a couple times during charging to make sure it was not getting hot. It seemed to charge the N900 battery just fine.</p> When purchasing after-market USB chargers, it is sometimes difficult to determine if they have the USB D+ and D- lines shorted together. In once case I purchased a car USB charger that did not work with my phones, but was able to “fix” it by disassembling it, and putting a solder blob between the D+ and D- signals on the USB connector.</p> </a></p> </a></p> </a></p> While shorting the D+ and D- pins usually makes a charger work, it is nice to know if the charger is compliant with the USB-IF specification and is designed to work in the constant current mode without shutting down, catching fire, etc.</p> Git Overview Screencast 2011-03-17T00:00:00+00:00 This screencast</a> (use Firefox to view the screencast) provides an overview of the Git version control system. There are 3 features of Git that are especially interesting:</p> many repositories (vs. one large repository)</li> distributed development</li> cheap branches</li> </ol> The fundamental driver for better tools is increasing system complexity. More and more we are required to manage and integrate more third party software, work in distributed teams, and more effectively re-use the software we do have. Git is a tool that helps you accomplish these goals.</p> Git and Why Multiple Repositories 2011-03-17T00:00:00+00:00 This is part of an ongoing series of articles</a> on the Git version control system.</p> This article discusses the trend in software configuration management toward multiple repositories, rather than one large repository. In the past when many companies used Subversion or comparable systems, there was typically one huge company repository (I’ve seen them in the 10’s of GB in size) that held a hierarchical tree of all the company source code. This worked fairly well, and is comfortable in that the organization is very similar to a file system directory structure. However, this model is not very flexible in that it does not have a consistent way to re-use components between different projects. Some people simply copy source code. Some have a “common” project that is included in all their other projects. Subversion externals can be used. With Git, typically a separate repository is created for each software component. There are perhaps several reasons for this, but one reason is that Git simply does not scale to huge multi-GByte repositories. However, this turns out to be a blessing in disguise as I think it is a better model in many cases. What we end up with is more of a catalog of software components rather than a rigid hierarchy.</p> </a></p> There is much emphasis these days on modular, re-usable software components (Object Oriented Programming, plugins, etc.). Why not keep things modular at the repository level? Another example of this type of organization is the Internet itself. It is not a hierarchy of information, but rather a flat system that is organized by hyperlinks.</p> One of the benefits of organizing your source code this way is that it encourages clean boundaries between software components. Each software component needs to stand on its own without being propped up by header files located in an unrelated source tree 3 levels up in the directory hierarchy. This type of organization forces us to make better use of standard build system practices.</p> How do you implement this type of repository infrastructure? Build systems such as OpenEmbedded, Gentoo, and the Android build system manage this fairly well. However, Git also includes a feature named “submodules” that provides a mechanism for one repository to reference source code from another. What you end up using really depends on your needs, and what you are trying to accomplish.</p> A screencast</a> is also available that covers this topic.</p> C++ callbacks to member functions 2010-12-16T00:00:00+00:00 How to properly do callbacks to C++ member functions is something that has intrigued me for some time now. There are a number of solutions, none of which I really liked. But now with the std::tr1::function</strong> and std::tr1::bind</strong> functions, there appears to be a clean solution.</p> In the past, if you wanted a callback to a C++ member function a typical approach was to create a static function wrapper as described here</a>. However, this is ugly, and a step backwards from the capabilities of plain old C.</p> std::tr1::function</strong> is a function wrapper that can be used to create function objects for a number of scenarios. We are interested in representing class member functions.</p> #include <tr1/functional> struct X { int foo(int); }; std::tr1::function<int (X, int)> f; f = &X::foo; X x; f(&x, 6); </code></pre> f() is now a simple function representation that can be passed around, etc. However, what we don’t like is the reference to the struct X type, which makes it much less generic. This is where the std::tr1::bind</strong> function comes in. With bind, we can dynamically bind one function to another, and re-arrange the arguments, etc. So to make a generic portable function that points to X::foo(), we can do something like the following.</p> #include <tr1/functional> struct X { int foo(int); }; std::tr1::function<int (int)> f; X x; f = std::tr1::bind(&X::foo, &x, _1); f(6); </code></pre> Now, f() is a very generic function that could be used in various interfaces without specifically referencing struct X.</p> A more complete example of how callbacks might work is given below:</p> #include <iostream> #include <tr1/functional> struct B { std::tr1::function <void ()> _callback; void reg_callback(std::tr1::function <void ()> callback) { _callback = callback; } void unreg_callback() { _callback = NULL; } void process() { // simply calls callback if (_callback) _callback(); } }; struct A { B * _b; A(B * b) : _b(b) { std::tr1::function<void()> callback; callback = std::tr1::bind(&A::callback, this); b->reg_callback(callback); } ~A() { _b->unreg_callback(); } void callback() { std::cout << "A::callback called\n"; } void process() { _b->process(); } }; int main() { B * b = new B(); A * a = new A(b); // the following is example where object b calls // back into a method // in object a a->process(); // now delete object a, and verify the callback // in B does not crash or still get called delete a; // now b should not execute the callback b->process(); } </code></pre> In the above example, the callback in the object a</em> will be called once.</p> Yocto and OpenEmbedded 2010-12-10T00:00:00+00:00 Recently, I attended an Embedded Linux summit hosted by the Linux Foundation to discuss the Yocto project. Overall, I thought the meeting was very positive and constructive. Having met and discussed at length the methods and goals of the Linux Foundation with some of their people, I’m impressed with their approach. They are there to help Linux succeed in the Embedded space in any way they can.</p> It is my understanding that the Yocto project has the goal of making Linux much more accessible and usable in embedded systems, and improve efficiencies. While the OpenEmbedded project has served many of us well for many years, we all can readily understand there are some deficiencies. These can be overcome by experienced developers, but there are a number of areas that can obviously be improved. Not only are we concerned with making it easier for new users to use Embedded Linux, I think there are areas where we can drastically improve the efficiency of implementing embedded Linux systems for those who are experienced. It was stated once that tools implemented must be useful to both the new developer as well as the experienced developer.</p> It should be noted that building an embedded Linux system is an inherently complex undertaking, and although we can improve tools and processes to make it more efficient, and somewhat easier, in the end it is still a very complex problem, and will require considerable skill to be an effective embedded Linux developer. There is no substitute for experience, and developer skill. Just as we would not slap a fancy GUI on top of a robotic surgery instrument, and tell a novice to have at it, likewise it is still going to require considerable engineering skill to be effective in developing Embedded Linux systems. But, if we improve the base technologies, and tools, we will spend less time messing with these and doing the same things over and over, and will have more resources toward implementing new things.</p> One example of the pain experienced in a typical OpenEmbedded system is getting Oprofile to run. Oprofile requires that ARM system be built with frame pointers (OE defaults to not), and that you have symbols in your binaries. Figuring out how to build a reasonably sized image with the needed symbols might be a 1-2 day process for me. Then there is the issue of kernel symbols, etc. I’m sure many other developers go through the same steps, but as many of us are paid to develop products on tight schedules, we don’t have the time to polish the Oprofile integration and tools.</p> As an extension of this, the application development story with OpenEmbedded is not all that great. Yes, we can generate a binary SDK, but again it may take a couple days of messing around to figure out the process, and get all the necessary libraries, etc. Then you still have the problem that application developers want Eclipse/Qt Creator integration, etc. Again, this can all be done, but takes time, and many people are doing the same things over and over again.</p> The Yocto project seems to have two primary thrusts:</p> stabilize a subset of the OpenEmbedded (called Poky) metadata and make it very high quality.</li> improve tools associated with the build process, and general embedded Linux development.</li> </ol> One thing the OpenEmbedded project has historically lacked in the past is industry funding and ongoing developer commitment. The OE project is characterized by a rotating group of developers. When someone needs OE for a project, they jump in and do a lot of work on OE, then they disappear again once their need is met. We have very few developers who work on the project consistently for long periods of time. This has been positive in some ways in that we have a very large number of contributors, and a very open commit policy. There are so many patches coming in right now, we can’t even keep up with processing them. Week after week, we have a healthy number of committers and changesets. The community behind OE is rather astounding, and it is amazing how OE succeeds almost in spite of itself as a self organizing project without much organization.</p> In the past the OpenEmbedded and Poky projects have existed as two independent trees, and things were shared back and forth manually. This works fairly well as recipes are self contained, and generally can simply be copied from one tree to another. However, some of the core recipes are fairly complex, and if they start to diverge, then sharing information can get more difficult.</p> It seems the vision that is emerging is that Poky could become the core component of OpenEmbedded. Poky would contain a very controlled subset of core recipes that must remain at a very high quality level. OpenEmbedded could then expand on this (through the use of layers) to provide the breadth and scope it has traditionally provided.</p> We may bemoan the lack of quality in the OpenEmbedded project as it has 1000’s of recipes, and many of them have rotted, etc. But as a consultant helping people implement products, I still find considerable value in this. For example, one of the products I support uses the HPLIP drivers from HP. Yes, the recipe is broke every time I go to use the latest version, but with a little work I can get it working again. Having something to start with provides value. The same is true for the geos mapping libraries. Very few people are going to use geos, so it will never be in a Poky type repository, but some of us do use geos, so having a common place like OpenEmbedded to put recipes like this is very important. Using Poky as the core of OpenEmbedded seems like a win-win. We are relieved of some of the burden of maintaining core components (like compilers, C library, tool integration, sdk generation, etc), but we can still have a very open culture, and provide a wide scope of platform, library, and application support that we have historically provided.</p> Richard Purdie is poised to become the Linus Torvalds of Yocto, and if OpenEmbedded choses to base on Poky, then the Torvalds of the OpenEmbedded core. I am personally fine with this as I don’t know anyone else who has contributed so much time and has the same level of skill in the OE/Poky area. He has proven over time to be levelheaded and able to deal with difficult situations. Also, as a Linux Foundation fellow, he is positioned to be company neutral, which is very important for someone in this position.</p> Yocto is using a “pull” model similar to the Linux kernel for accepting changes. It is planned to put maintainers in place for various subsystems. With the goal of providing a very high quality subset of embedded Linux technologies, it seems this makes a lot of sense. If OpenEmbedded choses to base on Poky, there is no reason OpenEmbedded can’t continue to still use the push model for its layer that has worked well in the past. But, as we see patches languishing in the patchwork, perhaps a pull model might actually be more efficient at processing incoming patches so they don’t get lost. This is also an area where layers might actually help in that we have dedicated people committed to processing patches for each layer. One of the problems with the OpenEmbedded project with its 1000’s of recipes, the structure is very flat, and its fairly difficult to divide it up into areas of responsibility.</p> So going forward, it seems that if OpenEmbedded can somehow use Poky as a core technology (just like it uses bitbake, etc), then we could have the best of both worlds. With Poky, we get strong industry commitment and dedicated people working on making the core pieces better, and the tools better (50 engineers are already dedicated). This is something those of us developing products don’t really have time for and we are not currently funded to do this. With OpenEmbedded we can still keep our broad involvement, and vast recipe and target collection. Yes, the quality of OpenEmbedded will be lower, but it still provides a different type of value. Poky provides the known stable core, and OpenEmbedded is just that–it is “open”, and provides the broad menu of components that can be used. Over time this could evolve into something different, but for now it seems a good place to start.</p> The last thing to consider is the OpenEmbedded brand. It is recognizable from the name what it is (unlike Yocto, and Poky). It has had broad exposure for many years. It has historically been very vendor neutral with very little corporate direction or influence. From this perspective, it seems the Yocto project can benefit greatly from having an association with the OpenEmbedded project. This topic was discussed at the summit, and there was general consensus on the strong awareness of the OpenEmbedded brand, as well as an openness to how branding for the Yocto core build system might look.</p> Qt Developer needed 2010-11-05T00:00:00+00:00 We are currently in need of a Qt developer to help on a project. If you are interested, please contact us</a>. Thanks.</p> OpenEmbedded srctree and gitver 2010-10-11T00:00:00+00:00 Recently an OpenEmbedded</a> class name srctree</em> became usable. The srctree.bbclass</a> enables operation inside of an existing source tree for a project, rather than using the fetch/unpack/patch idiom. The srctree.bbclass in combination with the OpenEmbedded gitver.bbclass</a> and git submodules provides a very interesting way to build custom software with OpenEmbedded.</p> One of the classic problems with OpenEmbedded is how do application and kernel developers use it. While OpenEmbedded excels at automating image builds, it is less friendly when used as a a cross development tool. Historically there are several options for iterative development:</p> develop in the working directory: cd /work/arm…/; ../temp/run.do_compile …</li> a variation of #1: bitbake -c devshell </li> manually set up an environment that uses the toolchain generated by OE. As example see this script</a> in the BEC OE template</a>.</li> a variation of #3: export a SDK that includes a toolchain and libs</li> </ol> While the above solutions work OK, the process can be a little cumbersome. Unless your OE recipe pulls software directly from the TIP of a SVN repository, you may have to manually update the recipe after you make changes, create patch files, etc. There is also the problem that if your recipe fetches the latest from SVN, it drastically slows down the recipe parsing as it has to check the repository for a new version every time the recipe is processed.</p> The optimal solution would to be to simply check a software component out of a version control system, and build it directly using OpenEmbedded. Icing on the cake would be if the package generated would automatically extract version information from the version control system. This would facilitate iterative development for software components that need to be cross compiled.</p> Although srctree can be used with any directory of source code, it really works best with a git repository. The gitver.bbclass provides a GITVER variable which is a (fairly) sane version, for use in ${PV}, extracted from the ${S} git checkout, assuming it is one (text from recipe). gitver uses the ‘git describe’ command to extract the last tag and uses that for the version.</p> The best way to illustrate the use of these tools is an example:</p> https://github.com/cbrake/autotools-demo</a></li> </ul> The easiest way to try this is clone the above project in your openembedded/recipes directory:</p> $ cd openembedded/recipes $ git clone https://github.com/cbrake/autotools-demo.git $ cd autotools-demo $ git describe 1.1 $ git tag -l 1.0 1.1 </code></pre> Notice that git describe</em> simple returns the latest tag. The recipe</a> can be located in the same directory as the source code and has the following contents:</p> # recipe to build the autotools-demo in the working tree inherit srctree autotools gitver PV = "${GITVER}" </code></pre> Can’t get much easier than that! If you build the recipe, you end up with a package:</p> $ bitbake autotools-demo $ ls tmp/deploy/glibc/ipk/armv5te/autotools-demo_1.1-r0.6_armv5te.ipk </code></pre> Now, what happens if you make changes and commit them?</p> $ cd .../autotools-demo $ (make a change and commit) $ git describe 1.1-1-gfbc1ecc (notice the count and hash automatically appended) $ (make another change and commit) $ git describe 1.1-2-g7ad3715 (notice the count is now 2) </code></pre> If we bitbake the recipe now, we end up with a packaged named:</p> tmp/deploy/glibc/ipk/armv5te/autotools-demo_1.1-2-g7ad3715-r0.6_armv5te.ipk </code></pre> The gitver</em> class in OpenEmbedded automatically takes care of creating a usable PV (package version) that always increments.</p> So in summary, srctree</em> and gitver</em> give developers a convenient way to handle custom components that change often in an Embedded Linux build without increasing parse times, requiring manual tweaks to version numbers, or creating a separate workspace for each version of the application that is built. As practices such as continuous integration become more common, OpenEmbedded features like this are increasingly needed. An added benefit is that the OpenEmbedded recipe can now be stored in the same location as the source code. Perhaps in the future, most applications will include an OpenEmbedded recipe as part of their source code and git submodules could be used to simple populate the components you want to use.</p> 2017-11-08 update: A tool named devtool</a> is now the preferred way to do much of the above.</em></p> Qt Creator for C/C++ development 2010-08-07T00:00:00+00:00 Recently, I’ve been evaluating Qt Creator</a> for general C/C++ development. I’m currently involved in the development of a rather large C++ application that is approaching 200,000 lines of code and 1000 source modules. In the past, I’ve typically used Vim for editing, and Eclipse as a gdb front-end when needed. Qt Creator is a rather new tool developed by Nokia’s Qt team. What initially attracted my attention was that one of the reasons the project was started was no existing tools effectively handled the Qt codebase, which is quite large. Things I like about Qt Creator:</p> it works fairly well with any C++ Make based project. This includes projects built with autotools as well as the Qt Qmake build system.</li> easy to import existing projects</li> it is very fast. Indexing 200,000 lines of code happens in around a minute or so.</li> Provides a Vim compatibility mode that actually works</li> provides fast methods for finding files, symbols, and seeing where symbols are used</li> did I mention yet it is fast</strong></em>?</li> </ul> I also recorded a screencast that demos Qt Creator with a large project</a> (can be viewed in firefox). As always, I’m interested in what others find interesting in this or other tools. Future efforts will be to use Qt Creator to build and remotely debug ARM binaries – I am interested in what others have done in this regard.</p> If you do try Qt Creator, I recommend the latest pre-release snapshot</a>.</p> Embedded Linux Presentation in Cleveland 2010-07-26T00:00:00+00:00 Chris Cole</a> and I will be giving a presentation</a> in Cleveland Tues evening (2010-07-26). Stop by if you are interested.</p> Git submodules: what to do when you commit to (no branch) 2010-06-17T00:00:00+00:00 When using git submodules, eventually you’ll run into a situation where a workspace has checked out a submodule as (no branch), and you unknownly make modifications and commits to (no branch). Now what? It turns out Git has a very useful feature named reflog. Reflog keeps a local time based log of all activity. That means that as long as you do not garbage collect your workspace, you can go back to any point in time and get any commit that was made. This is useful in a number of scenarios, but lets consider the case where we made two commits to a submodule in the (no branch) state. We are then ready to push our changes to the upstream repo and we get:</p> cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git push bec HEAD error: unable to push to unqualified destination: HEAD The destination refspec neither matches an existing ref on the remote nor begins with refs/, and we are unable to guess a prefix based on the source ref. error: failed to push some refs to 'git@git.bec-systems.com:openembedded.custom' </code></pre> If we do a git log, we find we are on the infamous (no branch).</p> cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git branch * (no branch) master </code></pre> At this point we may wonder what to do. How do we get these changes onto the master branch? Using git reflog, we can view all commits and changes to our local repository based on time.</p> cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git reflog 6d195b3 HEAD@{0}: commit: add recipe for qt-embedded image 4cf1bd6 HEAD@{1}: commit: add touchscreen utilities bcc673c HEAD@{2}: checkout: moving from master to bcc673c9824e916a7ec6c04f60a17116af539423 56d9def HEAD@{3}: commit: add more features 5286f87 HEAD@{4}: commit: rename debug image 90896de HEAD@{5}: pull : Fast-forward 5cc3f3b HEAD@{6}: checkout: moving from 90896deb2a25f0d40b1e6305a0ad994f932f22d8 to master 90896de HEAD@{7}: checkout: moving from master to 90896deb2a25f0d40b1e6305a0ad994f932f22d8 </code></pre> Git will keep our keeps commits to (no branch) even though we might switch to another name branch. And using git reflog, we can easily find these commits and merge them to the master branch.</p> cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git checkout master Previous HEAD position was 6d195b3... add recipe for qt-embedded image Switched to branch 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git reflog 56d9def HEAD@{0}: checkout: moving from 6d195b3909553b04cba3d3ac182076a2c6b866b2 to master 6d195b3 HEAD@{1}: commit: add recipe for qt-embedded image 4cf1bd6 HEAD@{2}: commit: add touchscreen utilities bcc673c HEAD@{3}: checkout: moving from master to bcc673c9824e916a7ec6c04f60a17116af539423 56d9def HEAD@{4}: commit: add more features 5286f87 HEAD@{5}: commit: rename debug image 90896de HEAD@{6}: pull : Fast-forward 5cc3f3b HEAD@{7}: checkout: moving from 90896deb2a25f0d40b1e6305a0ad994f932f22d8 to master 90896de HEAD@{8}: checkout: moving from master to 90896deb2a25f0d40b1e6305a0ad994f932f22d8 cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git merge 6d195b3 Updating 56d9def..6d195b3 Fast-forward recipes/images/bec-debug-image.bb \| 38 +++++++++++++++++++----------- recipes/images/bec-qt-embedded-image.bb \| 8 ++++++ 2 files changed, 32 insertions(+), 14 deletions(-) create mode 100644 recipes/images/bec-qt-embedded-image.bb </code></pre> At this point, the changes have been merged to the master branch, and we can now push to our public repository:</p> cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git push bec HEAD Counting objects: 14, done. Delta compression using up to 8 threads. Compressing objects: 100% (9/9), done. Writing objects: 100% (10/10), 1.13 KiB, done. Total 10 (delta 3), reused 0 (delta 0) To git@git.bec-systems.com:openembedded.custom bcc673c..6d195b3 HEAD -> master </code></pre> Git reflog can also be used to restore commits that you decided at one point to throw away with “git reset --hard HEAD~2”, etc. As long as you don’t garbage collect the repo, they are simply dangling state and always still exist. This is just another example of the ways git can be very confusing, but also very powerful if you understand how it works.</p> This screencast</a> covers the operations in this article.</p> Installing OMAP3 images on a SD card 2010-06-14T00:00:00+00:00 This article and screen-cast is a continuation of that last couple posts describing the BEC OE build template</a>. The purpose again for a build system is to automate tedious manual tasks, and in doing so, we end up documenting how the build system works. Having a good build system is important during product development so that you have an easy, repeatable process for building and deploying images. One of these tasks is setting up a SD card and installing images to the SD card that an OMAP system can boot from. This screencast</a> goes over the script and makefile targets that are used to automate this process.</p> In summary, the steps to set up a SD card and install an image using the build template are:</p> cd </li> determine the SD card device name (/dev/sdX)</li> unmount any file systems mounted on the SD card</li> sudo ./scripts/omap-sd-partitions.sh /dev/sdX (make sure you have the right device!!!)</li> make omap-install-boot</li> make omap-install--rootfs</li> </ol> Previous articles in this series:</p> OpenEmbedded build template</a></li> Creating a Custom OpenEmbedded Image</a></li> </ul> OpenEmbedded Build Template posted 2010-06-08T00:00:00+00:00 A password protected version of this article accidently got pushed into the feed before it was finished, so this post is simply to indicate the following article is now available. Sorry for the confusion.</p> /637/oe-build-template/</a></p> Creating a Custom OpenEmbedded Image 2010-06-08T00:00:00+00:00 In this article screencast, we’ll demonstrate how to create a custom Linux OS image using the OpenEmbedded build system. This demonstration builds on the earlier article about using the BEC OE build template</a>. The OpenEmbedded build system is similar to Linux distributions in that you can select from a wide array of components to install. One of the big differences is you can select these components when building the OS image, instead of after you have installed the “standard” image. This screencast</a> demonstrates how to set up a custom image (can be viewed with Firefox).</p> A related article</a> also covers this procedure.</p> In summary, the steps include:</p> Create an image recipe (typically in your meta data overlay). An example is located here</a>. An existing image can be referenced for simplicity.</li> Figure out what packages need to be added. This can be done by browsing the recipe tree</a> to see what is available.</li> Bitbake the recipe.</li> Look at what packages were generated. In this example they are found in the build/angstrom-2008.1/tmp/deploy/glibc/ipk/armv7a</em> directory.</li> Add the relevant package names to your image recipe.</li> </ol> As the OpenEmbedded project includes over 6000 recipes, it gives you a big head start in including standard components in your Embedded Linux build. Leveraging these existing components is why we use Linux, and having a way to easily build and deploy these components to your target hardware is what OpenEmbedded is all about.</p> OpenEmbedded Build Template 2010-06-05T00:00:00+00:00 Setting up an OpenEmbedded build is a fairly simple process if you carefully follow the instructions</a>. There are also a number of scripts available that automate the setup such as the OpenEmbedded Tools for Eclipse</a>, the Angstrom setup scripts</a>, the KaeilOS Openembedded Manager</a>, and I’m sure there are many more (feel free to add in comments section). As we have helped a number of clients use OpenEmbedded in commercial projects, we have the following requirements for a build template:</p> The build needs to be completely automated such that only one step is required to build images.</li> We need to be able to lock down versions of various components.</li> Needs to be as simple as possible.</li> </ol> As we continue to think about creating a build template, the primary reason for creating one is to get repeatable builds with very little effort. The build template serves as documentation; how did we build the 1.10 release. A good build system allows any developer to create any release on about any machine at any time. A good build system does not depend on the “golden build machine.” We have been using makefile wrappers around OpenEmbedded for several years now to accomplish much of this. These wrappers were typically stored in a Subversion repository with the rest of the project files. However, now that most open source projects have moved to Git repositories, there are significant advantages to using Git for the build system. Git has a feature called submodules which allows you to embed a git repository inside a source tree, always pointed at a particular commit. In concept, this is perfect. In practice, Git submodules are sometimes difficult to use, but considering the functionality provided, I think it is a reasonable trade-off.</p> The use of Git encourages a different development paradigm. Typically when people use Subversion within a company, there is one repository that stores everything. Subversion is actually quite good at this. I’ve run into a number of companies who have repositories many GB in size that have been running for many years. The stability is remarkable. Git does not scale in this way, so people tend to set up a separate repository for each component, application, etc. While there may be some disadvantages to this model, there are also significant advantages, and we can learn some of these by observing Open Source Software (OSS) development. Just as we strive for modularity in software development where each class can stand on its own and be generally useful, OSS maintains this dynamic at the project level. Each OSS project moves at its own pace, largely decoupled from other projects. This dynamic forces a level of organization that might be neglected otherwise. Likewise, in any project, if each major component is stored in its own source code repository, there will also tend to be less coupling, willy-nilly including of header files from other components, etc. It helps people think in a more modular manner at the component level. A component may be the Linux kernel, bootloader, each application in the system, etc. A similar concept may be thinking about storing documents in hierarchical filesystem directory structure versus documents in a Wiki, which are stored in a flat structure and then organized by cross linking. Yes, there are times to use a directory structure, but flexibility of a linked organization system (the essence of the Internet) is hard to beat. Git submodules (or similar concepts) provides a similar linking method of organization.</p> With this in mind, we look at a build system as a way to collect a number of independent components and build them in a repeatable fashion. While this may seem obvious to anyone somewhat familiar with OpenEmbedded, our vision is to extend this concept to easily facilitate product development so that we can move quickly and efficiently, but yet in a controlled and reliable way. The first step is a new build template that is available here</a>. The following video</a> provides a short overview of how to use the build template (can be viewed in Firefox).</p> </video></p> The functionality provided so far is fairly simple and provides a way to quickly set up a build environment. With Git submodules, the versions of Bitbake and the OpenEmbedded are locked down. Submodules are also used to provide an array of components to choose from, and the user initializes only the submodules they are interested in using. The first example of this is providing both the recipe meta data from the OpenEmbedded project or the modified version from Gumstix. In the future, examples will be provided for doing custom application and kernel development using the OE srctree and gitver classes. Feedback is welcome; please let me know if you have questions or suggestions.</p> MeeGo Review 2010-06-02T00:00:00+00:00 As we evaluate various technologies that might be applicable in embedded systems, MeeGo</a> is the subject of this article. MeeGo is a collaboration between Intel and Nokia, and is replacing the Moblin and Maemo efforts. For this review, MeeGo was installed to a USB flash disk and booted on a Asus EEPC. This was quite trivial to do, and the instructions on the MeeGo website are very good</a>. The following video</a> (can be viewed in Firefox) provides a quick overview of the MeeGo Netbook UI.</p> </video></p> Overall, MeeGo looks interesting. Hopefully with the collaboration between Intel and Nokia, there will emerge a number of components that become “standards” for embedded systems such as ways to manage wireless networks, cell phone radios, etc. It would also be nice to have better options for implementing GUI’s on embedded devices with small screens. While the MeeGo user interface is nice, it is not a radical departure from desktop applications. Maemo (used on the Nokia N900) is more advanced in that it has optimized widgets for a small screen. With Maemo, all applications run in a true full screen mode, which is generally desired for devices with smaller screens, or industrial devices where we want to keep user confusion to a minimum. MeeGo applications seem to run full screen, but you can still drag them around, and some applications like the terminal do not start full screen.</p> It seems there is considerable interest in providing distributions based on MeeGo. Novell has announced</a> it plans to provide MeeGo based distributions for netbooks, and Intel claims</a> there are a number of other companies planning to build on top of MeeGo.</p> Gumstix Overo review 2010-02-08T00:00:00+00:00 Based on the interest and number of embedded modules currently available, it appears that the OMAP3 CPU from TI will be very popular in the general purpose embedded Linux market. One of the OMAP3 modules available is the Overo from Gumstix</a>. As the company name suggests, this module looks about like a stick of gum, but smaller, as shown in the photo below. The Gumstix module provides a lot of functionality in a very small package. It is reasonably priced, and is an excellent way to quickly create a product that has advanced functionality such as a high powered CPU (OMAP3), high speed USB, DSP, 3-D graphics acceleration, etc. For an example of the OMAP3 performance compared to previous generations of ARM processors, see this article</a>.</p> </p> The module contains all the core components, and can then be attached to a custom baseboard using the two high density connectors shown in the above photo. There are 70 signals on each of the two connectors. BEC provides a spreadsheet</a> of the Overo signals to assist in designing a custom baseboard.</p> Due to the high level of component integration, the Overo has very few components on the module. As shown in the below photo, there are really only 3 major visible components. The flash and RAM is stacked on top of the CPU. The PMIC contains all the required power managment circuitry, as well as Audio and USB interfacing functionality.</p> </p> The Overo module must be used with a baseboard. Gumstix provides several development baseboards. For production, a custom baseboard is typically created. The Gumstix baseboards are reasonable cost, so they could be used for prototyping or low volume production if they have the required functionality. Below is a photo of the Summit baseboard that provides DVI video out, audio, USB host and OTG, and an expansion connector with a number of signals.</p> </p> Below is the Palo baseboard from Gumstix that can be used to interface with a LCD display. The Palo board contains a resistive touch screen controller, and circuitry required to interface with a LCD.</p> </p> </p> The Overo does not include an Ethernet controller, so you typically use a USB-Ethernet adapter for development. Below shows a typical development setup.</p> </p> One of the things about the OMAP3 that makes it very attractive for embedded development is the amount of software support, and the number of development and production solutions available. The OMAP3 is very well supported by the OpenEmbedded project, and TI is very active in making contributions to various open source projects. This greatly increases the quality and availability of advanced software functionality needed to support a complex system like the OMAP3.</p> </p> </p> There are many options for software development with the OMAP3 systems. Below is a photo of a very simple Qt application. GTK+ and Enlightenment are other popular GUI toolkits. With 256MB of both flash and RAM, this is more than enough memory for most embedded applications, and provides plenty of headroom for adding features for future product revisions. The Overo has the CPU processing capability to run advanced GUI applications with 3-D affects, as well as advanced web application frameworks like web2py that previous generations of ARM CPUs could not effectively run.</p> </p> The Gumstix Overo provides an excellent value for developing advanced products. Especially for low volume products, when you compare the effort and time required to develop a full custom OMAP3 solution versus a simple Overo baseboard, using a module can provide significant advantages in terms of time to market, and development cost.</p> The Go language for embedded systems 2010-01-06T00:00:00+00:00 As one of the things I do is evaluate new technologies for embedded systems, the Go language</a> from Google is an interesting development. I have been looking for a better “system” language for some time. What I mean by a better system language is one that is nearly as efficient as C, does not require a large runtime, has a fast start-up time, and yet supports modern language features. It is interesting that the constraints for very high performance server applications are often similar to those found in embedded systems in that applications must make very efficient use of memory and CPU cycles. This is why C++ is still used a lot by Google and is also popular in embedded systems, but is not as common in applications where CPU power is plentiful compared to the resources to implement the application such as business applications. One area of divergence is the focus on multi-core for high end server applications, but in the next few years, multi-core CPU’s such as the Atom and ARM Cortex-A9 will likely be common in low power embedded systems.</p> There are other options. Vala</a> is very nice to use, is efficient, offers advanced language features, and extensive language bindings. Vala is being used to implement some of the http://freesmartphone.org</a> software stack. C/C++ are the old standby languages, but are rather tedious to program in compared to some of the newer languages in that source and header files are separate, and memory management is manual.</p> My interest in Go at this time is for ARM systems, so I ran a few experiments. From what I can tell, there does exist an ARM Go compiler that is able to produce ARM code, but it currently lacks support for EABI soft floating point</a>. The lack of floating point support is pretty much a show stopper for now, but it is at least a start. The following is an example:</p> export GOARCH=arm cd $GOROOT/src/ ./make-arm.bash cd (create the following source file: hello.go) ================== package main import "fmt" func main() { fmt.Printf("Hello, world\n"); } ================= 5g hello.go (5g is the ARM compiler) 5l hello.5 scp 5.out root@n900: (copy to ARM system) ssh root@n900 ?:~# ./5.out Hello, world (modify with floating point code) ================== package main import "fmt" func main() { var f = 1.5; fmt.Printf("Hello, world\n"); fmt.Printf("float f = %f\n", f); } ================= (now run on ARM system) ?:~# ./5.out Segmentation fault </code></pre> As, you can see, floating point does not work as expected. I’m not sure yet if gogcc can be configured to compile ARM binaries. The other area where Go appears to need a lot of work is bindings to various libraries. Though it does not appear to be ready for embedded ARM systems at this time, Go will be an interesting language to watch over the next few years.</p> OpenEmbedded development activity 2009-12-29T00:00:00+00:00 Ever since I have been sending out weekly change logs</a>, I have been impressed by the consistent amount of development activity in the OpenEmbedded project. Every week there are consistently over a dozen developers making changes. Developers come and go, but the contribution level always seems healthy. While this amount of development leads to some amount of churn and issues, this amount of development is also required to keep pace with all the new developments in the OSS space. Not having to wait 2 years for the next big release is one of the big advantages of using Linux and OpenEmbedded. Having access to the latest technology provides a competitive advantage for many products, and makes dealing with the occasional issues that come up an acceptable trade-off. Coupled with a flexible and consistent build system, OpenEmbedded is the vehicle to build advanced products. Yes, its hard. Yes, you better budget for some system software development time. Yes, you should get some help if you’re not an embedded Linux expert. Yes, you can start with some canned vendor “BSP” that seems to work and is seemingly the low-cost way to go, but eventually you’ll hit a brick wall where you need some bit of functionality that is not there, and then you’ll start up the exponential curve of dumping lots of time into trying to make something work with little progress forward. It is much better to count the cost up front, and do things right.</p> OMAP3 Resume Timing 2009-12-07T00:00:00+00:00 One of the most common power management modes for ARM processors is the suspend mode. In this mode, peripherals are shut down when possible, the SDRAM is put into self-refresh, and the CPU is placed in a low power mode. A useful bit of information is to know how soon the system can respond to a resume (wake) event.</p> It turns out the answer to this question depends on where you want to do the work. In the first test, I created a simple application that continuously toggled a GPIO. This is an easy way to tell determine with a scope when the application was running. I then measured the time between the wake event, and when this application signal started toggling. It was a consistent 180ms.</p> For the next test I simply toggled a gpio in the omap3_pm_suspend()</em> first thing after resume. This tells me roughly how fast I can execute code in the kernel after a wake event. This turned out to be 250uS.</p> So the good news is we can run kernel code very quickly after a resume event (250uS), but it currently takes a long time until user space processes start running again (180mS). The 180ms can likely be reduced with some work, but this at least gives us a baseline. 180ms is fairly quick from a human perspective (seems pretty much instant), but for industrial devices that are responding to real-world events, then 180mS can be a long time.</p> Linux kernel stats, and long term advantages 2009-12-01T00:00:00+00:00 I just read an interesting interview</a> with Greg Kroah Hartman. According to Greg, we add 11,000 lines, remove 5500 lines, and modify 2200 lines every single day. This rate of change is something that few organizations have the resources to match. It is interesting that Google chose to use the Linux kernel in the Android project even though they implemented most other parts of the system. Another interesting thing to consider is the consequences of the Linux kernel license, which requires many drivers to be released as GPL. While initially, this may be viewed as a disadvantage, it leads to some interesting long-term implications. The obvious effect is that most Linux driver code is maintained in the mainline kernel source. This leads to other effects. One is that things are allowed to change, and get better. The USB stack in both Linux and Windows has been re-written several times, but the difference is that in Linux the old software does not need to be maintained, as most drivers are maintained in the kernel, and can simply be modified along with core changes. Another advantage is that drivers tend to be much simpler as common code is merged. As a result, a Linux driver tends to be about 1/3 the size of a driver in other operating systems. Yes, getting driver code accepted in the mainline kernel can be difficult at times, but it is clearly the best long-term option.</p> Linux PM: OMAP3 Suspend Support 2009-11-23T00:00:00+00:00 This article provides an overview of the Linux kernel support for the suspend state in the TI OMAP3. Power management has always been one of the more difficult parts of a system to get right. The OMAP3 power management is quite extensive. There are many levels of very granular control over the entire system. Initially, we are going to focus on a simple power state where we want to put the CPU in a low power mode (basically not doing anything), but resume very quickly once an event occurs and continue running applications where they left off. This is typically called “Suspend”.</p> Power States</h2> We must first match up terminology between the Linux kernel and the OMAP documentation. The Linux kernel supports several suspend power states:</p> PM_SUSPEND_ON</strong>: system is running</li> PM_SUSPEND_STANDBY</strong>: system is in a standby state. This is typically implemented where the CPU is in a somewhat static state. After all the peripherals have been shut down, and the memory put into self refresh, the CPU executes an instruction that puts the CPU into a low power state. After the next wake-event occurs, the CPU resumes operating at the next instruction. The CPU state is static in this case.</li> PM_SUSPEND_MEM</strong>: This goes one step beyond STANDBY in that the CPU is completely powered down and loses its state. Any CPU state that needs to be saved is stored to SDRAM (which is in self refresh), or some other static memory. When the system resumes, it has to boot through the reset vector. The bootloader determines that the system was sleeping, and re-initializes the system, enables the SDRAM, restores whatever state is needed, and then jumps to a vector in the kernel to finish the resume sequence. This mode is fairly difficult to implement as it involves the bootloader, and a lot of complex initialization code that is difficult to debug.</li> PM_SUSPEND_MAX</strong>: From browsing the kernel source code, it appears this state is used for “Off” or perhaps suspend to disk.</li> </ul> The OMAP3 documentation uses the following terms:</p> SLM</strong>: Static Leakage Management. Includes “suspend” functionality.</li> Standby</strong>: OMAP3x retains internal memory and logic. (uses 7mW)</li> Device Off</strong>: System state is saved to external memory(0.590mW). This could theoretically map to the PM_SUSPEND_MEM state.</li> WFI</strong>: Wait for Interrupt. This is an ARM instruction that puts the CPU in the standby state.</li> </ul> Based on the source code in the Linux kernel, there is currently only support for the Standby state:</p> static int omap3_pm_enter(suspend_state_t unused) { int ret = 0; switch (suspend_state) { case PM_SUSPEND_STANDBY: case PM_SUSPEND_MEM: ret = omap3_pm_suspend(); break; default: ret = -EINVAL; } return ret; } </code></pre> What kernel branches/versions support PM</h2> Kevin Hilman has been maintaining an OMAP3 power management branch at http://git.kernel.org/?p=linux/kernel/git/khilman/linux-omap-pm.git</a>. Bits are being merged upstream. Some of the things that don’t seem to be merged yet are SmartReflex support, and a number of other details. But, it seems that basic suspend/resume support is available in the upcoming 2.6.32 kernel.</p> More Details on the Standby Mechanism</h2> Due to the granularity and complexity of the the OMAP3 power management, relevant pieces of documentation are scattered throughout the OMAP35x Technical Reference Manual. The following were helpful:</p> Table 3-17: details the MPU subsystem operation power modes</li> Table 3-18: Power Mode Allowable Transitions. Standby mode can enter from active mode only by executing the wait for interruption (WFI) instruction.</li> SDRC, “Self Refresh Management”. The OMAP3 can automatically put the SDRAM in self refresh mode when the CPU enters the idle state.</li> MPU Power Mode Transistions: The ARM core initiates entering into standby via software only (CP15 - WFI).</li> </ul> The omap34xx_cpu_suspend</em> function is called to put the CPU into standby mode. The actual instruction that stops the CPU is the ARM WFI (wait for interrupt) instruction. This is a little different than previous generations of ARM CPUs that used coprocessor registers.</p> Testing</h2> With the 2.6.32-rcX kernels, suspend/resume appears to work at least from the console:</p> root@overo:~# echo mem > /sys/power/state PM: Syncing filesystems ... done. Freezing user space processes ... (elapsed 0.00 seconds) done. Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done. Suspending console(s) (use no_console_suspend to debug) <after key press on serial console ...> omapfb omapfb: timeout waiting for FRAME DONE CLIFF: going into suspend Wake up daisy chain activation failed. CLIFF: returning from suspend Successfully put all powerdomains to target state Restarting tasks ... done. root@overo:~# </code></pre> The CLIFF messages are debug statements placed in the low-level suspend code to make sure we were getting to that point. The console is disabled early in the suspend process, so we don’t get the debug messages until after resume. The no_console_suspend</em> kernel option can be used to leave the console enabled during suspend, but that feature does not seem to work.</p> Summary/Future work</h2> This covers the basics of Linux Suspend/Resume support for the OMAP3 CPU. There are many more options and details such as configuring wake sources, dynamic power management while running, etc.</p> Integrated CAN solutions for Linux 2009-11-19T00:00:00+00:00 I just received an email notification from EMS (http://ems-wuensche.com/</a>) that support for their CAN controllers is now in mainline Linux kernels. The EMS PCI products are supported in 2.6.31, and the CPC-USB product will be supported in 2.6.32. I’ve used various Linux CAN stacks in the past, but none were as well integrated as the SocketCAN</a> solution that has been merged into the mainline Linux kernel. Though I’ve not used the EMS products yet, the fact that support is in the mainline kernel source provides a good indication that they know how to do things correctly, and that the software is of good quality.</p> The Microchip MCP2515 is another option for interfacing CAN with a Linux system, but uses the SPI bus so it typically requires a custom hardware design. The availability of a USB adapter provides an easy way for CAN to be used with about any existing Linux system including small embedded systems. From looking at the kernel source code, it appears that EMS solutions use the SJA1000 CAN controller, which has been an industry standard controller for many years.</p> Message buffering is an important consideration when selecting a Linux CAN solution. Often in a system, the Linux computer is responsible for data processing, user interface, and other CPU intensive operations. The real-time control is often relegated to a separate microcontroller. The CAN bus is then just a convenient and robust way to get data from the control part of the system to the data processing part. The amount of hardware buffering in the CAN controller determines how quickly the system needs to respond to process the CAN data, so the CAN controller has space to receive new data. The SJA1000 provides a 64-byte receive fifo, or roughly space for 4 CAN messages. This provides a little more buffering than the 2 messages in the MCP2515. Dealing with full bandwidth CAN data in a Linux system can be a challenge with small amounts of buffering as you typically have to implement the Linux RT patch, and spend some time tuning it. Another upcoming CAN solution that looks nice is the TI High End CAN Controller (HECC) found in the new Sitara CPUs. This controller provides 32 hardware mailboxes which should provide even more buffering, which makes it even better suited for operating systems like Linux. As with any technology, there are tradeoffs, so its nice to have options.</p> One area the EMS solution may be very useful is for prototyping and development. The ideal development flow when developing an Embedded Linux product is to development as much as possible on a PC. The EMS product allows you to easily implement SocketCAN functionality on a PC so you can talk to the rest of your system directly from a PC. The final embedded solution may be based on a more cost effective solution like the MCP2515 or the TI HECC, but being able to interface to the rest of the system directly from a PC has the potential to improve development efficiency. This is part of the “Big Win” with Embedded Linux where the same technology scales from the embedded system, to the PC, to the server.</p> Notification at the end of builds 2009-11-05T00:00:00+00:00 I do quite a few OpenEmbedded project builds during the course of a week. This process usually takes 3-5 minutes. That is just enough time to get distracted doing something else and forget about the build until an hour later when you realize – oops, I was supposed to send out a release email once the build was finished and uploaded. It occured to me that it would be nice if my computer played a distinct sound at the end of the build. With Linux this is incredibly easy:</p> wget http://upload.wikimedia.org/wikipedia/commons/7/76/Ding_Dong_Bell.ogg</li> sudo mv Ding_Dong_Bell.ogg /usr/local/</li> sudo aptitude install cplay</li> sudo echo “cplay /usr/local/Ding_Dong_Bell.ogg” > /usr/local/bin/bell</li> sudo chmod 775 /usr/local/bin/bell</li> </ul> Now, when I have a long build, I simply do something like:</p> bitbake my-image; bell </code></pre> There is probably something better than cplay that does not start a UI – any suggestions?</p> A quick serial logging application 2009-10-21T00:00:00+00:00 Recently when debugging a device connected to a rs485 bus, I needed a simple application to dump the raw data coming over the rs485 bus. Minicom does all kinds of weird terminal stuff, plus it will not display binary data, so that was out. While looking for serial analyzer programs for Linux, and pondering writing my own simple C application, it occurred to me to use pyserial. With a few lines of code, I got exactly what I needed:</p> import serial import sys import struct if len(sys.argv) != 3: print "Usage: serial-dump.py <serial port> <baud rate>" sys.exit(-1) print "Dumping data on %s" % (sys.argv[1]) ser = serial.Serial(sys.argv[1], sys.argv[2], timeout=0.3) while(1): buf = ser.read(500) if len(buf) > 0: print "Read %i bytes:" % (len(buf)), for i in buf: value = struct.unpack('B', i)[0] print "%02x" % (value), print "" </code></pre> The full source is available here</a>. This could obviously be improved to do true blocking reads, automatically parse protocols, implement a gui, etc. Ideas for efficiently reading data using pyserial is available in a previous post</a>. Another example of using a scripting language for quick tasks is /226/a-really-nice-hex-calculator/</a>.</p> Apache and how to correctly use NameVirtualHost 2009-10-16T00:00:00+00:00 As I often get involved in server administration (SVN, git, redmine, etc setup), I deal with Apache on a regular basis. For simple configurations, the default Ubuntu/debian config works well. However, for a more complex setup with virtual hosts, multiple IP addresses, and SSL support, it is common to run into the following message:</p> mixing * ports and non-* ports with a NameVirtualHost address is not supported, proceeding with undefined results</em></p> Googling does not return a clear explanation to this problem, so that is why I’m writing this article. The impulsive approach of randomly changing things does not get you too far – especially with the Debian/Ubuntu Apache configuration that is composed of seperate files for each virtual host. After reading the documentation, things become clearer:</p> Note that the argument to the <VirtualHost></code> directive must exactly match the argument to the NameVirtualHost</code> directive.</em></p> And, by considering the fact that the seperate virtual host config files really just get concatenated into one big file, it makes sense that you can’t have more than one NameVirtualHost entry. Ubuntu by default puts the NameVirtualHost in the default virtual host file, which gets soft linked to sites-enabled/000-default. I presume the ‘000’ means it gets loaded first. I find it much more intuitive to put the NameVitualHost directives in the /etc/apache2/httpd.conf file. This way it is clear that these entries are common to all virtual host entries. So what I end up with is:</p> /etc/apache2/httpd.conf</p> NameVirtualHost 74.208.184.45:80 NameVirtualHost 74.208.184.46:80 NameVirtualHost 74.208.184.45:443 </code></pre> /etc/apache2/sites-available/bec-systems.com</p> <VirtualHost 74.208.184.45:80> ServerName bec-systems.com ... </code></pre> /etc/apach2/sites-available/bec-systems.com.ssl</p> <VirtualHost 74.208.184.45:443> ServerName bec-systems.com ... </code></pre> /etc/apach2/sites-available/svn.bec-systems.com</p> <VirtualHost 74.208.184.45:80> ServerName svn.bec-systems.com ... </code></pre> </code></pre> /etc/apach2/sites-available/site2.com (with different IP address)</p> <VirtualHost 74.208.184.46:80> ServerName site2.com ... </code></pre> Again, the points to remember are:</p> The argument to VirtualHost must match a NameVirtualHost directive</li> the NameVirtualHost directive is common, so there should only be one entry</li> </ol> Best practices for kernel development with OpenEmbedded 2009-10-13T00:00:00+00:00 A common question is how do you do kernel development with OpenEmbedded? Typically, OpenEmbedded builds a kernel by checking the source out of a git repository, or by applying patch files to a released version of the kernel. See the many recipes</a> for examples. This works very well for a Linux distribution build system, but is less than ideal for ongoing kernel development where you will have many iterations over a period of time. The OE work directory is considered transient, and should not be used for any ongoing development. Fortunately, the kernel build system is self contained, so it is a very simple matter to reference the toolchain built by OpenEmbedded and work outside the OpenEmbedded build system.</p> Use Git</h2> Trust me on this one, just use it! It will make your life easier. There is a learning curve to any powerful tool. Another way to state this is if a tool does not do much, then it would be easy to learn. So expect to spend a little time learning to use git. With modern Linux kernel development, much of your time is spent integrating work that others are doing. For example, if you are using the OMAP3 CPU, you might want to start with linux-omap branch, integrate some patches from the OE beagleboard build, merge the OMAP PM tree, and then merge some bits from the Linux-wireless tree. This is the way embedded Linux development is done today. Git makes it possible to have many parallel developments going at their own pace, including your own. Even in your own development, you will find significant advantages to being able to quickly create and merge branches, revise commits, etc. There are several articles about git</a> on this site.</p> Importing an OE kernel into git</h2> If you are starting with an existing OE kernel tree, then you typically need to set up an equivalent tree in your local git workspace. Typically you just look at the recipe to see how to manually reconstruct it. For example, lets consider the SRC_URI from the palm-omap1</a> recipe:</p> SRC_URI = "${KERNELORG_MIRROR}/pub/linux/kernel/v2.6/linux-2.6.22.tar.bz2 \ http://www.muru.com/linux/omap/patches/patch-2.6.22-omap1.bz2;patch=1 \ file://defconfig" </code></pre> To set this kernel up in git, we might do the following:</p> git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git</li> cd linux-2.6</li> git checkout -b my_branch v2.6.22</li> wget http://www.muru.com/linux/omap/patches/patch-2.6.22-omap1.bz2</code></li> bzip2 -d patch-2.6.22-omap1.bz2</li> git am patch-2.6.22-omap1</li> </ol> If the git am fails, then you might need to add a subject and author line to the patch in the following form and then retry git am.</p> From: Name <name@company.com> Subject: omap1 patch </code></pre> Now when you do a git log, you will see the above patch applied. git apply can also be used to apply patches, but I prefer git-am as it automatically handles new files and fills in the commit message and author.</p> Setting up the toolchain</h2> Now that you have a kernel, you must set up a toolchain so you can build it. I typically set up my kernel directory next to my OE build directory in my project space, so I can then source the following script</a> to set up the toolchain:</p> CROSS_COMPILER_PATH=`cd ../oe/build/angstrom-2008.1/tmp/cross/armv7a/bin; pwd` BUILD_ARCH=`uname -m` OE_STAGING_PATH=`cd ../oe/build/angstrom-2008.1/tmp/staging/${BUILD_ARCH}-linux/usr/bin; pwd` STAGING_KERNEL_DIR=`cd linux-2.6; pwd` export PATH=$PATH:$CROSS_COMPILER_PATH:$OE_STAGING_PATH export ARCH=arm export CROSS_COMPILE=arm-angstrom-linux-gnueabi- </code></pre> Adjust to whatever you are building for. Now you can simply type make. The reason the OE_STAGING_PATH is added to the PATH in the above example is so that the uboot mkimage utility can be used directly from the OE build if your build target is make uImage.</p> Installing modules</h2> Sometimes you need to re-install all modules on the target because you changed kernel version, etc. Typically OE packages up all modules and adds the one you specify to the rootfs. Because we are building the kernel outside the OE directory, OE can no longer do this. However, its still very easy to install modules from your kernel development directory:</p> rm -rf modules_install; INSTALL_MOD_PATH=modules_install make modules_install rsync -av modules_install/lib/modules/2.6.22 root@192.168.1.115:/lib/modules/ </code></pre> The first command installs the modules to a temporary directory, and the second command rsync’s the modules to your target system.</p> Creating a kernel recipe with your changes</h2> Once you get to the point where you want to create a recipe with your changes, you can easily export a patch, or series of patches using git diff or git format-patch. Simply add the patch files to your kernel recipe SRC_URI. You can also teach bitbake to fetch your kernel source directly from a git repository, but I’ve generally found the patch mechanism to be adequate and easier in most cases. As an example, you can create one large patch with all your changes that can be applied on the 2.6.22 released kernel:</p> git diff HEAD..v2.6.22 > my_kernel_changes.patch </code></pre> Other ways …</h2> This is just one flow and there are many variations. There is also a new mechanism in OpenEmbedded called srctree that allows you to use OpenEmbedded in an external source tree. srctree has the potential to further streamline this type of development.</p> As with any task, you want to use the tools that make sense for the task. OpenEmbedded makes sense for building Linux distributions, but it is not a kernel development tool. Git and the kernel build system make sense for kernel development.</p> converting digital camera movies to ogg format 2009-09-09T00:00:00+00:00 Now that firefox (v3.5) has built in support for ogg videos, ogg is now the most convenient format for “easy to view” video files (assuming you don’t want to use a service like youtube). Some digital cameras (like my Canon) records movies in AVI format. I created a script that uses gstreamer to convert the AVI file to an ogg file for easy viewing.</p> http://dev.bec-systems.com/svn/pub/avi_to_ogg/avi_to_ogg</a></p> And some of the results:</p> /video/</a></p> (hint, download firefox 3.5 to view these videos)</p> Best practices for building Gtk+ applications with OpenEmbedded 2009-09-08T00:00:00+00:00 I recentlywrote an article about best practices building Qt applications with OpenEmbedded</a>, and it occured to me that I should write an equivalent article for Gtk+ applications. The same points apply – put your application source in a SCM system, and put the install logic in the application source (read the above article). The difference is that Gtk applications typically use autotools where Qt applications use qmake to build the application. This article details how a minimal GTK+ application should be set up and built using OpenEmbedded.</p> Application Source</h2> I created a sample GTK hello application located at: http://dev.bec-systems.com/svn/pub/gtk_hello</a>. This is a SVN repository, so you can simply “svn co” the above URI to check out the code. If you are running Ubuntu, you can install the necessary tools to build a native Gtk+ application by:</p> sudo apt-get install libgtk2.0-dev build-essential autoconf automake pkg-config</li> </ul> To build on your x86 Linux PC, run the following steps:</p> svn co http://dev.bec-systems.com/svn/pub/gtk_hello</a></li> cd gtk_hello</li> autoreconf -i</li> ./configure --prefix=`pwd`/install</li> make install</li> </ul> This compiles and installs the application in to the “install” directory. If you look in this directory, you will notice the application binary is installed in the “install/bin” directory. Typically, the install directory is set to /usr/bin, but in this example we set it to install so we don’t need to run “make install” as root, but yet we can verify the install logic works properly.</p> OpenEmbedded Recipe</h2> Now that you have verified the application builds and installs properly on a x86 PC, it is trivial to build the application in OpenEmbedded. Create a recipe in your OE recipes directory named gtk-hello_svn.bb</em> with the following contents:</p> DESCRIPTION = "Sample Gtk+ Hello application, used to demonstrate build system" AUTHOR = "Cliff Brake <cbrake@bec-systems.com>" SRCREV = "17" PV = "0.0+svn${SRCREV}" PR = "r0" DEPENDS = "gtk+" SRC_URI = "svn://dev.bec-systems.com/svn/pub;module=gtk_hello;proto=http;rev=17" S = "${WORKDIR}/gtk_hello/" inherit autotools </code></pre> Now, run: bitbake gtk-hello. That is it! Building Linux applications is easy if you simply use the tools, whether it be autotools, qmake, etc. Too often there is the tendency to set up your own compile steps with ${CC} variables, etc. While this seems to be the simple approach at first glance (autotools is too hard), it quickly becomes unmaintainable and in the end is a lot more work than simply learning the basics of the industry standard tools. See previous autotools article</a> for more information.</p> Best practices for building Qt applications with OpenEmbedded 2009-08-06T00:00:00+00:00 This article describes how to cross compile a Qt application (named qt_tutorial) with OpenEmbedded, and several best practices you should consider. OpenEmbedded currently includes fairly good support for building Qt</a> – both Qt Embedded and Qt X11. OE also includes a number of qt classes</a> that make building Qt applications easy. One of the main considerations with embedded Linux application development is that you keep the build system flexible so that you can easily build on a PC or for your embedded hardware. This means that hand crafted Makefiles with hardcoded paths to cross compilers do not quality.</p> Put Your Application in a SCM system</h2> No matter what type of application you are building, it is a good idea to put your source in a SCM or revision control system and fetch it directly using OE. OpenEmbedded supports fetching sources from a number of revision control systems including Subversion, Git, Bazaar, Mercurial, CVS, etc. The not so obvious reason we do this is so you can easily check out the source code and build it on a PC as well as your target system in OE. In this example we fetch the application source from a SVN repository:</p> SRC_URI = "svn://dev.bec-systems.com/svn/pub;module=qt_tutorial;proto=http" </code></pre> For the above repository, the direct URI to the source would be: http://dev.bec-systems.com/svn/pub/qt_tutorial/</a></p> Put the Install logic in the Application Source</h2> Most Linux applications support “make install”, and this is the case with autotools, and qmake (Qt’s build tool). We could put logic in the OE recipe to install the application something like the following:</p> do_install() { install -d ${D}/${bindir} install -m 0755 ${S}/qt_tutorial ${D}/${bindir} } </code></pre> But, the problem with this approach is you can’t install the application in other environments (like a native x86 PC build) unless you are building with OE. So a better approach is to put the logic to install the application in the application source, so that in can be installed in both your PC environment, and your OpenEmbedded build. To accomplish this, you can set up the project as follows</p> application qmake project file (qt_tutorial.pro):</strong></p> TEMPLATE = app TARGET = qt_tutorial DEPENDPATH += . INCLUDEPATH += . # Input HEADERS += lcdrange.h SOURCES += lcdrange.cpp main.cpp target.path = /usr/bin INSTALLS += target </code></pre> OpenEmbedded recipe (qt-tutorial_svn.bb):</strong></p> inherit qt4e SRCREV = "${AUTOREV}" PV = "1.0+svnr${SRCREV}" PR = "r1" SRC_URI = "svn://dev.bec-systems.com/svn/pub;module=qt_tutorial;proto=http" S = ${WORKDIR}/qt_tutorial FILES_${PN}-dbg += "${bindir}/.debug" do_install() { export INSTALL_ROOT=${D} make install } </code></pre> Now, the same mechanism is used to install the application on both a PC native build, as well an OpenEmbedded build. If you look in the Makefile generated by qmake, you see the following:</p> install_target: first FORCE @$(CHK_DIR_EXISTS) $(INSTALL_ROOT)/usr/bin/ \|\| $(MKDIR) $(INSTALL_ROOT)/usr/bin/ -$(INSTALL_PROGRAM) "$(QMAKE_TARGET)" "$(INSTALL_ROOT)/usr/bin/$(QMAKE_TARGET)" install: install_target FORCE </code></pre> INSTALL_ROOT can be set to force the entire system to be installed in a subdirectory. This is required for build systems that generate packages, like OpenEmbedded. To build this example, put the qt-tutorial_svn.bb file in your OE recipes tree, and run: bitbake qt-tutorial. This will fetch the source code and build a package. To run the tutorial, install the package on a system that includes Qt Embedded, and then run: qt_tutorial -qws.</p> Gumstix Overo Pinout Spreadsheet updated 2009-07-30T00:00:00+00:00 The Gumstix Overo Pinout Spreadsheet</a> has been updated with the Palo board connector pinouts, and a few mistakes have been fixed.</p> </p> </p> How to capture source changes to an OpenEmbedded package 2009-07-14T00:00:00+00:00 One task that is a often confusing to new OpenEmbedded users is how to capture changes to the source code for a package/recipe. First, lets review the progression in tools use to capture source code changes.</p> Figure 1: Progression of tools</figcaption></figure> In days of yore, developers often used gnu diff to capture modifications to a source tree. The typical practice was to create two source trees: a pristine, and a dirty tree. Modifications were made in the dirty tree, and periodically gnu diff was run between the two trees to capture the changes. Then came quilt which allowed you to capture changes without a pristine tree – kind of a cheap version control system. The typical flow was:</p> cd </li> quilt new my_changes.patch</li> quilt add file1.c</li> edit file1.c</li> quilt refresh (now my_changes.patch records the edits to file1.c)</li> debug</li> edit file2.c</li> (oops forget to quilt add file2.c – now what ….)</li> restore file2.c from pristine source</li> quilt add file2.c</li> edit file2.c</li> quilt refresh</li> (now my_changes.patch records edits to file1.c and file2.c)</li> </ul> As you can see, things are much better than plain diff, but you still have to keep track of the files you edit. With git, we can do essentially the same things, only git keeps track of the files you edit, plus it is easier to organize patch files. The flow with git is something like this:</p> cd </li> git init</li> git add </li> git commit (now the entire source tree is recorded by git)</li> edit file1.c</li> edit file2.c ….</li> git status (shows you files that were modified)</li> git add file1.c file2.c</li> git commit</li> git log (shows you changes you have made)</li> git format-patch HEAD~1 (outputs the patch created by the last commit)</li> </ul> Once your source code is in git, then you can do all kinds of fancy things like merging, re-ordering, and re-naming patch files with the git-rebase --interactive</a> command.</p> One of the uses of this technique is when you need to record a small change to a package source in a build system such as OpenEmbedded, and apply that change for subsequent builds. In this case you might do something like the following:</p> bitbake -c devshell <recipe_to_patch></p> </li> in devshell capture changes in git as shown above to create my_changes.patch</em></p> </li> cp my_changes.patch <oe_tree>/recipes/<recipe_to_change>/files/</p> </li> edit <oe_tree>/recipes/<recipe_to_change>/<recipe_to_change>.bb</p> tell bitbake to apply the patch when building the recipe: SRC_URI += “file://my_changes.patch;patch=1”</li> increment the PR variable so that bitbake knows to rebuild the recipe.</li> </ul> </li> </ul> Figure 2 illustrates the above steps.</p> Figure 2: Flow to capture package source changes in OpenEmbedded</figcaption></figure> While the above steps might seem overly complex, you must consider what you are trying to do. Building a Linux distribution is not a simple task. We are dealing with large amounts of source code from many different projects. Putting all this source code in one tree is not an option, because then we would be defeating the very reasons open source is attractive in the first place: flexibility, largly decoupled projects, advanced technology, fast bug fixes, many options to choose from, developer support, etc. Each open source project must remain a self contained project that is largly decoupled from other projects so it can develop at its own pace. It is for these reasons that open source technologies are used to give products a competitive edge. So, like any complex process or task, there must be some understanding of how it works, an appreciation for what it can do for you, and an investment in tools and techniques required to be effective.</p> update 2023-09-23</strong>: recipetool</code> and devtool</code> are updated tools that can be used to make and capture changes.</p> Embedded Linux versus Windows CE 2009-06-20T00:00:00+00:00 Occasionally I am asked how Embedded Linux compares with Windows CE. I have spent the past 5 years doing mostly embedded Linux development, and the previous 5 years doing mostly WinCE development with a few exceptions, so my thoughts are no doubt a little biased toward what I understand best. So take this with a grain of salt :-) In my experience, the choice is often made largely on perception and culture, rather than concrete data. And, making a choice based on concrete data is difficult when you consider the complexity of a modern OS, all the issues associated with porting it to custom hardware, and unknown future requirements. Even from an application perspective, things change over the life of a project. Requirements come and go. You find yourself doing things you never thought you would, especially if they are possible. The ubiquitous USB and network ports open a lot of possibilities – for example adding Cell modem support</a> or printer support</a>. Flash based storage makes in-field software updates</a> the standard mode of operation. And in the end, each solution has its strengths and weaknesses – there is no magic bullet that is the best in all cases.</p> When considering Embedded Linux development, I often use the iceberg analogy; what you see going into a project is the part above the water. These are the pieces your application interacts with, drivers you need to customize, the part you understand. The other 90% is under water, and herein lies a great deal of variability. Quality issues with drivers or not being able to find a driver for something you may want to support in the future can easily swamp known parts of the project. There are very few people who have a lot of experience with both WinCE and Linux solutions, hence the tendency to go with what is comfortable (or what managers are comfortable with), or what we have experience with. Below are thoughts on a number of aspects to consider:</p> SYSTEM SOFTWARE DEVELOPMENT</h2> Questions in this realm include CPU support, driver quality, in field software updates, filesystem support, driver availability, etc. One of the changes that has happened in the past two years, is CPU vendors are now porting Linux to their new chips as the first OS. Before, the OS porting was typically done by Linux software companies such as MontaVista, or community efforts. As a result, the Linux kernel now supports most mainstream embedded cpus with few additional patches. This is radically different than the situation 5 years ago. Because many people are using the same source code, issues get fixed, and often are contributed back to the mainstream source. With WinCE, the BSP/driver support tends to be more of a reference implementation, and then OEM/users take it, fix any issues, and that is where the fixes tend to stay.</p> From a system perspective, it is very important to consider flexibility for future needs. Just because it is not a requirement now does not mean it will not be a requirement in the future. Obtaining driver support for a peripheral may be nearly impossible, or be too large an effort to make it practical.</p> Most people give very little thought to the build system, or never look much beyond the thought that “if there is a nice gui wrapped around the tool, it must be easy”. OpenEmbedded is very popular way to build embedded Linux products, and has recently been endorsed as the technology base of MontaVista’s Linux 6 product, and is generally considered “hard to use” by new users. While WinCE build tools look simpler on the surface (the 10% above water), you still have the problem of what happens when I need to customize something, implement complex features such as software updates, etc. To build a production system with production grade features, you still need someone on your team who understands the OS and can work at the detail level of both the operating system, and the build system. With either WinCE or Embedded Linux, this generally means companies either need to have experienced developers in house, or hire experts to do portions of the system software development. System software development is not the same as application development, and is generally not something you want to take on with no experience unless you have a lot of time. It is quite common for companies to hire expert help for the first couple projects, and then do follow-on projects in-house. Another feature to consider is parallel build support. With quad core workstations becoming the standard, is it a big deal that a full build can be done in 1.2 hours versus 8? How flexible is the build system at pulling and building source code from various sources such as diverse revision control systems, etc.</p> Embedded processors are becoming increasingly complex. It is no longer good enough to just have the cpu running. If you consider the OMAP3 cpu family from TI, then you have to ask the following questions: are there libraries available for the 3D acceleration engine, and can I even get them without committing to millions of units per year? Is there support for the DSP bridge? What is the cost of all this? On a recent project I was involved in, a basic WinCE BSP for the Atmel AT91SAM9260 cost $7000. In terms of developer time, this is not much, but you have to also consider the on-going costs of maintenance, upgrading to new versions of the operating system, etc.</p> APPLICATION DEVELOPMENT</h2> Both Embedded Linux and WinCE support a range of application libraries and programming languages. C and C++ are well supported. Most business type applications are moving to C# in the WinCE world. Linux has Mono, which provides extensive support for .NET technologies and runs very well in embedded Linux systems. There are numerous Java development environments available for Embedded Linux. One area where you do run into differences is graphics libraries. Generally the Microsoft graphical APIs are not well supported on Linux, so if you have a large application team that are die-hard windows GUI programmers, then perhaps WinCE makes sense. However, there are many options for GUI toolkits that run on both Windows PCs and Embedded Linux devices. Some examples include GTK+, Qt, wxWidgets, etc. The Gimp is an example of a GTK+ application that runs on windows, plus there are many others. The are C# bindings to GTK+ and Qt. Another feature that seems to be coming on strong in the WinCE space is the Windows Communication Foundation (WCF). But again, there are projects to bring WCF to Mono, depending what portions you need. Embedded Linux support for scripting languages like Python is very good, and Python runs very well on 200MHz ARM processors.</p> There is often the perception that WinCE is realtime, and Linux is not. Linux realtime support is decent in the stock kernels with the CONFIG_PREEMPT option, and real-time support is excellent with the addition of a relatively small real-time patch</a>. You can easily attain sub millisecond timing with Linux. This is something that has changed in the past couple years with the merging of real-time functionality into the stock kernel.</p> DEVELOPMENT FLOW</h2> In a productive environment, most advanced embedded applications are developed and debugged on a PC, not the target hardware. Even in setups where remote debugging on a target system works well, debugging an application on a workstation works better. So the fact that one solution has nice on-target debugging, where the other does not is not really relevant. For data centric systems, it is common to have simulation modes where the application can be tested without connection to real I/O. With both Linux and WinCE applications, application programing for an embedded device is similar to programming for a PC. Embedded Linux takes this a step further. Because embedded Linux technology is the same as desktop, and server Linux technology, almost everything developed for desktop/server (including system software) is available for embedded for free. This means very complete driver support (see USB cell modem and printer examples above), robust file system support, memory management, etc. The breadth of options for Linux is astounding, but some may consider this a negative point, and would prefer a more integrated solution like Windows CE where everything comes from one place. There is a loss of flexibility, but in some cases, the tradeoff might be worth it. For an example of the number of packages that can be build for Embedded Linux systems using Openembedded, see http://cgit.openembedded.org/cgit.cgi/openembedded/tree/recipes</a>.</p> GUI TRENDS</h2> It is important to consider trends for embedded devices with small displays being driven by Cell Phones (iPhone, Palm Pre, etc). Standard GUI widgets that are common in desktop systems (dialog boxes, check boxes, pull down lists, etc) do not cut it for modern embedded systems. So, it will be important to consider support for 3D effects, and widget libraries designed to be used by touch screen devices. The Clutter library</a> is an example of this.</p> REMOTE SUPPORT</h2> Going back to the issue of debugging tools, most people stop at the scenario where the device is setting next to a workstation in the lab. But what about when you need to troubleshoot a device that is being beta-tested half-way around the world? That is where a command-line debugger like Gdb is an advantage, and not a disadvantage. And how do you connect to the device if you don’t have support for cell modems in New Zealand, or an efficient connection mechanism like ssh for shell access and transferring files?</p> SUMMARY</h2> Selecting any advanced technology is not a simple task, and is fairly difficult to do even with experience. So it is important to be asking the right questions, and looking at the decision from many angles. Hopefully this article can help in that. For additional assistance, please do not hesitate to contact BEC Systems</a> – we’re here to help.</p> Dealing with large data structures efficiently in embedded systems 2009-05-26T00:00:00+00:00 I’m currently dealing with a programming problem where I need access to several 64MB, file-backed data structures concurrently on an Embedded Linux system that only has 64MB of RAM. The data structures are fairly sparse (mostly zero data), and I typically only need to access small portions of them at any particular time. There is always the brute-force approach where you write code to manually load sections of the file as you need them. But with a little thought, the realisation hits “this is what operating systems do.” This article explores using memory mapping, and the sparse file</a> support in file systems to solve this problem in a very efficient manner.</p> Sparse File Support</h2> Most Unix file systems support sparse files</a>. This means that sections of data that is zeros is not stored on the disk. Consider the following example where we create a 200MB file that is all zeros:</p> root@cm-x270:/media/mmcblk0p1# df Filesystem 1k-blocks Used Available Use% Mounted on /dev/mmcblk0p1 3917212 202448 3515776 5% /media/mmcblk0p1 root@cm-x270:/media/mmcblk0p1# dd if=/dev/zero of=sparse-file bs=1 \ count=1 seek=200M root@cm-x270:/media/mmcblk0p1# df Filesystem 1k-blocks Used Available Use% Mounted on /dev/mmcblk0p1 3917212 202456 3515768 5% /media/mmcblk0p1 root@cm-x270:/media/mmcblk0p1# ls -l -rw-r--r-- 1 root root 209715201 May 22 08:54 sparse-file root@cm-x270:/media/mmcblk0p1# time od sparse-file 0000000 000000 000000 000000 000000 000000 000000 000000 000000 1440000000 real 0m 17.23s user 0m 14.21s sys 0m 2.66s root@cm-x270:/media/mmcblk0p1# du sparse-file 68 sparse-file </code></pre> It this case we created a 200MiB file on a SD card formatted as ext3. Even though the file is 200M in size, it only uses a few KiB of disk space. This same test also worked fine with a JFFS2 filesystem. With sparse file support, we get a cheap form of run length compression (at least for zeros) with no effort.</p> mmap()</h2> The mmap() system call is used to map a file, or portions of a file into memory. The data in the file can then be accessed directly in memory. Because Linux is a demand paged system, portions of the file are paged in as needed, so the entire file does not need to be present in RAM at one time. There are several advantages to using mmap():</p> mmap() avoids the extraneous copy that occurs when using read() or write() as the data does not need to be copied to a user space buffer.</li> there is very little overhead</li> you can directly access any part of the file without doing a lseek() and keeping track of where you are.</li> the operating system takes care of paging in sections of the file you are using, and discarding sections that are not in use when memory is low. This includes flushing dirty portions to disk.</li> </ol> What this means, is mmap gives you an easy way to work on large, file backed data structures, and the OS takes care of loading the portions you need, and saving the modified portions back to disk. As most embedded systems are 32-bit, there is a limit to the file size you can use as virtual memory space is limited.</p> Test Application</h2> Next I wrote a small application that is used to create, and modify files using mmap(). I wanted to experiment with creating files of various sizes, writing non-zero data at various intervals, and test the performance of this. The test application source is located here</a>.</p> Usage: sparse_file_mmap_test [OPTION] -s, --filesize File size to allocate -o, --offset Will write a few bytes every offset bytes -m, --modify Modify existing file -d, --data Data to write to file at offset location (0-255) </code></pre> The application creates a file of size filesize</em>, and writes the value of data</em> to the file every offset</em> bytes. There is also an option to modify existing files, so we can test the performance of opening a large file, making a small change, and then closing it.</p> Test Results</h2> The results are pretty amazing, and the performance is beyond my expectations. This type of problem is where you learn to appreciate the performance of an advanced operating system, and file system. There is a reason for the complexity! There are 3 things I wanted to test: 1) does the sparse file support work as expected? 2) can mmap be used to easily modify large files? 3) can mmap be used to work on data structures that are larger than physical RAM?</p> 1. Sparse File Support</h2> The basic tests above confirm that sparse file support works for an empty file, but what about a file that has some data every X bytes? Below are the test results for creating a 10MB file, and writing data at various offset intervals.</p> Offset (bytes)</strong></td> File Size (KB)</strong></td> </tr> 1024</td> 9784</td> </tr> 2048</td> 9784</td> </tr> 4096</td> 9784</td> </tr> 8192</td> 4904</td> </tr> 16384</td> 2464</td> </tr> 32768</td> 1244</td> </tr> 65536</td> 632</td> </tr> 1048576</td> 60</td> </tr> 2097152</td> 40</td> </tr> 4194304</td> 32</td> </tr> </tbody></table> As soon as the offset was greater than the MMU page size (4KB), then the sparse file effect started to kick in and worked as expected.</p> 2. Can mmap() be used to efficiently modify large files?</h2> The test here was to open a large file, make a small change in the middle, and then close it. In this case a 100MB file was created with a data value of 1 written every 1MB, and then re-opened the same file and wrote a data value of 2 every 2MB. The od utility was used to examine the file to verify the contents were correct.</p> root@cm-x270:/media/mmcblk0p1# time sparse_file_mmap_test_arm -s104857600 \ -o1048576 -d1 Sparse File mmap() test Filesize = 104857600, offset = 1048576, data = 1 size = 102400 KB size on disk (KB): 508 sparse-file real 0m 1.10s user 0m 0.00s sys 0m 0.43s root@cm-x270:/media/mmcblk0p1# time sparse_file_mmap_test_arm -s104857600 \ -o2097152 -d2 -m Sparse File mmap() test Filesize = 104857600, offset = 2097152, data = 2 size = 102400 KB size on disk (KB): 508 sparse-file real 0m 0.37s user 0m 0.01s sys 0m 0.05s root@cm-x270:/media/mmcblk0p1# time od -x sparse-file 0000000 0002 0000 0000 0000 0000 0000 0000 0000 0000020 0000 0000 0000 0000 0000 0000 0000 0000 * 4000000 0001 0000 0000 0000 0000 0000 0000 0000 4000020 0000 0000 0000 0000 0000 0000 0000 0000 .... * 614000000 0001 0000 0000 0000 0000 0000 0000 0000 614000020 0000 0000 0000 0000 0000 0000 0000 0000 * 620000000 real 0m 12.93s user 0m 7.33s sys 0m 1.70s </code></pre> Creating and modifying large file-backed data structures is very fast and efficient, and takes on the order of 1 second for a 100MB file that contains data every 1MB. Dumping the data with od took considerably longer (13 seconds) as 100MB of data needed to be processed.</p> Possible Issues</h2> There are several things to watch out for when using sparse files and mmap()</p> With sparse files, there is the potential to run out of disk space as the files are using less space on disk than the file size. It is a good idea to monitor disk space when working with sparse files, so you don’t use up all of the disk space.</li> mmap() requires virtual memory space for the size file it maps. With embedded systems, this is less of an issue, because the physical RAM size tends to be much less than the 4GB virtual address space. With a system that only has 64MB of RAM, mmap()’ing files in the 10’s of MB in size makes a lot of sense because it insures that you will not run the system out of memory, and yet there should be plenty of virtual address space to map in files of this size.</li> </ol> Conclusion</h2> mmap() and sparse file support provide a very convenient solution for dealing with large, file-backed data structures. One example of this type of data structure is any type of large matrix such as maps used in mapping applications. Writing a “from scratch” solution to solve this problem would be a fairly large and difficult task. Processing large amounts of data efficiently is becoming more and more important in many embedded systems. This example provides another compelling reason why implementing modern, data-centric embedded systems using Linux makes a lot of sense.</p> MontaVista Linux 6 is based on OpenEmbedded Technologies 2009-05-13T00:00:00+00:00 Very interesting news from MontaVista yesterday as they announced MontaVista Linux 6</a>. It turns out they are using bitbake, which is the core of the OpenEmbedded</a> build system. Along with the adoption of OpenEmbedded in many leading Embedded Linux efforts (Beagleboard, Gumstix, Bug Labs, etc), this is a resounding affirmation that the OpenEmbedded way of building distributions is worthy of consideration. It is a well known fact that OpenEmbedded has some deficiencies: a steep learning curve, the testing/quality efforts could be improved, and the organisational aspects of the project are somewhat lacking. That said, OpenEmbedded is still the most effective tool for building Embedded Linux distributions that I’ve found yet, and is a very viable solution if you are willing to spend some time to learn it, or hire a consultant to get your project set up and work through the rough spots. MontaVista is attempting to address these difficulties with their MVL6 commercial offering.</p> Below are few a notes from watching a few videos on the MV web site:</p> Discussion with Jim Ready</a></p> around 2006 things changed</li> Semiconductor vendors now porting Linux to all their CPUs, so MV has moved up the stack</li> MVL6 first embedded linux distro produced in post-victory phase (Linux has won)</li> very friendly with other open source pieces (I assume OpenEmbedded)</li> enable open source</li> The decision the use Linux is the right decision technically. What is the right business decision to accompany the technical decision?</li> </ul> </li> MVL6 Demonstration</a></p> toolchains pre-packaged</li> sources are all downloaded from MV servers</li> </ul> </li> Interview with Joe Green</a></p> manager of developer tools team</p> </li> MLV6 is new approach</p> </li> previous editions were consistent for all targets</p> </li> MLV6 more flexible model. Market specific distros depending on market.</p> </li> source driven product. Previous products based on binary RPMs. MLV6 is a lot easier to build the whole environment. Very customized/fine-tuned distro.</p> </li> key problems MLV6 solves</p> Complete starting point for target hardware</li> complete build system, so you have total control vs pre-canned distribution that sets all the rules</li> </ul> </li> Integration platform based on Bitbake. Working closely with community. Enhanced in a number of ways. Customers have the option of bringing in software developed by the OpenEmbedded community and bringing it into their project.</p> </li> Gives you a system that you can configure at almost any level.</p> </li> Enhancements to make it easy to make changes, as well as provide a pre-built starting point.</p> </li> Features to control sources, and environment so the host environment does not contaminate your build.</p> </li> Essentially hardening, and commercializing the bitbake and openembedded projects.</p> </li> Why MVL6 vs roll-your-own.</p> Qualified starting point</li> Support. Community or Semi-conductor vendor support is hit or miss.</li> Consistent environment accross multiple systems.</li> Includes devrocket graphical tools</li> </ul> </li> </ul> </li> </ul> Overall, this seems like a very good approach. I’m glad they are not re-inventing the wheel, but instead have chosen to build on top of a proven open source solution, and leverage the work that is being done by a very active community of OpenEmbedded developers. The OpenEmbedded project currently has 61 developers (not all are active), and weekly contribution rate is very good. This type of effort would be very difficult to match by any but the largest commercial organizations. This should be a win-win situation for all involved. As a common base technology, the Bitbake and OpenEmbedded projects should improve due to MV’s involvement. Through the MVL6 product, the OpenEmbedded project and way of doing things will get a lot more exposure and there will be a lot more developers who understand how it works. Developers who are familiar with the OpenEmbedded build system will be able to apply their knowlege to a greater number of projects. And developers building products will have more options for getting the support and techology they need.</p> How to set up a NFS root filesystem for embedded Linux development 2009-05-11T00:00:00+00:00 Although ssh and friends work really well for embedded systems, occasionally you want to set up a NFS root for development. One of the scenarios where nfsroot is useful is if you are making a lot of rootfs changes, and you don’t want to spend the time to reprogram the flash on the target system. Fortunately, the Linux kernel includes complete support for NFS root, and does not require any userspace changes. This setup assumes the following:</p> the target system has Ethernet support built into the kernel</li> the target system is on the same network as a Linux workstation</li> said network includes a DHCP server</li> </ol> This configuration emphasises “simple” and does not require you to spend 4 hours trying to get a bootp server configured on a test network.</p> Kernel Support</h2> Make sure you have the following options turned on in the kernel:</p> CONFIG_IP_PNP_DHCP=y</li> CONFIG_ROOT_NFS=y</li> </ul> Then, add the following to your kernel CMDLINE:</p> ip=dhcp root=/dev/nfs nfsroot=<nfs server IP>:/path/to/nfsroot </code></pre> </li> </ul> Workstation Setup</h2> The following setup is for Ubuntu.</p> apt-get install nfs-user-server</p> (edit /etc/exports to contain something like the following)</p> /path/to/nfsroot 192.168.1.0/255.255.255.0(no_root_squash,sync,rw)</p> /etc/init.d/nfs-user-server restart</p> If you are using OpenEmbedded, then instruct OE to generate a rootfs tar image, and extract to your nfsroot directory:</p> cd /path/to/nfsroot sudo tar -xvf <path to OE images dir>/Angstrom-image...rootfs.tar </code></pre> And the result</h2> eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1 Sending DHCP requests ., OK IP-Config: Got DHCP answer from 192.168.1.1, my address is 192.168.1.112 IP-Config: Complete: device=eth0, addr=192.168.1.112, mask=255.255.255.0, gw=192.168.1.1, host=192.168.1.112, domain=hq.bec-systems.com, nis-domain=(none), bootserver=192.168.1.1, rootserver=192.168.1.11, rootpath= Looking up port of RPC 100003/2 on 192.168.1.11 Looking up port of RPC 100005/1 on 192.168.1.11 VFS: Mounted root (nfs filesystem) readonly. Freeing unused kernel memory: 220k freed Write protecting the kernel text: 2792k Write protecting the kernel read-only data: 788k INIT: version 2.86 booting </code></pre> Compulab cm-x270 kernel updated to 2.6.29 in OE 2009-04-24T00:00:00+00:00 The cm-x270 kernel support in OpenEmbedded has just been updated</a> to version 2.6.29.</p> Wi2Wi W2CBW003 Wifi/Bluetooth module review 2009-03-19T00:00:00+00:00 The Wi2Wi W2CBW003</a> is a highly integrated module that provides both Wifi and Bluetooth radios for embedded designs. This module is ideal for embedded designs, as it provides a lot of functionality in a small package and includes standard interfaces like SPI, SDIO and serial that connect with most embedded CPUs. With the availability of modules like the W2CBW003 and standard drivers in the Linux kernel, including radio functionality in an embedded device is very doable, even for low volume products. Wi2Wi provides an evaluation board for the W2CBW003 with a SDIO connector, UART connector, and BT Audio Connectors. For this review, the eval board was connected to a Marvel PXA270 ARM processor, and evaluated with current Linux and associated software.</p> W2CBW003 Overview</h2> The W2CBW003 module integrates both WiFi and Bluetooth functionality in a 12mm x 12mm x 1.6mm package. The WiFi portion is based on the Marvell 88W8686, and the Bluetooth on the CSR BC04. Both of these components are well supported by Open Source software. Some other features include:</p> Separate Antennas for WiFi and BT.</li> ROHS compliant</li> Single Supply at 3.3V</li> 802.11g support (54Mbps)</li> Both SPI and SDIO interfaces for WiFi</li> UART interface for BT</li> PCM audio interface for BT</li> </ul> Pictures of the module and the demo board are shown below.</p> </p> </p> Evaluation System Configuration</h2> The test software included with the Wi2Wi eval board is for a Windows PC, is provided in binary format only, and was not used in this review. For this review, I plugged the W2CBW003 demo board into a Compulab cm-x270 board (PXA270 cpu) running a 2.6.29-rc7 Linux kernel and a recent OpenEmbedded Angstrom distribution. With the exception of the Marvell Wifi Firmware, all software in this setup is Open Source and is available in the Linux kernel, and as packages in the OpenEmbedded Project.</p> When booting the kernel, you will see the following messages:</p> mmc0: new SDIO card at address 0001 libertas_sdio mmc0:0001:1: firmware: requesting sd8686_helper.bin libertas_sdio mmc0:0001:1: firmware: requesting sd8686.bin libertas: 00:19:88:06:0b:2e, fw 9.70.3p25, cap 0x00000303 eth2 (libertas_sdio): not using net_device_ops yet libertas: PREP_CMD: command 0x00a3 failed: 2 libertas: PREP_CMD: command 0x00a3 failed: 2 libertas: eth2: Marvell WLAN 802.11 adapter </code></pre> The “command 0x00a3 failed” messages are harmless, and have to do with features that are not supported. After the system boots, you will now see a new ethX network device:</p> root@cm-x270:~# ifconfig -a ... eth2 Link encap:Ethernet HWaddr 00:19:88:06:0B:2E BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:65902 errors:0 dropped:0 overruns:0 frame:0 TX packets:1758 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:13550002 (12.9 MiB) TX bytes:251627 (245.7 KiB) </code></pre> The “iwlist eth2 scanning</em>” command will list available access points.</p> Connecting to Open WiFi Networks</h2> A connection to an open wifi network can be accomplished by placing the following in /etc/network/interfaces:</p> /etc/network/interfaces: iface eth1 inet dhcp wireless_mode managed wireless_essid any </code></pre> And now execute “ifup eth2”:</p> root@cm-x270:~# ifup eth2 udhcpc (v1.13.2) started run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 Sending discover... Sending select for 192.168.1.115... Lease of 192.168.1.115 obtained, lease time 86400 run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 adding dns 208.67.222.222 adding dns 208.67.220.220 root@cm-x270:~# iwlist eth2 iwlist: unknown command `eth2' (check 'iwlist --help'). root@cm-x270:~# iwconfig eth2 eth2 IEEE 802.11b/g ESSID:"bec3" Mode:Managed Frequency:2.437 GHz Access Point: 00:18:39:C1:AD:4A Bit Rate:1 Mb/s Tx-Power=13 dBm Retry short limit:8 RTS thr=2347 B Fragment thr=2346 B Encryption key:off Power Management:off Link Quality=84/100 Signal level=-37 dBm Noise level=-87 dBm Rx invalid nwid:0 Rx invalid crypt:14707457 Rx invalid frag:0 Tx excessive retries:58 Invalid misc:3 Missed beacon:0 </code></pre> WPA Secured WiFi Networks</h2> The OpenEmbedded console image includes the WPA Supplicant packages which is used to manage wireless connections to secured networks. To set up the system for WPA encryption, modify the following files:</p> /etc/network/interfaces: iface eth2 inet dhcp wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf wpa-driver wext </code></pre> /etc/wpa_supplicant/wpa_supplicant.conf: ctrl_interface=/var/run/wpa_supplicant ctrl_interface_group=0 eapol_version=1 ap_scan=1 fast_reauth=1 network={ ssid="bec" proto=WPA2 key_mgmt=WPA-PSK pairwise=CCMP TKIP group=CCMP TKIP scan_ssid=1 psk="ascii passphrase" priority=10 } </code></pre> Then, execute “ifup eth2”, and you should see something like:</p> root@cm-x270:~# ifup eth2 WPA: Configuring Interface udhcpc (v1.13.2) started run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 Sending discover... Sending select for 192.168.1.115... Lease of 192.168.1.115 obtained, lease time 86400 run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 adding dns 208.67.222.222 adding dns 208.67.220.220 </code></pre> root@cm-x270:~# iwconfig eth2 eth2 IEEE 802.11b/g ESSID:"bec" Mode:Managed Frequency:2.412 GHz Access Point: 00:40:10:10:00:03 Bit Rate:1 Mb/s Tx-Power=13 dBm Retry short limit:8 RTS thr=2347 B Fragment thr=2346 B Encryption key:<too big> Security mode:open Power Management:off Link Quality=64/100 Signal level=-68 dBm Noise level=-89 dBm Rx invalid nwid:0 Rx invalid crypt:-1463809279 Rx invalid frag:0 Tx excessive retries:22524 Invalid misc:3 Missed beacon:0 </code></pre> Other Observations</h2> With the above networks, the bec</em> access point was much further away than the bec3</em>AP, so you will notice the difference in link quality. “iwlist eth2 rate</em>” can be used to list the current connection rate. When the network is idle, it sits at 1Mb/s. When downloading a large file, it will climb to 36 or 54Mb/s, depending on link quality.</p> Production Issues</h2> The review demonstrates that it is fairly simple to set up a demo quality Embedded Linux system with WiFi. Some of the issues that would likely need addressed for a production system include link management, test software for certification, and power management.</p> There are several ways to programatically control WPA Supplicant including linking to the wpa supplicant control interface, or using DBus. There are several WiFi management applications available including Gnome NetworkManager</a> (used in desktop systems), and connman</a> which seems a little better suited for embedded systems.</p> Unless you use a completely pre-certified module + antenna solution, you will likely need to do some level of agency certification. As many products use a custom antenna, this is an important issue to consider and plan for. While Marvell provides test software and firmware, it will likely require some work, as their software is designed to be used with their in-house drivers rather than the libertas driver which is available with modern kernels.</p> Also, if you want to minimize the power usage, some work will be required to figure out the power modes supported, and how to implement/control them. With this module, the Wifi and BT portions run off the same crystal, so if you only want the BT active, you will need to actively power manage the Wifi portion to a low power state instead of completely disabling it.</p> Conclusion</h2> TheW2CBW003 module is an attractive solution for products that need WiFi functionality. With the availability of modules like this, and mainstream open source software, the technology is available to about anyone, including low volume manufacturers. Standard interfaces such as SDIO make it possible to interface this module with about any modern ARM processor that can run Linux. Software support in the Linux kernel, wpa supplicant, and the Linux wireless tools provide the needed software support to implement a very complex system with relatively little effort.</p> Memory Performance on various Embedded Systems 2009-03-04T00:00:00+00:00 Marcin just published an interesting article</a> about memory performance on various embedded systems using the hdparm -T as a simple benchmarq. This test gives a pretty good indicator of memory performance in the system. From the hdparm man page:</p> Perform timings of cache reads for benchmark and comparison purposes. For meaningful results, this operation should be repeated 2-3 times on an otherwise inactive system (no other active processes) with at least a couple of megabytes of free memory. This displays the speed of reading directly from the Linux buffer cache without disk access. This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.</em></p> A few results I find interesting:</p> modern desktop systems have an order of magnitude more memory bandwidth than ARM systems.</li> the i.MX31 is the highest performing ARM device tested</li> the i.MX31 performs better than the OMAP3 in this test – why is this? As the ratio is 2, I’m guessing the bus is twice as wide?</li> </ul> Who is using OpenEmbedded? 2009-02-20T00:00:00+00:00 Having used OpenEmbedded for a number of Embedded Linux projects over the past 5 years, it is interesting watching new users come on board. The most recent announcement came from Koan</a> that the KaeilOS</a> distribution will be joining the OpenEmbedded project. In the announcement, they summed up my thoughts well:</p> We at KOAN wanted to avoid waste of energy converging KOAN efforts into a stable and solid Open Source project like OE is.</em></p> “Avoid waste of energy” – isn’t this what OSS is all about? Life is way too short to re-invent every wheel. Considering OpenEmbedded has dozens of active contributors, it would be very difficult for any one company to replicate the functionality of OpenEmbedded. Although there are many ways to build embedded Linux distributions, and some tools do some things better than OE, there still does not seem to be any other embedded Linux build tool that fulfills the following requirements as well as OE:</p> build from scratch on any x86 Linux system</li> large number of contributors</li> large number of packages</li> support a large number of architectures</li> Can modify images and integrate custom applications in a consistent way as part of the build system, and not a post processing hack.</li> Open Source</li> </ol> Some other companies using OpenEmbedded include:</p> TI</li> AMD</li> Intel (was OpenedHand)</li> Gumstix</li> Bug Labs</li> OpenMoko</li> Atmel</li> Techsol</li> Compulab</li> Toradex</li> 4G-Systems</li> Dream Multimedia TV</li> M&N Solutions</li> emQbit</li> Siemens</li> Embedded Alley</li> and the list goes on</a> …. (If you are using OpenEmbedded, please add your name to this list</a>.)</li> </ul> I’m looking forward to seeing the KaeilOS improvements such as mdev, and boot time improvements.</p> Sprint 598U USB Broadband Modem in Embedded Systems 2009-02-07T00:00:00+00:00 One thing that works really well in Linux is support for USB Broadband modems. While it usually takes a little fiddling with PPP scripts, it is generally not too difficult to get working. One of the reasons for this is most modems implement a USB serial interface, and then the modem is controlled with standard AT commands. A customer who currently supports a Verizon USB modem</a> in their product wanted to add support for a Sprint modem. After a little research, we settled on the 598U from Sierra Wireless</a>. Below is a picture of the 598U plugged into an industrial terminal.</p> </p> Why are USB Broadband modems useful in Embedded Systems?</h2> As systems become more complex, and time to market is shrinking (thus less testing), it is becoming increasingly important to be able to remotely diagnose problems. With a broadband modem like the Sprint 598U, developers can remotely log into a device using SSH, and debug problems. This is a huge benefit. The cost of the modem and a 1 year data plan becomes fairly insignificant when you consider the cost to fly a developer on-site to debug issues. While there are other ways to remotely access an embedded system, such as reverse ssh</a>, the simplicity of the broadband modem is attractive as it bypasses the firewall issues. Even with reverse ssh, I’ve run into paranoid sys admins who block the outgoing port 22 on their networks, which makes reverse ssh difficult. Once you have the IP address of the broadband connection, you can connect to it directly with ssh from anywhere in the world. I’ve accessed systems that were being field tested in New Zealand from the United States, and it worked great.</p> Increasingly, the value of complex embedded systems is shifting from control to data. While the control aspect is as important as it ever was, access to data is what is giving many products and businesses the competitive edge. Being able to collect data and visualize how the system is performing is becoming increasingly important to maximize performance. And sometimes this data must be collected in real-time. Scenarios where USB broadband modems might make sense:</p> remote sites that do not have internet connectivity</li> mobile applications such as vehicle terminals</li> temporary installations where the system is only located in one place for a limited time</li> situations where it is very difficult to get administrators of the local network to cooperate in adding the embedded system to their network</li> </ul> Finding PPP Config information</h2> I usually develop as much as I can on x86 systems (vs. an embedded target) as it is a lot more convenient, so I plugged the modem into a Ubuntu 8.10 laptop. While Ubuntu 8.10 recognizes and configures a UM150 Verizon modem, it does not automatically configure the 598U. Typically, the next step is to google around for information on various support forums on how to set up the PPP config files, as most manufactures don’t provide support for Linux. As I could not find any existing information for the 598U, I decided to see if Sierra Wireless published any information. Was I ever suprised when I found this page</a>. Sierra Wireless provides fairly comprehensive information on using the modem under Linux, and their PPP scripts actually worked – great job! But in the end, when I plugged the 598U into the embedded device that already supported the Verizon modem, it just worked with my existing PPP scripts. This is another indication of the value of using common code and interfaces in Linux.</p> The Linux Advantage</h2> When you run Linux in an embedded system, using a USB broadband modem becomes fairly simple. It is actually a little bit deceiving as to what all has to happen. Different modems use different chipsets, and thus different drivers. While they all eventually expose a USB serial interface, there are enough differences to require separate drivers. As an example, the 598U uses the “sierra” USB-serial driver. This driver is 782 lines, plus it utilizes a common USB serial framework that is many times this size. On the application side, the pppd framework is used to manage the modem and the network connection. Starting the connection is as simple as running “pppd call cdma” from your application. The pppd application manages getting an IP address, setting up nameservers in resolve.conf</em>, etc. Because all broadband modems are supported by a common application interface and very similiar drivers, supporting multiple modem vendors is actually quite easy. Contrast this to the Windows driver scenario where every vendor provides their “own” driver, and their “own” application user interface, and their “own” way to initiate an internet connection, and it is not obvious to me how you could easily control all this in a consistent, automated fashion in an embedded device. The way we set it up is you plug the modem in, and a few seconds later the system is automatically connected to the internet. There is nothing for the user to configure, initiate, etc. Printing from Linux in embedded applications</a> is similiar.</p> This is another example of how it is possible to leverage the work of many to add advanced functionality and features to a device with very little development cost. Getting more done with less should be an attractive (and perhaps critical) way to operate for any company seeking to remain competitive.</p> Socketcan utils and test apps added to OpenEmbedded 2009-01-28T00:00:00+00:00 The Socketcan test applications and utilities are now available in OpenEmbedded</a>. The socketcan kernel modules already exist in OE</a>. To build and deploy:</p> get the latest version of OE metadata</li> bitbake socketcan-utils-test</li> scp /build/angstrom-2008.1/tmp/deploy/glibc/ipk//socketcan-utils-test_0.0+svnr917-r0_armv5te.ipk root@:</li> </ul> And then on the target system:</p> opkg install socketcan-utils-test_0.0+svnr917-r0_armv5te.ipk</li> opkg files socketcan-utils-test</li> </ul> /usr/bin/tst_bcm_rx_sendto /usr/bin/tst_bcm_tx_sendto /usr/bin/canlogserver /usr/bin/tst_raw_filter /usr/bin/tst_raw /usr/bin/isotpsniffer /usr/bin/isotpsend /usr/bin/tst_bcm_throttle /usr/bin/tst_packet /usr/bin/vcan /usr/bin/isotprecv /usr/bin/cansniffer /usr/bin/tst_bcm_tx_read /usr/bin/tst_err /usr/bin/log2asc /usr/bin/tst_bcm_single /usr/bin/tst_filter_master /usr/bin/canplayer /usr/bin/cangen /usr/bin/slcanpty /usr/bin/log2long /usr/bin/tst_bcm_rtr /usr/bin/asc2log /usr/bin/slcan_attach /usr/bin/tst_bcm_filter /usr/bin/cansend /usr/bin/canecho /usr/bin/tst_filter_server /usr/bin/tst_bcm_cycle /usr/bin/tst_bcm_dump /usr/bin/isotpdump /usr/bin/candump /usr/bin/tst_raw_sendto /usr/bin/isotptun /usr/bin/tst_bcm_server /usr/bin/tst_proc /usr/bin/canbusload </code></pre> An impressive array of utilities!</p> Socketcan CAN-bus drivers added to OpenEmbedded 2009-01-26T00:00:00+00:00 I just added a recipe</a> to OpenEmbedded to build the Socketcan</a> kernel modules from the socketcan SVN. So if you are using the latest OpenEmbedded metadata, you can:</p> bitbake socketcan-modules</li> scp /build/angstrom-2008.1/tmp/deploy/glibc/ipk//socketcan-modules_0.0+svnr917-r0_cm-x270.ipk root@:</li> </ul> and then on the target system:</p> opkg install socketcan-modules_0.0+svnr917-r0_cm-x270.ipk</li> opkg files socketcan-modules</li> </ul> Package socketcan-modules (0.0+svnr917-r0) is installed on root and has the following files: /lib/modules/2.6.27/extra/net/can/can-bcm.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/ems_pci.ko /lib/modules/2.6.27/extra/net/can/can-raw.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/sja1000.ko /lib/modules/2.6.27/extra/drivers/net/can/softing/softing.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/ems_pcmcia.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/sja1000_platform.ko /lib/modules/2.6.27/extra/drivers/net/can/can-dev.ko /lib/modules/2.6.27/extra/drivers/net/can/softing/softing_cs.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/pipcan.ko /lib/modules/2.6.27/extra/drivers/net/can/mcp251x.ko /lib/modules/2.6.27/extra/net/can/can.ko /lib/modules/2.6.27/extra/net/can/can-isotp.ko /lib/modules/2.6.27/extra/drivers/net/can/vcan.ko </code></pre> Then to use the modules:</p> depmod</li> modprobe mcp251x</li> modprobe can-bcm</li> </ul> What is CAN? Can stands for Controller-area network and is popular in industrial and automotive applications. A convenient way to add CAN to your Embedded Linux system is with the Microchip MCP2515</a>. This device connects to a SPI bus which means it can be interfaced with a number of popular SOC’s such as the PXA270, OMAP3, 91SAM9xxx, etc.</p> More on Socketcan later …</p> BTW, this recipe really illustrates how easy it is to compile kernel modules outside a kernel in OpenEmbedded:</p> DESCRIPTION = "Socketcan kernel modules" HOMEPAGE = "http://developer.berlios.de/projects/socketcan/" SECTION = "kernel/modules" LICENSE = "GPL" DEPENDS = "virtual/kernel" PV = "0.0+svnr${SRCREV}" SRC_URI = "svn://svn.berlios.de/socketcan/trunk;module=kernel;proto=svn" S = "${WORKDIR}/kernel/2.6" inherit module </code></pre> This recipe tells OE to download the source, cross-compile it against your target kernel build dir, and then package it for easy install on the device. This is the way things should work – no messing around figuring out make options, kernel source paths, compiler env variables, etc.</p> Intel Atom vs TI OMAP3 2009-01-22T00:00:00+00:00 As we look at new projects, both the Intel Atom</a> and the TI OMAP3</a> processors generate considerable interest. As we have already shown</a>, the OMAP3 does offer a considerable performance improvement over earlier generations of ARM CPUs. The following video</a> I found on YouTube shows a similar comparison of a OMAP3 and Atom systems rendering web pages:</p> As one would expect, the Atom does perform better (about 14%), but considering the power differences, the OMAP does surprisingly well. It is also unknown in this demo if the screen size would make a significant difference in the results. Like most things, the choice depends on the application, and no two applications are the same, and each solution has advantages. Some things to think about:</p> Power: OMAP3 platform consumes on the order of <1-2W while the Atom is more in the range of 2-5W.</p> </li> High Speed I/O Interfaces: Atom supports PCI and PCI expansion interfaces where OMAP3 is limited to more special purpose user interfaces such as SD, Camera, Asynchronous bus, etc. Both Atom and OMAP support High Speed USB.</p> </li> Packaging: OMAP3 packaging is very aggressive with the stacked Package-on-Package. To get an idea how much space an OMAP3 solution takes, check out the module from Gumstix</a>. There are basically only two chips in the system: the OMAP3+stacked RAM/Flash and a power management+I/O chip. This is very high integration!</p> </li> Module availability: for many embedded systems with volumes in the 1000’s of units per year, a module solution is very attractive compared to a full custom design. This drastically reduces the engineering effort and time to market. A sampling of the modules available include:</p> OMAP3:</p> Gumstix Overo</a> ($117 for 3000 units)</li> LogicPD OMAP35x SOM-LV</a> (Cost unknown)</li> </ul> </li> Atom:</p> Eurotech Catalyst</a> (Cost unknown)</li> </ul> </li> </ul> </li> Software support: TI and the open source community have done a remarkable job of supporting the OMAP3 with the BeagleBoard</a> effort. Gumstix maintains open source software for their devices, and has a very active development community. Intel also has invested significantly in software with their Moblin</a> project. Other factors to consider is the boot software (bootloader vs BIOS, is it open?), are there 3D graphics libraries available, etc.</p> </li> Multimedia processing: The OMAP3 is available with an on-chip DSP. Intel has traditionally offered extensions for multimedia processing such as SIMD.</p> </li> </ol> Tradeoffs!</p> GTK performance on PXA270 vs. OMAP3 2009-01-21T00:00:00+00:00 Several of my customers have built applications using the GTK+</a> tookit. While GTK+ works fairly well for what we have done, I have been wondering how the performance compares on the new Omap3</a> processors from TI. As we are evaluating the OMAP3 for several projects, I did a simple comparison with an existing application. Below is a video</a> that shows a fairly complex application running on both a PXA270, and a OMAP3530. While the PXA270 gets the job done, the result on the OMAP3 is much more pleasing. With that advent of a OMAP3 module available for $117 in volume</a>, it seems like the OMAP3 will be a popular solution for upcoming Embedded Linux projects.</p> Fix for PXA270 MMC/SD Controller Write Corruption 2009-01-17T00:00:00+00:00 During system verification for a customer, we noticed occasional failures when writing to a SD card using the Marvell PXA270 MMC/SD controller. The failure is a 4KB block of data is shifted on byte, where the first byte is duplicated and the last byte is dropped. The test app that found this problem is available here</a>. This test app simulates an application that writes data to a SD card at a certain rate. We found that typically in about 2000 - 10000 hours of simulated testing, we would see a failure. Its very interesting that the field failure rates matched this rate as well.</p> This failure had all the signs of a hardware (likely DMA) problem, and sure enough there is a Marvell Eratta for this issue. A patch has been developed and tested by several people and is available in this mail thread</a>.</p> Installing Ubuntu on a Core i7 DX58SO motherboard 2009-01-15T00:00:00+00:00 UPDATE: 2009-03-17 – It appears that hard drive issues I encountered are likely due to the HW RAID formatting on the drive I tried. I just tried a brand new hard drive, and Ubuntu 8.10 installed flawlessly, and everything works.</strong></p> I’m not sure why computer upgrades are always such an epic struggle for me, but here I go again …</p> The goal is to upgrade my aging AMD X2 system to a new Intel Core i7 system. So after researching it a bit, I decided on the Intel DX58SO</a> motherboard. I liked the architecture, and have had good luck with Intel motherboards. And since I don’t upgrade all that often, I try to get something good when I do. As I learned later, I should have researched Linux compatibility a little more …</p> The plan was to simply replace the CPU/MB/RAM in my existing box as I already have a raid controller, decent case, etc. However, the first challenge was the existing PS does not supply a EPS/ATX12V8 pin connector – it only had the 4-pin version. So, I got to upgrade my PS to a Corsair CMPSU-650TX</a> which seems very nice, and quiet.</p> After I put everything together, the raid array (3ware 9500S-4LP PCI) was detected, but the system would not boot. So I booted from a Ubuntu install disk and selected boot from 1st hard disk, and then it seemed to boot fine. However, the Ethernet controller and CDROM don’t work under Ubuntu 8.04 (64-bit) which was currently installed on the raid array. However, everything else works pretty well, and it is quite fast. Because the core-i7 brought back hyperthreading, Linux now shows 8 cores.</p> Several days later in an attempt to get the system booting from the hard drive, I upgraded the BIOS to the latest version. However, this broke booting from the raid array; even with a Ubuntu install disk.</p> At this point, I decided to just rebuild the system by re-configuring the 4 hard drives currently in a hardware raid-5 array to two raid-1 software raid arrays. As the raid card is only PCI, I think this would provide better performance and be simpler to maintain in the future. So, I copy all my data off to a USB HD, and then try to install Ubuntu to a HD attached to the motherboard. To my suprise, I can’t get MB attached hard drives to work. Lots of messages like:</p> [ 361.943329] ata1.01: status: { DRDY ERR } [ 361.943332] ata1.01: error: { ABRT } [ 362.043760] ata1.01: configured for UDMA/133 [ 362.043771] ata1: EH complete [ 362.218194] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 [ 362.218200] ata1.01: BMDMA stat 0x45 [ 362.218207] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in [ 362.218209] res 51/04:08:00:00:00/04:00:1d:00:00/f0 Emask 0x1 (device error) </code></pre> I tried booting a 32-bit Ubuntu 8.10 live CD and I get the same thing when trying to fdisk a hard drive attached to the MB. So for now, the only way to get this running is to go back to the old RAID controller. Fortunately, Intel keeps old BIOS versions on their web site and after two tries, I was able to find one that works again with the HW RAID controller.</p> The Ethernet and CD ROM did work with the Ubuntu 8.10 32-bit live CD, so once I upgrade to 8.10, hopefully that will all work. So for now, I’ll limp along with my raid controller and a USB->Ethernet adapter for networking.</p> Anyone else have success with Linux on the DX58SO?</p> I’ve filed a bug on launchpad</a>.</p> Switching to Wordpress 2009-01-13T00:00:00+00:00 After using Joomla for several years, I’m switching my web site to wordpress. As I mostly use my site for writing technical articles, it makes sense to use something that works very well for just that. So far, I’m finding Wordpress very polished, and it has all the functionality I need built-in, so I don’t have to rely on 3rd party modules for very basic operations like tagging, RSS feeds, comments, etc. Now I have no excuse for now writing more as Wordpress works so well.</p> Mono 2.2 RC1 added to Openembedded 2009-01-10T00:00:00+00:00 I recently addedMono 2.2 RC1 to OpenEmbedded</a>. The last usable version of Mono in Openembedded was 1.2.6, so this is a big step up. Lots of interesting things have been happening since then (like Full Static Compilation</a>) that continue to make Mono interesting for embedded. There is still plenty to do yet such as clean up the packaging, and update the other related recipes such as mono-xsp, and gtk-sharp.</p> Using the Vala Programming Language in Embedded Systems 2009-01-10T00:00:00+00:00 Recently I’ve been following the Vala programming language</a> and using it some in embedded systems. Vala is a new programming language that aims to bring modern programming features to GNOME developers without imposing additional runtime requirements and without using a different ABI compared to applications and libraries written in C. A few notes and observations about Vala:</p> language syntax that resembles C#, so you can write code faster with less mistakes</li> Vala compiles to C, so it starts fast and runs fast like native applications</li> binaries are not platform independent (like Mono or Java)</li> lots of bindings already exist, as it is very easy to write bindings to existing C libraries</li> easy to write programs that are mixed C and Vala</li> documentation is still a bit sparse, so you end up reading the binding files to figure out how to use the libraries</li> </ul> Much of the core Vala functionality is built on top of Glib. Having programmed extensively with Glib, I can say programming in Vala is a lot more fun, and a lot less tedious. Vala also provides dynamic D-Bus bindings which makes if very nice for writing system daemons or other bits of software that need to implement a D-Bus server.</p> As far as real world experience, in a recent application, we ported a system monitoring application from C# to Vala. There were no real serious problems with the original application, but it started slow (7 seconds or so), and the customer wanted to reduce the boot time of the system. Once you get past the core language syntax, the libraries for Vala are all different than C#, so most of the library function calls needed tweaked a little. Most of the functionality was implemented with Vala, but there were a few minor functions that were implemented in C. It now starts fast, which is what we needed.</p> We are also using the fsod</a> application from the OpenMoko project in a customer project. This is an excellent example of a well written, advanced Vala project that uses features such as plugins and D-Bus.</p> Watching the releases from the Vala project, it is amazing the progress this language is making. I’m sure we’ll be hearing more about Vala in the future.</p> How to implement an interrupt driven GPIO input in Linux 2009-01-10T00:00:00+00:00 With Linux, some of the things that seem like they should be easy are not – at least at first glance. For example, how do you read an interrupt driven GPIO input in a Linux application? With simpler microcontroller systems, this is straightforward, but with a system like Linux, you have to navigate through several layers of software (and for very good reasons). You can’t handle interrupts directly in a Linux application, so this means you need a kernel component involved. This operation of reading a GPIO resembles a key press, so the Linux input subsystem might be a good place to start looking. Once we take that route, we discover the gpio_keys</a> driver.</p> Configuring the gpio_keys driver</h2> The gpio_keys driver is configured with a few lines of code in your Linux board configuration file. An example is below:</p> static struct gpio_keys_button svs_button_table[] = { { .code = KEY_RECORD, .gpio = PP_GPIO_MIC_EN, .active_low = 1, .desc = "MIC_EN", .type = EV_KEY, .wakeup = 0, }, }; </code></pre> In this application, we are reading button presses on a cell phone style headset. Once the gpio_keys driver is configured, a new entry will show up in /dev/input/eventX. An application can then do a blocking read on this device.</p> Reading the GPIO in an application</h2> To read the GPIO, we simply do a blocking read on the new /dev/input/eventX device. The read will block until there is a change in GPIO state. An example is show below:</p> #define MIC_INPUT_DEV "/dev/input/event0" static gboolean mic_button_callback(GIOChannel source, GIOCondition condition, gpointer data) { struct input_event ev; int bytes_read; g_io_channel_read_chars(source, (gchar )&ev, sizeof(ev), &bytes_read, NULL); if (bytes_read > 0) { if (bytes_read != sizeof(ev)) { s_debug(1, "warning, only read %i bytes from mic input"); return TRUE; } } else { return TRUE; } if (ev.type != EV_SYN && ev.value == 1) { /* button pressed, do something ... / } return TRUE; } void mic_button_init() { GIOChannel micbutton = g_io_channel_new_file(MIC_INPUT_DEV, "r", NULL); if (micbutton == NULL) { s_debug(TRUE, "Error initializing mic button"); return; } g_io_channel_set_encoding(micbutton, NULL, NULL); guint id = g_io_add_watch(micbutton, G_IO_IN, mic_button_callback, NULL); } </code></pre> The above example also shows how to incorporate the GPIO read into a GLib mainloop so that you don’t need to create a separate thread. ( As a side, GLib mainloop programming is worth learning!) Using this method, reading a GPIO interrupt is easy and requires very few lines of code. This is typical of complex systems like Linux – if you know how to do something, it is relatively easy, but getting started down the right path is sometimes the challenge.</p> Tips for reading a serial data stream in Python 2009-01-09T00:00:00+00:00 Interfacing with a RS232 serial device is a common task when using Python in embedded applications. The easiest way to get python talking to serial ports is use the pyserial project found at http://pyserial.sourceforge.net/</a>. This module works on most platforms and is straightforward to use (see examples on project web site). However, getting the read function in this module to operate in an optimal way takes a little study and thought. This article investigates how the pyserial module works, possible issues you might encounter, and how to optimize serial reads.</p> We start out with several goals as to how we want the application to behave in relation to the serial port:</p> application must block while waiting for data.</li> for performance reasons, we want to read decent size chunks of data at a time if possible. Python function calls are expensive, so performance will be best if we can read more than one byte at a time.</li> We want any data received returned in a timely fashion.</li> </ul> A key parameter in the pyserial Serial</em> class is the timeout</em> parameter. This parameter is defined as:</p> timeout=None, #set a timeout value, None for waiting forever </code></pre> The Serial</em> class read</em> function also accepts a size</em> parameter that indicates how many characters should be read. Below is the source for the read function on Posix systems (Linux, etc):</p> def read(self, size=1): """Read size bytes from the serial port. If a timeout is set it may return less characters as requested. With no timeout it will block until the requested number of bytes is read.""" if not self.fd: raise portNotOpenError read = '' inp = None if size > 0: while len(read) < size: #print "\tread(): size",size, "have", len(read) #debug ready,_,_ = select.select([self.fd],[],[], self.timeout) if not ready: break #timeout buf = os.read(self.fd, size-len(read)) read = read + buf if self.timeout >= 0 and not buf: break #early abort on timeout return read </code></pre> The easy way to use this module is to simply set the timeout to None, and read size to 1. This will return any data received immediately. But, this setup is very inefficient when transferring large amounts of data due to the Python processing overhead.</p> To meet our goal of reading multi-byte blocks of data at a time, we need to pass the read function a size greater than 1. However, if timeout is set to None, the read will block until size bytes have been read, which does not meet the goal of returning any data read in a timely fashion. The solution then is to:</p> set the read size high enough to get good performance</li> set the timeout low enough so that any data received is returned in a reasonable timeframe, but yet the application spends most of its time blocked if there is no data.</li> </ul> As an example, a size of 1000 and a timeout of 1 second seems to perform well. When used this way, the pyserial module performs well and returns all data read quickly.</p> How to Optimize Python reads for general data 2009-01-09T00:00:00+00:00 The Python file object read() function acts a little differently than the standard read() found in C. This article describes some of these differences and how to optimize reads for general continuous data streams such as reading data from a collection device through a pipe.</p> Python read()</h2> The Python read function seems to be optimized for reading files and text oriented streams. By default, a read() call will block until a EOF is encountered. This is very handy for reading files from a disk. You can just slurp them up with one read() statement. If you pass a size parameter to read(), it will block until size bytes have been received. This is less than ideal for reading a continuous data stream where some of thye data may be stuck until the size threshhold is reached.</p> Non-blocking read()</h2> The way you get read to return with whatever data is available even if it is less than the read size parameter is to set the file object up in non-blocking mode. This can be done using the fcntl module:</p> flags = fcntl.fcntl(fp, fcntl.F_GETFL) fcntl.fcntl(fp, fcntl.F_SETFL, flags \| os.O_NONBLOCK) </code></pre> However, now the application does not block while waiting for data and spins using CPU resources.</p> select()</h2> Enter the select call. The Python select module does much the same thing as the C select() function. In this case, it can be used to block waiting for data from a non-blocking file object with the added benefit of a timeout. So the resulting code might look like:</p> fp = os.popen(<application that returns data to stdout>, 'r') flags = fcntl.fcntl(fp, fcntl.F_GETFL) fcntl.fcntl(fp, fcntl.F_SETFL, flags \| os.O_NONBLOCK) while 1: [i, o, e] = select.select([fp], [], [], 5) if i: s_ = fp.read(1000) else: s_ = '' if s_: logging.debug("received %i bytes of data, total = %i" % (len(s_), total)) <do something with s_> </code></pre> The above select statement blocks until data is available from the fp object or times out after 5 seconds. Reading continuous data streams in Python is very possible, but usually requires the file object to be set in non-blocking mode and a select used to block while waiting for data.</p> Implementing Asterisk 2009-01-09T00:00:00+00:00 Over the past month, we have been working on implementing Asterisk in our office. Asterisk (http://www.asterisk.org/</a> ) is a complete software PBX (Private Branch Exchange) that runs on top of Linux. Asterisk supports a number of features commonly found in a PBX including voicemail, hosted conferencing, call queuing, and many other features. Having a real PBX provides many benefits for a business and has the potential to improve our service and lower our costs. Over the next couple months we will be writing about our experiences implementing Asterisk, tips for setting it up etc.</p> One of the most amazing things about Asterisk is its flexibility. You can mix and match IP phones, traditional analog phones, soft phones, traditional PSTN lines, and VOIP services in about any way you want. Our setup is fairly simple – IP phones and connections to the outside world using our PSTN land line. We are also evaluating several VOIP providers and trying to determine if VOIP is viable. Stay tuned …</p> How to Implement a Web Application Framework in an Embedded Linux System 2009-01-09T00:00:00+00:00 As devices are increasingly more networked, an embedded web server is becoming a standard way for users to interact with and configure an embedded device using a standard web browser. As an example, most pieces of networking equipment (such as routers and wireless access points) are configured this way. There are many ways to implement a web server. One way is to just write a monolithic program that handles the requests and outputs HTML using print statements. Without a lot of planning, you usually end up with something where logic is intermixed with presentation. The disadvantage of such an approach is that it quickly becomes very difficult to maintain your web application as it grows and changes. The current best practice with web application frameworks (such as Ruby on Rails) splits a web application into 3 distinct components (Model, View, Controller – MVC) so that changes to one component can be made with minimal impact to others. This article details a solution used in a recent project to implement such an architecture using the following open source components: SQLite, Clearsilver, and Python. The challenge was to find a solution that performed acceptably on a resource constrained 130MHz ARM-Linux system.</p> Components Used</h2> The requirements for the web application framework for this system are:</p> Footprint must be fairly small – less than 10MiB.</li> Must enable us to implement a clean MVC type architecture.</li> Must support a high level language like python for rapid development.</li> Includes a database.</li> Reasonable performance – less than 1 second to render a typical page.</li> </ul> After researching and testing several options, the following components were chosen to implement the framework:</p> Web Server: Cherokee (1MiB)</li> Model: SQLite (290KiB)</li> View: Clearsilver (170KiB)</li> Controller: Python (2-3MiB)</li> </ul> </p> Total size for the above components is about 4MiB.</p> The following sequence occurs during a typical web transaction:</p> An HTTP request is received by Cherokee</li> If the request matches the URI for the web application, Cherokee forwards the request to the web application framework via CGI.</li> Clearsilver parses the HTTP headers (including CGI GET and POST parameters) and provides the data to python in a easy to access HDF format.</li> Python looks at the URI and dispatches the request to the appropriate function. Data is extracted from the SQLite database and an HDF datastructure is created.</li> Clearsilver is called to render the appropriate template. Data from the HDF datastructure is used in the template to provide the dynamic content in the page.</li> The rendered HTML is passed back to Cherokee, and is then returned to the user’s web browser.</li> </ul> Each component is discussed in more detail in the following sections.</p> Performance</h2> One of the issues with modern web frameworks is that most require a lot of processing power. Many frameworks are written in an interpreted language which tend to be not very efficient on embedded systems such as the 130MHz ARM-Linux system used in this project. As most of the application development for this project is done in Python, I tried several other Python based solutions with the following results:</p> Django</strong>: takes 10</strong> seconds to render a page</li> webpy</strong>: takes 5</strong> seconds to render a page</li> Clearsilver/Python</strong>: takes about 1</strong> second to render a page</li> </ul> Anything over 1 second is very slow for a web interface. With fastcgi, webpy could probably have been sped up to be acceptably fast, but I don’t think there is much hope for frameworks like django on this type of system. I have read of similar experiences trying to run Ruby-on-Rails on ARM-Linux systems. As I’m currently just using a basic CGI interface, the Clearsilver solution could also be sped up significantly as well, if the Python portion could be kept running between requests with fastcgi or a similar mechanism.</p> Cherokee Web Server</h2> A web application typically consists of a web server (like Apache) and a program that provides the dynamic web content. Some of the functions of a web server are:</p> receives HTTP requests</li> handles the request by serving static files/images or routes the request to a program that outputs dynamic content</li> authentication</li> encryption (SSL, TLS)</li> </ul> There are many web servers that can be used in embedded devices. Some of them are:</p> Lighttpd (http://www.lighttpd.net</a> )</li> thttpd (http://www.acme.com/software/thttpd/</a> )</li> Appweb (http://www.appwebserver.org/</a> )</li> Boa (http://www.boa.org</a> )</li> Cherokee (http://www.cherokee-project.com</a> )</li> SHTTPD (http://shttpd.sourceforge.net/</a> )</li> </ul> Cherokee was chosen for this application because it provides a good balance between the size and functionality I am looking for. It is also included in the Openembedded (http://openembedded.org</a> ) build system I am using. Other systems will have different requirements – use the one that fits your application best.</p> Clearsilver</h2> Clearsilver (http://www.clearsilver.net/</a> ) is the real gem that was discovered during this exercise. Clearsilver is a language-neutral HTML template system written in C. It is used as the templating system for many high volume sites such as Google Groups 2. Clearsilver also provides CGI handling functions and bindings to several languages including Python. The fact that it is written in C and is fast for high volume sites also makes it fast enough</em> on slower embedded systems. Because Clearsilver is written in C, the templating system is already fast. Parts or all of the application can also be moved to C as needed to get the required performance. Having a performance upgrade path is nice. Clearsilver also forces a strict separation of application logic and presentation templates, which keeps things in line with the MVC architecture.</p> SQLite</h2> SQLite (http://www.sqlite.org/</a> ) is a small C library that implements a self-contained, embeddable, zero-configuration SQL database engine. SQLite works as expected and also has bindings to a number of languages including Python. A SQLite database is just a single file that requires no configuration, making it very easy to use. The database engine runs in the same process as the application using it, which increases performance because there is no context switch when running database operations. Because SQLite does not have its own process, it does not handle concurrency as well as other databases such as MySQL, but in an embedded system you seldom have more than one process using the database. SQLite provides some support for concurrency with reader/writer locks for the entire database. If your application is not real busy, this is often adequate if you need two processes to share some data.</p> Python</h2> Python is the glue that ties things together in this web application framework. It sits between Clearsilver and the database. The advantages of using a language like Python are rapid development and an extensive library. Python is considerably slower than C, but there is always the option to move parts of the application into C as needed.</p> Openembedded Build System</h2> How does one put together a system with all these components? The answer is the Openembedded Build System (OE) (http://openembedded.org</a>). OE includes support for all of the components discussed in this article.</p> Summary</h2> The combination of Python, Clearsilver, and SQLite provides a compelling solution for a web application framework in an Embedded Linux System. The solution is reasonably small, performs well, and provides a path to improve performance if needed. We also get a clean architecture to create clean, maintainable web applications.</p> About the author</strong> – Cliff Brake owns BEC Systems, a consulting operation that helps customers utilize modern computer technologies in their products. BEC offers a range of services to make your embedded project a success including technology selection, development, troubleshooting, and training. Please visit our website (http://bec-systems.com</a> ) for more information and other free resources.</p> Do you have an Exit Strategy? 2009-01-09T00:00:00+00:00 Some time ago, eWeek published in interesting article by Scott McNealy named “Technology’s Barriers to Exit</a>”</em>. In this article, Scott makes an interesting point about considering the cost of technology. There are three costs to consider:</p> cost of acquisition</li> ongoing cost</li> cost to exit</li> </ul> Scott states that most people analyze the first two costs fairly well. What is not thought about as much is the the cost to exit – how do I move on to the next thing. I think these basic questions apply very much to embedded technology decisions.</p> Most of the time we carefully consider our requirements, think about ongoing manufacturing costs, etc. There may be some thought given to upgrading (the exit), but we encounter several difficulties when planning for the future with complex embedded systems:</p> we often can’t predict what technologies will be available in the future</li> if may be difficult to predict our product requirements in the future</li> </ul> As an example, we may have a product with an ethernet networking interface, and two years after product launch we have an opportunity that requires a wireless network interface. How much is it going to cost to add wireless support to our product? If the product is based on technologies that do not fundamentally support wireless networking, we are looking at very costly exit strategy. Another example is application-specific, off-the-shelf solutions. While these are great for many problems and provide a quick way to get something working, what happens when we want to upgrade? Do we have to re-implement the entire system?</p> How does one hedge against the unknowns in the future? One way is to try to anticipate every possible requirement and design it in. I think this is bad strategy because you usually end up spending a lot of time working on things you don’t really need and the quality of the product suffers. A better approach is make sure your product does what it needs to do well and is based on a flexible platform where significant parts of your design and software can be re-used in future versions. This approach has many parallels to recent trends in software development and manufacturing such as Agile Software Development, and Lean Manufacturing.</p> Follow Scott’s advice and make technology selection decisions with the following questions in mind:</p> Is the technology based on open standards?</li> Are programming interfaces and protocols open, published, and been adopted by industry?</li> Is there a community process behind the technology (not just one company)?</li> Can the same technology be sourced from multiple companies?</li> </ul> The use of Linux in embedded systems is an excellent example of a technology that meets the above criteria. You can develop your prototype on a x86 PC. Move it to an embedded ARM single board computer (SBC) for the first production run. Switch to a different SBC or design your own board when you run into new product requirements. Sure, it will cost you more to develop the product initially due to the complexity of a system like Linux, but you now have options to expand and change your product while directly re-using the work you have already done.</p> But wait, as long as I write my application in C, it is portable – right? Most embedded systems support development in C so I will just port my application to a new platform when needed. C is a great language and the language</em> is more or less portable between systems. But as systems get more and more complex, embedded engineers find themselves spending less time writing original code and spending more time putting together pieces that already exist. A modern embedded application is often little more than some glue between various libraries and system functions. The fact that you can develop in C for a particular system does not really mean a lot. What about the threading model, driver model, library APIs, etc. I just spent a significant amount of time recently porting a large C application from a proprietary platform to new hardware platform as the original system was going obsolete. It was a huge amount of work. The driver model was completely different, APIs were different. Changing platforms can be very expensive.</p> It rarely pays to be short sighted when choosing technologies for complex embedded systems. The best strategy is to stay flexible and choose a platform with a future.</p> Gumstix Overo Connector Spreadsheet 2008-12-15T00:00:00+00:00 One of the first things I do when designing a system based on a processor module is create a spreadsheet listing all the connector I/O. This is the easiest way I’ve found to make sure nothing gets missed. My first pass at a I/O spreadsheet for the Gumstix Overo is available at:</p> http://redmine.bec-systems.com/bec/documents/1</a></p> How to disable SSH host key checking 2008-12-12T00:00:00+00:00 This article</a> presents a very nice overview of options to disable ssh host key checking. When working with embedded systems where you are constantly reloading the rootfs, it is really annoying to have to edit your ~/.ssh/known_hosts</em>every time you update the rootfs on the target device. The solution is to simply disable host key checking for your local subnet by adding something like the following to the beginning of /etc/ssh/ssh_config</em>:</p> Host 192.168.1.* StrictHostKeyChecking no UserKnownHostsFile=/dev/null </code></pre> Anything not on the 192.168.1.* subnet is still checked.</p> Update 2009-09-26:</strong></p> Another place to put these config settings is: ~/.ssh/config</em></p> Printing from Embedded Systems 2008-12-02T00:00:00+00:00 How does one implement support for printing in embedded systems? I recently had the opportunity to add printing support to an embedded Linux system. The device is an industrial touch screen powered by a Compulab cm-x270 module (PXA270 CPU), and runs a GTK+ application. The customer is implementing a device calibration system where customers bring their equipment in to get calibrated, and the system prints out a report on a local printer. This article describes how components of Hewlett Packard’s HPLIP solution along with Cairo can be used to implement printer support in a non-desktop Linux system.</p> Requirements</h3> The requirements for this project were fairly basic – we needed to print a single page report that contained text and some elementary graphics. We initially wanted to support several low-end Inkjet and Laser USB printers. As the system was powered by a fairly slow ARM processor (slow compared to modern desktop systems), the solution needed to be efficient, and not require excessive amounts of memory.</p> Desktop Linux Print Flow</h3> With desktop Linux systems, the standard printing flow looks something like this:</p> Application -> PS output -> Ghostscript -> Rasterized output -> Printer Driver (filter) -> I/O Backend -> Printer</p> CUPS is typically used to manage this flow, provide queuing, etc.</p> Print Architecture</h3> As the application is written in GTK+, we decided to generate the report using Cairo, which is a 2D graphics library. Cairo is easy to use, and is well suited for this application. With a Cairo generated report, we already had a raster image of the report, so it seemed that the PS output/Ghostscript steps were not really needed and only added more processing to the data flow. Also the PXA270 CPU does not have a FPU, and PostScript processing can be floating point intensive. So the need was now to figure out how to get a Cairo generated raster image to a USB printer.</p> Hewlett Packard offers lots of interesting software for their printers. Their APDK is a OS independent library in source code form. However, implementing this would have required quite a bit of integration, and writing the I/O layer. It seemed like there should be something available that would work with a little less effort. The HPLIP project</a> is a comprehensive set of software for Linux printing, and is used by desktop Linux systems. It is not obvious at a glance how all the components of HPLIP fit together, but after spending some time digging through source code, and asking questions on several maillists/forums, we were able to figure out that the basic flow is:</p> Ghostscript -> HPIJS (driver) -> HP backend -> Printer</p> In this application, we simply launched the HPIJS driver directly from our application instead of Ghostscript.</p> HPLIP</h3> There were several challenges using HPLIP in our ARM system. The HPLIP is not cross-compilation friendly out of the box, so we had to fix a few issues, and ended up disabling all the Python pieces as we only needed the HPIJS and backend components. We also had to figure out the data flow from the application to HPIJS, and then to HP backend. In a nutshell, the process is:</p> start the hp backend process by forking, and obtain a file handle to STDIN for the hp backend process</li> start the HPIJS process, send various parameters to it including the file handle for the backend STDIN</li> send HPIJS the print raster and tell it to print</li> </ul> There are some additional details such as handling margins, error conditions, cancel support, etc. Some of the details can be gleaned from the Ghostscript source code, and the IJS reference implementation</a> provides some very useful library code for implementing the IJS client functionality in the application that is doing the printing. When finished, our printing module was 403 source lines of code (SLOC) – not bad considering the functionality.</p> How well does it work?</h3> Overall, we are very pleased with the result. With a simple industrial terminal, we can now support just about every USB printer made by HP with the exception of some of their very low end LaserJet printers that require a binary plugin. The HP backend can be used to detect what printer is attached at run time, so everything is plug-n-play with no</em> user configuration. Let me repeat as this is significant – on a simple Industrial terminal, we can support about every HP USB printer available with no user configuration. This is easier for users than their desktop as they simply need to buy a printer and plug it in! Kudos to HP for their excellent open source software. Because the source code is available, we were able to customize it for our application with very little support from HP. Performance wise, the printing process is quite fast; the printer starts almost immediatly after the user initiates the print. Now if I could just get HP to build a ARM version of their binary plugin for the few low end lasers we can’t support … but this is not a critical issue for this vertical application.</p> Future Direction</h3> Hopefully, this type of solution can evolve into a standard printing solution for Embedded Linux systems. Epson also supports the IJS driver model, so adding support for their printers should be possible. It may eventually make sense to integrate portions of CUPS for queing, and other management tasks. Some of the tasks I hope to implement in the future:</p> Clean up the HPLIP build for inclusion in OpenEmbedded.</li> Clean up the IJS reference code library build and packaging in OpenEmbedded.</li> Figure out portions of CUPS that may make sense.</li> Keep conversations going with the Linux printing group so we can someday have a “standard” printing solution for embedded Linux systems.</li> </ul> Thanks to Matt Gessner for helping implement this solution, and for providing feedback on this article.</p> Things that work! 2008-11-19T00:00:00+00:00 Running a small business has a special set of challenges, and if you are not careful, the overhead can quickly consume you. This article is a list of things that I use in my business that work well for me, and help me to be efficient when there is so much to do …</p> Entire article at: /web/content/view/101/9/</a></p> Asterisk</h3> While I can’t say the same for VOIP proiders, the Asterisk PXB is a very solid product. I run it on an ancient 1GHz celeron system, and use it for my office PBX. In combination with a VOIP service, I can:</p> handle multiple calls at once</li> easily conference multiple callers</li> forward calls to my cell phone or other numbers</li> provide extensions for associates who are working with me</li> </ul> and all at a very low cost. To mitigate some of the reliability problems I’ve had with VOIP providers, I set the VOIP provider to use my cell phone as a backup if the VOIP link is down. I also use multiple VOIP providers for outgoing calls so that if one is not working, I can usually use the backup. Overall, I really like the features, and for my business the occasional hassle is worth it.</p> SQL-Ledger</h3> SQL-Ledger is a web based accounting program that is open source. It provides all the features I need to run my business, and offers the following advantages over more traditional programs such as Quickbooks:</p> web based so I can access it anywhere, even remotely.</li> subcontractors and associates can log time remotely directly in my accounting system.</li> granular permissions for different people who need to use it.</li> upgrades just work.</li> consistent, reliable operation that I expect from an accounting system.</li> </ul> OpenOCD</h3> We finally have a low cost, open source JTAG solution that works reasonably well. While there are still issues, it gets the job done for most of my projects, and offers my customers a low cost way to program their systems using JTAG. Often customers are not doing extensive development, and it is nice to be able to provide a solution for the occasional need. One customer fought with a very expensive commercial unit for weeks. I finally shipped them a $50 Olimex unit that did everything they needed. There can’t be enough said for everyone on a team using the same tools, and they only way this can happen easily is if the tools are fairly low cost.</p> Tracks</h3> Tracks is a nice web-based task managment tool to help implement the GTD methodology. The user interface is supurb and better than most native applications I’ve used. I use the “starred” items for my “next action” on each project. Very nice!</p> Printable CEO Emergent Time Tracker</h3> I use David Seah’s handy form</a> for tracking my time during the day. It forces me to use a pen and paper every day which is a pretty rare thing, and is a nice compromise between simplicity and automation.</p> OpenEmbedded</h3> I’ve written a lot about the benefits</a> of OpenEmbedded. It makes the huge task of creating a build system for Embedded Linux systems very manageable and repeatable. When build systems are fully automated, things are so much easier. As an example, I’ve been working on one project for over two years, and have built 313 releases. We have had almost no image failures due to forgetting to put something in the image, or some other manual step you tend to find in build systems. This is the power of a fully automated build system with _no_ manual steps.</p> Ubuntu</h3> Once I become convinced that Ubuntu could do distro upgrades, I finally quit using Gentoo. Between servers, workstations, and home computers, I now maintain about 10 Ubuntu systems. Over the past several years, I’ve had very few problems using the distro upgrade mechanism. Also, the fact that Ubuntu includes fairly recent Linux technology is a big plus for experimenting with components I use in embedded Linux systems.</p> DHL/UPS Pickup</h3> Working from a small office where I do not regularly make shipments, it is really convenient to be able to schedule a pickup with shipping companies like DHL and UPS. This can be done over the web in 15 minutes or so, and a shipping label then prints out on your printer. The DHL website will also provide commercial invoices for overseas shipping. This sure beats driving to the local UPS shipping place, filling out paper work with a pen, waiting in line, etc. Just another thing that makes it easier for a small business to run efficiently.</p> Intel Bare Bones Server</h3> Last year, I was looking for a cost effective 1U rack server. I selected a Intel SR1530 Bare Bones server. I installed it over a year ago and have not touched it since. Last time I checked, the uptime was 254 days.</p> 3-ware Raid Controllers</h3> (I guess it is AMCC now …) They just work with Linux – currently running in two systems.</p> Sandusky Cabinets</h3> Doing embedded system development means I get a lot of equipment from customers. As some of the development stretches over several years with occasional ongoing support, its nice to be able to store this equipment and then find it efficiently. I’ve found the “Sandusky Standard Storage Cabinets” from Staples to work very well. They are fairly cheap, yet durable enough for what I need, and keeps everything enclosed which looks neat, and keeps the dust problem under control. And, Staples offers free delivery.</p> Bob Pease organization system</h3> Bob Pease had an article</a> in Electronic Design some time ago about organizing things in his refrigerator. He used small boxes to group things in his refrigerator. I use the same method in my Sandusky cabinets and it makes it very easy to find something when I need it.</p> </p> GTD Labeling System</h3> I’ve slowing been adopting some of David Allen’s organization ideas from “Getting Things Done”. One of his best ideas is to get a labeller. I used to use a Brother handheld labeller, but lately, I’ve been using a Dymo LabelWriter 400 left over from a project. The LabelWriter connects to your computer via USB, works well with Linux, and allows me to use my standard keyboard, and OpenOffice to create labels. Note, the labels on the boxes in the above picture. The high contrast black on white makes it much easier to find things versus hand writing notes on cardboard boxes, file folders, etc.</p> GTD Filing System</h3> I used to use hanging file folders, but no more. David Allen’s suggestion of simply using plain manila file folders with nice labels works very well. Once again the large, high contrast labels makes it easy to find things.</p> </p> Snaketray Cable Management</h3> I’ve already written about my cable management system</a>. A year later, I still like it and no longer dread digging through cables.</p> Steelcase Leap Chair</h3> I’ve sat in various chairs over the years – mostly the cheap office store variety that most companies buy. I must be getting old enough now that they no longer cut it. I tried a Herman Miller Aeron for a week as that is what everyone recommends, but I found it too hard. I then tried a Steelcase Leap chair and it works great for me. I no longer even think about the chair I’m sitting in, even after a long day. Expensive, but well worth it if you spend as much time sitting as I do.</p> Tomboy Notes + gnome-do</h3> Tomboy notes is a Linux desktop note-taking application that is essentially a personal wiki. It automatically tracks links to other pages, links to file locations, etc and organization is done mostly through links instead of hierarchy. It is simply the best note taking system I have ever used. Coupled with gnome-do, I can instantly bring up any note with only a few keystrokes. In this day and age where information abounds, being able to take and recall notes efficiently is critical to being effective.</p> Konsole</h3> Being able to easily name the tabs in the KDE console application is very handy as I’ll typically have 5-10 named consoles going at one time.</p> </p> 8-port USB serial expander</h3> As I work with a lot of different devices, it is very handy to have lots of serial ports available. The SerialGear device</a> available from http://www.usbgear.com works out of the box with modern Linux distributions.</p> </p> Git</h3> I’m starting to really appreciate the benefits of distributed source control systems. For commercial embedded development, it allows me to easily track upstream projects, yet have a local source control system for tracking project changes. Having everything in one version control system makes it much easier to extract patches to push back upstream, etc.</p> Redmine</h3> I’ve been a happy trac</a> user for years, but have recently switched to using Redmine</a> for project management. Redmine offers some addition features such as multiple project support, and the user interface is a little more polished. As all of my projects are accomplished with distributed teams, it is very important that there be clear visibility as to what is going on in the project. It also helps to give customers the “Warm Fuzzy Feeling</a> TM” as they have clear visibility into the project, and they can easily verify there is progress being made on the project.</p> Benefits of OpenEmbedded switching to Git 2008-10-22T00:00:00+00:00 Recently the OpenEmbedded project has switched to the Git version control system. This is good news for many reasons. The obvious reasons are Git is faster than Monotone, handles branching better, has lots of nice features, larger user base, etc. Monotone has served us well, but as new tools become available, it is time to change. For those of us helping customers use OE for embedded projects, Git will make using OE much easier. Companies want to be able to track OE with minimal effort and typically maintain a few minor tweaks to OE. As most Embedded Linux projects now implement Git infrastructure for kernel development, the OE source tree can be maintained using the same infrastructure. Branching is very natural in Git, so it is fairly easy to create a OE “topic branch” and periodically merge with upstream changes as needed. The ability for easy repo hosting and branching allows OE users to lock down a version of OE for periods of project development, and still maintain a connection to the upstream OE repository for easily updating to new versions, or cherrypicking changes. So a big thank you to all who helped make this change happen.</p> more fun with git branches 2008-08-26T00:00:00+00:00 One thing you learn after using git for awhile is you get used to trying random things, and it will often just work. For example, if I want to know the differences between Linus’s kernel tree, and the Wolfson dev branch for a particular directory:</p> cbrake@happy:/build/linux-2.6$ git diff --stat origin/master..wolfson/dev sound/soc/codecs sound/soc/codecs/Kconfig \| 125 +++ sound/soc/codecs/Makefile \| 34 + sound/soc/codecs/ad1939.c \| 690 +++++++++++++++ sound/soc/codecs/ad1939.h \| 70 ++ sound/soc/codecs/ad1980.c \| 309 +++++++ sound/soc/codecs/ad1980.h \| 23 + sound/soc/codecs/cs4251x.c \| 771 +++++++++++++++++ </code></pre> Another way to get similar information is:</p> git log origin/master..wolfson/dev sound/soc/codecs </code></pre> If I’m working on a topic branch, and I want to see a summary of all the changes in my Topic branch:</p> git log --stat origin/master..origin/my_topic_branch > my_topic_branch_changelog.txt </code></pre> One of the neat things about git branches is you don’t have to be on a branch to interact with it. You can diff, log, and checkout from branches other than the one you are currently in, and it is all very fast. For instance, if I want to grab the latest copy of a file in the Wolfson dev branch, I can do something like:</p> git log checkout wolfson/dev sound/soc/codecs/wm9713.c </code></pre> Not that this is a very good idea, but just to illustrate a point. Git branches are amazing and make development work so much less tedious.</p> Linux Input Testing and Debugging 2008-08-22T00:00:00+00:00 The Linux input layer has made a lot of progress in recent years. When writing a new input driver (such as keyboard, trackball, etc), it is useful to be able to monitor input events using a test application. This article describes two ways to accomplish this using kernel input debugging, and the evtest utility.</p> kernel input debugging</h3> The Linux kernel includes some support for printing input events as debug messages. To use, set the following in the kernel config:</p> CONFIG_INPUT_EVBUG=y </code></pre> Because the print messages are at the KERN_DEBUG level (7), we typically need to change the debug level:</p> echo 8 > /proc/sys/kernel/printk </code></pre> At this point, you will see messages like the following on the console:</p> evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 108, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 106, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 1 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 1 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 </code></pre> evtest</h3> The linux-input project (http://linuxconsole.sourceforge.net/input/input.html</a>) includes a neat little tool named evtest that is very handy for testing input devices. If you are using OpenEmbedded, simply:</p> bitbake linux-input </code></pre> If you are not using OpenEmbedded, you can glean enough details about how to get and build the this package from the OE recipe:</p> http://gitweb.openembedded.net/?p=org.openembedded.dev.git;a=blob;f=packages/linux-input/linux-input_cvs.bb</a></p> Install the resulting package on your device. To use, run evtest with a /dev/input/eventx argument:</p> root@machine1:~$ evtest /dev/input/event0 Input driver version is 1.0.0 Input device ID: bus 0x19 vendor 0x1 product 0x1 version 0x100 Input device name: "gpio-keys" Supported events: Event type 0 (Sync) Event type 1 (Key) Event code 103 (Up) Event code 105 (Left) Event code 106 (Right) Event code 108 (Down) Event code 116 (Power) Testing ... (interrupt to exit) Event: time 1009.433605, type 1 (Key), code 108 (Down), value 0 Event: time 1009.433737, -------------- Report Sync ------------ Event: time 1010.735605, type 1 (Key), code 108 (Down), value 1 Event: time 1010.735740, -------------- Report Sync ------------ Event: time 1011.075586, type 1 (Key), code 106 (Right), value 1 Event: time 1011.075720, -------------- Report Sync ------------ Event: time 1011.103379, type 1 (Key), code 108 (Down), value 0 Event: time 1011.103550, -------------- Report Sync ------------ Event: time 1011.333924, type 1 (Key), code 108 (Down), value 1 Event: time 1011.334058, -------------- Report Sync ------------ Event: time 1011.339880, type 1 (Key), code 106 (Right), value 0 </code></pre> Notice that evtest lists the keymap when it is initially run which is very handy, and then lists key events as they occur. The key descriptions are also included with events so you don’t have to manually decode the keycodes.</p> Using a Verizon USB720 modem in an Embedded Linux system 2008-06-02T00:00:00+00:00 </p> The Verizon USB720 Cellular modem is a quick and easy way to add wireless internet connectivity to your embedded system. Why might you want to add cellular connectivity to your embedded device? Remote access and diagnostics is a big reason. One customer I have is planning to supply all their distributors with a USB720 so they can easily upload log files from devices in the field. The alternative is copying the log files to a flash drive, finding a PC connected to the internet, and then uploading the files – too much hassle, therefore it does not get done. Also, think about the cost of sending a field support person on-site to troubleshoot problems versus shipping them a USB720 and accessing the device remotely.</p> The USB720 is a fairly small device that plugs into the ubiquitous USB port. This articles covers some of the details on how to configure an embedded Linux system to work with this device.</p> Why is the USB720 a good fit for wireless connectivity in Embedded Systems?</h3> Some reasons the USB720 (or other USB based cellular modems) is an attractive option:</p> fairly fast: provides typical download speeds of 600-1400kbps, and upload speeds of 500-800kbps</li> convenience: it plugs into a USB port. No taking apart the box and installing an embedded module, etc.</li> affordable: with plans around $60/mo, its a lot cheaper than sending field support people on-site to debug every problem.</li> uses standard PPP protocols for connection. That means it works with Linux.</li> supported by the mainstream Linux kernel</li> </ul> </p> Kernel Configuration</h3> The USB720 looks like a USB-serial device to the host system. If you plug it into your Ubuntu desktop Linux system, all the needed kernel pieces are there and it will just work. With your embedded system, you need to make sure a number of kernel options are enabled including:</p> CONFIG_PPP=m CONFIG_PPP_ASYNC=m CONFIG_PPP_DEFLATE=m CONFIG_SLHC=m CONFIG_USB_SERIAL=m CONFIG_USB_SERIAL_OPTION=m </code></pre> The CONFIG_USB_SERIAL_OPTION selection enables a driver that is specifically optimized to work with the USB chip found in many cellular modems. Once you enable the above modules, and if you are running udev, simply plugging in the device will cause the appropriate modules to load and a /dev/ttyUSB0 device will now appear.</p> PPP Scripts</h3> The most difficult part of this exercise is figuring out how to configure the PPP scripts. Verizon of course does not tell you how to configure their device and only supplies software for Windows and MAC. There are a number of howtos available on the internet, but most of them are tailored to using kppp or wvdial. Neither are very practical for embedded systems. Fortunately, the standard pppd daemon is fairly easy to use, is easy to build for embedded systems (it is already included in OpenEmbedded), and seems to work very well. The following scripts are needed:</p> /etc/ppp/peers/verizon</p> noauth connect "/usr/sbin/chat -v -f /etc/ppp/peers/verizon_chat" defaultroute usepeerdns ttyUSB0 921600 local usepeerdns debug -detach </code></pre> /etc/ppp/peers/verizon_chat</p> '' 'ATZ' 'OK' 'ATDT#777' 'CONNECT' '' </code></pre> /etc/ppp/chap-secrets /etc/ppp/pap-secrets</p> "<phone number>@vzw3g.com" * "vzw" </code></pre> At this point, you can run pppd call verizon</em> and you should see something like:</p> root@cm-x270:~$ pppd call verizon Serial connection established. using channel 9 Using interface ppp0 Connect: ppp0 <--> /dev/ttyUSB0 Warning - secret file /etc/ppp/pap-secrets has world and/or group access sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x539657b7> <pcomp> <accomp>] rcvd [LCP ConfReq id=0x0 <mru 1500> <asyncmap 0x0> <magic 0xb9b652e5> <pcomp> <accomp>] sent [LCP ConfAck id=0x0 <mru 1500> <asyncmap 0x0> <magic 0xb9b652e5> <pcomp> <accomp>] rcvd [LCP ConfAck id=0x1 <asyncmap 0x0> <magic 0x539657b7> <pcomp> <accomp>] sent [CCP ConfReq id=0x1 <deflate 15> <deflate(old#) 15>] sent [IPCP ConfReq id=0x1 <compress VJ 0f 01> <addr 192.168.1.119> <ms-dns1 0.0.0.0> <ms-dns3 0.0.0.0>] rcvd [LCP DiscReq id=0x1 magic=0xb9b652e5] rcvd [LCP ProtRej id=0x2 80 fd 01 01 00 0c 1a 04 78 00 18 04 78 00] rcvd [IPCP ConfReq id=0x0 <addr 66.174.61.4>] sent [IPCP ConfAck id=0x0 <addr 66.174.61.4>] rcvd [IPCP ConfRej id=0x1 <compress VJ 0f 01>] sent [IPCP ConfReq id=0x2 <addr 192.168.1.119> <ms-dns1 0.0.0.0> <ms-dns3 0.0.0.0>] rcvd [IPCP ConfNak id=0x2 <addr 70.210.82.191> <ms-dns1 66.174.95.44> <ms-dns3 66.174.92.14>] sent [IPCP ConfReq id=0x3 <addr 70.210.82.191> <ms-dns1 66.174.95.44> <ms-dns3 66.174.92.14>] rcvd [IPCP ConfAck id=0x3 <addr 70.210.82.191> <ms-dns1 66.174.95.44> <ms-dns3 66.174.92.14>] not replacing default route to eth0 [192.168.1.1] local IP address 70.210.82.191 remote IP address 66.174.61.4 primary DNS address 66.174.95.44 secondary DNS address 66.174.92.14 Script /etc/ppp/ip-up started (pid 2303) Script /etc/ppp/ip-up finished (pid 2303), status = 0x0 </code></pre> root@cm-x270:~$ ifconfig ppp0 Link encap:Point-to-Point Protocol inet addr:70.210.82.191 P-t-P:66.174.61.4 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:4 errors:0 dropped:0 overruns:0 frame:0 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:3 RX bytes:64 (64.0 B) TX bytes:94 (94.0 B) </code></pre> You are connected :-)</p> Using the OE ppp package</h3> Add ppp support to your image using OE is fairly straightfoward:</p> add the kernel modules to your image</li> addppp</em> to your image</li> add a custom package that includes the configuration scripts listed above</li> </ul> One thing that may trip you up is the ppp scripts are expecting resolv.conf to be in the following location: /var/run/resolv.conf. The reason for this is we want resolv.conf in a RAM file system to prevent unneeded flash wear. Therefore, /etc/resolv.conf should be configured in your rootfs as a soft link to /var/run/resolv.conf.</p> Summary</h3> The Linux kernel support for USB720 and the pppd package offer a quick, robust way to add cellular network connectivity to about any embedded Linux system that has a USB host port. As systems become more complex, it will be increasingly important to be able to remotely access systems in the field to troubleshoot problems, help users with configuration, etc. The USB720 offers a quick way to provide this type of connection without going through the expense of designing cellular connectivity into your product. One thing I usually encourage all my customers to do when designing new products is expose a USB host port, even though they may not see any immediate use for it. This is one of those reasons.</p> Below is a picture of a USB720 connected to a Compulab cm-x270 development system.</p> </p> Autotools quick reference 2008-05-03T00:00:00+00:00 At some point, most Linux developers need to master Autotools. For most of us, this is a fairly painful process, but like any good tool, Autotools is extremely useful and well worth learning. For example, if your program is set up correctly using Autotools, it will cross compile with almost zero effort in OpenEmbedded, or with slight effort manually. Compare this to the tedious effort of manually referencing toolchain locations, cross toolchain library and header file locations, etc. Despite first impressions, Autotools will make your life easier. This article contains a collection of tips on how to do standard operations using Autotools.</p> Minimal Autotools files required for a simple C application</h2> configure.ac</strong></p> AC_INIT([fooprog], [1.0], [email@company.com]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_PROG_CC AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([Makefile]) AC_OUTPUT </code></pre> Makefile.am</strong></p> bin_PROGRAMS = foo foo_SOURCES = foo.c </code></pre> The above files will create a binary name foo (compiled from foo.c) when you run the following steps:</p> touch NEWS AUTHORS ChangeLog</li> autoreconf -i</li> ./configure</li> make</li> </ul> How to add libraries to the link step</h2> For example, if you want to link in the pthread library to your application, add the following line to Makefile.am:</p> foo_LDADD = -lpthread </code></pre> How to add compiler options</h2> You can easily add compiler options using the foo_CFLAGS or foo_CPPFLAGS variables.</p> foo_CFLAGS = -DMY_DEFINE</p> How to use pkg-config in Autotools</h2> pkg-config is a helper tool that gives you library and include file information for various libraries installed. For example, to link to alsa and glib in your project:</p> configure.ac</strong></p> PKG_CHECK_MODULES(PACKAGE, alsa glib-2.0) AC_SUBST(PACKAGE_CFLAGS) AC_SUBST(PACKAG_LIBS) </code></pre> Makefile.am</strong></p> foo_CFLAGS = @PACKAGE_CFLAGS@ foo_LDADD = @PACKAGE_LIBS@ </code></pre> How to add a command line option to the configure script</h2> AC_ARG_ENABLE(option-foo, [ --enable-option-foo help for option foo], [CFLAGS="$CFLAGS -DOPTION_FOO_DEFINE"], []) </code></pre> The above example sets a define on the compiler command line if configure is run with the --enable-option-foo</em> command line option.</p> Disable shared libraries in your project</h2> Add the following to configure.ac:</p> AC_DISABLE_SHARED </code></pre> Pass a variable from configure.ac to Makefile.am</h2> Often there is a need to pass a variable in configure.ac to a Makefile. This is done with the AC_SUBST macro:</p> configure.ac</strong></p> AC_SUBST(VAR_FOO) </code></pre> Makefile.am</strong></p> foo2 = @VAR_FOO@</p> Installing data files</h2> By default, Autotools will install executables built when you run “make install”. To install other non-executable files, you need to tell autotools explicitly about those files. The standard place to install data files for an application is in $(datadir)/appname.</p> Makefile.am</strong></p> fooextradir = $(datadir)/@PACKAGE@ </code></pre> fooextra_DATA = \ foo_data1.txt \ foo_data2.txt </code></pre> The above will install the foo_data1.txt</em> and foo_data2.txt</em> files in the $(datadir)/fooprog</em> directory. $(datadir) is typically /usr/share/ or /usr/local/share, but it can be anything.</p> How does my program know where $(datadir) is located</h2> Because $(datadir) can be anywhere in the system, we need some way to tell the program where it was specified to be at time of compilation so that it can locate foo_data1.txt and foo_data2.txt. This can be done by passing a define at compile time:</p> Makefile.am</strong></p> AM_CFLAGS = -DDATADIR=\"$(datadir)\" </code></pre> How to install data files during development without cluttering up my system directories</h2> If you use the above method to install data files needed by your project, you may run into a case during development where you want to run the program from the build directory, but the program needs the data files at run time. Since, the program is looking for them in the location specified by the DATADIR define, they need to be installed. You could just run “make install”, but this would install them into /usr/local/share, require you to run as root, and clutter up your system directories. A much more elegant solution is to specify the install prefix at configure time to be a temporary “install” directory in your source tree. The --prefix option for configure requires an absolute directory, so the `pwd` trick is used to figure out what the current directory is.</p> ./configure --prefix=`pwd`/install make install </code></pre> Your project will now be installed in a directory in your source tree named ./install. This is also a nice way to test that the install for the project is working correctly as you can easily see everything that gets installed. Now, the program has been compiled to reference extra files in ./install and will run just fine from the build directory.</p> The Linux kernel container_of macro 2008-04-02T00:00:00+00:00 The Linux kernel contains a very useful macro named “container_of” that is used extensively in back-casting a data structure to its containing data structure. This article includes a simple program that illustrates how this macro is used, and explains why it is so useful.</p> If you do a lot of C programming, this program is worth figuring out :-).</p> The program source can also be downloaded from the following location: /linux/linux_container_of_example.c</a></p> /* test code to illustrate use of Linux kernel container_of macro * * Copyright (c) 2008 Cliff Brake, BEC Systems LLC * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. * / #include <stdio.h> #include <string.h> #include <stdlib.h> / This program illustrates how the container_of macro works. * The container of macro is very useful in multi layered * software systems where you have progressivly more detailed * software layers. Below is an example of a bus layer, * and then a device layer where a number of different * devices might register with the bus. * The device registers itself with the bus subsystem, and * then the bus subsystem makes a callback into the device. * Normally if there are multiple devices registered, the * bus subsystem must store and pass a device structure * when making callbacks. With the container_of macro, this is * no longer necessary, and the bus subsystem only has to * know about one generic device structure, and does not need visibility * into lots of different device structures, or do tricks * by casting void pointers, etc. With the container_of macro * we can backcast from the generic data structure, to the containing * datastructure. This forces good separation of code in that * that bus layer cannot modifiy data structures that are specific * to the device layer. * / /* * (from Linux kernel source) * container_of - cast a member of a structure out to the containing structure * @ptr: the pointer to the member. * @type: the type of the container struct this is embedded in. * @member: the name of the member within the struct. * / #define container_of(ptr, type, member) ({ \ const typeof( ((type )0)->member ) __mptr = (ptr); \ (type )( (char )__mptr - offsetof(type,member) );}) #define offsetof(TYPE, MEMBER) ((size_t) &((TYPE )0)->MEMBER) /========================================================== BUS layer code ==========================================================/ /* generic bus device structure / struct bus_device { int general_device_param_a; int general_device_param_b; void (device_callback)(struct bus_device * bd); }; /* the following is a global list of * devices that have registered with the * bus subsystem. Normally this would * be something like a dynamic linked list. / struct bus_device bd_list[5]; /* function to register a device with the bus / void register_with_bus(struct bus_device bd) { /* since this example only deals with one * device, will put it in slot 0 / bd_list[0] = bd; } void start_bus() { int i; struct bus_device bd; /* make callbacks to all devices on bus / for (i=0;i<sizeof(bd_list)/sizeof(bd_list[0]);i++) { bd = bd_list[i]; if (!bd) continue; / call device callback with generic * bus device structure / bd->device_callback(bd); } } /========================================================== * device X specific code * this would normally be in a different module ==========================================================/ /* structure that holds device X specific stuff, as well as * generic bus_device structure / struct device_x { int device_x_specific_param_a; int device_x_specific_param_b; struct bus_device bd; }; void device_x_callback(struct bus_device bd) { /* if we know the structure type that contains the bus_device structure, * we can extract a pointer to the containing structure using the container_of * macro / / ptr type member / struct device_x devx = container_of(bd, struct device_x, bd); /* the above statement expands to * struct device_x * devx = ( * { * const typeof( ((struct device_x )0)->bd ) __mptr = (bd); * (struct device_x )( (char )__mptr - ((size_t) &((struct device_x )0)->bd) ); } * ); / printf("device_x_callback called!, device_x_specific_param_a = %i\n", devx->device_x_specific_param_a); } void device_x_init() { / dynamically allocate structures / struct device_x devx = malloc(sizeof(devx)); memset(devx, 0, sizeof(devx)); /* set a parameter in the device_x structure so * we can test for this in the callback / devx->device_x_specific_param_a = 1001; / set up callback function / devx->bd.device_callback = device_x_callback; / we register the generic bus device structure * as the bus layer does not need to know * about the device_x stucture. Note, the * devx structure is not stored anywhere, yet * its location is being preserved without * specifically passing it to the bus * layer. / register_with_bus(&devx->bd); } int main() { / test the above system / / first, initialize device_x / device_x_init(); / now, start the bus. This should make * a callback into the device_x / start_bus(); } / when run, this program returns: * device_x_callback called!, device_x_specific_param_a = 1001 / </code></pre> Should you be using monotonic timers? 2008-03-19T00:00:00+00:00 In a previous article</a>, I covered some of the basics of Linux timers. Any time you are doing any type of fixed time delay in a program, you should really be using monotonic times, so the delay will not be affected by system time changes. In an effort to save cost, some embedded systems today do not have a battery backed up RTC, and instead get the time via GPS, NTP servers, or other clever means. What this means is your applications had better be able to handle the system time changing as the system time may not be set until well after the unit boots. This article describes how you can quickly test your system for timer problems.</p> There are two cases where delays may fail if you are using non-monotonic timers. The first is if the time advances forward by a large amount. Delays will expire immediately in this case. The other case is if the time advances backward by a large amount. Delays will never expire in this case. To test for these situations, write a simple test application (sample included below) that rapidly changes the time and then test your application while the time is changing.</p> #include <stdio.h> #include <time.h> int main() { time_t system_time; static int count = 0; printf("Starting test application...\n"); while(1) { // Sleep for 0.01 seconds usleep(10 1000); system_time = time(NULL); system_time += 30; stime(&system_time); printf("Cycle %d - system date/time set to %s", ++count, ctime(&system_time)); } } </code></pre> Compulab cm-x270 kernel update to 2.6.24 2008-02-29T00:00:00+00:00 The kernel support</a> in OpenEmbedded for the Compulab cm-x270 has been updated to version 2.6.24. The 2.6.24 kernel opens up several possibilities including better real time and high resolution timer</a> functionality, an improved SD Card driver, and a full SDIO stack with support for SDIO Wifi devices</a>. Also included is a patch to set the default framebuffer depth to 16 for several commonly used displays. Xfbev really does not like when you give it a 8-bit framebuffer when it is expecting 16, and typically returns a very unhelpful error message like: “error: Invalid argument”.</p> GESBC-9302E kernel update to 2.6.24, and reboot fix 2008-02-18T00:00:00+00:00 I recently updated the GESBC-9302 machine support in OpenEmbedded to include the 2.6.24 released kernel. Also, Glomation has kindly provided me with a patch for the software reboot problem which I have integrated into the OpenEmbedded build for the GESBC. The patch files are part of the OpenEmbedded dev branch or can be obtained here:</p> http://cgit.openembedded.net/cgit.cgi?url=openembedded/tree/recipes/linux/linux-2.6.24/gesbc-9302</a></p> Other related OpenEmbedded bits:</p> http://cgit.openembedded.net/cgit.cgi?url=openembedded/tree/recipes/linux/linux_2.6.24.bb</a></li> http://cgit.openembedded.net/cgit.cgi?url=openembedded/tree/conf/machine/gesbc-9302.conf</a></li> </ul> Future features to be integrated include RTC driver configuration, and support for loading Initramfs images from NOR flash.</p> Glomation GESBC-9302E Review 2008-01-29T00:00:00+00:00 Recently, I’ve had the opportunity to work with a Glomation GESBC-9302E single board computer. This SBC contains a Cirrus Logic EP9302 ARM processor and options for plenty of RAM and Flash memory which means you have many options for writing your applications including high level languages like C# and Python. In this review, I’ll cover some of basic features of this board and where this board might be useful.</p> Why the GESBC-9302?</h3> One of the most attractive features of the GESBC-9302E is cost. The base model with 32MB of RAM, and no NAND flash costs $95 at quantities of 1 unit. For a model with 64MB of RAM, and 128MB of flash, the cost is $120. Looking at the board, its simplicity is very evident. There is basically a CPU, flash, ram, a few linear power supplies, and a very limited amount of support circuitry. There really is not much to it.</p> </p> We are helping a customer use the GESBC-9302E in a project where it is functioning as a gateway between a local Zigbee wireless network, and a server located on the internet. The customer had originally considered using a Linksys NSLU2 for the gateway device during the alpha testing, but upon consideration it seemed like the GESBC-9302E is a better choice in this application for the following reasons:</p> the GESBC does not cost a lot more than a NSLU2</li> you can buy the GESBC with plenty of NAND flash.</li> the GESBC provides two serial ports</li> the GESBC provides many signals on headers than can be used with cables or a daughter card.</li> </ul> Having raw NAND flash on the board is a significant advantage in my opinion. I’ve run into considerable hassles with consumer grade flash devices in the past (SD, USB Flash, etc). Although there are industrial grade flash devices, they tend to be more expensive. My preference at this point for reliable storage in an embedded Linux system is JFFS2 on a 64-128MB raw NAND flash device. With 128MB of NAND flash, there is plenty of space</a> for a full featured Linux distribution.</p> With 2 USB ports, plenty of general pupose I/O pins (GPIO), and lots of memory, there is not a lot you can’t do with the GESBC in your typical headless control or gateway type application. Being able to interface to a number of USB peripherals, and support for high level languages makes this SBC+Linux ideal for implementing a device that may need to talk web services or other complex network protocols.</p> Software Support</h3> Most of the GESBC functionality is fairly well supported in the mainstream Linux kernel. Simply configure the kernel for the EDB9302 machine and most things will just work. There are several pieces missing from the mainstream kernel for the GESBC that are useful: NAND flash support, and a change needed to make the Ethernet work. For the NAND flash, I extracted a MTD driver from the Glomation kernel source code. The Ethernet change is a small tweak to the platform files. These changes are available for the 2.6.24 kernel in a public git repository at: http://dev.bec-systems.com/linux-2.6-gesbc-9302.git.</p> The GESBC uses Redboot for the bootloader. No complaints with the bootloader so far – it does everything I need including loading images from a TFTP server. The bootloader does not support writing images to NAND flash, so to load an OS image into NAND flash you typically have to boot a Linux OS from NOR flash, USB flash, or NFS root, and then use the userspace mtdutils to write an image into NAND flash.</p> Glomation provides a sample Linux image, but I prefer to use images generated by OpenEmbedded so that I have support for things like C#</a> , Python, and thousands of other packages. Support for the GESBC has been added to OpenEmbedded and I’m in the process of updating the kernel support to 2.6.24.</p> Summary</h3> Overall I’m very pleased with this SBC. Glomation has been very responsive to requests and has been a good company to work with. I still need to apply a fix for a known soft reboot issue with the 2.6 kernel and enable a driver for the RTC on the SBC. Future articles will detail how to load Linux images built with OpenEmbedded on this device.</p> Marvell Embedded SDIO Wifi Success 2008-01-09T00:00:00+00:00 As detailed in the article I wrote back in September of 2007 (/web/content/view/75/9/</a>), getting embedded wifi modules functioning is not a simple task. However, due to recent advances in the Linux kernel, it looks like a viable solution for low-mid volume products is emerging. This article provides a few details on how to get a Zcomax (Zcom) XG-180MU module working with a PXA270 processor.</p> System Setup</h3> The setup used includes:</p> Zcomax XG-180MU SDIO Wifi module</li> Compulab cm-x270 + sb-x270 system (includes the PXA270 processor)</li> Linux kernel 2.6.24-rc5</li> OpenEmbedded</li> </ul> Support for the Marvell 8385 and 8686 based Wifi chipsets is now included in the mainstream Linux kernel 2.6.24 release candidate source code. To use this driver, enable the CONFIG_LIBERTAS_SDIO option. I also had to enable the CONFIG_WIRELESS_EXT option to prevent a compile error, but this is probably required anyway to build a functioning wireless system.</p> Firmware</h3> The firmware for the Marvell SDIO wifi solutions is downloaded at runtime, so the next challenge is to find firmware files. I am working on a project that has access to a driver from Marvell, so I was able to extract this firmware from their source code header files. I’m sure there are other places where this firmware is available. The firmware files must be placed in the following location:</p> root@cm-x270:~$ ls /lib/firmware/sd* /lib/firmware/sd8385.bin /lib/firmware/sd8385_helper.bin </code></pre> The version of udev used in OpenEmbedded determines the location of the firmware files. The Libertas Linux driver requests these firmware files, and a userspace component of udev provides these files to the kernel.</p> Success</h3> The following kernel modules are required:</p> root@cm-x270:~$ lsmod Module Size Used by pxamci 7008 0 libertas_sdio 8488 0 mmc_core 46868 2 pxamci,libertas_sdio libertas 85288 1 libertas_sdio ieee80211 29956 1 libertas ieee80211_crypt 4768 1 ieee80211 root@cm-x270:~$ </code></pre> And the result:</p> root@cm-x270:~$ modprobe pxamci root@cm-x270:~$ mmc0: new SDIO card at address 0001 libertas: eth1: Marvell WLAN 802.11 adapter </code></pre> ifconfig -a </code></pre> ... </code></pre> eth1 Link encap:Ethernet HWaddr 00:60:B3:34:77:66 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) </code></pre> Very nice!</p> So it looks like there is finally an embedded Wifi solution for products that sell less than 500,000 units per year. I also briefly tried this on a AT91SAM9260 system, but it did not work. So there is still some work to do on the Linux SDHC driver for the AT91SAM9260.</p> C# on embedded ARM Linux systems is now practical 2007-12-15T00:00:00+00:00 Update</strong>: Michael Dominic K. has kindly provided feedback on this article and has some additional points</a> about Mono and embedded systems.</p> Mono has included support for ARM systems for some time, but with the release of Mono 1.2.6, and its inclusion in OpenEmbedded, it is now practical and easy to build and run C# applications on your everyday ARM Linux system. Several ARM issues in 1.2.5 have been fixed, and thus far it seems the average program runs quite well. In this article, we’ll explore what advantages C# offers over more traditional languages such as C or Python, and how OpenEmbedded simplifies adding C# support to your embedded system.</p> Why C#</h3> Why is C# an attractive language for embedded Linux systems? There are many reasons including memory protection and exception handling, native types, good threading support, extensive library, and developer productivity. In the past, I’ve typically used C or Python for implementing large applications on embedded Linux systems. This section will compare C# to my experiences with both languages.</p> A typical experience when implementing a large application in an embedded system is to deploy the system for field testing (before it is ready) and start getting reports that the application is crashing. Somewhere in the mass of C code your team just wrote there is a stray pointer clobbering something it should not. Of course this does not happen in the lab where you have access to the machine, but in some piece of industrial equipment many miles away. And, you cannot reproduce the problem in the lab because with embedded systems, simulating a real world situation is challenging because it is difficult to simulate real I/O. This is less of a problem with desktop applications where you are less dependent on external I/O. Sound familiar? Even if you can reproduce the problem on a machine you have access to (perhaps even remotely</a>), the problem can be caused by a stray pointer corrupting data in a completely unrelated piece of code. So while the debugger may tell you what got clobbered, you still don’t know exactly who did it. If you have lots of time and a team to do lots of testing, this is less of an issue, but available resources and time to market sometimes don’t give you all you need to do the job right before it gets to the field. One solution to this problem is use a language that has exception handling (I’ve used Python a lot in the past</a>), and log all exceptions. If something bad happens in the field, I ask for the log. With the stack trace in the log, and some focused testing I can usually find the problem relatively quickly. Add in the memory protection and things get a lot easier. For small applications this is not so much of an issue, but lets face it, now that we have access to very capable hardware at a low cost</a>, applications in embedded systems tend to get large and complex. C# gives you exception handling and memory protection that makes problems much easier to find and debug; especially during field testing.</p> While its hard to beat Python or Ruby for developer productively and ease of use, there are performance issues like threading support, and support for static types that must be weighed in the balance. With Python, everything is an object or a reference. The checking is done at run time to make sure a variable is of the correct type. With C#, this is done at compile time (much like C). While it is generally more work to write code in C# than Python, the compiler tends to catch more errors for you. There is probably also a performance advantage to being able to implement simple variables statically. Threading support is another consideration and is often critical in embedded systems because you are usually tending to I/O, processing data, running a user interface, etc. This type of problem is well suited to a threaded application. Threading in Python is implemented using a global interpreter lock mechanism that is inefficient in some situations. C# threading support is more similar to C. While it may require you to do more manual locking than Python, it gives you more control.</p> When it comes to developer productivity, in my experience C# is more work to write than Python or Ruby, but less work than C. So it seems to fall somewhere in the middle of the spectrum.</p> So, C# seems to fall nicely into the middle ground between C and very high level languages such as Python and Ruby. It offers many of the advantages at both ends of the spectrum and seems to be a nice balance for programming embedded systems.</p> Mono</h3> What is Mono? From the Mono web site</a> :</p> Mono provides the necessary software to develop and run .NET client and server applications on Linux, Solaris, Mac OS X, Windows, and Unix.</em></p> Mono includes all the necessary components to build and run C# applications on Linux systems including embedded ARM Linux systems. The Mono source tree is fairly clean and is nicely segmented between native and managed code. This is critical when it comes to cross compiling. The Mono team has also stuck with traditional build tools such as autotools which makes the project a lot easier to cross compile for an embedded target.</p> Mono includes a very extensive C# library that is comparable to Microsoft’s implementation. Most of what you might ever need is built right into the Mono class library; much like Python or Java. With C or C++, you typically need to add a number of additional libraries to the system.</p> Mono Support in OpenEmbedded</h3> OpenEmbedded includes comprehensive support for Mono including crosscompiling support, granular packaging of the Mono class library, and automatic run-time dependency generation for Mono applications. In this section, we’ll see how OpenEmbedded can make your life a lot easier.</p> Mono is typically cross compiled in two steps</a> :</p> The managed portions are generated by compiling Mono for the host machine architecture. A very nice feature of the Mono build process is that it can bootstrap itself.</li> The native only portion of Mono is then cross compiled over top of the above output and the native host binaries will then get replaced with the cross compiled target binaries.</li> </ul> To implement this, there are two recipes implemented in OpenEmbedded. The mono-mcs-intermediate</em> is used to generate all the managed binaries, while the mono</em> recipe builds only the native portion of Mono. The first step when building the mono</em> cross recipe is to extract the results of the mono-mcs-intermediate</em> in the install directory and then proceed to overlay the native binaries produced by the mono</em> recipe in that same directory. As mentioned before, the use of standard tools like autotools makes all this work pretty well. All of this is automated in OpenEmbedded and happens when you run “bitbake mono”.</p> One issue with embedded systems is you typically have limited flash/disk space. So you only want to include that pieces of Mono in your target image that your application needs. The OE build of Mono makes this very easy by packaging up the Mono class library into a number of different packages. The dependencies between packages is automatically tracked so that if you add a Mono package to an OE build, its dependencies will automatically be included with no extra work on your part. This mechanism uses the monodis</em> tool to figure out what each Mono library provides and the dependencies of each library. This saves a huge amount of very tedious manual work to have all this automated.</p> Building and Running your Mono application</h3> Now that we have Mono running on our target system, there are a number of ways to build Mono applications to run on the target. The quick-n-dirty method is to simply build the mono binary on any system, copy it to your embedded system and run it. Building and running your applications is easy as Mono binaries are platform independent and will run on any system that supports Mono. So, you don’t need to worry about cross compiling, toolchains, etc.</p> The quick-n-dirty method works well for getting started, experimenting, etc, but there are several advantages to integrating your Mono application into an OpenEmbedded build, even though you technically don’t need to worry about cross compiling. The first reason is packaging. A very simple OpenEmbedded recipe provides an easy way to automatically fetch your project from a separate version control system, build it, and package it properly into the OS image for your target. It also gives you the ability to deploy updates to target systems using the ipkg package manager. The second reason is automatic run-time dependencies. If you inherit the mono</em> OpenEmbedded class, all of the Mono components your application needs are automatically added to the image. An example OpenEmbedded recipe for a Mono application is shown below:</p> DESCRIPTION = "My Application" DEPENDS = "mono" SRCREV = "${AUTOREV}" PV = "1.0+svnr${SRCREV}" PR = "r1" SRC_URI = "svn://svn.mycompany.com/svn/myproject/;module=myapplication;proto=https" S = ${WORKDIR}/myapplication </code></pre> inherit autotools mono </code></pre> The above recipe assume you have set up your application to be built with autotools, which is a good idea. This page</a> provides lots of good information on building Mono applications using autotools. You can also look at a number of open source applications such as Tomboy</em> for additional clues about how to use autotools with Mono. For a simple application that I built, the following run time dependencies where automatically determined by OpenEmbedded: mono, libmono0, libmono2.0-cil, libmono-corlib2.0-cil.Each of these dependencies may have additional dependencies, and in this case a dozen or so mono packages ended up being installed.</p> Summary</h3> As embedded systems become larger and more capable, we can expect that higher level languages such as C# and Java will start to become more common. In the past, the focus on embedded systems programming was often efficiency, minimizing resource utilization, etc. Now that very capable hardware is becoming cheaper</a>, the focus in some systems is shifting to features and time to market. Operating systems like Linux and languages like C# help us reach those goals and develop competitive products quickly with small teams. C# provides the features needed to rapidly develop, deploy, and debug complex applications that still perform reasonably well. OpenEmbedded provides a way to manage builds, so that you can focus on real work instead of tedious build issues.</p> Mono 1.2.6 added to OpenEmbedded 2007-12-14T00:00:00+00:00 I recently updated the Mono recipes in OpenEmbedded to include the Mono 1.2.6 version. Several critical bugs that were present in 1.2.5 have been fixed, and its looking very good. More on this later …</p> Embedded Linux development on Windows??? 2007-11-30T00:00:00+00:00 From http://www.embedded.com/design/opensource/204300381</a> :</p> Host platform</strong> Although everything I will describe can be also done on a Windows host, I recommend using Linux. It’s more convenient, and more tools and utilities are available. And if you rely on a few Windows applications such as Word and Outlook, you can still run them on Linux in emulation using VirtualBox, Wine, or other commercial package. If you’re new to Linux, using a Linux host will also force you to learn the new platform faster.</em></p> </blockquote> How true. Those who think they can do embedded Linux development without learning the basics of using a Linux desktop computer are in for a long hard road, and they are missing out on a valuable opportunity to learn a lot about embedded Linux development. This is the primary advantage of embedded Linux – there are many things in common with desktop and server systems. There is a tremendous amount of reuse. Things can be easily prototyped on your desktop Linux system before deploying to an embedded system. Instead of viewing the suggested that a Linux workstation be used for embedded Linux development as stumbling block, it should be viewed as a huge advantage.</p> Compulab EM-X270 Review 2007-11-07T00:00:00+00:00 The EM-X270 is a full featured computer board from Compulab (http://www.compulab.co.il/x270em/html/x270-em-datasheet.htm</a>) designed for handheld/mobile applications. The board includes options for about anything you might need in a portable system including a PXA270 processor, GSM radio, GPS, Wifi, Bluetooth, Audio, SD/MMC, USB, battery circuitry, and the list goes on. The EM-X270 is designed to give you a big head start in designing a handheld computing device, and can significantly reduce time to market and design costs. This article provides a high level overview of this product, how it might be used, and lots of photos.</p> EM-X270 Overview</h3> The EM-X270 is a computer board about the size of a display you might find in a typical PDA. While the size is probably a little large to be used in consumer devices like phones and PDAs, it seems like a nice size for slightly larger products used in vertical industries.</p> </p> As already mentioned, the EM-X270 includes about all the base features you might need in a mobile computing device. An expansion connector is also included that allows you to easily add custom circuitry to the system. The stack-up between the EM-X270 and an expansion board is shown below:</p> </p> The EM-X270 differers from most other embedded computer modules in that it is a little more application specified, and quite a bit more circuitry is included in the design such as power supplies, battery charging circuitry, radios, etc.</p> Why EM-X270?</h3> Why are we interested in the EM-X270? If all you need is a generic handheld computer, there are plenty of options from PDAs to off-the-shelf industrial handhelds. However, if you need a handheld that provides some additional features not typically found in existing devices, then you can use the EM-X270 plus a custom baseboard and packaging to add your high value features. Depending what you need to add, the expansion board can be a simple 4 layer board which is very easy to design and manufacture. All of the complex PCB design and assembly is already done for you on the EM-X270 module. This allows low to moderate volume manufactures design products where they would not otherwise have the time and budget for designing a full custom processor board. Some example applications might be handheld products for the medical, industrial, and test and measurement industry.</p> Compulab’s pricing model is also very interesting. Compulab will custom build EM-X270 modules with just the options you need. The base price without any extra features (such as radios) is $122 for 1000 pieces.</p> The EB-X270 Evaluation Kit</h3> Compulab provides a EB-X270 Evaluation kit for getting started with EM-X270 development. Like their other Evaluation klits, is is very full featured and very well done. An extender board is provided that adds Ethernet, JTAG, and several other useful connectors for development.</p> </p> The eval kit also includes all necessary cables, a display, battery, etc.</p> </p> Software for the EM-X270</h3> Compulab provides software support for both Windows CE and Linux. This includes very up-to-date support for kernel version 2.6.23. EM-X270 machine support has also been included in the OpenEmbedded project for building full featured Linux images. This combination will drastically reduce development costs compared to solutions provided by many other vendors. All too often, single board computer vendors provide an ancient kernel, a hacked up root file system and call it a “Linux BSP”. Actually, any time the term “Linux BSP” is used, you should probably be a bit wary. A much more attractive solution for those of us building products using Linux is for processor and module vendors to say “our products are supported in the mainstream kernel and OpenEmbedded sources”. Compulab is doing it right.</p> OpenEmbedded provides many interesting options for devices like the EM-X270. One of the most interesting might be the ability to run the OpenMoko software on industrial devices.</p> Compulab provides a Linux demo image built with OpenEmbedded that runs the GPE Palmtop Environment. The method for loading this image is interesting, and similar to the method I have been using for some time with systems based on the cm-x270 module (/web/content/view/82/9/</a>) in that it used mtd-utils from a running Linux system to erase and write to the NAND flash. The bootloader in the EM-X270 is able to boot a kernel and rootfs directly from a USB Flash disk. This “LiveDisk” image, as Compulab calls it, contains a small utility that programs an image from the USB flash disk directly into NAND flash.</p> As with any modern Linux based embedded system, there is a lot going on in a very small package. Managing this complexity effectively requires some amount of experience. Stay tuned for future articles that discuss using OpenEmbedded to build software for the EM-X270.</p> Additional Photos</h3> </p> </p> </p> Linux 2.6.23 for the Compulab cm-x270 2007-10-22T00:00:00+00:00 Updated 2.6.23 kernel patches for the Compulab cm-x270 are now available in OpenEmbedded:</p> http://www.openembedded.org/filebrowser/org.openembedded.dev/packages/linux/linux-2.6.23/cm-x270</a></p> The cm-x270 support in OpenEmbedded has undergone a lot of clean-up in recent weeks. Now is a good time to give OpenEmbedded a try if you need a full featured Linux distribution for the cm-x270.</p> Compulab cm-x270 NAND flash eraseblock sizes 2007-10-08T00:00:00+00:00 If you are having trouble mounting a JFFS2 file system on a Compulab cm-x270 module, you may have a device that has a NAND flash with 16KiB eraseblocks. All of the devices I have personally used to date have had 128KiB eraseblocks, but I just helped a cm-x270 user through some flash issues and after much pain discovered the eraseblock size was 16KiB on his module. A variable has been added to the cm-x270.conf file in OE to set the eraseblock size that can be overridden in your local.conf file if you are using OpenEmbedded to generate jffs2 images:</p> http://git.openembedded.net/?p=openembedded.git;a=blob;f=conf/machine/cm-x270.conf;h=3c21c546b1320914c9b5d2c44e7332782fc58748</a></p> Howto load Openembedded on the Compulab cm-x270 computer module 2007-10-08T00:00:00+00:00 There are quite a few people using (or would like to use) OpenEmbedded on the Compulab cm-x270. The cm-x270 is a high performance, low cost computer module that can be used with a custom baseboard in embedded systems (see /web/content/view/62/9/</a> for a review). I get a lot of questions on how to load OpenEmbedded on this system, so this howto attempts to detail a procedure that can be used to load OpenEmbedded.</p> The NAND Flash wrinkle</h3> The Linux solution provided by compulab includes a proprietary NAND flash driver. Most developers are interested in using standard Linux MTD drivers with the JFFS2 filesystem, but its not readily obvious how to flash a JFFS2 NAND image using the cm-x270 bootloader. Fortunately, you can boot a small Linux image stored in NOR flash, and then use standard MTD utilities to then flash the NAND jffs2 image.</p> Creating the needed images</h3> For this method, you need to generate two images: a small image to run from NOR flash, and a larger full featured image that will run from NAND flash. Because you only have about 2.3MB of NOR flash, you want to use uclibc in the NOR image as it is much smaller. A base Linux image that will boot into a shell using uclibc and contains a few utilities is about 1MB in size, which leaves plenty of space for custom apps should you need them in this partition. The local.conf settings to build this image are:</p> MACHINE = "cm-x270" DISTRO = "angstrom-2008.1" ANGSTROM_MODE = "uclibc" IMAGE_FSTYPES = "jffs2 tar cpio.gz" </code></pre> At this point, “bitbake angstrom-minimal-image-with-mtd-utils”, and OE will generate a “minimalist-image-mtdutils-cm-x270.cpio.gz” image that can be programmed into the NOR ramdisk partition using the cm-x270 bootloader, and tftp download. A kernel will also be built that should be programmed into the kernel partition.</p> To build the NAND image, simply remove the ANGSTROM_MODE line, and “bitbake angstrom-console-image”. This will generate a “console-image-cm-x270.rootfs-summary.jffs2” that can be programmed into NAND flash.</p> Programming Images</h3> Loading the software onto the cm-x270 is a multi-step process. The first step is to the load the kernel image and the minimalist-image into NOR flash using the cm-x270 bootloader and tftp. After this is accomplished, run the following commands to boot into the NOR flash rootfs:</p> ramdisk on</li> bootos</li> </ul> If successful, you will be presented with a prompt – log in as user root, and no password.</p> You now need to download the jffs2 NAND image into RAM on the cm-x270. One way to do this is to use the tftp download feature in busybox:</p> tftp -g -r console-image-cm-x270.rootfs-summary.jffs2 <tftp server IP> </code></pre> At this point you can program the jffs2 image into NAND flash:</p> flash_eraseall -j /dev/mtd3</li> nandwrite /dev/mtd3 console-image-cm-x270.rootfs-summary.jffs2</li> </ul> After programming completes, reboot the system and enter the following in the cm-x270 bootloader:</p> ramdisk off (this will disable the initramfs in NOR flash)</li> bootos</li> </ul> At this point, your system should boot into the main NAND rootfs. With a little imagination, it should be obvious that you can do a lot of neat things with this type of setup by extending the functionality in the NOR rootfs including field upgrades and system recovery.</p> Mono support in OpenEmbedded and Openmoko packages 2007-10-04T00:00:00+00:00 Original article published</em>2007-10-04</p> As of today, the Mono build in OpenEmbedded is working pretty well. Mono support for ARM and other embedded architectures should add some interesting capabilities for embedded Linux devices. Having written a number of large embedded Linux applications, I can really appreciate the advantages of high level languages for exception handling. It is also interesting to note that Java support in OpenEmbedded is making progress as well (http://jalimo.org/wiki/doku.php</a>). Windows CE developers have already taken to the .net compact framework in a big way, so its nice to see similar capabilities for embedded Linux systems.</p> A feed has been set up for installing mono packages on your Openmoko phone. Add the following line to a /etc/ipkg/-feed.conf file:</p> src/gz mono-armv4t http://dev.bec-systems.com/feed/openmoko/mono </code></pre> And then:</p> ipkg update </code></pre> ipkg install mono </code></pre> The packaging is fairly granular, so look at the listing in the above URL to see a complete list of what is available.</p> How to implement realtime periodic tasks in Linux applications 2007-09-25T00:00:00+00:00 (see also should you be using monotonic timers?</a>)</p> Have you ever wondered what is the best way to implement periodic tasks in Linux applications – something better than usleep()? This article covers a number of issues related to this subject including real-time tasks, the different timers available, timer resolution, and how to implement periodic tasks accurately so that error is not accumulated. The recent inclusion of the high-resolution timers in the mainstream kernel makes this a very interesting subject.</p> High Resolution Timers</h3> Historically Linux has done all timing off the OS timer tick which is usually between 1ms and 10ms. While this is adequate for many tasks, it is really nice to have a high resolution timer for timing tasks. With the integration of the high resolution timers into the mainstream Linux source tree, this is now possible. From a user space perspective, there are no API changes. The only difference you will notice is that now you can sleep for less than OS timer tick period. clock_getres() can be used to check the timer resolution and will tell you instantly if you have high resolution timer support. If the clock resolution is 1ns, you have high res timer support. Realistically, you can’t delay for 1ns in a Linux application, but delays in the range of 100us should be possible, and depending on the configuration, much better performance is possible. The kernel config entry for high resolution timers is CONFIG_HIGH_RES_TIMERS.</p> The difference between CLOCK_REALTIME and CLOCK_MONOTONIC</h3> Some of the Linux timer functions take a clockid_t</em> argument that can be specified as CLOCK_REALTIME, or CLOCK_MONOTONIC. The big difference between these two is that changing the system time will have an affect on CLOCK_REALTIME, thus affecting your timers. Changing the system time will have no affect on CLOCK_MONOTONIC – it will always count upward. For periodic tasks, CLOCK_MONOTONIC may be more applicable. The best way to get burned using CLOCK_REALTIME is when your application takes a timestamp, does something, takes another time stamp and then compares them. If you are using CLOCK_REALTIME, and the system time gets changed between the two timestamps, your comparison will not be valid. For most timeouts and relative timekeeping in Linux, use CLOCK_MONOTONIC. This issue becomes more important as many systems now have a process that periodically sets the time automatically from network time servers, and you have no idea when this might happen.</p> Preemption</h3> Kernel preemption (/web/content/view/69/9/</a> ) makes a huge difference in the performance of real time applications by allowing the kernel to be preempted by higher priority application processes. Historically, any kernel code that was runnable ran before the kernel returned control to applications. This all changes with kernel preemption. Kernel preemption has been available in mainline kernels for some time now (I think since 2.6.16). The worst offender I’ve found for locking the kernel for long periods of time have been flash drivers – especially proprietary ones. But, even jffs2 can cause problems in realtime applications without kernel preemption.</p> The PREEMPT_RT patch</h3> Much of the realtime work being done for Linux is maintained in the PREEMPT_RT patch. Bits of this patch have already been merged into the mainline kernel, but there is still a lot of very useful functionality in the patch and it should be considered if you are doing any type of realtime work. More details will be presented in a future article.</p> Absolute vs Relative timekeeping</h3> When implementing a periodic task, you really want to base your timekeeping off an absolute time, versus relative delays such as usleep(). It is not possible to achieve precise periodic activation with a relative sleep such as usleep(). The reason for this is you must first get the current time, make a calculation to determine how long to sleep, and then call the relative sleep function. If your process gets preempted between the time you acquire the timestamp, and the sleep, your relative sleep time will probably be wrong. This problem is solved by using the clock_nanosleep() function. clock_nanosleep() can be called with the TIMER_ABSTIME value in the flags argument. If the TIMER_ABSTIME flag is set, then clock_nanosleep() will sleep until the absolute timer value is reached. It does not matter if you get preempted between the time you take a timestamp, and the sleep function.</p> Summary</h3> In summary, if you want to implement accurate, realtime, periodic tasks in a Linux application, consider the following:</p> Use high resolution timers. Although Linux is still limited in its response time, at least the scheduling resolution is now quite high.</li> Enable kernel preemption. This makes sure long kernel processes don’t get in the way of your real-time application process.</li> Use the CLOCK_MONOTONIC for relative timekeeping. You don’t want your application to lock up due to the system time changing.</li> use the clock_nanosleep() function instead of relative delays like usleep(). This is the only way to accurately schedule periodic tasks.</li> if needed, apply the PREEMPT_RT patch.</li> </ul> The clock_nanosleep man page also includes a lot of useful information about timer functions.</p> The Embedded WiFi module Quest 2007-09-19T00:00:00+00:00 original article written in Sept, 2007</em></p> How does one implement WiFi functionality in vertical, low volume portable products? This is a good question, and one I’ve been struggling with for the past 3 months. I have a customer who is designing a portable data acquisition system based on a AT91SAM9260 processor, and needs WiFi functionality. The fundamental problem is that no one has time to deal with low volume customers, and the task of implementing WiFi functionality is obviously complex. At volumes of 5000 units a year or less, it makes a lot of sense to go with a WiFi module rather than trying to integrate the WiFi chipset on the board. This article covers some of the options we have looked at and some of the possible solutions.</p> How is WiFi done on x86 systems?</h3> With desktop Windows, most devices are well supported by device manufacturers. With x86 Linux some of the devices are well supported by OSS drivers, but these tend to be PCI or USB devices. Some of the devices that are not supported can still be used by running the Windows NDIS drivers inside of ndiswrapper, which allows you to run the Windows driver on a x86 Linux system, but this obviously does not work very well on non-x86 processors such as ARM.</p> WiFi solutions for portable, embedded systems</h3> While there are many off-the-shelf solutions for PCs, Notebooks, etc, the current solutions are generally in the USB formfactor. The ideal solution for portable systems running non-x86 processors are the small modules that connect to the host processor using the SDIO or SPI bus – like the modules you would find in newer cell phones and PDAs. Older modules tend to use the Compact Flash interface, but these modules tend to be larger and are not packaged as nicely for deeply embedded applications. So Ideally we want:</p> a modules with a SDIO or SPI interface</li> packaging options such as solder down for robust packaging</li> </ul> The scope of this article is limited to modules that implement a SPI or SDIO interface. Why are we interested in modules? Putting a WiFi chip down on a custom PCB is a lot of work. Because the WiFi silicon includes a processor, if things do not work, it may be very difficult to debug. There is also the issue of factory calibration which usually requires proprietary PC based tools that can communicate directly with the WiFi silicon. If the WiFi silicon is buried in your product, getting all this to work can be a challenge. The test equipment required for calibration is expensive. Add to this the difficulty in getting support from anyone who makes WiFi silicon …</p> Complexity</h3> It should be noted that WiFi modules are fairly complex devices. They are typically based on a highly integrated IC that includes an ARM processor, Flash memory, and radio circuitry. The fact that these modules include a significant amount of firmware on the module contributes significantly to the complexity of getting WiFi solutions to work properly. Hopefully no bugs are encountered in the module firmware, because I can only imagine how difficult it would be to get these fixed for a small customer. The fact that the module firmware is involved in many of the WiFi functions like authentication further complicates the problem with the abundance of authentication and encryption options available for WiFi. The worst possible problem is the case where you must design an embedded WiFi system that must operate at the enterprise level in every environment. It is one thing to design a system that works most of the time in most environments where WiFi is more of a convenience (like a Cell phone or PDA). It is another thing to design an industrial grade system where it must work all the time and in all environments.</p> The Fundamental Problems with low volume embedded WiFi</h3> I have never worked with a technology with so many dead ends as embedded WiFi modules. Many of the companies I email or call never even bother to return emails and phone calls. This is certainly true of the chipset manufactures. From what I can gather, the fundamental problems are:</p> WiFi implementation is difficult; therefore, a manufacturer’s scarce resources are dedicated to high volume customers. Low volume customers are a significant distraction.</li> WiFi companies are very busy right now – they are not hungry for business from small customers.</li> WiFi solutions are very competitive and highly proprietary. Many companies are very secretive and will not release driver source code. This is certainly true at the module firmware level.</li> </ul> There are many WiFi module manufacturers and resellers out there, but very few of them offer any type of driver solution for Windows CE or Linux other than binary modules that are supposed to work on perhaps one reference platform. Sorry, this is not going to cut it. So, with plenty of hardware available, the gating item is the availability of software drivers and knowledge of how to use the software. It would seem to me that this is the perfect opportunity for chipset vendors to open source some drivers and develop vibrant support communities so that their modules can be used without a lot of hand-holding by the chipset vendors.</p> So What are the options?</h3> Atheros AR6001</h4> The Atheros AR6001 seems like a nice solution. The driver situation for the AR6001 is progressing. For Windows CE, there is an opensource WinCE 6.0 driver available at: http://www.codeplex.com/CEWifiDriverAR6000/</a>. Unfortunately, this project appears to be a snapshot of code and does not have any significant amount of community activity or development. Perhaps once more AR6001 modules are available, this will change.</p> For Linux, there are several options:</p> Atheros has released a driver that is available: http://sourceforge.net/project/showfiles.php?group_id=186068</a>. There are several issues with this driver in that it is written for a SDIO stack that will likely never be part of the mainstream kernel.</li> A driver is being developed as part of the OpenMoko project.</li> </ul> Embedded Works supplies Atheros based modules and development boards: http://www.embeddedworks.net/wlan/oem_sdio_80211g.html</a>. AR6001 development boards are also available from Cardaccess: http://www.cardaccess-inc.com/products/index.php?a=wlan_sdio</a>.</p> Marvell</h4> The Marvell 88W8385 seems to be a very popular IC in WiFi modules available from a number of different companies. OSS Linux drivers are in progress and are reportedly somewhat functional at this point, so it is probably just a matter of time before these devices are well supported in Linux. This chipset/driver is often referred to as “libertas”. The SDIO stack being developed for the Linux mainline (http://lwn.net/Articles/242747/</a>) is where support for this device is being developed.</p> For Windows CE, it seems the only option is to get the drivers from Marvell, which is a very difficult and time consuming process – at least for low volume customers. First you have to find a module reseller that can get you the source code from Marvell, and then the process takes about 3 months.</p> Other Options</h4> There are a number of other WiFi Silicon manufacturers and module vendors, but the driver options from them seem to be very limited in the form of binary only drivers that will only work with certain processors/operating systems. As noted before there is almost no hope of getting anyone to even talk with you if your volumes are low. So your only hope is finding a module vendor that can do any driver work for a NRE fee, obtain the source from the silicon manufacturer, or use OSS drivers.</p> Summary</h3> The Embedded WiFi situation is changing fast, so by the time you read this, it is probably already out of date. We are currently at the phase where it is very difficult to deeply embedded WiFi in low volume products. I expect this will change during the next year. Eventually, WiFi modules re-sellers will figure out that supplying hardware is just one side of the equation. As things get more and more complex, the software availability and support is becoming the gating item.</p> Many thanks to James Nahra, Dave Anders, and Erik Strack for sharing information and their WiFi experiences with me over the past few months.</p> The correct way to add packages to an OpenEmbedded Image 2007-08-30T00:00:00+00:00 Update 2007-10-14: use IMAGE_INSTALL in image recipe</strong></p> Update 2007-10-22: a few corrections, added full path for include and comments</strong></p> Update 2010-06-07: use recipes instead of packages directory</strong></p> As more and more OpenMoko developers are coming on-line, it is becoming obvious that my previous post (/web/content/view/59/9/</a>) about adding packages to OpenEmbedded could use some improvements. This post suggested using the DISTRO_EXTRA_RDEPENDS variable, which was intended to only be used in distro.conf files. A better approach is to create a custom image recipe that includes the packages you want. This article covers how to create a custom image recipe.</p> What are the problems with the DISTRO_EXTRA_RDEPENDS approach?</h3> As detailed by Marcin Juszkiewicz in his blog post “Why using of DISTRO/MACHINE variables in local.conf is wrong</a>”, and from discussions with OE developers, the following issues surfaced:</p> You have to remember to rebuild task-base every time you update the DISTRO_EXTRA_RDEPENDS variable.</li> The image is no longer angstrom-console-image or whatever you have extended, so when asking for support, other developers do not realize exactly what you are building. If it is a custom image, then that is obvious by the image name.</li> Changing DISTRO_EXTRA_RDEPENDS in local.conf requires that you bitbake reparse the entire recipe tree (which takes minutes). Changing a custom image recipe only requires one file to be reparsed.</li> </ul> Creating a custom image file</h3> Creating a custom image file is quite easy, and works much better than modifying DISTRO_EXTRA_RDEPENDS in your local.conf file. A custom image is created by simply copying or including an image file closest to what we want to use, and then adding additional packages to IMAGE_INSTALL variable. In the following example, we require angstrom-console-image.bb and extend it with a few lines:</p> require recipes/images/console-image.bb IMAGE_INSTALL += "\ xset xserver-kdrive-fbdev xinit \ kernel \ kernel-module-mmc-block \ kernel-module-mmc-core \ kernel-module-pxamci \ kernel-module-ac97-bus \ " export IMAGE_BASENAME = "my-custom-image" </code></pre> Note, this method requires your custom image file to be in the same location as the original image recipe. If you use a bbcollections overlay, then you may want to copy the original recipe.</p> Note the use of a full path specified in the require statement. This allows you to have your custom image recipe in a bbcollections overlay, and bitbake will still find the required recipe in the main OE tree.</p> Thats it! With the many advantages to this method, there is no reason not to create your own image recipes – it works better.</p> Howto use a Git topic branch to maintain a patchset, and isolate problems 2007-08-16T00:00:00+00:00 The more I use Git, the more I am really impressed with some of its capabilities. What is Git? Git is a very advanced distributed source code control system that can do some very neat things. For the average developer using open source software, we are mainly concerned about maintaining patch sets (often called a topic branch). Git includes a rebase capability that is very useful for a number of different operations related to maintaining a branch of code including moving a branch forward, moving a branch around on an upstream branch to look for breakage, and merging changesets to create patch files. This article provides a brief overview of some of these operations and several tips.</p> When in doubt, make a branch</h3> The most important concept to understand when working with Git is to get used to creating and destroying branches. This is different than most other SCMs where all branches are stored on a central repository, and you don’t want to clutter the branch namespace. With Git, get used to branching without even thinking about it. For example, the following are the branches in a Git repository I use to maintain patch sets for the Compulab cm-x270:</p> git branch cmx270 cmx270_2.6.20 cmx270_2.6.21 cmx270_2.6.22 master svs test v2.6.20_test </code></pre> Most of the names are self explanatory. The cmx270 branch is where I track the latest kernel version. The svs branch is for a customer platform based off the Compulab cm-x270. To view the changes on any particular branch:</p> git-log cmx270_2.6.22 </code></pre> an Git will start showing you a list of changes on that branch. This works even if you are not working on that branch, so it is very handy for seeing what changes are in other branches. With Git is also very easy to move changes between branches using git-cherry-pick</em>. For example, if we are working on the cmx270 branch, and we want to pull the 3rd changeset from the HEAD of the cmx270_2.6.22 branch into the cmx270 branch, we would do something like:</p> git-cherry-pick cmx270_2.6.22~2 </code></pre> Git is really good at maintaining patch sets</h3> Many developers use quilt to maintain patch sets. Once you understand how Git works, Git is even better at maintaining patch sets. Although there are some add-on tools available for maintaining patch files in a git repository like Patchy GIT and Stacked GIT, I’ve found that core Git works well enough. Actually the Patchy GIT maintainer has quit working on Patchy GIT because core GIT now meets his needs. The fundamentals are you simply export each Git commit as a patch using the git-format-patch command as shown below:</p> git-format-patch -o tmp/ master </code></pre> Produces the following files (all of the commits since the branch has left the master): </code></pre> tmp/0001-cm-x270-base2.patch tmp/0002-cm-x270-match-type.patch tmp/0003-cm-x270-ide.patch tmp/0004-cm-x270-it8152.patch tmp/0005-cm-x270-pcmcia.patch tmp/0006-ramdisk_load.patch tmp/0007-mmcsd_large_cards-r0.patch tmp/0008-cm-x270-nand-simplify-name.patch </code></pre> Pretty slick! HEAD~8 means to back up 8 revisions from the current HEAD and then start outputting changes as patch files. The name of the patch file is the commit comment.</p> Moving your patch set forward in time</h3> If you are maintaining a patch set for the Linux kernel, you might want to move your patchset up to the latest version for testing, and also maintain patchsets for each stable version of the kernel. Lets say we already have a patch set for the 2.6.20 version. The first thing we need to do is create a branch. In this example, I am maintaining a patch set for the Compulab cm-x270 computer module.</p> git-branch cmx270_2.6.20 v2.6.20 (branch name, tag name in repository) </code></pre> git-checkout cmx270_2.6.20 </code></pre> At this point, I would manually apply each patch, and then commit each patch as its own changeset. Now, I want to move the changeset forward to kernel version 2.6.21. The easiest way to do this is create a new branch, and simply rebase your patchset to the new kernel version. Make sure you create a new branch first, or your existing branch will be moved!</p> git-checkout -b cmx270_2.6.21 cmx270_2.6.20 </code></pre> git-rebase --onto v2.6.21 master (tag point to relocate branch, upstream branch) </code></pre> The upstream branch is required to be specified so that Git knows where our branch changes start – specifically where the cmx270_2.6.21 branch diverges from the master branch.</p> There may be a few things to clean up along the way if changes don’t apply cleanly. The most critical thing to know is that if you want to skip a patch, you must run “git-reset --hard</em>” before running “git-rebase --skip”</em>. This is often handy if you have an especially troublesome patch that you don’t want to apply now. You can skip it and then cherry-pick it later. Otherwise the process is fairly straightforward. After you are finished, you can now export a set of patch files that can be applied to the 2.6.21 kernel.</p> Bisecting the master branch to determine when things broke</h3> You may run into a situation where something broke on the main branch, and you don’t know where. Most of the time, you would just run git-bisect</em> to do a binary search to find where things broke. This does not work so well if the platform you are testing is not supported in the main branch – the changes in your topic branch are required to make the system run. Once again, git-rebase</em> to the rescue. The first thing we must do is find the mid-point between the known good and bad points.</p> git-rev-list --bisect v2.6.20 v2.6.21 13f7e5acc8b329080672c13f05f252ace5b79825 </code></pre> Now, rebase out changes to the above rev and test.</p> git rebase --onto 13f7e5acc8b329080672c13f05f252ace5b79825 master </code></pre> The build failed, so now we have a new “bad” point and can continue our binary search.</p> git-rev-list --bisect v2.6.20 13f7e5acc8b329080672c13f05f252ace5b79825 </code></pre> If you can build working kernels from the main tree, then you can directly use git-bisect – another good reason to get your changes into the mainstream source, but this is not always possible.</p> Another option for doing this type of binary search is to use git-bisect, and then cherry pick your changes at each bisect point. It may also be possible to use git-merge, but I have not figured out how yet.</p> Summary</h3> That is it for now. Git is very useful for maintaining multiple patch sets for different versions of an open source project, and is just another thing that makes working with open source projects a real pleasure. Future articles will cover how to further manipulate patches such as how to combine two patch files.</p> Linux 2.6.22 for the Compulab cm-x270 2007-08-13T00:00:00+00:00 I just completed porting the cm-x270 Linux kernel patches to the 2.6.22 kernel. Kernel build is available in OpenEmbedded. Patches are available in the OE tree:</p> http://www.openembedded.org/filebrowser/org.openembedded.dev/packages/linux/compulab-pxa270-2.6.22</a></p> Shortly I will be making 2.6.22 the default kernel for the cm-x270 in OpenEmbedded and posting instructions on how to use JFFS2 with the NAND flash on the cm-x270.</p> Design tips for your embedded Linux product 2007-07-20T00:00:00+00:00 When designing a system that runs embedded Linux, there are number of things you can do to make your life a lot easier and provide for future expansion to cover cases you did not anticipate. Following these simple guidelines can drastically improve the efficiency of developing the system, and troubleshooting problems during field testing, and provide the flexibility needed in todays products.</p> #1 Have a dedicated serial console</h3> This is the single most important interface to provide. A serial console is your gateway into the system, and allows you at any time to poke around at the internals of the system, debug problems, run applications, change configuration, etc. A serial console does not have to be a big DB9 connector accessible outside the system. It can can be a simple 3 pin header and you make a special cable to connect to a PC. Also, do put the RS232 level translators on the board. This saves so much hassle during the development and testing cycle. You can always not populate them later, but you will probably find that the debug console is so useful, you will leave it in – even in production.</p> Another consideration is to make sure you use a serial port provided by the processor if possible. The advantage of this is very little hardware has to be working for the serial console to work. If you use a serial port off an external UART, then the bus interface between the processor and the UART, the UART and the processor need to be working. If something is wrong with the system, you are less likely to get debug messages out the serial console which will help you debug the system.</p> #2 Implement a USB Host port</h3> A USB host port is so useful it should be considered for about any system. Some of the things USB can be used for include:</p> keyboard and mouse if you are designing a display based system</li> USB networking</li> Wifi networking</li> extra serial ports</li> WAN modems</li> </ul> Many embedded processors that can run Linux include a USB Host interface, so assuming you have one of these, it is very cheap to provide this interface as it only requires two wires, and a 5V power supply.</p> Once again, you don’t have to put a full size USB connector in your device – a simple header wired to a connector might work in most situations.</p> I can’t begin to describe how useful a USB keyboard was on a recent touchscreen based product. The product is used in agricultural vehicles, so anytime in the field we can connect a USB keyboard and look at logs, run commands, and other operations that are very difficult to do though a touch interface.</p> #3 Implement a Network Interface</h3> Working on any Linux system is very painful without a networking interface. Even if you don’t plan to use Ethernet in the final product, make sure a networking interface is available for development. Some reasons why a network interface is critical:</p> can easily log into the system remotely for development and debugging</li> easily copy files to and from the device quickly</li> install software updates easily during the development process</li> </ul> Trust me on this one. The neat thing about Linux, is if you have a USB Host interface, and a reasonably modern kernel, you can use off-the-shelf USB->Network devices for Ethernet.</p> #4 Provide a fast way for the bootloader to download SW updates</h3> This is important. Don’t depend on JTAG or a serial port for programming large OS images. Some options for BL image download include:</p> network (preferred)</li> MMC/SD</li> USB Client</li> USB Host (using a USB Mass storage device)</li> PCMCIA/CF</li> </ul> The good news is if you did steps #2 & #3, this might be taken care of – just make sure you check what the bootloader actually supports. One project I worked on recently only had a serial interface and a USB client interface that the BL did not support. We ended up creating a mini rootfs that included just enough smarts to download the main rootfs over the USB Client interface. So, there are ways to work around some of these issues, but its a lot easier if you do a little planning up front.</p> Summary</h3> There is a lot to be said for using industry standard interfaces and components like Linux, USB, and Ethernet. When you design a system using standard building blocks, your product will have the flexibility to implement features you never thought of when the product was originally designed. And, your development and testing will go so much smoother.</p> Open JTAG debugging tool for the PXA270 2007-07-10T00:00:00+00:00 Looking for a low cost, high functionality JTAG solution for the PXA270? As of several weeks ago, the openocd project now works with the Intel/Marvell PXA270 processor. There are a number of USB based JTAG debuggers available for under $100, so the cost is minimal. This article provides a brief overview of JTAG, suggestions for your hardware design, and how to use OpenOCD (Open On-chip-debugger) with the PXA270.</p> OpenOCD</h3> Over the past 8 years, I’ve used a number of JTAG solutions for programming and debugging embedded processor systems. At the low end, there are JTAG solutions for flash programming only that just wiggle pins on a PC parallel port which in turn wiggle pins on the scan chain of the processor. This way of doing things works, but is very slow because you have to fabricate the bus cycles in software and clock the entire jtag chain just to change the state of one pin. The jtag tools from the openwince project on Sourceforge fall into this category.</p> More advanced JTAG tools use the debugging capability of a processor to speed up flash programming and add debugging features. This typically involves downloading a tiny bit of code into the cache or internal SRAM in the CPU. This bit of code then can receive blocks of data or instructions and run bus cycles directly through the memory controller in the device. This makes flash programming very fast as most of the work is being done by the CPU, instead of manually controlling CPU pins using a jtag chain.</p> In past years, there have been various attempts at creating an open source JTAG solution, but none of them have gotten very far. When I had a project that needed to get done quickly, I simply bought or upgraded a BDI2000 debugger – which is an excellent tool. The OpenOCD project now provides much of the same functionality as a BDI2000, but as in the case of the PXA270, it sometimes takes some work to get it all working.</p> OpenOCD was created by Dominic Rath as part of a diploma thesis (http://openocd.berlios.de/web/</a>). It provides support for a wide range of target processors and JTAG debuggers. OpenOCD includes functionality for On-Chip Debugging, In-System Programming, and Boundary-Scan Testing.</p> Hardware design considerations</h3> Although openocd has worked with other Xscale parts for some time, functionality for the PXA270 has not worked until recently. The primary reason for this is the way the power supplies are sequenced in most PXA270 designs. There is one supply to the PXA270 that is always on (VCC_BATT). All of the other supplies are often kept off until reset to the processor is de-asserted and the SYS_EN and PWR_EN signals go active. Because of this, some special considerations need to be taken during the sequencing of the reset signals, and the setting of the hold_rst bit in the PXA270 DCSR register.</p> Make sure you power the JTAG connection (Pin #1 of your JTAG connector) with VCC_BATT or some power that is always active. If you power the JTAG connector with a power supply that is controlled by the SYS_EN or PWR_EN signal, then you will loose JTAG power during the connection process and things obviously will not work.</p> JTAG Debugger Hardware</h3> Most of the JTAG debugger debuggers used with OpenOCD are based on the FTDI FT2232 USB chip. The debugger I used is the arm-usb-ocd device from Olimex. At a cost of $80, you can’t afford not to have one.</p> </p> OpenOCD Installation</h3> OpenOCD can use two different libraries that provide access to the FTDI chip: an open source library named libftdi</em>, and a closed source library from FTDI named libftd2xx</em>. I could not get the opensource libftdi library to work, so I suggest you use the FTDI library for now. Otherwise, installation is pretty straightforward. Find yourself a reasonably modern Linux computer and follow instructions on the following page: http://openfacts.berlios.de/index-en.phtml?title=Building_OpenOCD</a> . Make sure you use the --enable-ft2232_ftd2xx</em> option when running configure</em>. On Ubuntu 7.04:</p> Install the FTDI library:</p> wget http://www.ftdichip.com/Drivers/D2XX/Linux/libftd2xx0.4.13.tar.gz</li> mkdir libftd2xx; cd libftd2xx; tar -xzf ../libftd2xx0.4.13.tar.gz</li> sudo cp .h /usr/local/include/</li> sudo cp libftd2xx.so.0.4.13 /usr/local/lib</li> cd /usr/local/lib/; ln -s libftd2xx.so.0.4.13 libftd2xx.so; sudo ln -s libftd2xx.so.0.4.13 libftd2xx.so.0</li> cd /usr/lib; ln -s /usr/local/lib/libftd2xx.so.0.4.13 libftd2xx.so.0</li> </ul> Install OpenOCD:</p> sudo apt-get install build-essential</li> sudo apt-get install automake</li> sudo apt-get install autoconf</li> svn checkout svn://svn.berlios.de/openocd/trunk openocd</li> cd openocd</li> ./configure --enable-ft2232_ftd2xx</em></li> make</li> sudo make install</li> </ul> OpenOCD Configuration</h3> OpenOCD is configured using a configuration file. An example is:</p> # config for Intel PXA270 # not, as of 2007-06-22, openocd only works with the # libftd2xx library from ftdi. libftdi does not work. telnet_port 3333 gdb_port 4444 interface ft2232 ft2232_layout olimex-jtag ft2232_vid_pid 0x15BA 0x0003 ft2232_device_desc "Olimex OpenOCD JTAG" jtag_speed 0 # set jtag_nsrst_delay to the delay introduced by your reset circuit # the rest of the needed delays are built into the openocd program jtag_nsrst_delay 260 # set the jtag_ntrst_delay to the delay introduced by a reset circuit # the rest of the needed delays are built into the openocd program jtag_ntrst_delay 0 #use combined on interfaces or targets that can't set TRST/SRST separately reset_config trst_and_srst separate #jtag scan chain #format L IRC IRCM IDCODE (Length, IR Capture, IR Capture Mask, IDCODE) jtag_device 7 0x1 0x7f 0x7e #target configuration daemon_startup reset target xscale little reset_halt 0 pxa27x # maps to PXA internal RAM. If you are using a PXA255 # you must initialize SDRAM or leave this option off working_area 0 0x5c000000 0x10000 nobackup run_and_halt_time 0 30 #flash bank <driver> <base> <size> <chip_width> <bus_width> # works for P30 flash flash bank cfi 0x00000000 0x1000000 2 4 0 </code></pre> Programming Flash</h3> After OpenOCD is installed, copy the above config file contents to a file name openocd.cfg, and run openocd in the same directory as your configuration file.</p> Currently, there is an issue in that the openocd is expecting a debug handler file at a specific relative path from the openocd binary. The easiest way to get around this for now is to to run openocd from the src directory in the openocd source code.</p> If you get the following message: Error: ft2232.c:1338 ft2232_init_ftd2xx(): unable to open ftdi device: 2, there is a permissions problem with the USB device. To get around this, run openocd using sudo, or add the appropriate rules to udev to allow the usb device to be used by any user.</p> OpenOCD runs as a daemon waiting for telnet or GDB connections. After openocd starts, you can connect to it by running the following command: telnet localhost 3333</p> At this point you should be presented with a OpenOCD prompt. The following is an example of how to program a bootloader into flash memory:</p> > flash probe 0 flash 'cfi' found at 0x00000000 > flash protect 0 0 5 off cleared protection for sectors 0 through 5 on flash bank 0 > flash erase 0 0 5 off erased sectors 0 through 5 on flash bank 0 in 4s 615249us > flash write 0 /build/tmp/u-boot-holon.bin 0 wrote 121832 byte from file /build/tmp/u-boot-holon.bin to flash bank 0 at offset 0x00000000 in 6s 138592us (19.381735 kb/s) > </code></pre> Debugging</h3> In this example, OpenOCD waits for GDB connections on port 4444. To debug code running on the target, you would use gdb in the same way you would use a BDI2000, or most any other hardware debugger with GDB.</p> Summary</h3> OpenOCD is a robust, full-featured JTAG debug solution that supports many processors and low cost JTAG debuggers. There are still a few rough edges to clean up with PXA270 support, but OpenOCD is already a very useful tool for embedded software development.</p> How to control the cable nightmare 2007-06-20T00:00:00+00:00 Do you ever wonder if there is a better way to manage cables in your lab? I typically have have 5-10 embedded systems on the bench powered up at any give time. When you start thinking of all the cables required to connect these systems (every one requires at least power, serial console, and network), it starts to add up. I recently ran across a product called snaketray (http://snaketray.com/products/wall.html</a> ) and decided it might be the solution to my rats-nest of cables. This article includes a few pictures of my implementation, and some ideas. I’m quite pleased with the result and have high hopes that I’ll keep my sanity for a while longer.</p> I’m sure there are many solutions to cable management. The IT folks managing racks of computers no doubt have lots of good solutions. There are lots of different cable ties available, etc. But, the needs of a developer are a little different in that you need a solution that can easily be changed. You need something that is not permanent where you can quickly add, change, and re-route cables as needed. Most cable management solutions are more permanent solutions not designed for flexibility. Then I read the following article: http://www.joelonsoftware.com/articles/BionicOffice.html</a>. It occurred to me that snaketray might be the perfect solution for an embedded systems development lab. The idea of snaketray is very simple as shown in the picture below:</p> </p> I installed two runs about 1.5 feet above my lab work space. There are power strips along the back of each lab surface. All cables now run in the snaketray. The idea is cables go straight into the snaketray and exit exactly where the are needed. If the cable is too long, I loop it back in the snaketray. The result is pretty amazing when you compare it to what it was before. My wife is not real impressed with how it looks – apparently she has not spend enough time under tables untangling cables :-)</p> Some of the benefits I’ve discovered so far:</p> even though it still looks pretty messy, I’m finding it very easy to pull cables out of the tray and re-route them – even network cables which are notorious for getting hung up due to the little plastic clip on the connector. I think the reason is the cables are always running orthogonal – in the tray and the come out at a 90 degree angle where they are needed. So if cables are not crossing each other they do not get tangled.</li> bench space is not used for cables, so I have lots of space for more systems</li> I’ll be more likely to try builds routinely on more targets. One of my goals is to routinely do OpenEmbedded builds for a variety of embedded systems to keep my finger on the OE pulse, and help with QA. Obviously crawling around on the floor untangling cables to connect various systems is a negative motivator for such activities.</li> I have a place to put cables when I’m not using them – just put them in the tray. Before, they were left on my lab bench taking space, or falling on the floor. Now I have a few extra network and serial cables waiting in the tray for when I need them.</li> </ul> The bottom line is this makes me more efficient at what I do, so that I can better serve my customers and the projects I work on. I imagine snaketray would also be very useful in factory setups and other situation where you routinely need to change cabling.</p> What you you found that works in your lab?</p> </p> </p> </p> Atmel AT32AP7000 development board review 2007-06-09T00:00:00+00:00 I recently purchased an Atmel NGW100 development board which includes an Atmel AT32AP7000 CPU. This CPU is based on the new Atmel AVR32 architecture, and is capable of running full Linux. This review covers the basics of this development board, the AVR32 architecture, options for building Linux for this system, as well as opinions on how a new architecture can succeed in a field of well established architectures like ARM and MIPS.</p> What is AVR32?</h3> The AVR32 architecture defined by Atmel is a 32-bit RISC load/store architecture. There are several different versions of the core. Atmel currently supplies two classes of AVR32 cpus:</p> UC3 Flash MCUs - low end devices with integrated RAM and flash, no MMU, and look similar in capability to ARM7 microcontroller type devices.</li> AP7 Application Processors - faster devices, include MMU, seem similar in capability and speed to ARM9 application processors such as Atmel’s AT92SAM devices.</li> </ul> As I am most interested in CPUs that can run Linux, this review will focus on the AP7 (AVR32 AP) class devices.</p> It seems that AVR32 is Atmel’s answer to ARM Cortex. Some features of the AP7 devices:</p> good code density</li> DSP instructions that execute in a single cycle</li> Full MMU</li> Instruction and Data caches</li> SIMD extensions</li> dynamic branch prediction</li> </ul> The MMU in the AP7 devices is interesting in that page tables are handled in software similar to a MIPS device. In a ARM CPU, the page tables are located in physical RAM and are strictly defined by the ARM architecture and page faults are handled in hardware. With MIPS (at least some devices) and AVR32, the MMU basically consists of a TLB (Translation Lookaside Buffer) that maps virtual addresses to physical address. As the TLB is limited in size, any misses are handled in software.</p> NGW100 Development board</h3> The NGW100 development board is available from a number of distributors (including Digikey) for about $70. The board includes the following major components:</p> Atmel AT32AP7000 CPU</li> 32MB of SDRAM (MT48LC16M16A2)</li> 8MB Parallel NOR Flash (AT49BV642D-70TU)</li> 8MB Serial NOR Flash (AT45DB642D)</li> 2 Ethernet ports</li> SD/MMC slot</li> RS232 port</li> USB Client port</li> Microcontroller for board control (ATtiny24-20SSU)</li> </ul> </p> Linux Support</h3> Atmel is directly supporting the development effort for the AP7 processors. They seem to be doing it right in that they are pushing support directly into the mainstream kernel. Currently, Atmel provide patches for GCC and uClibc. There is already efforts underway to provide AVR32 support in Openembedded, which will enable developers to quickly build full featured Linux distributions for AVR32 based products.</p> Summary</h3> It will be interesting to see how the AVR32 based products work out. On one hand, one could argue that a custom architecture like AVR32 can’t succeed as it does not have an extensive ecosystem of tool vendors in place like ARM and MIPS. But, the availability of open source software such as GCC and Linux is rapidly lowering the barrier to entry for creating tools for new architectures. No longer do you need to wait for commercial tool vendors to provide software support. The model for software development is changing.</p> In comparison to Atmel’s AT91SAM parts, its difficult initially to see what the AVR32 parts offer over the SAM parts. For one, they are lacking a USB Host port which is becoming increasingly important to have in embedded Linux systems. Perhaps they offer other advantages such as lower cost and power. As we gain experience with these parts, I’m sure we’ll find applications where they are an excellent fit.</p> The Embedded Industry Transformation 2007-06-09T00:00:00+00:00 I just ran across an interesting article written by Doug Gaff titled “Is the embedded industry dead?</a>”. The article points out the transformation that is happening in the “embedded” industry. For many systems, we are no longer constrained by minimal resources (/web/content/view/67/9/). Embedded Systems are no longer isolated systems, but parts of larger systems.</p> A few of my own observations:</p> There is more to modern embedded systems than control and logic. We now have to worry about data. In the past the task might be to get a microcontroller to run a state machine, control motors, etc. Today, we need to present a calendar to the user through a web interface, collect information about the location of the Sun, upload statistics to a remote server, support a rich GUI application on a VGA color display. The control aspect is suddenly the small part of the problem, and managing data is often the primary concern.</p> Embedded system development now encompasses many disciplines including OS development, web services, advanced application development in high level languages, and the traditional hardware debugging and low level coding tasks. It is becoming less about writing code and more about integration of the right pieces. As Embedded Systems become more complex, access to specialized knowledge is becoming more important. There is just too much for one person (or even one company) to know. Companies developing competitive products in the future will be the ones who have access to experts. Is your company taking advantage of this transformation?</p> Can you afford not to use Linux in your product? 2007-05-29T00:00:00+00:00 How much does hardware capable of running Linux cost? Recently, I priced components for a fairly minimal system.</p> CPU: Atmel AT91SAM9260 (< $10)</li> 32MBytes of SDRAM ($6)</li> 256MBytes of NAND flash ($6)</li> </ul> These are prices for relatively low volumes on the order of 1000 pieces. So, you can get a processor, RAM, and flash that is more than capable of running Linux for about $20. Think about the possibilities. The AT91SAM9260 is a rather impressive part and includes interfaces for about anything including USB, SD/MMC, Ethernet, and the list goes on. It also includes that capability to boot from NAND flash which eliminates a dedicated NOR boot flash and reduces cost. If you don’t want to do a full processor design, processor modules can be obtained for less than $100</a> .</p> So, at this price you have to ask: can I afford not to embed Linux in my product? Can I afford not to take advantage of standard interfaces like Ethernet and USB? Can I afford not to take advantage of 1000’s of high quality applications and drivers? Can I afford to not implement a system with virtually unlimited storage</a> and capability for expansion? Think about this carefully because your competitors are. Have questions or need help getting started? Contact us</a> – we can help.</p> Is Linux ready for real-time applications? 2007-05-12T00:00:00+00:00 Have you ever wondered about the real-time response of Linux and how well it performs? I recently developed a system where I had the requirement to send a packet of data out a serial port every 30ms. This article describes how stock Linux can be used to accomplish this and solutions to several problems I encountered. There are several concepts that need to be understood such as how to set a Linux process to real-time, and kernel preemption.</p> All of the work referenced in this article was done on a Compulab cm-x270</a> system which is based on the PXA270 ARM processor. Before utilizing any real-time optimizations, the packets sent out the serial port where anything but consistent. There was a significant amount of jitter in the packet timing and occasionally we would see delays of up to several seconds. Obviously this was not good enough.</p> Real Time Process Priority</h3> The first step was to partition the application into several threads, and give the serial communication thread a priority of real time. Linux supports several scheduling policies:</p> SCHED_OTHER: default universal time-sharing scheduler policy used by most processes.</li> SCHED_BATCH: intended for “batch” style execution</li> SCHED_FIFO: First In-First Out scheduling. Can only be used with static priorities higher than 0. A SCHED_FIFO process will always run before a SCHED_OTHER or SCHED_BATCH process. A SCHED_FIFO process runs until it is blocked by an I/O request, or is preempted by a higher priority process.</li> SCHED_RR: Round Robin scheduling. SCHED_RR is similar to SCHED_FIFO, except the process will only run for a maximum time quantum if there is a process of equal priority waiting to be run.</li> </ul> Practically, in a simple system there is really not much difference between SCHED_FIFO and SCHED_RR as you will typically only have at most a couple real time processes and you will probably assign them different priorities.</p> The sched_setscheduler() system call can be used to set a process or thread priority – see the Linux man pages for more details.</p> After setting the communication thread to real time, there was a noticeable improvement in the packet scheduling, but there was still way too much jitter and the occasional several seconds of silence with no packets.</p> Preemptible Kernel</h3> After some experimentation, and reading, it was suspected that the NAND flash driver in the system was locking the kernel for long periods of time. With older Linux kernels, the kernel was not preemptible. Any process running in the kernel had to finish before something else could run. Recent 2.6 Linux kernel include an option to enable Kernel Preemption (CONFIG_PREEMPT). This allows a real-time process to preempt the system even if it is running in kernel space. After enabling the CONFIG_PREEMPT option in the kernel, the scheduling of packets was very consistent most of the time with almost no perceptible jitter.</p> Be careful what you put in your real-time thread</h3> Now the system was running fairly well, but we still noticed at system startup that there were still some long delays between packets. It turns out the application was writing log messages to a file on NAND flash in the real-time thread using the standard glib logging functions. At system startup, the NAND driver blocks for long periods of time (up to several seconds). After the system has been running for a short time, this went away. The solution in this case was to send any logging messages from the communication thread to a separate normal priority logging thread. This fixed any remaining issues and the packet scheduling is now rock solid all the time. The leads to an fundamental real-time concept which seems very obvious in hindsight:</p> Do not access any I/O (such as storage or network interfaces) that may block for long periods of time**in a real time process.</strong></p> Although we had designed all the primary I/O to be handled this way, we had not given any consideration to log file writes, which are an important part of any program.</p> The glib library provides several very nice mechanisms for sending data between threads.</p> Linux Real-time</h3> With improvements in the 2.6 Linux kernel, Linux works very well for some real-time applications. While I would still be very careful in using Linux for critical, hard-real time requirements due to its overall complexity, Linux works very well for multi-media, communication, and data acquisition systems.</p> NAND Flash is almost "free" and what it means for Embedded Systems 2007-02-26T00:00:00+00:00 original article p</em>ublished in March, 2007</p> EETimes recently ran an article titled “Plummeting prices make NAND appear almost free” (http://www.eetimes.com/news/latest/showArticle.jhtml?articleID=197002923</a>). 1GBytes parts cost around $55 in Nov 2005. As of 2007-03-28, a 1GBytes part costs around $8. This is almost an order of magnitude reduction in a little over a year. This article examines the impact this price drop will have on embedded systems and the changes we will likely see including: storage is now at post-scarcity, will “embedded” distributions become obsolete, will uclibc become obsolete, and challenges with large NAND flash parts.</p> 3 major components in an embedded computer</h3> The 3 major components in an embedded computer are:</p> processor</li> RAM</li> Flash Storage</li> </ul> All of these components have been gradually decreasing in price over the years, and there has been a corresponding gradual increase in the use of complex embedded systems. However, the drop in NAND flash in the last year represents more of a step change, and will likely result in some more drastic changes.</p> We are to the point of post-scarcity for Flash storage</h3> Up to this point, most full featured embedded Linux systems contained 16-64MB of NOR flash. A full featured graphical Linux terminal requires about 20-30MB of flash disk space just for applications and libraries. So, in the past we were still concerned about application footprint. The was a lot of incentive to not cross the 32MB to 64MB boundary, etc. With 1GiB of flash now costing less than $10, we essentially have unlimited flash storage in the context of an embedded Linux distribution. We will probably see the following:</p> faster time to market as we need to spend less time optimizing</li> increased use of GTK+ as the standard GUI toolkit. We no longer care how much space GTK+ takes.</li> more debugging features can be left in the production system. Why not leave VIM, Minicom, and SSH in the build if we have plenty of space. These non-critical features will lower the cost of maintaining a product.</li> </ul> Will “embedded” distributions become obsolete?</h3> There has always been a great deal of interest in running Debian and other “standard” distributions in embedded systems. With 1GiB of storage, this is going to be a lot more practical. However, with a little experience, you soon realize that distribution size is only one of the many factors. Another equally important factor is the ability to easily automate the build process for the entire project, including custom applications and image generation for programming. With complex systems like Linux, it is easy to get into the hack-n-slash mode where it takes a lot of manual steps to build the system. While this may seem OK when you building the system, it ends up causing a lot of pain and errors once you get in a hurry, of down the road when you are trying to maintain the system. Ability to easily customize the distribution is another requirement. OpenEmbedded (http://openembedded.org), and other build systems provide these features. I think we will see a big increase in Debian and other standard Linux distributions in Embedded systems, but OpenEmbedded and other embedded build systems will continue to grow in use as well because there are many factors in the equation. This is the nice thing about open source – use what works best for you.</p> Building and maintaining an embedded system is a lot different than setting up and maintaining a desktop or server system. Other articles related to this topic include: /web/content/view/63/9/</a> and /web/content/view/35/37/</a>.</p> Will uclibc become obsolete?</h3> Because flash is now cheap, why do we need to use uclibc (small footprint libc) – can’t we just use glibc. For full featured embedded Linux systems, this argument makes sense. However, there is push toward higher integration at the low end. Many companies are offering 32-bit “microcontrollers” with integrated flash and RAM. The processors in these devices are more than capable of running Linux. The limitation to date has been the RAM and Flash size. The available memory in 32-bit microcontrollers will increase over time enabling them to run a low end uclibc based system. The idea of running Linux on a system that is composed of a single chip is very interesting.</p> Challenges with large NAND flash parts</h3> What challenges will using large NAND parts present? In general, dealing with NAND is a lot more complex than with NOR flash. For reading, NOR parts present a simple memory mapped interface. A NAND interface is not memory mapped and looks more like a FIFO or a Hard Drive interface. This makes booting from NAND flash much more difficult. Some parts present a memory mapped interface for an initial loader. Some processors build NAND flash bootloading into the processor. Other systems boot from a small NOR flash and use NAND after the system has booted.</p> Filesystems are also a challenge with NAND flash. The standard method is to use JFFS2 which was designed for smaller NOR parts. JFFS2 works well, but is a log based filesystem. This means it has to scan the entire file system on mount and store the entire inode chain in memory. This takes lots of time and uses lots of RAM. For small devices (64MB), this is tolerable, but for large devices the mount time can take 10’s of seconds. YAFFS2 is another log based flash filesystem designed specifically for NAND flash which includes a feature called checkpointing that bypasses the normal mount scanning. Another development is LogFS which looks promising, but will probably not be ready until mid/late 2007.</p> Programming NAND flash can also be a challenge. Because NAND flash can contain bad blocks, programming is a lot more complex. Traditionally, flash programming in a NOR system was done by the bootloader. For a system that boots from NOR flash, programming the NAND flash can be done from a small Linux system (see /web/content/view/66/9/</a>). If you are booting from NAND flash, programming may be done with a JTAG programmer that supports NAND flash, or by downloading a small bit of bootstrap code into the processors internal memory using JTAG, and this bootstrap code then has enough functionality to program the NAND flash.</p> Summary</h3> The lower NAND prices will bring about change in the way embedded systems are built. Please comment if you have addtional ideas or observations.</p> Do you need "software update" functionality in your Embedded Linux system? 2007-02-21T00:00:00+00:00 In this day and age, most embedded systems include a way for users to easily update software once the device has been deployed. This article discusses the requirements for a field update mechanism along with pointers for how to implement.</p> Update Mechanism Requirements</h3> Requirements for a field update mechanism might be:</p> easy for users to perform updates.</li> little chance of “bricking” a unit if reset occurs during update.</li> ability to update all of the software in the unit</li> in this case, update from files on a USB Storage device</li> ability to program NOR and NAND flash</li> </ul> Other systems may have requirements to update from a network, Compact Flash card, etc.</p> The USB wrinkle</h3> System update is typically done by the bootloader. In this case, we are using a Compulab cm-x270 module (/77/compulab-cm-x270-pxa270-module-review/</a>) that has a proprietary boot loader and does not support updates from a USB storage device. Therefore, we need to run the updates from the context of the Linux operating system which has drivers for USB and the NAND flash. We decided that the best way in this case is to have a small Linux OS image that contains enough functionality to update the main filesystem. This also provides us with the capability to recover if the main filesystem ever gets corrupted. The flash layout for the system is:</p> Partition</strong></td> Flash Device</strong></td> Size</strong></td> File System</strong></td> Description</strong></td> </tr> Kernel</td> NOR</td> 1536KiB</td> NA</td> Linux kernel</td> </tr> Update rootfs</td> NOR</td> 2304KiB</td> Initramfs (RO)</td> Small rootfs that contains just enough functionality to update the system</td> </tr> Main rootfs</td> NAND</td> 512MiB</td> JFFS2 (RW)</td> Main rootfs including application</td> </tr> </tbody> </table> There are other schemes that could work as well. One way might be to have two identical application partitions and only update one at a time. A reset while updating the kernel or the update rootfs does have the potential of “bricking” the system, but most of the updates will be for the Main rootfs, so we think this risk is low enough to be acceptable in this system. The update rootfs is loaded into RAM before it is used, so there is little chance of it being corrupted during normal operation. Running the update rootfs from ram is also convenient in that we can easily re-program the update rootfs in flash as we are not directly mounting the filesystem in flash.</p> Building the Update rootfs</h3> The base update rootfs is built with OpenEmbedded as described in /77/compulab-cm-x270-pxa270-module-review/</a>. Once we had the system booting into the update rootfs, we had to add the following functionality:</p> Boot the main rootfs from NAND flash</li> look for update files on a USB Storage device and flash the kernel, update rootfs, and main rootfs as needed.</li> </ul> Booting the NAND flash image is accomplished using the busybox switch_root</em> application. This utility allows you to easily switch to another rootfs from an initramfs. There are a few gotchas to be aware of when working with switch_root:</p> switch_root must be “exec’d” from the init process with PID=“1”.</li> switch_root requires a “/init” file to be present</li> </ul> The above functionality was accomplished by writing a shell script called /init. If an error occurs, the /init shell script then launches the standard init process in /sbin/init which provides a terminal and allows for easy debugging. The mtd-utils package provides a handy utility called nandwrite</em> which can be used to write jffs2 images to NAND flash (handles bad blocks, etc). At some point, we may remove all the extra functionality from the update rootfs like the standard sysvinit, terminal login, etc, but for now it is handy to have for debugging. To give you an idea how easy it is to write to flash from a shell script, consider the following snippets:</p> # write Image into update rootfs parition </code></pre> cat /usb/nor_rootfs.img > /dev/mtd2 </code></pre> # write Image to NAND rootfs partition </code></pre> flash_eraseall /dev/mtd3 </code></pre> nandwrite /dev/mtd3 /usb/nand_jffs2.img </code></pre> # and to launch rootfs in nand flash </code></pre> exec switch_root -c /dev/console /jffs2 /sbin/init 5 </code></pre> Summary</h3> Having typically used bootloaders for system update in the past, I’m very pleased with how this mechanism worked out. Thanks to the flexibility of Linux and OpenEmbedded, this only took several days to implement and debug. Writing update programs with shell scripts at the application level gives you a lot of flexibility and allows you to use standard Linux drivers for USB and Flash access which are very robust. At some point if there is interest, we may look at cleaning up our code and contributing functionality to OpenEmbedded to generate a more generic update initramfs. Please contact us or leave a comment if you are interested in collaborating on something like this.</p> Building a tiny "safe" boot image using OpenEmbedded 2007-02-16T00:00:00+00:00 We are currently working a project that contains cm-x270 module (/web/content/view/62/9/</a> ) which contains 4MB of NOR flash and 128MB of NAND flash. The Linux kernel and a small root file system (rootfs) will be stored in NOR flash, and the main rootfs in the NAND flash. The rootfs in the NOR flash will be used to update the system from a USB Flash drive in the field. This article describes how to build a small boot rootfs using OpenEmbedded. We also look at various ways to store the image in flash.</p> Requirments</h3> In many embedded systems, field upgrades is a requirement. There are many ways to organize this, but one of the most reliable is to have two rootfs partitions. The first partition is a “safe” rootfs that is only used to update the second partition. The second partition contains the main application and can be updated in the field. If a programming error, or power loss occurs while updating the 2nd partition, you can always boot into the first rootfs partition and restart the process.</p> The kernel and the bootloader in this system take up about 1.5MB (out of 4MB total) of the NOR flash. This leaves us with about 2.5MB for the update rootfs. Because we are building the rest of the system using OpenEmbedded (OE), it may make sense for us to also build this update rootfs using OE to keep everything in one build environment.</p> uclibc, glibc, or klibc?</h3> There are several libc libraries that are commonly used to build Embedded Linux systems. glibc is the standard that is used in desktop and sever systems and is what is being used in the main NAND rootfs partition. uclibc is smaller than glibc is often used in space constrained systems. klibc is a very minimal libc subset that supports operations typically used in an initramfs. Because we are already using glibc, we decided to try this first as we could then use the same toolchain and build tree and just generate two images – one for NAND, and one for NOR.</p> OpenEmbedded support for small images</h3> OE includes a task-base-minimal</strong> meta target that can be used as a basis for a small images, or you can create your own based on task-base. You will then need to create a custom image recipe that will use this meta task. An example is shown below:</p> PR = "r0" export IMAGE_BASENAME = "svs_nor" export PACKAGE_INSTALL = "task-min" # keep extra language files from being installed export IMAGE_LINGUAS = "" RDEPENDS = "task-min" IMAGE_FSTYPES = "tar jffs2 ext2 cramfs squashfs squashfs-lzma cpio.gz" IMAGE_ROOTFS_SIZE_ext2="10240" #EXTRA_IMAGECMD_jffs2="--pad=0x1000000 --eraseblock=0x40000" EXTRA_IMAGECMD_jffs2="" inherit image </code></pre> In this case, I had defined a custom recipe for task-min based on task-base, which installs the following packages:</p> root@compulab-pxa270:~$ ipkg list_installed base-files - 3.0.14-r58 - base-passwd - 3.5.9-r2 - busybox - 1.2.1-r11 - initscripts - 1.0-r86 - ipkg - 0.99.163-r1 - ipkg-collateral - 1.0-r5 - libc6 - 2.5-r4 - libgcc1 - 4.1.1-r10 - libipkg0 - 0.99.163-r1 - makedevs - 1.0.0-r2 - sysvinit - 2.86-r32 - sysvinit-inittab - 2.86-r32 - sysvinit-pidof - 2.86-r32 - task-min - 1.0-r1 - tinylogin - 1.4-r3 - update-rc.d - 0.7-r0 - </code></pre> Storing the NOR filesystem in flash</h3> OE can generate images for many different flash filesystems and initrd mechanisms. These are specified in the IMAGE_FSTYPES</strong> variable in the above recipe. In this case, I chose to generate many different filesystems so I could see evaluate how much compression I would get with each filesystem type. The results:</p> Rootfs Type</strong></td> Size (bytes)</strong></td> </tr> squashfs-lzma</td> 1785856</td> </tr> tar.bz2</td> 1918243</td> </tr> cpio.gz</td> 2024558</td> </tr> squashfs</td> 2109440</td> </tr> cramfs</td> 2265088</td> </tr> jffs2</td> 2455396</td> </tr> ext2 (uncompressed)</td> 5481000</td> </tr> </tbody></table> I ended up choosing the cpio.gz format as it can be loaded directly into an initramfs filesystem by the kernel. There are several advantages to using an initramfs for this task:</p> You don’t have to mess around setting up tmpfs filesystems for directories that need to be writable.</li> You can update the NOR flash partition while you are running out of the initramfs filesystem.</li> It is fairly small with only squashfs-lzma being smaller.</li> </ul> Future Optimizations</h3> For now, the glibc based solution is good enough</em> because it fits in the flash space we have and it is easy to build with our existing build environment. In the future, we may move to a uclibc or klibc based solution and try to reduce our flash size (OpenEmbedded supports both klibc and uclibc). The above image sizes can also be reduced by removing ipkg and other components that are not needed.</p> This exercise illustrates many of the advantages of Linux and OpenEmbedded. With very little work, I can produce exactly what I need. The ability of the 2.6 Linux kernel to load a cpio archive into an initramfs is a very elegant solution for small boot images.</p> Compulab cm-x270 PXA270 module review 2007-01-30T00:00:00+00:00 We are in the process of supporting a customer who is designing a product that uses the Compulab cm-x270 “Computer-on-module”. The cm-x270 is a small computer module 66x44x7mm that contains an Intel PXA270 ARM processor similar to those found in many PDAs or smart phones. This review provides an overview of the cm-x270, why it was selected, and how we are using it.</p> Project Needs</h3> For this project, we needed a low cost embedded computer that could drive a VGA (640x480) color display. The graphical user interface for the device was fairly advanced, so we were also looking at a system that could run a modern OS with an advanced graphical toolkit. After discussing the display requirements with the customer, we concluded that we needed a 32-bit processor with a LCD display controller. As there was some custom circuitry needed, a configuration where the customer could design a custom baseboard and use an off-the-shelf computer module seemed optimal. The schedule was also very aggressive and we did not have the time or budget to develop a full custom processor board.</p> CM-X270</h3> After researching a number of computer modules, we chose the cm-x270 from Compulab. The cm-x270 is a module that includes the following components:</p> Intel PXA270 ARM processor (up to 520Mhz)</li> 802.11b Interface</li> up to 128MB SDRAM</li> 128-512MB NAND flash</li> up to 4MB NOR flash</li> PCMCIA</li> PCI</li> Serial Ports</li> Touch panel controller (UCB1400)</li> Host and Slave USB</li> and other features …</li> </ul> For more details visit the compulab web site (http://http://www.compulab.co.il</a> ). The cm-x270 provides a very impressive number of features in a very small space.</p> Below is a picture of the cm-x270 mounted on the Compulab SBC-X270 baseboard with a VGA display. The system is running the GPE Palmtop Environment built with OpenEmbedded.</p> </p> Low Cost</h3> Perhaps the most impressive feature of the cm-x270 is the cost. When ordering in volume, Compulab will build the module with only the components you need. For prototyping, you can order a standard offering with all components populated. Starting a price of $50, it is the lowest cost module we found. The configuration we will be using costs around $98 in 2K quantities (64MB DRAM, 4MB NOR Flash, 512MHz + 128MB NAND Flash + Audio/Touchscreen + RTC). Eventually, we may be able to reduce some of the features and get the price down around $70 per module.</p> At this price, the cm-x270 provides a very cost effective way to develop a low volume product (a few thousand per year) with advanced features provided by the PXA270 processor.</p> Documentation</h3> The bootloader and hardware design of the cm-x270 is proprietary. Compulab does not supply schematics for the design or source code to the bootloader. The documentation provided is adequate and provided all the information we needed to complete the design. Compulab also provides schematics for their development baseboard, which is helpful.</p> Bootloader</h3> The cm-x270 bootloader provides typical features found in an embedded processor bootloader, including: a serial console, NAND and NOR flash programming, download images from a tftp server, downloading images over a USB connection to a PC, and a number of other configuration commands. The tftp download works very well and is fast. The USB download is much slower, but is convenient for systems that do not have a network connection. The USB download requires a windows PC with a specific version of Microsoft ActiveSync installed, which is unfortunate. A more general solution that uses something like libusb (http://libusb.sourceforge.net/</a>) instead of ActiveSync would be preferred as it could then run on both Windows and Linux computers.</p> Linux Support</h3> Compulab provides a fairly complete port of the 2.6.16 Linux kernel for the cm-x270 in the form of a kernel patch. Most of the peripherals we tested seem to work with the following exceptions:</p> The SD card driver is not preempt safe. In this application, we need kernel preemption turned on to meet certain real-time requirements.</li> The Touch driver contains a small bug that does not allow applications using it to exit properly</li> </ul> Overall, support is pretty good and enabled us to get started with development quickly.</p> NAND Flash Driver</h3> As the Compulab module only provides 4MB of NOR flash, using the NAND flash is a requirement when implementing a full featured Linux system with an advanced graphical user interface. Compulab provides a binary module that is linked into a kernel driver to support the NAND flash. This driver provides a block interface on which standard filesystems like ext2 can be used. The NAND driver has worked fairly well, although we have seen several cases where we get I/O errors if we remove power to the device without shutting down the system properly. The way to recover is to do a low level format the NAND flash. This should not be an issue with this product as we will have a backup battery that will be used to power the system during shutdown if power is removed. We may eventually move to an open source JFFS2 flash driver as time permits.</p> The proprietary NAND driver presents an additional wrinkle in that the driver cannot be linked into the kernel due to GPL licensing requirements. One solution is to boot a small root file system from NOR flash that includes the NAND driver module, and then proceed to boot from NAND flash.</p> GNU/Linux Root File System</h3> Compulab provides an example root file system (I think based on Debian) and suggests using scratchbox to build applications. We looked at it briefly, but then decided to build our own rootfs using OpenEmbedded for the following reasons:</p> We can easily add additional packages from the extensive OpenEmbedded catalog as needed.</li> Allows us to have an rootfs independent of the module – we can more easily move to other processor modules or architectures in the future.</li> OpenEmbedded provides us with mechanisms for package management and building complete images that include our applications and tweaks to the OS.</li> </ul> For more information on OpenEmbedded, read our “Introduction to OpenEmbedded</a> “. With OpenEmbedded, we were able to quickly build a toolchain and a root file system that included the kdrive version of Xwindows and GTK+ libraries. This rootfs fits in less than 32MB of flash disk space, which is much smaller than Compulab’s OS image.</p> We have also added machine configuration files to the OpenEmbedded project to support the cm-x270: http://www.openembedded.org/repo/org.openembedded.dev/conf/machine/compulab-pxa270.conf</a></p> Design Support</h3> Compulab seems mainly interested in customers who are interested in purchasing modules in volume. They do not sell low cost development boards such as LogicPD’s $500 board. Compulab requires you to purchase a fairly high cost development system ($1900) in order to get started. Part of this cost is to pay for Compulab’s support costs. For companies who do not have experience designing systems with PXA270 class processors, we highly recommend getting some up-front help from someone who has experience with these systems. Even though the module takes care of a lot of the design complexity, it helps to have a detailed understanding of the PXA270 and associated components, as you still need to interact with them. Consulting companies like BEC Systems can provide assistance in the form of hardware design reviews and Linux OS support to accelerate projects and help avoid costly mistakes and delays. As Compulab designs only the module, there is only so much they can do when you run into problems – as shown in the next section.</p> Issues Encountered</h3> Overall, the project using the cm-x270 has gone well. The only real challenge we have faced is a latch-up issue with the ITE PCI controller on the cm-x270 module. It appears that this part is very sensitive to the characteristics of the 3.3V supply to the module. After a considerable amount of debugging and testing, we were able to solve this problem with some filtering and slowing down the rise time of the 3.3V supply. This is a good example of where a project can get difficult very quickly when things don’t work as expected.</p> Summary</h3> So far, the cm-x270 has worked very well in this project. With its low cost, it is a very attractive option for companies wanting to implement advanced functionality in their products, but don’t have the time, sales volumes, or experience to justify a full processor board design. Feel free to post comments about your experiences using embedded computer modules.</p> Introduction to OpenEmbedded part 1 2007-01-12T00:00:00+00:00 This is the first in a multi-part series on the OpenEmbedded development environment. This and subsequent articles will provide an introduction to OpenEmbedded, why to use it, how it works, and will provide examples of how BEC has used OE in various projects.</p> Part 1 – Why do we need OpenEmbedded? This article discusses some of the problems developers face when building embedded Linux systems, such as collecting the pieces needed, setting up a repeatable build environment, cross-compiling, customizing a system, and licensing issues. Future articles will cover how OE addresses some of these issues.</p> Embedded Linux = Complex System</h3> The fundamental problem with building Embedded Linux systems is the sheer complexity. Lets face it, there is a lot going on in a typical Embedded Linux system. Millions of lines of code are being compiled. Typically there are tens to hundreds of applications and libraries that need to be built. Linux supports many different processor architectures. The size of Linux leads to the conclusion that a good deal of automation is required to manage all this. In my paper “Tips for Planning your Embedded Linux Project” (/web/content/view/35/37/</a> ), the analogy of the Iceberg is presented. The application specific software in the project is the part of the Iceberg that is above water – the part you see. Most of the system (Linux and system applications) is below water – you don’t see it. The part under water is the big unknown and has the potential to add a lot of variance to a project. Note: this problem is not specific to Linux – any system of this complexity that is targeted to so many different applications faces the same problem. As a side, one might ask why use Linux if it is so complex? The answer is simple – features. Linux has the features and device support required for many Embedded products to be competitive.</p> Collecting and building all the pieces</h3> There are a lot of pieces of software that need to come together to build an Embedded Linux system. This may include a toolchain, kernel, system libraries, and many system applications and configuration scripts required to make a system work. The is not something you want to do manually, or building an embedded Linux system quickly becomes impractical for most developers.</p> Repeatable Builds</h3> With any software project, it is highly desirable to have a build system that is easily repeatable – which means it can be reproduced at any time on a wide variety of development workstations. Some of the of the situations I have run into that are less than desirable are:</p> Being dependent on a vendor to provide a pre-compiled root file system image for a system. What happens when you need to add something two years down the road (such as foreign language support) and you don’t have a build system where you can easily reproduce the current build?</li> Being dependent on a golden build machine that takes a lot of work to set up and is very difficult to reproduce. If you have a build environment where all the developers use one machine to build software, you may want to investigate if you are in this situation.</li> </ul> What is really needed is a build system where you can check the build system out of a source control system onto about any Linux workstation at any point in time, type make, and a few hours later you have an image.</p> GPL Compliance</h3> If you distribute binaries in your product compiled from GPL software, you must offer to provide the source code for these binaries. Do you have easy access to the source for everything that is in your product? Do you even know what all is in your product? It is not good enough to simply point to someone else’s distribution or web site and say “I use that”. As the founder of the MEPIS distribution discovered</a>, the GPL license explicitly states the distributor of the GPL code is obligated to provide source code for up to 3 years. So if you build a distribution by copying binaries from a pre-build distribution, you may have a problem. In one case I am familiar with, a company was in the last stages of a multi-year product development cycle. They then discovered that they did not have the source code used to build the GPL binaries in the system. The company that built the distribution for them was out of business and the open source project the binaries where originally obtained from no longer maintained the sources. Although it may be possible to find all of the old sources, it is a huge amount of work and would probably be easier to just start over and build new distribution.</p> Customizing the system</h3> For most embedded systems, the distribution needs to be customized. There are often requirements such as custom software update schemes. Ideally, a good build system should easily allow you to integrate changes into the build system in a automated way, so that you don’t have any manual steps in the process of creating an image for the embedded system.</p> Cross Compiling</h3> The above “repeatable build” requirement that you be able to build an image for your embedded target on about any Linux workstation leads to a requirement for cross-compiling. Cross-compiling is the process of creating executable code for a platform different than the one on which the cross compiler is run (http://en.wikipedia.org/wiki/Cross_compiler</a>). Cross compiling many standard open source applications and libraries can be difficult on a good day, and near impossible on a bad day, as many of them were not written with cross compiling in mind. Therefore, a good build environment should have tools that automate the process of cross-compiling as well as provide the ability to easily cross compile the many common programs and libraries that may be needed in an embedded Linux system.</p> Summary</h3> As we examine some of the challenges, it should be obvious that putting together a production embedded Linux system is not something to be done in an ad-hoc fashion. There are several ways to approach this problem. Stay tuned for future articles where we explore how the OpenEmbedded build system solves these problems (subscribing to our blog or newsletter is a good way to do this).</p> Also, please post comments about your experiences – we like to hear and learn from you. We are also interested in what problems you face when building Embedded systems. This input will be used when writing future articles. Thanks!</p> Remote access for embedded systems behind a firewall 2006-12-19T00:00:00+00:00 When deploying embedded systems, wouldn’t it be nice if you could easily set up remote access for troubleshooting and updates? With Embedded Linux systems, this is quite easy to do. This article will explore how to use SSH to set up remote access to an embedded system behind a firewall without using VPN or special network setup and presents a few tips for making this setup work well.</p> The Problem</h3> With today’s complex embedded systems, testing and deployment can be a challenge. During the initial testing and roll-out, a new system may need to be carefully monitored to assure it is operating correctly. Embedded Systems programming is different than application programming in that often you are interfacing with external I/O. The behavior of things external to the embedded system are often very difficult to model during development; therefore time must be spent in the field testing the system in real-world situations. Being able to remotely access the system during testing is very convenient.</p> As with most networked computer systems, networked embedded systems are typically deployed on a network behind a firewall. It is very easy to log in and administer any Linux system you have direct network access to – the challenge is getting to a system behind a firewall.</p> Options for remote Access</h3> There are several general options for remote access to networks:</p> VPN</li> Port forwarding at the firewall</li> PC remote access technologies. These are typically single session type connections that are initiated by the PC user.</li> </ul> The disadvantage of the above technologies is they require involvement from the IT department or users of the remote network. BEC is currently involved in the deployment of a system that uses embedded computers in six different buildings. Each building has its own firewalled network. Getting remote access to all of these networks is a big task and everyone is busy. An ideal solution would be a way to communicate directly with each embedded computer from a central location, which brings us to SSH …</p> SSH</h3> SSH (Secure Shell) is a set of standards and an associated network protocol that is used to establish a secure channel between a local and remote computer. SSH encrypts all traffic (including passwords) and is generally considered to be very secure if used properly. Some uses for SSH are:</p> remote terminal access.</li> transfer files between computers.</li> tunnelling and port forwarding between computers</li> </ul> The fact that SSH can be used to forward ports over an encrypted tunnel is what makes remote access possible using what is called a Reverse SSH</strong> tunnel. Because the embedded system is behind a firewall with no direct access from the Internet, the connection must be initiated from the embedded system. A reverse SSH tunnel is set up as follows:</p> the embedded systems make a SSH connection to an external server that is internet accessible.</li> The SSH connection is set up so that a port on the external server (let’s pick 2900) is forwarded back over the SSH tunnel to port 22 (port used for SSH) on the embedded system.</li> The developer connects to 2900 on the external server using SSH. Because port 2900 on the external server gets forwarded over the SSH tunnel to port 22 of the embedded system, the developer is actually logging into the embedded system.</li> </ul> This connection is illustrated in the figure below:</p> </p> In the above figure, the external server is typically a system located in the developer’s office with only port 22 exposed to the Internet. Port 2900 is only accessible to developers on the same private network as the server.</p> Security</h3> When talking about any type of remote access technology involving the internet, it is important to consider the security implications. The reverse SSH mechanism is fairly secure, as everything is encrypted. SSH is widely used for remote access to servers; therefore it receives a lot of testing and scrutiny. With the reverse SSH remote access setup, the most likely attack is probably to compromise the external server and then try to access the remote network the embedded system is located on through the embedded system. If the external server is properly secured, then this mechanism should be fairly safe. One of the advantages of this scheme is that the embedded system is not directly exposed to the Internet, therefore securing the embedded system itself is not all that important.</p> It is obviously very important that the people administering the remote network understand what you are doing – that is where this article can come in handy. If there are concerns about external developers accessing a network, one mode of operation may be for the the SSH tunnel to be manually started and stopped so that administrators of the remote network know exactly when remote access is enabled.</p> TIPS</h3> A few ideas for making this work well:</p> It may be useful to put the reverse SSH connection in a cron job on the embedded system so that the connection is automatically set up. If you do this, I advise killing the connection and restarting every so often. Reverse SSH connections are somewhat fragile and occasionally need restarted if there is a network glitch.</li> Use the SSH TCPKeepAlive option. Some firewalls/routers will drop a network connection if there is no activity.</li> </ul> Summary</h3> Using SSH for remote access is another example of the “Big Win” with Embedded Linux – how industrial grade technologies developed for use in server and desktop systems can be readily used in embedded systems (/web/content/view/35/37/</a> ). Remote access during the deployment and testing of embedded systems can drastically reduce the amount of time and effort required to roll out an embedded system.</p> How to build GNU/Linux for an embedded x86 computer 2006-11-29T00:00:00+00:00 Recently, I had a customer who needed an embedded Linux distribution running on an Advantech PCM-9371 single board computer (SBC). The PCM-9371 contains a low voltage Celeron or Pentium III processor. This article describes why the Openembedded build system was chosen and a few tips for running Openembedded on a x86 system.</p> The hard way!</h2> There are many ways to put Linux on a x86 embedded system. There are hundreds of distributions out there and the temptation is always to do something “quick” and just try to trim down one of the standard distributions. Other shortcuts are often taken, such as compiling your application on a workstation that is running a different distribution and different versions of the libraries used in the embedded systems. This approach can work, but there are many possible pitfalls:</p> Several years down the road, it will be difficult to compile applications for your target system as the libraries on the target system are very dated compared to tools and libraries used in standard desktop distributions. What this means is you will need to keep a “golden” compile machine around for the life of your project – not fun! I went through an experience like this on a previous version of this same project where it took us a week to set up a build machine to compile applications using a very old version of Debian.</li> It is difficult to gather GPL sources used to build the distribution you are using. You are required to supply GPL source code upon request to customers who purchase your product.</li> To build applications correctly, you really need a build machine that is running the same distribution as the target, unless you set up a cross-compilation or scratchbox environment.</li> Most distributions are quite large and have difficulty fitting on CF card.</li> Most distributions are difficult to customize in a clean, systematic way.</li> </ul> OpenEmbedded</h2> Considering the disadvantages of using an standard x86 distribution for this project, I chose to use OpenEmbedded (OE). The OpenEmbedded project (http://openembedded.org</a> ) is a full featured build system for building embedded Linux distributions. The development is done on a Linux workstation of your choice and the entire distribution is cross-compiled to the architecture of your choice. The entire build process is automated, including building the toolchain and an image that can be installed on the target embedded system. Some of the features of OpenEmbedded are:</p> Over 3000 packages that can be built. About anything you would ever need for an embedded system.</li> Support for a wide variety of target architectures. ARM and x86 processors are well supported.</li> Build system is very self contained. This means you can do the builds on a variety of different host machines. Many of the tools OE needs are built by OE, so there is very little dependence on host tools besides a few basics such as Python, make, and other standard tools. This also means it is much easier to pull the build system out of the archives in several years, load it on a current machine, and do a build.</li> All source code used in the build is downloaded, so GPL compliance is easy.</li> and the list goes on …</li> </ul> Build and Installation</h2> Setting up an OE build is beyond the scope of this article, but it basically involves selecting a machine and distribution. In this case, I chose the x86</em> machine and the generic</em> OE distribution. Several hours later, I had an image in the form of a tar.gz file that I decompressed to a Compact Flash (CF) card. After installing grub on the CF card, I had a booting Linux distribution that included all the basics (like ssh, ipkg package manager, shell, standard file utilities, etc) in less than 4MB of disk space.</p> Adding GTK+ and Xwindows</h2> This system required support for GTK+ and Xwindows. To add this involved running the following commands: bitbake xserver-kdrive</em> and bitbake gtk+</em>. The OE build system then automatically downloaded and compiled the need packages. The image size is now around 20MB – still fits very easily on a CF card.</p> Getting X running</h2> There are several options for running X on a x86 system. In this case, I chose the quickest way so that my customer could get started with application integration as quickly as possible. Linux and Xwindows support the standard VESA Framebuffer mode. Then can be enabled by selecting the CONFIG_FB_VESA kernel config option. The kdrive version of X can then be started by running Xvesa. In the future, we may move to an accelerated X driver if needed.</p> Summary</h2> We now have Linux running off of a CF card with full X/GTK+ support and the customer can proceed with integration, testing and application development. Getting to this point took about 1 day of development as we did not encounter any major problems – much less time than it took to resurrect an old build environment used in previous versions of the project. Future work will involve integrating applications into the OE build system, and customizing the distribution as needed. Openembedded has proved to be a very valuable tool for getting embedded Linux systems running quickly and maintaining them in a controlled way.</p> Adding software to a Moxa UC7408 using OpenEmbedded 2006-10-30T00:00:00+00:00 One of the big advantages of using Linux in embedded systems is the thousands of applications and drivers available for it (/web/content/view/35/37/</a> ). One of the challenges of Embedded Linux is building these applications. Building applications for an embedded system often involves cross-compiling which is frustrating on a good day. This article details how to use OpenEmbedded (OE) with the existing toolchain Moxa provides to build a number of applications for the Moxa UC7408.</p> Web Application Support</h2> My needs for this project were to build a web application that ran on the UC7408 (See /web/content/view/55/9/</a> for a review of the Moxa UC7408). The Moxa firmware is fairly full featured and provides php for web application development. This is probably fine for many applications, but I prefer to build web applications as detailed in this article: /web/content/view/46/9/</a> . Because the rest of the application is written in Python, there are advantages to also implementing the web application in Python. I also wanted to use the sqlite database. A quick look at the php supplied with the Moxa firmware suggested it was built without sqlite support. In summary, I needed the following components:</p> Python</li> Clearsilver</li> sqlite</li> minicom</li> vim</li> strace</li> subversion</li> wget</li> </ul> The above is a fairly small list, but what you don’t see are all the dependencies that also need to be built and installed. This ends up being many more components. Some of the items are for development only, but this is Linux – why not have nice tools running on the target system?</p> OE to the rescue</h2> Ideally, the Moxa system would have a complete root file system built with OE running on it. But I did not have the time or budget to implement this, so I did the next best thing – use the rootfs and toolchain supplied by Moxa along with OE to compile just the applications I needed. Typically, OE builds the toolchain and Glibc, but fortunately OE provides a very slick way to use an external toolchain and glibc with OE. How to do this is detailed in the OE manual: http://www.openembedded.org/user-manual&dpage=commonuse_prebuilt_toolchain</a>. Below is the setup I used:</p> Environment variables:</p> TOOLCHAIN=/usr/local/mxscaleb/bin/ export PRE_BUILT=/usr/local/mxscaleb/armv5b-linux export CCACHE_DIR="${TOPDIR}/ccache" export PYTHONPATH="${TOPDIR}/bitbake/lib" export BBPATH="${TOPDIR}:${TOPDIR}/openembedded:${TOPDIR}/bitbake" export PATH="${TOPDIR}/bitbake/bin:${TOOLCHAIN}:${PATH}" export LD_LIBRARY_PATH= export LANG=C </code></pre> Local.conf file:</p> DL_DIR = "/build/sva_oe/downloads" BBFILES := "/build/sva_oe/openembedded/packages//.bb" BBMASK = "" PREFERRED_PROVIDERS = "virtual/qte:qte virtual/libqpe:libqpe-opie" PREFERRED_PROVIDERS += " virtual/libsdl:libsdl-qpe" PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}gcc-initial:gcc-cross-initial" PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}gcc:gcc-cross" PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}g++:gcc-cross" PREFERRED_PROVIDER_virtual/libx11 = "diet-x11" ASSUME_PROVIDED += " virtual/${TARGET_PREFIX}gcc " ASSUME_PROVIDED += " virtual/libc " MACHINE = "moxa" IPKG_ARCHS = "all arm armv4 armv5te xscale ${MACHINE}" TARGET_CC_ARCH = "-mcpu=xscale" TARGET_ARCH = "arm" PACKAGE_ARCH="xscale" TARGET_OS = "linux" TARGET_FPU = "soft" DISTRO = "moxa-disro" DISTRO_NAME = "moxa-distro" DISTRO_VERSION = "0.0.1" DISTRO_TYPE = "release" INHERIT += " package_ipk package_tar debian" IMAGE_FSTYPES = "jffs2 tar" PARALLEL_MAKE = "-j 4" BBINCLUDELOGS = "yes" CVS_TARBALL_STASH = "http://www.oesources.org/source/current/" export CC = "mxscaleb-gcc-3.3.2 ${HOST_CC_ARCH}" export CXX = "mxscaleb-g++ ${HOST_CC_ARCH}" export CPP = "mxscaleb-gcc-3.3.2 -E" export LD = "mxscaleb-ld" export AR = "mxscaleb-ar" export AS = "mxscaleb-as" export RANLIB = "mxscaleb-ranlib" export STRIP = "mxscaleb-strip" TARGET_CPPFLAGS_append = " -I${PRE_BUILT}/include " TARGET_LDFLAGS_prepend = " -L${PRE_BUILT}/lib -Wl,-rpath-link, ... </code></pre> ${PRE_BUILT}/lib -Wl,-rpath-link,${PRE_BUILT}/qt2/lib " </code></pre> With the above setup, I was able to build the applications I needed with very little effort. There were a few little problems I ran into, but I was able to quickly work around them. The result is a number of packages that can be installed on the target system:</p> bigreqsproto-dev-X11R7.0-1.0.2-r0.tar.gz python-crypt-2.4.3-ml0.tar.gz busybox-1.2.1-r1.3.tar.gz python-curses-2.4.3-ml1.tar.gz busybox-udhcpd-1.2.1-r1.3.tar.gz python-datetime-2.4.3-ml0.tar.gz clearsilver-0.10.3-r0.tar.gz python-db-2.4.3-ml0.tar.gz clearsilver-dbg-0.10.3-r0.tar.gz python-devel-2.4.3-ml0.tar.gz clearsilver-dev-0.10.3-r0.tar.gz python-distutils-2.4.3-ml0.tar.gz clearsilver-doc-0.10.3-r0.tar.gz python-email-2.4.3-ml0.tar.gz inputproto-dev-X11R7.0-1.3.2-r0.tar.gz python-fcntl-2.4.3-ml0.tar.gz ipkg-0.99.163-r1.tar.gz python-gdbm-2.4.3-ml0.tar.gz ipkg-dbg-0.99.163-r1.tar.gz python-hotshot-2.4.3-ml0.tar.gz ipkg-dev-0.99.163-r1.tar.gz python-html-2.4.3-ml0.tar.gz kbproto-dev-X11R7.1-1.0.3-r0.tar.gz python-idle-2.4.3-ml0.tar.gz libapr-0-0-0.9.12-r0.tar.gz python-image-2.4.3-ml0.tar.gz libapr-0-bin-0.9.12-r0.tar.gz python-io-2.4.3-ml0.tar.gz libapr-0-dev-0.9.12-r0.tar.gz python-lang-2.4.3-ml0.tar.gz libaprutil-0-0-0.9.12-r0.tar.gz python-lib-old-and-deprecated-2.4.3-ml0.tar.gz libaprutil-0-bin-0.9.12-r0.tar.gz python-logging-2.4.3-ml0.tar.gz libaprutil-0-dev-0.9.12-r0.tar.gz python-mailbox-2.4.3-ml0.tar.gz libcrypto0.9.7-0.9.7g-r1.tar.gz python-math-2.4.3-ml0.tar.gz libexpat-bin-2.0.0-r1.tar.gz python-mime-2.4.3-ml0.tar.gz libexpat-dev-2.0.0-r1.tar.gz python-mmap-2.4.3-ml0.tar.gz libexpat-doc-2.0.0-r1.tar.gz python-netclient-2.4.3-ml1.tar.gz libexpat1-2.0.0-r1.tar.gz python-netserver-2.4.3-ml0.tar.gz libgcrypt-dbg-1.2.2-r0.tar.gz python-pickle-2.4.3-ml0.tar.gz libgcrypt-dev-1.2.2-r0.tar.gz python-pprint-2.4.3-ml0.tar.gz libgcrypt-doc-1.2.2-r0.tar.gz python-profile-2.4.3-ml0.tar.gz libgcrypt11-1.2.2-r0.tar.gz python-pydoc-2.4.3-ml0.tar.gz libgdbm-dbg-1.8.3-r2.tar.gz python-pyserial-2.2-r1.tar.gz libgdbm-dev-1.8.3-r2.tar.gz python-pysqlite2-2.2.2-ml1.tar.gz libgdbm-doc-1.8.3-r2.tar.gz python-pyxml-0.8.4-ml0.tar.gz libgdbm3-1.8.3-r2.tar.gz python-re-2.4.3-ml0.tar.gz libgnutls-bin-1.4.0-r1.tar.gz python-readline-2.4.3-ml0.tar.gz libgnutls-dbg-1.4.0-r1.tar.gz python-resource-2.4.3-ml0.tar.gz libgnutls-dev-1.4.0-r1.tar.gz python-shell-2.4.3-ml0.tar.gz libgnutls-doc-1.4.0-r1.tar.gz python-stringold-2.4.3-ml0.tar.gz libgnutls-extra13-1.4.0-r1.tar.gz python-subprocess-2.4.3-ml0.tar.gz libgnutls-locale-en+boldquot-1.4.0-r1.tar.gz python-syslog-2.4.3-ml0.tar.gz libgnutls-locale-en+quot-1.4.0-r1.tar.gz python-terminal-2.4.3-ml0.tar.gz libgnutls-locale-pl-1.4.0-r1.tar.gz python-tests-2.4.3-ml0.tar.gz libgnutls-openssl13-1.4.0-r1.tar.gz python-textutils-2.4.3-ml0.tar.gz libgnutls13-1.4.0-r1.tar.gz python-threading-2.4.3-ml0.tar.gz libgpg-error-dbg-1.3-r1.tar.gz python-tkinter-2.4.3-ml0.tar.gz libgpg-error-dev-1.3-r1.tar.gz python-unittest-2.4.3-ml0.tar.gz libgpg-error-locale-de-1.3-r1.tar.gz python-unixadmin-2.4.3-ml0.tar.gz libgpg-error-locale-fr-1.3-r1.tar.gz python-xml-2.4.3-ml0.tar.gz libgpg-error-locale-pl-1.3-r1.tar.gz python-xmlrpc-2.4.3-ml0.tar.gz libgpg-error-locale-ro-1.3-r1.tar.gz python-zlib-2.4.3-ml1.tar.gz libgpg-error-locale-vi-1.3-r1.tar.gz sqlite3-3.3.7-r2.tar.gz libgpg-error0-1.3-r1.tar.gz sqlite3-dbg-3.3.7-r2.tar.gz libice-dbg-X11R7.1-1.0.1-r0.tar.gz strace-4.5.14-r3.tar.gz libice-dev-X11R7.1-1.0.1-r0.tar.gz strace-dbg-4.5.14-r3.tar.gz libice6-X11R7.1-1.0.1-r0.tar.gz strace-doc-4.5.14-r3.tar.gz libipkg-dev-0.99.163-r1.tar.gz subversion-1.3.1-r1.tar.gz libipkg0-0.99.163-r1.tar.gz subversion-dbg-1.3.1-r1.tar.gz liblzo-dbg-1.08-r14.tar.gz subversion-dev-1.3.1-r1.tar.gz liblzo-dev-1.08-r14.tar.gz subversion-doc-1.3.1-r1.tar.gz liblzo1-1.08-r14.tar.gz subversion-locale-de-1.3.1-r1.tar.gz libneon-bin-0.25.5-r0.tar.gz subversion-locale-es-1.3.1-r1.tar.gz libneon-dev-0.25.5-r0.tar.gz subversion-locale-fr-1.3.1-r1.tar.gz libneon-doc-0.25.5-r0.tar.gz subversion-locale-it-1.3.1-r1.tar.gz libneon25-0.25.5-r0.tar.gz subversion-locale-ja-1.3.1-r1.tar.gz libpython2.4-1.0-2.4.3-ml5.tar.gz subversion-locale-ko-1.3.1-r1.tar.gz libreadline-dbg-4.3-r3.tar.gz subversion-locale-nb-1.3.1-r1.tar.gz libreadline-dev-4.3-r3.tar.gz subversion-locale-pl-1.3.1-r1.tar.gz libreadline-doc-4.3-r3.tar.gz subversion-locale-pt-br-1.3.1-r1.tar.gz libreadline4-4.3-r3.tar.gz subversion-locale-sv-1.3.1-r1.tar.gz libsm-dbg-X11R7.1-1.0.1-r0.tar.gz subversion-locale-zh-cn-1.3.1-r1.tar.gz libsm-dev-X11R7.1-1.0.1-r0.tar.gz subversion-locale-zh-tw-1.3.1-r1.tar.gz libsm6-X11R7.1-1.0.1-r0.tar.gz tcl-8.4.11-r3.tar.gz libsqlite-bin-2.8.17-r2.tar.gz tcl-dbg-8.4.11-r3.tar.gz libsqlite-dbg-2.8.17-r2.tar.gz tcl-dev-8.4.11-r3.tar.gz libsqlite-dev-2.8.17-r2.tar.gz tcl-doc-8.4.11-r3.tar.gz libsqlite0-2.8.17-r2.tar.gz time-1.7-r0.tar.gz libsqlite3-0-3.3.7-r2.tar.gz time-dbg-1.7-r0.tar.gz libsqlite3-dev-3.3.7-r2.tar.gz time-doc-1.7-r0.tar.gz libssl0.9.7-0.9.7g-r1.tar.gz tmp libx11-6-X11R7.1-1.0.1-r1.tar.gz update-rc.d-0.7-r0.tar.gz libx11-dbg-X11R7.1-1.0.1-r1.tar.gz util-macros-dev-X11R7.1-1.0.2-r0.tar.gz libx11-dev-X11R7.1-1.0.1-r1.tar.gz vim-7.0-r0.tar.gz libx11-doc-X11R7.1-1.0.1-r1.tar.gz vim-doc-7.0-r0.tar.gz libx11-locale-X11R7.1-1.0.1-r1.tar.gz vim-help-7.0-r0.tar.gz libxau-dbg-X11R7.1-1.0.1-r0.tar.gz vim-syntax-7.0-r0.tar.gz libxau-dev-X11R7.1-1.0.1-r0.tar.gz vim-tutor-7.0-r0.tar.gz libxau-doc-X11R7.1-1.0.1-r0.tar.gz wget-1.9.1-r6.tar.gz libxau6-X11R7.1-1.0.1-r0.tar.gz wget-dbg-1.9.1-r6.tar.gz libxdmcp-dbg-X11R7.1-1.0.1-r0.tar.gz wget-doc-1.9.1-r6.tar.gz libxdmcp-dev-X11R7.1-1.0.1-r0.tar.gz wget-locale-bg-1.9.1-r6.tar.gz libxdmcp6-X11R7.1-1.0.1-r0.tar.gz wget-locale-ca-1.9.1-r6.tar.gz libxml2-2.6.22-r3.tar.gz wget-locale-cs-1.9.1-r6.tar.gz libxml2-dbg-2.6.22-r3.tar.gz wget-locale-da-1.9.1-r6.tar.gz libxml2-dev-2.6.22-r3.tar.gz wget-locale-de-1.9.1-r6.tar.gz libxml2-doc-2.6.22-r3.tar.gz wget-locale-el-1.9.1-r6.tar.gz libxml2-utils-2.6.22-r3.tar.gz wget-locale-es-1.9.1-r6.tar.gz libxt-dbg-X11R7.1-1.0.2-r0.tar.gz wget-locale-et-1.9.1-r6.tar.gz libxt-dev-X11R7.1-1.0.2-r0.tar.gz wget-locale-fr-1.9.1-r6.tar.gz libxt-doc-X11R7.1-1.0.2-r0.tar.gz wget-locale-gl-1.9.1-r6.tar.gz libxt6-X11R7.1-1.0.2-r0.tar.gz wget-locale-he-1.9.1-r6.tar.gz libz-dbg-1.2.3-r1.tar.gz wget-locale-hr-1.9.1-r6.tar.gz libz-dev-1.2.3-r1.tar.gz wget-locale-hu-1.9.1-r6.tar.gz libz1-1.2.3-r1.tar.gz wget-locale-it-1.9.1-r6.tar.gz minicom-2.1-r0.tar.gz wget-locale-ja-1.9.1-r6.tar.gz minicom-dbg-2.1-r0.tar.gz wget-locale-nl-1.9.1-r6.tar.gz minicom-doc-2.1-r0.tar.gz wget-locale-no-1.9.1-r6.tar.gz ncurses-5.4-r8.tar.gz wget-locale-pl-1.9.1-r6.tar.gz ncurses-dbg-5.4-r8.tar.gz wget-locale-pt-br-1.9.1-r6.tar.gz ncurses-dev-5.4-r8.tar.gz wget-locale-ro-1.9.1-r6.tar.gz ncurses-doc-5.4-r8.tar.gz wget-locale-ru-1.9.1-r6.tar.gz ncurses-terminfo-5.4-r8.tar.gz wget-locale-sk-1.9.1-r6.tar.gz ncurses-tools-5.4-r8.tar.gz wget-locale-sl-1.9.1-r6.tar.gz openssl-0.9.7g-r1.tar.gz wget-locale-sv-1.9.1-r6.tar.gz openssl-dbg-0.9.7g-r1.tar.gz wget-locale-tr-1.9.1-r6.tar.gz openssl-dev-0.9.7g-r1.tar.gz wget-locale-uk-1.9.1-r6.tar.gz openssl-doc-0.9.7g-r1.tar.gz wget-locale-zh-cn-1.9.1-r6.tar.gz python-audio-2.4.3-ml0.tar.gz wget-locale-zh-tw-1.9.1-r6.tar.gz python-bsddb-2.4.3-ml0.tar.gz xcmiscproto-dev-X11R7.0-1.1.2-r0.tar.gz python-codecs-2.4.3-ml0.tar.gz xextproto-dev-X11R7.0-7.0.2-r0.tar.gz python-compile-2.4.3-ml0.tar.gz xf86bigfontproto-dev-X11R7.0-1.1.2-r0.tar.gz python-compiler-2.4.3-ml0.tar.gz xproto-dev-X11R7.1-7.0.5-r0.tar.gz python-compression-2.4.3-ml0.tar.gz xtrans-dev-X11R7.0-1.0.0-r1.tar.gz python-core-2.4.3-ml1.tar.gz </code></pre> Installing the needed packages is just a matter of uncompressing a subset of the above files on the target system.</p> Summary</h2> This exercise demonstrates how a developer can quickly (took me less than 4 hours) add a large number of applications (and dependencies) to about any Embedded Linux system. Leveraging embedded Linux is being able to use the components you</strong></em> need. Please contact us if you would like additional details or assistance.</p> Moxa UC7408 Review 2006-09-29T00:00:00+00:00 I am currently evaluating a Moxa UC7408 for one of my customers. The UC7408 is a small fanless industrial computer that runs Linux or Windows CE. This article provides a basic overview of this unit and a review of some of the UC7408 features and the Linux distribution Moxa provides.</p> </p> The basic specifications for the UC7408 are:</p> relatively small size and fanless</li> Intel Xscale IXP-422 266MHz Processor</li> 8 RS232/422/485 serial ports</li> 8 digital Input and 8 digital output ports</li> dual 10/100 Ethernet</li> PCMCIA, CompactFlash, Wireless LAN Expansion (supports 802.11b/802.11g)</li> Runs Linux or WinCE</li> Console serial port</li> </ul> Documentation</h3> The documentation Moxa provides is adequate and fairly detailed. The moxa documents include a quick install guide, hardware user’s manual, and a 114 page user’s manual. As good as this documentation is, a general knowledge of Linux is always helpful when working with embedded Linux systems. Basics like using SSH, FTP, Telnet, and basic Linux system administration are essential when working with embedded Linux.</p> Linux Distribution</h3> The Linux Distribution provided by Moxa is fairly full featured. A few highlights with the v1.8 Linux firmware:</p> Provides a 26MB user JFFS2 flash partition. This partition is read/write and is about 45% full leaving about 14.5MB for user files. It is probably also possible to delete some of the files that are not required, freeing up more space.</li> Apache & PHP support</li> SSH</li> a number of other utilities are provided such as telnet, ftp, iptables, etc.</li> VI editor</li> Many command line utilities found in most Linux systems.</li> 2.4.18 Kernel</li> </ul> The Linux distribution seems to be based on MontaVista Linux and seems fairly solid. I ran through some of the basics like mounting a CF card and everything seems to work.</p> Firmware Updates and Recovery Mode</h3> Moxa provides a mechanism to update the firmware in the system. This update is a global flash update that programs the entire flash and will erase all user changes. The update works by enabling a RAM file system and copying the new flash image to the ram file system. Running an update command that copies the firmware update file into flash.</p> The 7408 contains a stripped down version of Linux in a separate flash partition that can be booted in a recovery mode if the User flash partition becomes corrupted or unusable. This mode can be enabled by pressing the “Reset to Default” button on the unit and powering it on. In this mode, there is no ssh support and you must ftp an image from an ftp server to the device and then reflash it. This requires you to set up an ftp server that the Moxa system can access. While this is an entirely workable solution, it may be a challenge for inexperienced Linux users.</p> Tools</h3> Moxa provides a Linux and Windows toolchain to use with the device. I have built a number of packages with the Linux toolchain and it seems to work well. It is based on GCC 3.3.2. Cross compiling applications for embedded Linux is always a challenge and in a future article I will present a way to use OpenEmbedded to compile a number of packages using the Moxa toolchain.</p> Possible Improvements</h3> There would be some benefits if the system was a little more open in the following areas:</p> access to the bootloader console and documentation. This would allow developers to more easily flash their own software.</li> readily provide source for GPL components. Moxa provides a form that you can fill out to receive GPL source code on a CE. The cost is $100.</li> </ul> One of the big advantages of using Linux in an embedded system is the ability to use the many open source components available. The easier an embedded solution provider makes this for developers, the more value their system provides. See the white paper Tips for Planning Your Embedded Linux Project</a> (/web/content/view/35/37/) for more ideas on this subject.</p> Summary</h3> Overall, the Moxa system seems very usable and the hardware is quite nice. Stay tuned for future articles about how to get the most out of this system.</p> </p> </p> </p> How to evaluate VoIP providers with Asterisk 2006-09-20T00:00:00+00:00 There are many options when selecting a VoIP provider to terminate your Asterisk PBX. One of the most important factors in this selection is the quality of the internet connection between your PBX and the VoIP provider. The route between you and your VoIP provider is one of the most important considerations when selecting a provider. This article details a simple method for evaluating this connection using Asterisk.</p> Asterisk includes a feature that can be used to monitor the latency between your system and a peer</em> or friend</em>. This feature is enabled by setting the qualify</em> setting in the iax.conf</em> configuration file. Valid options are yes</em>, no</em>, or a time in milliseconds. If qualify is enabled, NOTIFY messages are sent periodically to the peer and the latency between replies is measured. The peer is determined unreachable if the number of milliseconds is greater than the qualify value or 2,000 if qualify is set to yes</em>. If a peer is unreachable, events are logged in /var/log/asterisk/messages as shown below:</p> Sep 19 21:48:19 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now UNREACHABLE! Time: 49 Sep 19 21:50:09 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now REACHABLE! Time: 49 Sep 20 07:16:38 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now TOO LAGGED (769 ms)! Sep 20 07:16:48 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now REACHABLE! Time: 50 Sep 20 12:09:31 NOTICE[752] chan_iax2.c: Peer 'teliax' is now TOO LAGGED (2923 ms)! Sep 20 12:17:27 NOTICE[752] chan_iax2.c: Peer 'teliax' is now REACHABLE! Time: 83 </code></pre> I am currently using two different VoIP providers and I get qualify notice messages about once every day or two with the qualify set to 400ms. This is a very useful information as it allows me to determine which provider has the most reliable route from my office. If you get very few qualify NOTICE messages, you can probably assume your VoIP connection will be fairly reliable.</p> Have connections with two different providers also tells me when the congestion is with my Internet connection or if it is something beyond my connection. If both providers go out at the same time, it is most likely the internet connection at my office or problems with my ISP. I rarely if ever see this scenario, so I can conclude that my internet connection is fairly reliable and the congestion problems are most likely between me and the VoIP provider or problems at the VoIP provider.</p> Overall, things are now working fairly well. As we continue to resolve problems, VoIP is now starting to become a very usable option for us.</p> Implenting QoS for VOIP in a Linksys router 2006-08-28T00:00:00+00:00 In our quest to implement an Asterisk PBX in our office, we are evaluating several VOIP (Voice Over IP) providers. VOIP provides several benefits over traditional phone lines including:</p> low cost (typically around $0.02 per minute)</li> available in pre-paid plans</li> can support concurrent concurrent calls on the same number</li> </ul> However, implementing VOIP successfully is not trivial. You must</strong> have a reliable internet connection and we have found you must implement some type of Quality of Service (QoS) in the router. The first problem I ran into was calls sounded great to me, but people I called mentioned the calls sometimes sounded garbled to them. This article discusses the diagnosis of this problem and how to implement QoS in a Linksys router to fix this issue.</p> Our connection is a standard DSL connection – 5Mb/sec down and 640Kb/sec up. There is plenty of bandwidth for incoming voice, but bandwidth for outgoing voice is much more limited. A few tests confirmed this – when a ftp upload or some other bulk transfer was occurring, the outgoing voice sounded really bad.</p> The solution to this problem is to implement some type of Quality of Service (QoS) mechanism that prioritizes the VOIP traffic over all other types of traffic. Vonage and other “boxed” solutions no doubt do this as they recommend placing their box between the modem and the router. It turns out that QoS can be implemented in a standard Linksys router/firewall. I spent some time trying to get QoS working using a Linksys WRT54G v5 router with no success, I ended up loading the dd-wrt firmware (http://www.dd-wrt.com</a>) on an older Linksys router I had laying around. The dd-wrt firmware is quite nice and is worthy of consideration if you have a v4 or earlier Linksys router or the WRT54GL version. QoS may have worked on the WRT54G v5 with Linksys firmware with some more effort, but after trying dd-wrt, I’m not going back.</p> Setting up QOS</h2> The QoS settings can be found under Applications & Gaming -> QoS.</p> There are many ways to assign priority to your VOIP system. I just used the IP address as shown below. 192.168.1.10 is the LAN private IP of the Asterisk system.</p> </p> With the priority of the Asterisk box set to high, we should be all set. It turns out that unless you cap the bandwidth, you will still have QoS problems. Enter around 85% of your up and down bandwidth in the Uplink and Downlink boxes. Capping the bandwidth is required to keep the latency low. Otherwise, your bandwidth is filled up with large packets and the priority settings are not effective for VOIP traffic. See http://www.dd-wrt.com/wiki/index.php/Quality_of_Service</a> for more information.</p> </p> With the above settings, VOIP calls are now clear even with other concurrent activities.</p> UPDATE:</strong></p> Several people have reported being able to install dd-wrt on v6 WRT54 routers and the QoS configuration is the same.</p> Embedded is a Service Industry 2006-06-02T00:00:00+00:00 Linuxdevices.com has recently published an article titled: Let’s call Embedded what it is – a Service Industry</em>(http://linuxdevices.com/articles/AT6989225710.html</a>). In this article, Curt Schacker discusses how little growth there has been in the Embedded Systems industry. He claims that most embedded companies are trying to establish themselves as product companies, when the market is really a services market. Having worked for two Embedded companies in the past that went out of business, I think he is right. I have often heard the argument that service companies are too much work, they don’t scale, etc. But, to me services has always been a more natural fit for myself and an embedded systems company – one of the reasons BEC Systems exists.</p> TRAC and Distributed Development 2006-05-29T00:00:00+00:00 I have long been a big fan of source control tools and issue tracking systems. The “concurrent” model made popular by CVS is a tremendous benefit any time there are multiple people working on the same project. For most of the projects BEC participates in, we end up setting up a source control system (usually Subversion these days) and an issue tracking system (typically Bugzilla). On one of my recent projects, I decided to try the trac integrated SCM & Project Management tool. Trac brings together several functions into a clean, easy to use web based tool for managing projects. This article relates some of my experiences and the benefits of using trac.</p> What is trac?</h2> Trac provides the following functions (from the trac website - http://www.edgewall.com/trac/</a> ):</p> An integrated system for managing software projects</li> An enhanced wiki</li> A flexible web-based issue tracker</li> An interface to the Subversion revision control system</li> </ul> Why is trac so interesting?</h2> Although all of these functions are available as separate tools, having them in one tool has several benefits:</p> The tool is more likely to be used if everything is in one place</li> The ease of cross referencing wiki pages, issues, and Subversion changesets compounds the usefulness of each tool.</li> </ul> I have been using and implementing collaboration tools for years and one thing you eventually learn is that getting people to use collaboration tools is a big challenge (see http://www.linux-watch.com/news/NS9450464356.html</a> for some more thoughts on this subject). Having mutiple tools in one place greatly increases the likelihood they will be used. A typical session might go something like this:</p> A developer might want to see what has changed recently in the code base, so he logs into trac to view the timeline – after all, most developers are mainly concerned with code.</li> As the trac timeline lists all changes (wiki and issues as well as source code changes), he notices that a wiki page with notes about driver X has changed – that looks interesting, so he view the page. While viewing the page, the developer notes there is some missing information, so he adds the information by simply editing the page.</li> Once this habit is established, it can greatly improve the productivity of the team as the project becomes more transparent to all team members.</li> </ul> Any collaboration tool must be easy to use and provide an obvious benefit to those using the tool.</p> The wiki markup used in trac is used for both wiki pages as well as issue/bug reports. Referencing an issue in a wiki page or another bug report is simple as typing #. It is also very easy to reference code changsets in the wiki markup by simply typing r (see image below).</p> </p> Clicking on a issue or changeset number will take you directly to the information. The trac interface is very clean and very easy to use.</p> Distributed teams</h2> Tools like trac allow for very effective distributed teams. The success of Open Source software has proven that development can occur efficiently in a distributed manner. BEC has integrated this model and we regularly work with people all over the country and in Europe. When there is a process for all people involved in the project to keep up with its status and progress, distributed development becomes very possible. Often, the customer is part of the development team.</p> Project Transparency</h2> Another benefit of trac for BEC, is that we can easily provide project transparency to our customers. It is BEC’s practice to provide customers with access to the project as it develops, so we give our customers trac logins and they can follow the progress by watching the timeline and milestone pages. This gives the customer an assurance that progress is being made on the project and some insight into some of the issues encountered. Having a comprehensive view into the project also allows customers to provide early feedback and reduces BEC’s project management and reporting overhead.</p> Summary</h2> Tools like trac improve our customer relationships, increase efficiency and reduce our costs. If you are interested in implementing tools like trac, BEC can assist you in selecting and implementing the right tools to make your projects run smoothly.</p> Opencores SPI Controller Review 2006-05-08T00:00:00+00:00 Recently, I had the opportunity to use a SPI controller from Opencores in a customer design based on an Intel PXA255 processor and a Xilinx CPLD. The customer did not want to change the processor or the CPLD family, but needed an extra SPI interface in the device. I had my doubts, but mentioned I would look into a SPI controller from Opencores. To my surprise, the core fit in a Xilinx CPLD and worked out very well. We have since upgraded to a Altera MAXII CPLD. This review details this effort.</p> SPI Controller Overview</h3> The SPI controller is based on the interface for a standard Motorolla SPI controller and is available at http://www.opencores.org/projects.cgi/web/spi/overview</a> . The core includes the following features:</p> Full duplex synchronous serial data transfer</li> Variable length of transfer word up to 128 bits</li> MSB or LSB first data transfer</li> Rx and Tx on both rising or falling edge of serial clock independently</li> 8 slave select lines</li> Fully static synchronous design with one clock domain</li> Technology independent Verilog</li> Fully synthesizable</li> </ul> A datasheet type document is provided with the core that provides the necessary information for developers writing software for this core.</p> The Wishbone bus</h3> Most of the cores from Opencores uses the Wishbone bus standard. The Wishbone bus is a synchronous bus designed to be used inside an IC. The CPLD was connected to the PXA255 processors asynchrous bus, so most of the effort required was to create an PXA255 <-> Wishbone interface module. At some point, I hope to release this module on Opencores.</p> Simulation</h3> The simulation was done using the Icarus Verilog simulator under Linux. Below is a screen shot of the simulation displayed in GTKWave:</p> </p> Conclusion</h3> The Opencores SPI core worked out very well in our design and I was very impressed with quality of this core. The documentation is more than adequate. Standards like Verilog make it very easy to move designs between different vendors devices and allows designers to use tools of their choice such as the Icarus Verilog simulator. As FPGAs and CPLDs become more larger and more standand, Opencores is valuable resource for adding standard interfaces to these devices.</p> A Really Nice Hex Calculator 2006-05-02T00:00:00+00:00 When working with embedded software, I often have a need for a hex calculator. Over the years I’ve used a number of different tools. Through college and for several years after, I used my HP48 calculator. It is a great tool. Around 1998, I discovered the Python programming language. As Python supports an interactive session, it works well as a desktop calculator – I don’t think I have used my HP48 since I discovered Python. Recently, I’ve been using Ruby more in place of Python and am finding it is even nicer. This article presents several tips and snippets of code for using Ruby as a very effective hex calculator.</p> For those of you not familiar with Ruby, Ruby is an interpreted scripting language for quick and easy object-oriented programming – similar in concept to Python or Perl.</p> Ruby provides an interactive Ruby shell called “irb”. After you install Ruby, you can run irb from a command prompt. You can immediately do useful calculations like the following:</p> irb(main):001:0> a = 10 => 10 irb(main):002:0> b = 23 => 23 irb(main):003:0> a+b => 33 </code></pre> Hexadecimal numbers are prefixed with “0x” as in many other languages:</p> irb(main):005:0> c = 0x1000 => 4096 </code></pre> It is easy to convert from hex to decimal – just type the hex number, hit return and Ruby will give you the decimal equivalent as shown above. To convert from decimal to hex, I often just use the printf</em> statement. The syntax is similar to C, so it is very natural for embedded programmers.</p> irb(main):006:0> printf "0x%x", (310241024) 0x300000=> nil </code></pre> Ruby provides standard shift operators similar to those found in C. For example, if you want to know the hex value when bits 15 and 23 are set high:</p> irb(main):019:0> printf "0x%x", (1<<15)\|(1<<23) 0x808000=> nil </code></pre> It is often useful to know what bits are set in a particular number when determining if a bit in a status register is high, what address bits are used, etc. Ruby provides a bit reference method in the form.</p> fix[n] </code></pre> n is the nth bit in a binary number fix</em> where fix[0] is the least significant bit. Some examples:</p> irb(main):010:0> 2[0] => 0 irb(main):011:0> 2[1] => 1 irb(main):017:0> a = 310241024 => 3145728 irb(main):018:0> 31.downto(0) {\|n\| print a[n]} 00000000001100000000000000000000=> 31 </code></pre> For convenience, I wrote a simple module called bits.rb</em> that extends the basic Integer number class and provides a nice display of the bits in a number. To use this, create a file called bits.rb with the following contents and drop it in your Ruby site_ruby</em> directory.</p> M = 10241024 K = 1024 class Integer def bits print "3322222222221111111111n" print "10987654321098765432109876543210n" print "---\|---\|---\|---\|---\|---\|---\|---\|n" 31.downto(0) { \|x\| print self[x] } end end </code></pre> And to use this extension to see what address bits are used to decode 8MB of address space:</p> irb(main):020:0> require 'bits' => true irb(main):021:0> (8*M).bits 3322222222221111111111 10987654321098765432109876543210 ---\|---\|---\|---\|---\|---\|---\|---\| 00000000100000000000000000000000=> 31 </code></pre> From this example, we can see that address bits 0 through 23 are required to decode 8MB of address space.</p> ===========================================</p> Update 2009-07-30</strong></p> The following was posted as a comment to my old web site:</p> An even easier way of converting between bases in Ruby:</p> irb(main):001:0> puts 300.to_s(16) 12c => nil irb(main):002:0> puts "12c".to_i(16) 300 => nil irb(main):003:0> puts 255.to_s(2) 11111111 => nil </code></pre> Very nice!</p> Specialization or not 2005-11-15T00:00:00+00:00 I have often wondered about specialization. We can often find ourselves in ruts if we specialize too much. But on the other hand, specialization can lead efficiencies in some cases. In my work as a consultant, I often find myself in new situations where I have to learn fast. As one of my associates states – problem solving is the key skill. Kevin Barnes article Three theories on how to use developers efficiently</a> is a good read about the subject.</p> Total Phase USB Protocol Analyzer review 2005-11-03T00:00:00+00:00 I recently purchased a USB protocol analyzer from Total Phase</a>. The device will analyze full speed USB traffic and works well with Linux. On Linux, the software makes use of libusb, so there is very little setup required to get the device working – very nice. At a price of $400, there is really no reason not to have one if you are doing any type of USB development.</p> </p> Economics and Finding Outsourcing Talent 2005-11-03T00:00:00+00:00 Economics In One Lesson by Henry Hazlitt presents an interesting and logical view of economics. The book presents many ideas that seem right (like saving) and provides very logical arguments to back them up. Even though the book was originally published in 1946, the topics covered seem very applicable to today (The Blessings of Destruction, Public Works Mean Taxes, Taxes Discourage Production, Credit Diverts Production, The Curse of Machinery, Spread-the-Work Schemes, Disbanding Troops and Bureaucrats, The Fetish of Full Employment, Who’s “Protected” by Tariffs?, The Drive for Exports, “Parity” Prices, Saving the X Industry, How the Price System Works, “Stabilizing” Commodities, Government Price-Fixing, What Rent Control Does, Minimum Wage Laws, Do Unions Really Raise Wages?, “Enough to Buy Back the Product”, The Function of Profits, The Mirage of Inflation, The Assault on Saving)</p> The book can be found:</p> online: http://jim.com/econ/contents.html</a></li> download: http://www.fee.org/pdf/books/Economics_in_one_lesson.pdf</a></li> amazon.com</li> </ul> The following article presents some of the realities of finding talent in places like India. It is encouraging that the basic economic laws of supply and demand still work.</p> Finding coders on the subcontinent</a></h3>}

Survey of device ID mechanisms

2025-04-15T00:00:00+00:00

Are you implementing an IoT system that requires you to identify many connected devices? This post reviews different types of IDs and provides some technical details to help you decide if any of these are appropriate for your application.

</figure>

Note: some of the below information was generated by Perplexity and may not be 100% accurate. The goal of this article is to get a sense of all the different types of IDs in use. To implement anything, use the official documentation.

UUIDs</h2>
UUIDs (Universally Unique Identifiers) are often used in distributed software systems where multiple distributed systems create objects that must be uniquely identified. UUIDs are typically 128-bit numbers represented in a 36-character format that might look like: d50ae174-b65f-4ab9-b938-5f99b67ac50b. Base64 encoded UUIDs can be as short as 22 characters and look like: 1QrhdLZfSrm5OF-ZtnrFCw. UUIDs are typically constructed using the time and the MAC address of the device generating the UUID.
These IDs are attractive because no central organization or database is required to prevent conflicts - they can be created anywhere at will with a statistically insignificant possibility that there will ever be a collision. However, these IDs are large and best processed by machines, not humans.

MAC Address/EUI-48</h2>
Network devices use a MAC (Media Access Control) address, also called (EUI-48), to uniquely identify a device on a network. These are 48-bit numbers typically displayed as 12 hexadecimal digits and grouped into 6 pairs (`00:1A:2B:3C:4D:5E</code>). The first half of the MAC address is called the OUI (Organizationally Unique Identifier).` `IEEE is the registration authority</a> for MAC address blocks. You can purchase 4K, 1M, and 16M size blocks; the cost ranges from $1,080 to $3,680 at the time of writing.`
`You can also purchase EEPROM integrated circuits, like the 24AA02E48</a> from Microchip, that have a EUI-48 number already programmed into it. This is a nice option for lower volume products where you don’t want to mess around with managing the EUI-48 numbers.`
`One advantage of using a MAC address is that it may already be programmed into the device you are using, so it does not require any extra effort to allocate and program the ID.`

Bluetooth Low Energy (BLE)</h2> BLE uses “MAC” (EUI-48) hardware addresses from the same pool as Ethernet. Additionally, the following IDs are used: Identifier Type</th> Size</th> Purpose/Usage</th> Example</th></tr></thead> Bluetooth Address (EUI-48)</td> 48 bits</td> Device-level unique ID (public or random)</td> 12:34:56:78:9A:BC</td></tr> UUID (16/32/128-bit)</td> 16/32/128b</td> Service/characteristic identification</td> 0x180D, 12345678-...</td></tr> Company Identifier</td> 16 bits</td> Manufacturer/vendor identification</td> 0x004C (Apple)</td></tr> System ID (Device Info)</td> 64 bits</td> Encodes device's MAC address (optional)</td> (varies)</td></tr> Local Name</td> Variable</td> Human-readable device name</td> "Charge HR"</td></tr> Appearance/Class Codes</td> 16 bits</td> Device type/classification</td> 0x0340 (Heart Rate Sensor)</td></tr></tbody></table></figure> Bluetooth Mesh</h2> Bluetooth Mesh networking introduces several unique identifiers (IDs) at various layers of the protocol to manage devices, messages, and network security. These IDs serve different purposes than traditional Bluetooth or BLE device addresses. Unicast Address (Node Address/Element Address) Each node (device) in a Bluetooth Mesh network is assigned a unique unicast address during the provisioning process. This is a 16-bit value that uniquely identifies the node or its elements within the mesh network.</li> A node can have multiple elements, each with its own unicast address in a contiguous range.</li> </ul> </li> Group Address 16-bit values used to address multiple nodes simultaneously (e.g., all lights in a room). Nodes can subscribe to one or more group addresses, enabling efficient publish/subscribe communication.</li> </ul> </li> Virtual Address 128-bit values derived from a UUID. They allow addressing of a logical group of nodes, regardless of their physical location or provisioning sequence. Virtual addresses are mapped to a 16-bit address space for use in messages.</li> </ul> </li> Device Key (DevKey) A unique 128-bit cryptographic key assigned to each node during provisioning. It is used for secure communication between the provisioner and the node, especially for configuration and management messages.</li> </ul> </li> Network Key (NetKey) and Application Key (AppKey) Network Keys are shared among all nodes in a mesh network and are used to secure network layer messages.</li> Application Keys are used to secure application-specific messages, allowing for separation of different functionalities (e.g., lighting vs. sensor data).</li> </ul> </li> Model Identifier Each mesh model (defining device behavior, such as a light switch or sensor) is assigned a unique identifier:</li> SIG-adopted models: Identified by a 16-bit model identifier.</li> Vendor-specific models: Identified by a 16-bit vendor ID and a 16-bit model ID.</li> </ul> </li> Message Sequence Number Every message sent by a node includes a sequence number to prevent replay attacks and ensure message uniqueness.</li> </ul> </li> Device Property ID For sensor and property models, each property (e.g., temperature, humidity) is assigned a unique Device Property ID as per the Bluetooth Mesh device properties specification.</li> </ul> </li> Node ID Some documentation refers to a Node ID (or nodeId), typically a 2-byte value, as a unique identifier for a mesh device within the network, used for addressing and control.</li> </ul> </li> </ol> IPv6/EUI-64</h2> IPv6/EUI-64 addresses are a 64-bit number commonly used in IPv6 and typically displayed as 8 groups of four hexadecimal digits: 2001:0db8:85a3:0000:0000:8a2e:0370:7334</code>. Consecutive groups of zeros can be replaced with a double colon. For example, 2001:0db8:0000:0000:0000:0000:0000:0001</code> can be shorted to 2001:0db8::1</code>. Note, leading zeros in the group are also omitted. IPv6 address blocks are assigned by the Internet Assigned Numbers Authority (IANA), which allocates large blocks of addresses to Regional Internet Registries (RIRs). These RIRs then distribute smaller blocks to Internet Service Providers (ISPs) and other organizations. The ARIN (American Registry for Internet Numbers) provides blocks of IPv6 addresses. The annual fee for a 3X-Small block (288 addresses) is $262.50 at the time of writing. There are also EEPROM and other ICs that have EUI-64 numbers preprogrammed, such as the Microchip 24AA02E64</a>. QR Codes</h2> QR codes provide the opportunity to easily read a UUIDs or some other long ID on every device using a phone camera. This relieves some of the complexity of humans having to deal with long IDs. Entering a known device in the field into an application is easy with QR codes. However, finding a device (one among many) in the field can be difficult unless the application has a feature designed just for this. </figure> There is no central registry for QR codes. Instead, the process of QR code generation and use is decentralized and relies on the vast number of possible QR code combinations to avoid conflicts. SNMP/Private Enterprise Numbers (PEN)</h2> A SNMP Private Enterprise Number (PEN) is a unique identifier assigned by the Internet Assigned Numbers Authority (IANA) to organizations, individuals, or entities. It is primarily used in network management protocols like Simple Network Management Protocol (SNMP) to identify objects defined by an enterprise within the Management Information Base (MIB) structure. PENs are part of the Object Identifier (OID) hierarchy under the prefix 1.3.6.1.4.1</code>, which corresponds to .iso.org.dod.internet.private.enterprise</code>. This prefix ensures that each PEN is globally unique. Any entity can request a PEN assignment from IANA at no cost</a>. Automobile VIN</h2> The modern 17-character Vehicle Identification Number (VIN) format was first conceptualized and standardized by the United States National Highway Traffic Safety Administration (NHTSA) in 1981. This standardization aimed to unify the previously inconsistent vehicle identification practices across manufacturers and facilitate more effective tracking of recalls, registrations, and thefts. The VIN system introduced by NHTSA was later adopted globally and is now governed by the International Organization for Standardization (ISO), specifically under ISO 3779 (content and structure) and ISO 4030 (location and attachment) An example of an automobile VIN (Vehicle Identification Number) is: 1P8ZA1279SZ215470 VINs are always 17 characters long and use a combination of capital letters and digits. Each section of the VIN provides specific information about the vehicle: The first three characters (World Manufacturer Identifier) indicate the manufacturer and country of origin.</li> Characters 4–8 describe features such as model, body type, and engine.</li> The 9th character is a check digit for security.</li> The 10th character indicates the model year (e.g., “S” for 1995, “R” for 1994 or 2024).</li> The 11th character shows the assembly plant.</li> The last six characters are a unique serial number for that vehicle.</li> </ul> KNX Serial Number</h2> KNX is a popular building automation standard. The KNX serial numbers</a> follow the following format: Length: The KNX serial number is always 6 bytes (octets) long. Manufacturer Code: Octets 6 and 5 (the first two bytes, when read from left to right) contain the official KNX Manufacturer Code assigned to each manufacturer by the KNX Association.</li> </ul> </li> Device-Specific Portion: Octets 4 to 1 (the remaining four bytes) are managed by the manufacturer.</li> This part may encode information such as production site, date, series, batch, or simply a sequential unique number.</li> </ul> </li> </ul> BACnet</h2> BACnet Device IDs, also known as Device Object Identifiers or Device Instance Numbers, are unique numerical values assigned to each BACnet device on a network to identify them distinctly. The Device ID is a 22-bit unsigned integer ranging from 0 to 4,194,302 (decimal). The value 4,194,303 (0x3FFFFF) is reserved and cannot be used as a valid Device ID. This Device ID corresponds to the object instance number portion of the BACnet Device Object’s Object_Identifier property. The full Object_Identifier is a 32-bit value composed of: Object Type (10 bits): For a Device object, this is fixed as 8.</li> Object Instance (22 bits): The Device ID itself.</li> </ul> BACnet IDs are not assigned during manufacturing, but rather during setup/installation. Matter</h2> Matter devices include a Device Attestation Certificate (DACs) generated through a hierarchical Public Key Infrastructure (PKI) process, ensuring each device is uniquely authenticated. Each device is cryptographically verified, making spoofing difficult. The DAC is part of a chain of trust: Product Attestation Authority (PAA): Root certificate issued by the Connectivity Standards Alliance (CSA) or an approved third party.</li> Stored in the Distributed Compliance Ledger (DCL), a global registry of trusted certificates.</li> </ul> </li> Product Attestation Intermediate (PAI): Intermediate certificate signed by the PAA.</li> Typically includes the manufacturer’s Vendor ID (VID).</li> </ul> </li> Device Attestation Certificate (DAC): Unique per-device certificate signed by the PAI.</li> Contains the device’s VID, Product ID (PID), public key, and serial number.</li> </ul> </li> </ol> Additionally, devices use rotating keys, which makes tracking difficult and preserves privacy. End users scan a QR code to onboard devices. Matter attempts to be: Authentic (not counterfeit)</li> Secure (resistant to hacking)</li> Private (resistant to tracking)</li> Interoperable (works with any ecosystem)</li> </ul> Serial Numbers</h2> Other custom serial numbers take many different forms. Some examples are shown below: Format Type</th> Example</th> Description</th> Typical Use Cases</th></tr></thead> Sequential</td> D000001</td> Incremented by one, often padded with zeros</td> Manufacturing, inventory</td></tr> Alphanumeric</td> TBL-101A234</td> Letters and numbers, often with category or model encoded</td> Electronics, equipment rental</td></tr> Random</td> X9F4-7T32-ZY6</td> Unpredictable, generated by algorithm</td> Security-sensitive items</td></tr> Date/Batch Embedded</td> 20230715-001-123</td> Encodes date and batch with sequence number</td> Quality control, traceability</td></tr></tbody></table></figure> Systemd network interfaces</h2> Systemd’s predictable network interface name encodes the physical path of the device. Example of how enp0s13f0u2c2 is constructed: en: Ethernet device (prefix for Ethernet interfaces).</li> p0: PCI bus 0</li> s13: PCI slot 13 (device 13 on the bus)</li> f0: PCI function 0</li> u2: USB port 2 (the device is connected via USB, possibly a USB Ethernet adapter)</li> c2: USB configuration or child port 2 (sometimes used to indicate a specific port or configuration on a USB hub)</li> </ul> Summary</h2> I’m sure there are many other examples – please leave a comment if you’d like to share other interesting formats. In a future post, we’ll explore a new standard for device IDs that is optimized for human interaction in the field. Subscribe below to stay informed of the latest developments. How to implement Zephyr web applications 2025-04-08T00:00:00+00:00 Are you considering implementing a web UI in your Zephyr RTOS</a> application and perplexed by the endless number of web technologies available today? This article traces my journey through trying several options and presents some of the tradeoffs. </figure> Modern microcontrollers (MCUs) have many connectivity options (Ethernet, WiFi, Bluetooth, Cellular, etc.), and when coupled with the Zephyr OS, implementing a web application on these devices often makes sense. Once you start doing this, you soon realize that this is not Ruby-on-Rails. Rather, Zephyr is still a very constrained environment (see my article on the differences between MPUs and MCUs</a>). While we can still leverage modern web technologies, we need to be selective because of the constraints of MCU systems. An example of a web UI in the Zephyr SimpleIoT project</a> is shown below: </figure> Primary concerns in an MCU system</h2> Size</h2> The memory resources on MCU and MPU systems are vastly different</a> – a difference of over 1,000x. </td> MCU (Internal)</td> MCU (External)</td> MPU (Server)</td></tr> Flash</td> 0.5-2MB</td> 16MB</td> 500GB</td></tr> RAM</td> 0.25 - 1MB</td> 16MB</td> 16GB</td></tr></tbody></table></figure> While it is possible to use larger memory devices with MCUs, like an SD card or SDRAM, the above table is for the more common scenarios. If we want to serve the web application entirely from a Zephyr device, then the web assets must fit in flash. The RAM size also limits the number of connections we can handle and the size of the data payloads we can transmit and receive. Stability (includes maintainability and security)</h2> The constraints of an MCU drive us toward doing more in the browser (frontend) because resources there are free and plentiful. This typically ends up being a Single-Page Application (SPA) architecture, where the web application is a free-running application that fetches and sends data to the MCU (backend). All rendering happens in the browser instead of the MCU. Because of this, the web application needs to be stable. Extra care must be taken so that the application does not crash and freeze or pop up annoying messages telling the user to reload the page, which can easily happen with JavaScript. We also need to consider how the application is hosted. A typical web application is hosted in the cloud in one place. If there is a problem, we have complete control over updating and fixing it as needed. Embedded applications are a completely different matter – there may be thousands of devices. If there are problems, it is not as simple as updating a single server. We may have a firmware update mechanism in the device, but it is often up to the user to decide when to update. Users expect embedded devices to be stable. Many of them are used in critical applications. An instability in the web UI of the device does not give a positive impression, even though it may only be a web frontend crash, and the device is still functioning fine. </figure> It seems simple to write an initial prototype, but maintaining a web application over time is a different matter. Security can also be a concern as many web frameworks pull in mountains of JavaScript dependencies via NPM, which are difficult to audit for security problems. State Management</h2> One of the hardest parts of a web application is maintaining state. The reason for this is that user interfaces can be deeply nested and complex. The naive approach is to distribute the state into each UI component. The difficulty with this is keeping everything in sync. There has been a continual stream of new frontend state management libraries (Redux, MobX, VueX, React Context, XState, react-easy-state, etc.). This is not an easy problem. Options</h2> Serve site/assets from external server</h2> One way to work around the asset size constraint in MCUs is to serve large assets (like a Bootstrap CSS file) from an external server. </figure> There are several potential issues with this: ❌ The external asset must always be available, and if it goes away or changes, then the web application on the device must be updated to point to the new location. ❌ If the entire site is served from an external server, then you must host all versions of the web application or be very careful to maintain backward compatibility to all previous versions of firmware running on the Zephyr device. ❌ You may need the Zephyr UI to function if the Internet is down or unavailable (example: air-gapped secure setups or during device setup). In many systems, device state is mirrored to an upstream server, so the only time you would use the local UI is during setup or when the Internet is down. ✅ The most reliable way to host a Zephyr web application is on the device itself. You certainly have many more options if you host it externally, but there are tradeoffs that must be considered. No Build, just HTML, JS, CSS</h2> DHH and others are advocating No-Build</a> web solutions. However, they have Rails for a backend, and we have Zephyr. I investigated this approach, but manually mutating HTML elements in JavaScript in a complex app is messy and scattered – you’ll eventually create a framework anyway, so you may as well start with something close to meeting your needs. </figure> HTML/JS/CSS development is powerful and flexible, but is tailored to design, not programming. In its raw state, it is like developing a program where everything is a global variable. If you have a dynamic web application responding to real-world events, this quickly becomes a programming problem that is difficult to manage with 200 global variables. You need an abstraction on top of this. Again, a powerful templating engine on the backend helps a lot, which we do not have in Zephyr applications. React, Vue, Angular (non-compiled frameworks)</h2> Solutions like React are too large for many MCU applications as they typically require a megabyte or more of web assets to be transferred from the host. Additionally, you have the issue of JavaScript crashes and the continual bloat and churn of the NPM package ecosystem. Updating to new versions of packages can be a lot of work as APIs sometimes change. While this solution can work for large-scale cloud-hosted apps, it is not ideal for embedded web UIs where you often don’t have control over the device and can’t easily patch the system if there is a problem. </figure> I’ve been writing/maintaining a 50K SLOC Meteor/React app for the past 10 years – it works and has been very successful. However, we’ve had a handful of frontend crashes. Being a cloud app, we were able to deploy fixes for this quickly, but it is still an inconvenience for users. In another project developed by a very experienced JavaScript developer, we also had a number of JavaScript crashes in the frontend. There are likely ways to reduce or avoid these stability problems, but it is not inherent in the system/language. It requires a high degree of carefulness and discipline. It is not easy to refactor JavaScript and know that you have fixed all the issues caused by your code change. </figure> There are slimmer solutions like Preact which may solve the size issue, but you likely lose some of the integration with all the libraries written for React. htmx</h2> htmx is the cool kid on the block right now – supposedly simple and easy to use. I tried this for the first iteration of a Zephyr web application and got a prototype running, but concluded it works best if you have a powerful backend that can do most of your rendering, and the frontend just handles a little bit of reactivity. Additionally, with htmx, you typically open up a network connection for every widget to get updates, and this requires a lot of connection resources on the backend, which are expensive in a Zephyr system. </figure> Svelte</h2> Svelte is a compiled language that generates small assets, so it works well from a size perspective. It also appears to be a well-maintained project with good tooling. However, there are two downsides: 1) it is still JavaScript, so you have potential stability issues when maintaining applications long-term. 2) It puts the “JS” in the “HTML” instead of the “HTML” in the “JS”</a> (like React or Elm). This is largely a matter of preference, but as a programmer, I prefer the latter. The success of React indicates that many others do, too. Functional Languages</h2> Functional languages are becoming more common in frontend development because they solve many reliability and refactoring challenges. Some options include ClojureScript, Elm (discussed below), PureScript, ReasonML, F#, Scala.js. Some of these options likely have smaller asset sizes than non-compiled options. Elm</h2> Elm</a> is a compiled language that produces small assets. It is like React JSX in that it puts the “HTML” in the “Elm</a>“. Elm is a functional language that solves many of the challenges with web programming, including stability and managing state. Runtime errors in an Elm application are very rare. I wrote the Simple IoT UI in Elm, which is now 18,610 lines of code, and have maintained it since 2018. It has never crashed, and it is easy to add new features. Code density/reuse in Elm is very high. If it compiles, it generally works, so very little time is spent in the browser console debugging stuff. The elm-ui</a> package offers a powerful style and layout system that eliminates most of the pain with web programming and allows someone (like me) who is not a dedicated web developer to maintain a pretty nice web application without learning the intricacies of CSS. </figure> As you maintain an Elm project, you’ll find that the language pushes you to a cleaner design. It nudges you in the right direction. The opposite generally happens in JavaScript projects, where a lot more discipline is required to keep things from becoming a mess. Elm allows for creating powerful functions like the following for a text input form: textInput Point.typeBitRate "Bit rate" "250000" </code></pre> This line of code renders a form element and automatically posts a new point to the backend of type BitRate</code> when the content is changed, nothing else in the system needs to change for this to happen. There are no “handlers” that need to be attached to this element. Elm also has some of the best tooling in this industry – elm-review, elm-land, etc. This tooling is largely made possible by the language. Elm is a small project and community, but it is a stable/friendly community that has existed for many years, and many companies are successfully using Elm</a>. The Elm package system is fairly comprehensive and has most of what you might need to implement web apps. Since Elm packages are pure Elm, the entire ecosystem is stable and relatively secure</a>. There is some concern about the future of Elm as the compiler does not get frequent updates. Some have produced new versions of the compiler to meet special needs. The packages and tooling around Elm are where most innovation is happening right now, and it is phenomenal. Elm got it mostly right, so there is not much that needs immediate improvement in the core. Summary</h2> There is no silver bullet in technology. With any choice, there are tradeoffs. Most people stick with mainstream JavaScript for frontend programming as the “safe” approach. However, in Zephyr/MCU-based systems, there are constraints that we don’t face in server environments. There are significant challenges in the HTML/JS/CSS world. While it is nice to have options, the continual churn of new frameworks (jQuery -> Backbone -> Ember -> Angular -> Meteor -> React -> Angular2 -> Vue -> Svelte -> Next.js) (and many others) indicates there is some fundamental problem with the Javacript approach that has not been solved in a general way. The number of NPM package dependencies pulled into a typical project is staggering and nearly impossible for small teams to audit for security and longevity. NPM installs routinely break for various reasons. NPM suffers from some of the same problems as Python, as many packages compile C code in non-standard ways during the package install process, which can be fragile. There is also the issue of what might be viewed as an impedance mismatch – what the technology mega-companies produce and use is not always the best fit for small companies/teams with limited resources. Facebook faces different problems on a different scale than a small embedded team. While React is right for them, we can’t assume it is right for everyone. Most of us are not facing Google-scale problems</a> that require a system as complex as Kubernetes. Ansible and a few servers work just fine. Facebook has near-infinite resources. They have ways to deal with the inherent stability problems of JavaScript. A small company/team does not have these resources, so different technologies like Elm may be a better fit. When faced with the need for a Zephyr web application, I tried htmx first, then looked at “no build”. Neither of these worked very well for me. Having worked on decent-sized React and Elm projects, Elm looked like a better fit for the next try due to asset size, reliability, and maintainability. Additionally, I had already solved the problem of how to effectively deal with data in a distributed IoT system</a>, and had frontend code already written</a> to interface with this dataflow in Simple IoT. It was easy to port the Simple IoT frontend to a smaller version tailored for MCUs, and thus far it seems to be working well. Check out the Simple IoT Zephyr Networking Example</a> to learn more. Struggling to scale your IoT system? Simplify your data. 2025-03-29T00:00:00+00:00 Have you struggled with adapting your IoT system to new applications and requirements? Is handling configuration updates at both the cloud and edge a challenge? Is synchronizing required data between cloud and edge instances a challenge? This article describes a simple way to model data that eliminates many of these difficulties. We can scale an IoT system both horizontally (deploy more units) and vertically (add features and address new applications). While the ideas presented in this essay are focused on vertical scale, any simplification will likely help with horizontal scale as well. Previously, we explored data-centric architectures in IoT systems</a>. This post expands on this by describing how to represent data using Nodes and Points in a way that drastically simplifies IoT data storage, exchange, and synchronization. This is in contrast to encodings and mechanisms used in traditional web and cloud systems. How are IoT systems different?</h2> IoT systems 🌐 have the following properties, which differentiate them from traditional computer/web/cloud applications: 🌐 IoT systems are inherently distributed (equipment is located in many physically different locations). 🌐 Data must exist concurrently in multiple places (cloud and many edge instances). 🌐 Data is not fully replicated on every instance (a cloud server instance might contain the entire dataset, whereas an edge instance only has a tiny fraction of the dataset that is relevant at that site). 🌐 Instances are not symmetrical (edge instances are very different than cloud instances. It is not a cluster). 🌐 There can be many instances (thousands of edge devices). 🌐 The network is even less reliable (edge systems are often in uncontrolled environments). 🌐 The system must continue to operate during network interruptions (it is not a cluster). We can attempt to address these challenges by adding constraints such as: 1) data only flows in one direction, 2) device configuration is only changed in the cloud, and 3) the data payload format will not change over time. However, if we think realistically about these constraints, they are impractical for many systems. In distributed computer systems, change is the constant. Because a system can be networked, its capabilities are endless. The ability to connect to new and different systems means new systems and new value can always be created, and things will change. If you don’t, somebody else will. The “struct”</h2> Cloud/Web APIs have traditionally used various encoding schemes and patterns such as Protobuf, JSON, REST, gRPC, etc. Tools exist to manage schemas, such as GraphQL and JSON Schema. Synchronization mechanisms can be used to manage conflicts (Raft, CRDT, etc.). Most of these schemes assume that the complex struct with multiple and sometimes nested fields is the fundamental element of data representation, storage, and transmission. An example is shown below: type EmployeeData struct { ID int Name string Age int Email string Phone string Street string City string Country string PostalCode string EmploymentHistory []struct { Company string Position string Years int } Skills map[string]struct { Proficiency string YearsExperience int } LastUpdated time.Time } </code></pre> Structs are great – they are a powerful tool for organizing data in a computer program. They are so useful that we have also assumed structs are the best way to store and transfer data outside a program. Thus, we have tables in SQL databases, REST APIs, Protobuf/JSON encoding, etc. This works fairly well and is likely the right solution in many domains. However, we have the following problems with using the struct</code> outside a computer program: ❌ Updating the struct</code> tends to be an all-or-nothing proposition. Fetching the data from a REST API gives you the entire struct</code>. Likewise, when updating, you send the entire struct</code>. There are tools, such as GraphQL, that attempt to solve this problem. ❌ Schema changes are painful. Many very elaborate tools are dedicated to solving the problem of database migrations. ❌ API schema changes are even more painful. ❌ Distributed systems are hard, especially if the struct</code> may be updated by multiple systems concurrently and then requires merging. CRDTs and various other distributed system algorithms attempt to solve this problem, but they are complex. ❌ Nested data is problematic and compounds issues with schema changes and synchronization. Data in IoT Systems</h2> What are desirable features for handling data in a distributed IoT system? ✔️ We can generate/change data anywhere (in the cloud, at the edge, etc.). ✔️ Synchronization between instances is granular, simple, and reliable. ✔️ Data can flow in any direction. ✔️ One mechanism/transport can be used for all types of data (config, sensor samples, etc.). ✔️ All data changes in the system are recorded in history. On the surface, this seems like a hard problem. If you have a struct that exists in multiple places, and one field gets changed in one location, and another field gets changed in a second location, how do you reconcile the differences and record what has changed? As mentioned before, there are solutions such as CRDTs or sending data changes over the wire. Some systems only store changes (mutations) and then compute the current state from all the changes. However, all these schemes are mostly just adding layers of complexity on top of an existing paradigm and are rarely a good fit for IoT systems where you accumulate many thousands of data samples (changes) over time. Maybe it’s time to change the paradigm … Points and Nodes</h2> What if we rethink our concept of data representation? Instead of trying to make a generic struct</code> the fundamental unit of data, we take a more granular approach and focus on each piece of data in a struct. Can we create a simple unit of data that is easy to store and synchronize, and then build on top of this? This same unit of data is used for everything, including configuration, sample data, notifications, etc. The data unit we have arrived at in the Simple IoT project</a> (after many iterations) is the “Point.” </figure> The data type is typically a string</code>, int</code>, or float</code>, but it can be anything you want. Points are organized in a hierarchical tree of nodes, where each node contains an array of points. </figure> Because a point</code> is defined by both a type</code> and a key</code>, it is possible to construct arrays and maps by populating the key</code> field with either an array index or a map key. This allows us to easily represent one level of structure (arrays and maps) in a node as shown in the following example: </figure> If a deeper structure is required, then nest additional nodes. Benefits</h2> There are multiple benefits to this data architecture. ✔️ Data is granular. An edge instance can update a node’s sensor reading, and the user can update a node’s config setting in the cloud simultaneously. The points are easily merged as they are received into the node at both locations, without conflicts. This works very well because different users/instances tend to update different points. The exception to this may occur when a user at the edge and a cloud user update a config item at the same time. In this case, we simply use the point with the latest timestamp. ✔️ The storage and communication infrastructure does not change. One of the biggest problems in traditional storage mechanisms and network APIs is that schema and encoding changes can be very painful. If you add a field to a data structure, every piece of code that touches the data as it travels throughout the system needs to be updated. With nodes and points, only the ends of your system need to be updated. Everything in between (transmission, storage, sync, etc) can stay the same. </figure> ✔️ Introspection and monitoring are easy. Since our data format is simple and fixed, it is easy to watch and decode traffic anywhere to see what is going on. ✔️ Integration is easy. Because we have standardized the data to be very simple, it is easy for one service to provide data that another can use. ✔️ Everything is data that can be acted upon. In the past, we might have had a rules engine that could respond to sensor data changes. Since config changes are now the same kind of data, the rules engine can now respond to and make config changes as well. ✔️ History is easily logged for all changes. Likewise, since config and sensor sample data are now the same type of data, everything that happens in the system can be easily recorded in history. The system is automatically auditable. ✔️ It is efficient. Because the data is granular, only what changes is transmitted and processed. ✔️ It works both internally and externally. While this data scheme was primarily developed to efficiently communicate between systems connected by networks, it also works very well for communicating between threads in an application. When an application is architected this way, it is very easy to extend, it as the communication scheme internally and externally is the same. This architecture scales. Objections</h2> 🚩Objection #1: But isn’t this a lot of overhead to store a type/key/timestamp/etc with every bit of data? We need to focus on where the bottlenecks/hard/expensive parts of the system are. Synchronizing data from many different instances over networks is the hard part of IoT systems. Local storage and CPU processing are relatively cheap in comparison. Networks are also relatively slow compared to the additional processing time required to process this data. CPU and storage are now abundant, even on microcontrollers. Development, operations, and deployment are not cheap. 🚩 Objection #2: Nodes and Points are hard to deal with. True, dealing with raw nodes</code> and points</code> is not as convenient as a struct</code>, but we can create tools</a> that easily convert structs</code> to and from nodes/points</code>. We do it all the time – JSON, Protobuf, etc. Computers are very good at encoding/decoding data – let them do it. 🚩 Objection #3: How do you protect your data if anyone can update anything? AuthZ/AuthN is a valid concern and has not been fully solved yet. One idea is to authorize access to data at the node level – each user or instance would have access to a set of nodes. However, this says that an edge device could change its own config parameters. Perhaps this is OK because, without physical security, you don’t have true security. It may also be useful to differentiate betweenread and write access</a> to node data. Ideas are welcome! 🚩 Objection #4: If points for a single struct are flowing in at different times, how do I know when a change is complete and when to process it? This is also a valid concern. One option is to set a timer that gets reset any time the config changes, and the changes are only processed once the timer expires. With a little creativity, we can probably come up with other schemes to signal the change is complete – perhaps a special point that is sent at the end of a sequence of points. 🚩 Objection #5: What if the data is more complex than what can be represented with a simple data type like a number or string? You can store anything you want in the point data field (JSON, Protobuf, etc.) This should provide enough flexibility for things like messages, where it probably does not make sense to separate the data into separate points. 🚩 Objection #6: How do you deal with schema changes? There may be instances where there are point type name changes, node topography changes, etc. In this case, each client, which provides the backend code for a node type, should be responsible for handling any schema changes. This localizes the migration logic to each client (including external clients that plug into the system at runtime) rather than requiring a central system to understand all migrations. A node may contain a version point if necessary. 🚩 Objection #7: Do I have to represent all my data as points? If data needs to be distributed between instances on a network, then it is very beneficial to use points. However, for local processing of data, there is nothing preventing you from using more conventional techniques, storage, etc. 🚩 Objection #8: How do you store data as nodes and points? SQL databases are not designed for this. That is an excellent question and part of our research in the Simple IoT project. We are currently using SQLite, but we are moving to NATS Jetstream</a>. Stay tuned! 🚩 Objection #9: Will this scale to a bazillion devices? Probably not. If you need to connect a bazillion devices, then perhaps AWS IoT or something else is a better fit. However, what we are presenting here is a flexible architecture that will scale over time to many different applications and configurations as your needs evolve and change. How do you want to scale? Think carefully about this, as scaling in one dimension sometimes limits how far you can scale in another. 🚩 Objection #10: This will not work for data sampled at 100 bazillion samples per second. For moderate data rates, transmitting blocks of data with every point by putting an array of data in a point data field may work. For high-rate data that flows, a custom data encoding and transport will likely be required. 🚩 Objection #11: You can’t depend on time in a computer system, so last-write-wins (LWW) is not reliable. True, time, like networks, is not always reliable. However, in IoT systems, most of the time, you do not have the same data being edited in multiple locations, so LWW is only used in the rare case where two human users make changes at different locations. A little clock drift is not a huge deal. You can also do things like check if two systems have similar times before you synchronize data between them, and verify a system has a valid time before writing any data. You can also require that every instance in the system has its own RTC clock and that some engineering effort is made to ensure it is reliable and reasonably accurate. 🚩 Objection #12: Why not send commands, events, changes, etc.? Sending “commands” is a common paradigm in IoT systems, but it is the siren song into complexity. How do you deal with cases where the command does not make it due to a network problem? There are valid cases for events such as orders or notifications, but in most cases, commands/events are an unnecessary abstraction and only complicate things. A sensor is at value X. The config setting is Y. Keep things declarative. Just send the data itself, keep some amount of history, and if you want to know when something changed, look at history. Utilizing new technologies like NATs Jetstream</a> is a critical piece of this puzzle. 🚩 Objection #13: But this is different !?!? Yup, I’ve never seen anything like it before either. But it works. Summary</h2> When we want to advance something, we often need to simplify it first. Piling on layers of complexity will only get us so far – at times, we need to go back to ground zero and re-think things. There are many ways to specify contracts between systems, such as APIs and schemas. A more fundamental contract focuses on a more granular unit of data. Points in an IoT system are like atoms in matter. A Node tree in an IoT system is like the DNA in a living organism. The composition of points and structure of nodes (like atoms and DNA) is infinitely flexible. It is this simplicity and flexibility that allows atoms and DNA to define so many different types of matter and species of organisms. It is time to throw out the custom struct as the defined unit of data between systems and focus on something more granular, flexible, and scalable. Want to learn more? Check out and follow the Simple IoT project</a>. The Ability to Improve 2024-03-05T00:00:00+00:00 </figure> (while this article contains many general ideas, it is written from the perspective of product development.) If you don’t continually improve, you soon lose the ability to do so … This thought came after an associate described a company that is struggling to manufacture one of their products: They have been making a product since way back in the last century. EOL (end-of-life) is catching up with them. The circuit is really hairy analog stuff with ridiculously high precision. Pots everywhere to get it dialed in. But their voltage regulator is drifting in value significantly, whereas the old parts are rock steady. </blockquote> How do we get into situations like this, and how can they be avoided? A few more examples</h2> Below are a few more examples of situations that are very difficult to deal with in the lifecycle of a product: A part becomes obsolete in a printed circuit design, and no one knows how to use the CAD system it was designed in. Rather than updating to an open/modern CAD system</a> years ago at a relaxed pace, exorbitant efforts are now spent trying to track down a few more parts or find a replacement. When this fails, it is now a frantic crash-and-burn project to revise the design with no time for testing and verification.</li> A serious security problem is found in a product, but the software build system has fallen into disrepair and no one knows how to set it up, and it will not run on current operating systems.</li> A new Linux kernel driver is needed to support a new USB peripheral, but the kernel we are shipping is 7 years old, and the SOC/SOM vendor has not provided any updates. Updating to the latest kernel would take months. Backporting the new driver is nearly impossible due to the many changes in the kernel and its subsystems.</li> Manufacturing was outsourced to another country years ago. Then COVID happened and the manufacturer shut down for an extended time. No one knows where the design files are or how to build the product to move it to another manufacturer.</li> The manufacturing process for a complex product has been the same for years. No one knows why things are done the way they are, but are afraid to change anything for fear that problems may crop up.</li> A code base has not been refactored as it grew and no tests were written. Adding new functionality is very difficult and usually causes bugs in unrelated areas, requiring an exorbitant amount of manual testing with each release.</li> An application does not have a cohesive architecture. Global variables and unrestrained function calls between various subsystems are common. As time goes on, the application becomes less and less stable. Fixing one bug often causes another. All developer resources are consumed in a circle of support, continuous bug fixing, and damage control.</li> A new customer is interested in purchasing a product, but it is missing a critical feature. The original development was contracted to an outside development firm, and there is no knowledge in-house on how to modify the software. The managers at the company are scrambling to find someone who can figure out how to make the changes needed.</li> </ol> Technical debt is another term to describe systems that are not adequately maintained – eventually, it takes so much effort just to keep things going that no time is left for improvements. Maintaining Systems</h2> How do you maintain the ability to react to changes, security problems, part shortages, and feature requests? How can you add value to your products over time? Maintaining systems (both hardware and software) is what keeps them malleable and flexible. If you don’t maintain stuff, then it soon becomes brittle and harder and harder to change. Modern systems are not ship-it-and-you-’re-done, but rather more similar to a living organism. If it is not maintained, it is dying. The only way to make sure you can maintain something is to maintain it. Make it better. Replace a part with a cheaper/better one. Improve something. Add a feature. Make it faster. Streamline and simplify manufacturing. If we don’t continually improve, then it becomes difficult to do anything without starting over. It is much easier to absorb a series of small costs over time than a large expense all at once. Small changes over time allow us to improve our processes. If we are not under pressure and something goes wrong, then we simply revert to the previous version or write tests to keep it from happening again. This does not have to be hard, but it does require an investment in methodology and workflow. It requires a commitment to open tools, automation, and testing. It requires process – not the type that results in stifling bureaucracy but rather improves efficiency through automation and transparency. There are several concerns related to change including return on investment (ROI), quality, and efficiency. The ROI Objection</h2> There are two financial mindsets: View R&D as a one-time capital expense where you design something, lock it down, and then try to extract as much value as possible – the “minimizing costs” mindset.</li> View R&D as an ongoing operating expense, improve things over time, and continually increase value – the “maximizing value” mindset.</li> </ol> The first is easier to measure and forecast. Financial people like and understand this model. The second requires intuition and vision. It involves some risk. As your processes become more efficient, the value you get from an improvement begins to outweigh the costs of making the change. Over time, these improvements have a compounding effect. </figure> It depends if you are playing the short or long-term game … The Quality Objection</h2> Stability and Quality is always a concern with any change in a system. With changes, mistakes will be made – especially early on. When this happens, there is an opportunity to learn and proactively improve the testing and verification process for next time, or reactively place blame, lock things down even harder, and remove responsibility. When someone introduces a bug, the question should be: “Did you create a test so it does not happen again?” If no mistakes are being made, not much is likely being done. The upside of growth and improvement should far exceed the costs associated with the occasional mistakes made in the course of changes. Done right, quality should follow the same value curve as the value shown in the graph above. The key is to not get discouraged by an initial drop in quality and focus on improving the processes, not minimizing or restricting change and responsibility. The Efficiency Objection</h2> Efficiency is always a concern, and again there are two schools of thought: It is more efficient if we just do things the way we’ve always done them, not risk change, and get what we can out of our investment. Change brings risk.</li> Efficiency can always improve, bringing lower costs and more value.</li> </ol> Again, the key is to be patient with the initial declines and persevere until you get to the point where efficiency is increasing and the cost and value lines cross. Case Studies</h2> Tesla is an interesting case study on continuous improvement</a>. From a design and manufacturing perspective, their accomplishments are notable. There is always a better way to design or manufacture something. Google’s blameless postmortem</a> practice enables proactive improvements for the future instead of reactively placing blame for the past. Amazon’s “You build it, you run it”</a> philosophy allows people who know the most about the system to improve how it is run instead of going through endless rounds with gatekeepers who don’t understand the system. Toyota’s implementation of “kaizen</a>” has produced excellent results</a>. Kaizen is a concept referring to business activities that continuously improve all functions and involve all employees from the CEO to the assembly line workers.1</a> These practices are used at large successful companies, but in concept, they are simple and can be applied at any scale. The Irony of Kaizen</h2> We can develop products with the thought – we are going to get this perfect and then go into production. Logically it would seem that if we expect to make improvements after it is in production, then we must be starting with an inferior product with flaws. However, Toyota has proved the opposite. Even when a model is first introduced, their quality is likely better than the average in the automobile industry. Kaizen does not mean we slop something together of inferior quality, but rather we do the best we can but have the humility to realize it can always be better. Kaizen is not limited to the assembly line, but all aspects including design, prototyping, quality control, testing, etc. If all of these processes are continually getting better, then the first item produced will also be better. Requirements for Improvement</h2> Every person and every organization is different, so there is no one-size-fits-all recipe that will always work. But there are several ingredients that are generally beneficial including: Metrics</li> Testing</li> Openness</li> Simplicity</li> Leveraging outside improvements</li> Iterate faster</li> Owning your platform</li> The right people</li> </ol> Metrics</h2> Not all change is an improvement – change for the sake of change is just churn and generally negative. You have to be able to measure something to know if things are improving or not. Improvement is very difficult without some metrics, because we are then just guessing and going off our intuition, and human intuition is often distorted by emotion, social pressures, and a host of other distractions. Testing</h2> Testing is trendy today and many talk about it, but fewer individuals and teams have the discipline to implement testing consistently. If you are not confident enough to deploy new software to production after your tests pass</a> without doing some manual checking, then you possibly need better tests. Automated end-to-end and performance testing is critical. Consider the following statement about SQLite: The reliability and robustness of SQLite is achieved in part by thorough and careful testing. As of version 3.42.0 (2023-05-16), the SQLite library consists of approximately 155.8 KSLOC of C code. (KSLOC means thousands of “Source Lines Of Code” or, in other words, lines of code excluding blank lines and comments.) By comparison, the project has 590 times as much test code and test scripts - 92053.1 KSLOC. https://www.sqlite.org/testing.html</a> </blockquote> Most of SQLite’s tests are not open-source. This is the real IP of SQLite. Openness</h2> Openness is another necessary ingredient for improvement. This takes many forms: Open access to tools, design files, and source code. Open-source software and tools are preferred when available.</li> Good documentation is available to all involved.</li> Work being done is in the open and can be reviewed by anyone at any time. Pull requests are created when work starts to provide visibility into work being done rather than when the work is finished. Transparency is a powerful ingredient for quality.</li> Metrics, including financial information</a> and customer feedback, are accessible to the team.</li> Anyone on the team to allowed and encouraged to fix or improve anything. There are no “turf battles” where only certain people are allowed to do certain things. Within practical limits, anyone can do any work, but the process (transparency and testing) ensures that substandard work will not be automatically used.</li> </ul> The above does not mean everything is open-source to the world, but rather those involved with the product – a team, organization, company, customers, partners, etc. We can learn a lot from the success of open-source projects and their associated culture and methodologies. Simplicity</h2> As the complexity of systems continues to increase, we need to also be aware that humans interacting with these systems are finite – we can only handle so much complexity. Thus, efforts need to be made to simplify where possible. We often hear the phrase: “keep it simple!” But, from whose perspective? Design, Manufacturing, Users, Service, Ops, Support, Sales, Financial, CEO, etc. If we look at how much time these various groups interact with the product, we can arrange them in an inverted pyramid: </figure> The CEO is only one person and hopefully, there are many Users, so if we can simplify things for the Users, that will probably provide more benefits to the product than simplifying things for the CEO. Likewise, if Design can spend a little more time making Manufacturing and Support simpler, that is probably a good investment. Simplifying a system’s architecture often provides the biggest wins. This can take the form of common data structures, how data is structured, how data is communicated and synchronized, lifecycle management, etc. A simplification at this level has the potential to improve the entire system. Faster Iterations</h2> The most efficient way for humans to design complex systems is through iteration</a>. Some call this failing fast</a>. The faster you can design/develop, test, get feedback, and repeat, the more likely you will be successful. </figure> There is certainly a place for some up-front planning, but you don’t want to get stuck there</a>. Build a minimal slice of your product</a> as soon as possible, even if it only has a few features. This can even be done with hardware – put down only the bare minimum parts on the PCB and build something. The rest can be added later. If it takes too long to build a PCB prototype, find ways to speed it up, or switch to another vendor. Automation and process is very important for fast iterations. CI (continuous integration) tells you quickly if your code fails tests. CD (continuous delivery) continually builds and deploys your work so that it can be tested in the complete system and evaluated by others. CAD and PLM automation</a> allows you to quickly package, track, and build new versions of PCBs. A focus on iteration compensates for our many human weaknesses. It reveals our blind spots. It improves teamwork and project transparency. It helps avoid the scenario where the lone genius works for three months behind closed doors while the rest of the team is largely stalled. It ensures we are building what the customer actually needs. It encourages automation to avoid the tedium of manual testing and releases. It accelerates learning. Above all, it improves the efficiency of the entire process. It ensures we can quickly make changes and release them when needed, with confidence. Owning Your Platform</h2> Owning your platform means you “own” and take responsibility for key parts of whatever it takes to build and maintain your products. It might be a printed circuit design, a build system, an application architecture, automation, infrastructure, process, etc. Owning your platform likewise is an investment for the future. It gives you more control and the ability to improve things. You are not locked into what another company provides at one snapshot in time. Certainly, there are limits to this and we cannot create an entire system from scratch so we all must use software and components that other companies create. However, we can build a system that has flexibility so that we can easily swap out one component for another. We can understand it well enough to test it. We can manage the build systems so that if something needs to be fixed or updated, we can do that. We build our device software when possible instead of relying on binary images that we can’t control. We are not afraid of developing our own automation and process software instead of relying entirely on outside vendors. Perhaps the most important thing is to own the integration points. This might include: Top-level PCB (printed circuit board) designs (we can change electronic components and submodules as needed)</li> Software build systems (we can add, remove, and update software components)</li> PLM (project lifecycle management) (we write our own or use a system that can be easily extended through programming to meet our needs)</li> Core application architecture (can be adapted to specifically meet our needs)</li> Hosting (we can run our own servers or move applications between hosting providers as needs change)</li> CI/CD (can be adapted to any task/technology)</li> </ul> Integration is the most difficult (but also the most valuable) part of product development, so it is tempting to outsource integration. But this often locks you into one vendor and does not give you the flexibility to scale to higher volumes or additional products, as this running commentary</a> illustrates. Ideally, you can reuse parts of your platform across many products, thus compounding your efforts over time. Even if your design files and source code are open-source, owning your platform (which includes so much more than design files) is what gives you the ability to efficiently design, test, maintain, and produce something. This is the hard part. If you are moving fast, a static snapshot of your product design at any point in time is of marginal value to competitors. Leveraging Outside Improvements</h2> Leveraging outside improvements is another important activity. This can include upgrading to a new open-source software component, designing in a new integrated circuit, etc. It is easy to get stuck in the mindset that the “old” was “better.” In some aspects of life, this may be true, but with technology, things are generally getting better. Open-source software is generally more stable as time goes on due to better testing. Newer integrated circuits are more capable and lower cost. Many products today are largely an integration of many outside technologies, many of which are now open-source</a>. To improve, we must continually track and integrate these improvements. Developing using the mainline</a> (upstream) branch of open-source projects is a good strategy that companies are increasingly adopting</a>. Knowing when to adopt</a> new technology is a key skill. If you use an outside development firm to develop parts of your product, it is critical that you receive a deliverable that is maintainable. Several suggestions for ensuring this include: run the project like an open-source project. All activity is transparent and is committed to Git often.</li> own the integration points and testing.</li> use small iterations when possible. This ensures you are engaged and what is being built is what you need.</li> use open tools</a>/technology that everyone has easy access to where possible. Ensure that you can build or generate usable outputs from the design/source files. Even if you don’t have the expertise to use the tools, there is a better chance you can find someone else who can.</li> </ol> The Right People</h2> There is a saying often used by hikers: “Leave it better than you found it.” If we are working on a section of code that does not have tests, do we write them? If a manufacturing problem is found, do we improve the process to try to prevent the problem in the future, or just write it off? Do we give more responsibility and opportunity to everyone involved to improve things, or do we let a few mistakes restrict responsibility to only a few people? For this to all work, you have to have people with integrity – who are genuinely interested in improvement and not just gaming the metrics. You have to have people with the discipline to write tests, which goes above and beyond just getting the job done and the ticket closed. This all applies personally too … do you want to improve how you use your time? Then measure how you spend your time and share these measurements with someone. Use one of the many tools available to help track your time. Create your own tooling and process, and then improve it. Look for better ways to do all this. Fundamentally, people don’t usually change much, so if someone is not interested in improving themselves, they likely will not be interested in improving the organizations they are a part of and be a drag on the process. So perhaps the most fundamental thing to learn in an interview is not only past accomplishments, but also if the candidate still has that burning desire to learn and improve themselves. It is a frustrating situation when a person who is wired for improvement is part of an organization that is not, or vice-versa. Both types of people and organizations exist. Small Changes vs Large Changes</h2> We can make small or large changes. This might be the difference between an improvement vs. a pivot, slow and steady vs drama. The drama captures attention in the news – the new charismatic CEO brought in from the outside to shake things up. You typically don’t see great companies in the news, but you do see them in investment portfolios. In his book “Good to Great,”</a> Jim Collins studied several companies that produced exceptional results and concluded the following: We keep looking for change in the wrong places, asking the wrong questions, and making the wrong assumptions. There’s even a tendency to blame Wall Street for the “instant results” approach to change. But the companies that made the jump from good to great did so using Wall Street’s own tough metric of success: a sustained leap in their stock-market performance. Wall Street turns out to be just another myth—an excuse for not doing what really works. The data doesn’t lie. Now picture a huge, heavy flywheel. It’s a massive, metal disk mounted horizontally on an axle. It’s about 100 feet in diameter, 10 feet thick, and it weighs about 25 tons. That flywheel is your company. Your job is to get that flywheel to move as fast as possible, because momentum—mass times velocity—is what will generate superior economic results over time. Right now, the flywheel is at a standstill. To get it moving, you make a tremendous effort. You push with all your might, and finally you get the flywheel to inch forward. After two or three days of sustained effort, you get the flywheel to complete one entire turn. You keep pushing, and the flywheel begins to move a bit faster. It takes a lot of work, but at last the flywheel makes a second rotation. You keep pushing steadily. It makes three turns, four turns, five, six. With each turn, it moves faster, and then—at some point, you can’’t say exactly when—you break through. The momentum of the heavy wheel kicks in your favor. It spins faster and faster, with its own weight propelling it. You aren’t pushing any harder, but the flywheel is accelerating, its momentum building, its speed increasing. This is the Flywheel Effect</a>. It’s what it feels like when you’re inside a company that makes the transition from good to great. Take Kroger, for example. How do you get a company with more than 50,000 people to embrace a new strategy that will eventually change every aspect of every grocery store? You don’t. At least not with one big change program. Instead, you put your shoulder to the flywheel. That’s what Jim Herring, the leader who initiated the transformation of Kroger, told us. He stayed away from change programs and motivational stunts. He and his team began turning the flywheel gradually, consistently—building tangible evidence that their plans made sense and would deliver results. “We presented what we were doing in such a way that people saw our accomplishments,”Herring says. “We tried to bring our plans to successful conclusions step by step, so that the mass of people would gain confidence from the successes, not just the words.” Think about it for one minute. Why do most overhyped change programs ultimately fail? Because they lack accountability, they fail to achieve credibility, and they have no authenticity. It’s the opposite of the Flywheel Effect</a>; it’s the Doom Loop. Companies that fall into the Doom Loop genuinely want to effect change—but they lack the quiet discipline that produces the Flywheel Effect</a>. Instead, they launch change programs with huge fanfare, hoping to “enlist the troops.” They start down one path, only to change direction. After years of lurching back and forth, these companies discover that they’ve failed to build any sustained momentum. Instead of turning the flywheel, they’ve fallen into a Doom Loop: Disappointing results lead to reaction without understanding, which leads to a new direction—a new leader, a new program—which leads to no momentum, which leads to disappointing results. It’s a steady, downward spiral. Those who have experienced a Doom Loop know how it drains the spirit right out of a company. https://www.jimcollins.com/article_topics/articles/good-to-great.html</a> </blockquote> Change is easiest to implement and absorb if it is incremental and not a drastic overhaul. Incremental change allows you to measure, adjust, and backtrack if needed. Incremental improvements can be implemented by anyone without permission and can then be evaluated and improved by others. It is also important to understand the difference between foundational truths/values (things you should not quickly change) and inefficiencies that and ripe for improvement. The foundational principles of your business provide stability and need to be understood and valued. Summary</h2> Do you want to be able to respond quickly to unexpected opportunities or issues that come up? The solution is simple – practice doing so now. If you regularly practice on the little things that by themselves may not really be worth the time to do, several things will happen: you will be ready and practiced to make critical changes that will make a big difference such as fixing a critical security problem, responding to supply chain shortages, or making a change necessary to land a big sale.</li> the compounding effect of small improvements will eventually provide a sizable return in the future.</li> each small improvement is a small push on the flywheel of momentum. This adds positive energy to your organization and this momentum helps carry you through difficult times.</li> </ol> A culture of improvement is a long-term game. There will be some up-front pain as you develop your processes, but this is much easier than the long-term pain of accumulated technical debt which cripples your ability to do anything. You can compound the value from improvements or the cost of technical debt – take your choice. ^1https://en.wikipedia.org/wiki/Kaizen</a> </div> Think Horses, not Zebras (Part 2) 2024-03-01T00:00:00+00:00 There is a popular quote</a> in medical circles: When you hear hoofbeats, think of horses, not zebras. — Dr. Theodore Woodward </blockquote> I recently posted several debugging experiences</a> where it was beneficial to examine simple scenarios before complex ones. Below are a few more … NVMe slot not working</h2> An NVMe slot is fairly simple – it has a clock and one to four PCIe lanes. However, in a recent new hardware design, the slot was not working. We reviewed the kernel boot logs, scoped the clock signals, etc. We did notice the clock amplitude was lower than other designs (see below). The initial suspect was signal integrity as the PCIe data lines run at very high speeds, so we reviewed the routing, board stackup, impedance calculations, etc. Eventually, I compared the PCIe connector pinout to another design and it appeared that the RX and TX data signals were swapped. Sometimes RX vs TX (receive vs transmit) can be a little confusing based on which perspective the signals are referenced to (the host system or the target device). In the case of PCIe, it appears the signals are all named from the perspective of the host. After modifying to swap the RX and TX signals, the NVMe slot worked fine, even though the modification wires drastically violated signal integrity and impedance requirements. PCIe must be a fairly robust transport! </figure> </figure> PCIe clock amplitude</h2> While verifying the hardware of a design, we noticed that the PCIe clock was at a lower amplitude than the specification required: </figure> What we were measuring: </figure> The Vcross was about 170mV, and the spec required it to be between 250 and 500mV. To verify our measurement technique, we measured the PCIe clock on another system and it it looked correct. The default output of the clock chip should have given us the correct amplitude. The clock chip was connected via an I2C bus, so we read back the registers to make sure they were set to the defaults. We then set the amplitude to the maximum the chip would output, and it still was not within range. Finally, we noticed the following in the datasheet: </figure> Sure enough, we had terminations resistors for these signals on the PCB. After the resistors were removed, the clock amplitude was correct. Cellular connection problems</h2> Cellular IoT systems can be tricky to troubleshoot. They are often installed in harsh environments (rain, cold, hot, damp, caustic sewer gasses, etc.). Occasionally a modem goes bad and connections fail. In one recent case, we replaced a modem and it still would not connect. The antenna, cable, and connections were the next suspect. After spending considerable time swapping components and not making any progress, I reviewed the configuration and noticed the APN was not set. After setting the cellular APN, it worked. Zephyr Nanopb build error</h2> We recently updated the version of Zephyr we were using in a project. In the process, a protobuf generation process broke and another developer found a solution – nanopb was now an optional dependency that needed to be activated. However, the build still failed in my workspace. We discussed diffing our build workspace, but in the back of my mind, I was thinking – there has to be a simple reason. While reading the documentation, it occurred to me to check the west</code> version – sure enough, I was running 1.0 and the latest was 1.2. After updating, it worked. What can we learn?</h2> Again, in the above cases, the initial assumptions were more complex than the actual problem. Suggestions from the previous article</a> are relevant here: Double-check connector and component pinouts.</li> Compare to working designs.</li> Re-read the entire datasheet carefully.</li> </ul> Additional suggestions learned from these examples include: Verify the configuration of the system is correct, never assume it is.</li> Make sure you have the correct version of the tools installed (ideally your build system would check this for you).</li> Re-read documentation for tools and software components. This will often spark ideas on where to look for problems.</li> </ul> Why Open Source for Product Development 2023-12-01T00:00:00+00:00 (this topic is also discussed in a podcast episode</a>) One of the paradoxes in product development is Open Source. How can you personally or as a company benefit from participating in Open Source projects? Why should you share your great ideas and code? How can you build a business or a career by giving things away? How can an open-source project be a reliable supplier without contractual guarantees? Most successful technology companies today participate in open-source projects. Why do they do this? Open source is difficult to fully understand and much has been said about it. There are many approaches to licensing and funding. Many question the equity and sustainability. Many get bogged down in ethical, moral, and political arguments. There are messy problems that don’t seem to have good answers. But we need to go deeper and examine the core issues of why companies are turning to open-source. We will consider several product development perspectives: Our motivation is to produce something of value that society can use and benefit from.</li> Software technology is advancing at a rapid rate of change.</li> Technology in modern systems is increasingly complex.</li> How can we maintain some level of control over the resources we use to ensure they will meet our needs?</li> And how does organizational culture impact our ability to deal with these issues and build modern systems?</li> </ul> </figure> Value</h2> The value of software is hard to quantify, and breakthroughs are difficult to predict. New software algorithms, methods, languages, and structures are sometimes much more effective than previous ones. It is these non-linear breakthroughs that we must leverage. Software is an unbounded sphere of opportunity, free from the constraints of physical locality, physical devices, duplication costs, and, in many cases, computing resources. Modern computers, even those we hold in our hands, are capable of running more software than we can write in multiple lifetimes and storing a lot of information. Systems that improve human efficiency and capability potentially add value and wealth to society. The best software can do more with simpler expressions and less effort. The best software can scale to larger problems. The best software is easier to maintain and thus the rate at which value is added over the long term increases. The best software is more reliable and can be easily deployed without worrying about it crashing or malfunctioning. Complexity</h2> We should clarify our use of the term complexity – we are talking about inherent and necessary complexity, not artificial, human-created complexity (some call this accidental complexity</a>). The human body is incredibly complex, but this is necessary for it to do all the things it does. Modern microprocessors are complex, but they need to be in order to do the amount of processing they do. The Linux kernel is complex, but only because this is required. In modern systems, software is potentially orders of magnitude more complex than hardware. We might spend X hours designing the hardware and XY hours writing and maintaining the software over the lifecycle of the product. This is hard to comprehend because the hardware can be seen, but the software is largely invisible. Software is increasingly the driver of value in modern systems, whether the systems are products or processes. Change</h2> The development of hardware versus software requires a different mindset and process. With hardware, the effort is mostly in the initial development as we design, specify, and integrate the components needed. Once designed or qualified, a hardware component can be used for 5 or 10 years with very few changes. Its value is driven by minimizing variance (change), which is how we define hardware quality. Software systems are completely different – the inherent value in software is that it can be changed over a product lifecycle and its value subsequently increases, so the value in software comes from maximizing (instead of minimizing) changes. Software in a product may be updated hundreds or even thousands of times over its life. This is why it is so difficult for a company that is really good at hardware to also be good at software, or vice versa – a different culture and mindset are required. Control</h2> How can we secure and have some level of control over the critical software components we need to build products? First, we should clarify the scope of the problem. Modern systems are increasingly assembled from hardware and software components we did not create ourselves. In a Linux-based system, there are many millions of lines of source code. The few thousands of lines of custom application code we might write for a product, which may represent many man-years of effort, are only a drop in the bucket compared to the rest of the software running in the system. However, it is easy to ignore the “rest” of the software, as it is largely invisible. How you manage the “rest” is critically important and perhaps the hardest part. And this “rest” is increasingly being supplied by open-source projects. Many engineers are good at creating things from scratch, but fewer are good at understanding, modifying, debugging, and managing software technology that others create. This skill is increasingly valuable. Ideally, we would create everything we need and not be dependent on anyone else. However, we won’t get very far on our own if we need to create our own microprocessors, operating systems, programming languages, databases, and the myriad of other components needed to build systems. Likewise, these components by themselves are of little value unless they are combined into a larger system to do something useful. No one person, company, or even nation can build and maintain all the technology building blocks required to build modern systems. Human success is largely dependent on the degree to which we are able to work with others. The scope of the Linux kernel is beyond even the largest companies or governments. Even if an organization had the manpower, it would likely not have the culture required to pull off a project like the Linux kernel, the PostgreSQL database, or many other useful OSS projects. For smaller organizations, the capability to build even the most basic building blocks is often beyond our reach. Culture</h2> Thus far we have established that to create value with modern systems, we must be able to build complex systems using software technology, much of which we did not create, and be able to change this software over time to deliver more value. In all this, we must have some level of control over this software. If there are problems, can we fix them? Can improvements be implemented and new features added? Secondly, can we influence development over time if we are not doing it ourselves? When we source software technology, there are many options, but let’s reduce it to two for the purpose of discussion: Find a vendor with a proprietary software product. Initially, it fits our requirements – at least during the sales cycle. As changes/fixes are needed in the future, this vendor is hired to provide these updates or we work around them.</li> Find an open-source software project. We may purchase support from a company supporting this project or we may make needed changes ourselves.</li> </ol> There are no doubt times when #1 is the best solution, but increasingly #2 is being chosen. Why is this? Let’s again review the challenges with software development: Obtaining the software we need (value), as we only have the resources to write a tiny fraction of it.</li> Making changes over the product lifecycle, as this is where value is added.</li> Having some level of control and influence over the software we use.</li> Managing the extreme complexity of modern systems.</li> Providing an environment and finding the people where software innovation can happen (culture).</li> </ol> As we review these challenges, we start to realize that the primary benefits of open-source software are cultural. Software innovation is not created with a manufacturing-type mentality but is more art than science – a process of discovery, experimentation, collaboration, inspiration, and simplification. Developing great algorithms and code is a creative process that cannot be easily planned or scheduled, other than allocating the resources and time to work on it. The development of software innovations is not always predictable and cannot be forced. It requires the rare combination of the right people, the right environment, the right level of collaboration, and the freedom to experiment and innovate. All these conditions rarely exist inside corporations. Companies are realizing they cannot always hire the people needed or provide the environment for the level of innovation required. So the next best thing is to support open source projects where this innovation is happening. There are two ways to do this: time and funding. Many authors of innovative software projects are driven by the craft itself, not wealth. Funding is necessary, but they are not working on a project to become rich – there are much easier ways to make money. They are driven by the intrinsic challenge, the joy of discovery, and the reward of building something useful. The first step in making sure an open-source project will be maintained and available in the future is to ensure that the authors have the funding they need. There will never be perfect equity in that authors will receive exact compensation proportional to the value they create, but this matters little to most OSS authors</a>, and in reality, this is also the case with most developers employed by companies. Many developers are more than satisfied with earning a good living and having the freedom to work on whatever they choose. Most are generous and glad to see their work being used. However, there are cases where OSS authors do not receive funding comparable to what they could earn working elsewhere, so it would bewise for companies to audit all the OSS components they are dependent on and ensure that these projects are adequately funded. Contributing financially also makes it more likely the authors of the project will respond to the company’s needs, but we need to be careful that this process does not become restrictive to the point where the environment for innovation is destroyed. Contributing time (code, documentation, support, feedback, etc.) to open-source projects is another way to help ensure a project will be useful and available to companies needing it. Contributing to the community around an open-source project is critical for the project’s health and is part of the flywheel of innovation. If someone is a valuable contributor, then they will also have some influence on the direction and priorities of the project. They will become skilled at understanding and using the project. They will also become better programmers, as they work with some of the best in the world, and are exposed to well-written software. Open source projects provide an environment for innovation that is often not possible inside a company, so having your engineers do some work in OSS projects provides them with an alternative environment to do software development. Why Open-Source Works</h2> It is somewhat of a puzzle why the open-source environment works so well for software innovation. Transparency is a powerful tool for great work. When you do things in public, you do them better. In OSS projects, there are no barriers to working with the best people in the world. Freedom is likely another ingredient. When there is the latitude to experiment, it is more likely that new and better methods will be discovered. Formal software development methodologies are good at managing more predictable efforts to solve business problems, but they seem to be of less value in OSS projects at driving innovation. Each successful OSS project is different than the previous one and is a unique combination of various factors that cannot be easily replicated. Many people still stumble at the fact that if my company contributes to an open-source project, we are helping our competitors. This is true. However, if we don’t participate in open-source projects, we likely won’t get to where we need to go. It may no longer be optional. Open source is not like walking into a discount store and picking something off the shelf at a tremendously good deal. It is not a one-time decision where we select something and are done. It is an ongoing process of continual change and improvement. And the better we are integrated into this process, the more we can leverage and use open source. Economic opportunity is not a fixed-sized pie, where we are all competing for the largest slice. With software, the duplication costs are near zero, so this is a perfect opportunity for economic growth or an overall increase in society’s wealth. We can work to grow the size of the pie, so everyone can have a larger slice. Often in writing, the author learns and understands things he did not know before, and this essay is no exception. The benefits of open source are not primarily in getting stuff for free or pooling resources in a socialist sense for the greater common good, but rather in providing an environment and connecting the right people for innovation to happen.This dynamic has happened occasionally in the past inside companies such as Bell Labs</a> and Lockheed Martin’s Skunk Works</a>. But this environment is obviously difficult to replicate or more companies would do so. Even the largest and most successful technology companies, such as Google and Facebook, aggressively support and promote open source. At the time of this writing, Google employs over 27,000 engineers – why would they need open-source projects? With that number of developers, why can’t they create anything they would ever need? Google’s support for OSS projects is not an altruistic gesture, but rather a calculated business decision – they are supporting the processes and people that will produce the software technology they need. This is a long-term game that requires experience, skill, understanding, and patience. It is an example of humility, which is the source of all true greatness – the realization that there are some things I cannot do, even if I have huge financial resources. It is the realization that there are people out there who are much better at certain tasks than anyone in my company or anyone I can hire. It is the realization that my company organization is not the perfect environment for the development of needed software breakthroughs and highly refined building blocks, so we need to support projects that are. Because it is counterintuitive, many companies will not make this leap, especially those who have been successful in the past without it, or those building products that in the past did not require advanced software systems. Ironically, it is often the smaller companies that need OSS components the most that are resistant to adopting and contributing to OSS projects. However, the opportunity is available for anyone to participate, support, influence, and use the most advanced software innovations available today. And these innovations are largely happening in OSS projects. Your engineers can work with some of the most gifted developers in the world and learn skills that will greatly enhance their careers and benefit your organization. You will be plugged into an engine of innovation that produces value. You will have more control over the needed building blocks. You can be part of the process that produces the valuable software you need, rather than just a consumer. Open-source projects are far from perfect. They are messy at times, as human efforts tend to be. Their future is not guaranteed, but nothing in this world is guaranteed. The lack of formal business contracts/methods is perplexing to most business people. However, open source appears to be the best model we’ve discovered thus far for software innovation, and the results are amazing. Open source is the only supply chain that can deliver the needed value, innovate and change, manage the increasing complexity, give you some amount of control, and provide the culture necessary for innovation. Will you be a part of it? Think Horses, not Zebras 2023-09-22T00:00:00+00:00 There is a popular quote</a> in medical circles: When you hear hoofbeats, think of horses, not zebras. – Dr. Theodore Woodward </blockquote> “Zebra” is the American medical slang for arriving at a surprising, often exotic, medical diagnosis when a more commonplace explanation is more likely.1</a> What does this have to do with product development? Like the medical profession, we often diagnose problems – we call it debugging</a>. Below are three recent cases where I would have been helped by applying this approach a little more rigorously. Simple IoT Startup</h2> I recently encountered a bug</a> where the compiled version of Simple IoT would not work on one of my computers. Interestingly, go run</code> worked fine. What was the difference? At first, it seemed that a race condition was at play – the compiled version was faster, did not have the race detector enabled, and would likely have different characteristics. SIOT is a highly concurrent application with many clients running in parallel. However, after spending some time debugging, I could not find any issues with startup, rather the NATS client in SIOT could not get data from the embedded NATS server. Again, race conditions were assumed, so I tried an external vs. embedded NATS server – no difference. I finally observed the network traffic with Wireshark and could not observe any NATS traffic in the failing version. The NATS client was not even sending any requests. I then looked at /etc/hosts</code> and there was no localhost entry. After adding this, everything worked properly. See this discussion</a> for more analysis of this issue. In this case, a race condition was the zebra and the missing entry in /etc/hosts</code> was the horse. i.MX8 Stability</h2> We recently ran into a problem with a new product that uses a Variscite VAR-SOM-MX8</a> in a custom baseboard. The software was stable on the Variscite development baseboard, but the same SOM (System on Module) would crash in the custom hardware. The system would never crash until after the Yoe updater initramfs ran and the main rootfs was loaded. Hardware was the immediate suspect – the power supply is probably not able to handle the dynamic load when multiple cores spin up, etc. However, the power looked stable and we could not find any hardware issues. In the end, we found a missing device tree line ("fsl,imx8qm")</code> in the Linux kernel DTS file for the new design. This missing construct caused some critical kernel initialization code not to run. In this instance, a subtle power problem was the zebra and a missing line in the DTS file was the horse. STM32H7 USB Instability</h2> In a recent STM32H7 design, we wanted to use a High-speed USB Phy with the STM32H7 and Zephyr. This is not supported by Zephyr yet, but with a few hacks</a>, we got it working. We are trying to push a fair amount of data over this link (10’s Mb/sec) so some optimization and performance work was required on both the sending and receiving end. In the mix of all this, we got a new revision of the target hardware, and at some point started seeing an increase in error rates and the target not enumerating and connecting properly to the host. With a lot of moving parts and a lot of data flowing through the system, it is sometimes hard to pinpoint the problem. In this situation, we tried to narrow the scope by testing older/newer versions of Zephyr and comparing STM32H7 reference designs to the customer hardware. We found the reference designs performed much better. We also learned that a previous version of the custom hardware also performed like the reference designs, so it appeared there was a hardware problem with the latest custom hardware. When something worked before, and now is experiencing high levels of communication errors, the first thought was signal integrity. On the PCB design, there is a fairly long trace from the USB Phy to the connector, so this was an obvious suspect. Although we were careful to design the PCB such that the impedance of these traces was controlled, there is always the possibility that the PCB vendor did not implement the board stack-up correctly or some other strange issue. One of the engineers analyzed the trace impedance on the old and new boards with a network analyzer. This is a fairly involved test procedure that requires cutting traces, soldering coax cable to the PCB, and a fair amount of expertise. The USB traces between the two revisions looked equivalent and the impedance measured was correct. The next suspect was the high-speed signals between the MCU and Phy – we had changed some of the length matching between the two versions, so perhaps we messed something up. In the process of looking at these signals, the following was observed when scoping the data line: </figure> The data line was running at two different voltage levels. The problem ended up being the power to the VDDIO pin on the phy was floating, so the signals from the phy to the MCU were not being driven at the right voltage levels. Simply scoping all the power pins on the Phy would have found this problem in a few minutes. In this instance, signal integrity was the zebra, and a floating power pin was the horse. What can we learn?</h2> In all of the above cases, the initial assumptions about the cause of the problem were considerably more complex than the actual problem turned out to be. Problem</td> Zebra (complex)</td> Horse (simple)</td></tr> SIOT Startup</td> Race conditions</td> Host networking config</td></tr> i.MX8 Stability</td> Dynamic power supply problem</td> Missing line in device tree file</td></tr> SMT32H7 USB errors</td> Signal integrity</td> Floating Phy IO power</td></tr></tbody></table></figure> Hindsight is always 20/20, so it’s easy to say we should have solved all these problems more quickly than we did. But there is always something to be learned, and I think as humans, our minds generally go to the more exotic scenario. Part of this may be overconfidence in our ability to get the simple things right, like copying a DTS file from a reference design, or supplying power to a chip. How can we possibly mess up something so simple? So we tend to overlook the simple and obvious and jump to the obscure and exotic, which exercises our most advanced abilities and knowledge. The more experienced we are, the more likely we may be to think this way. To overcome these human tendencies, it is helpful to acknowledge first that they exist (humility), and then put in place a process for future endeavors to help guide us in our debugging efforts. Consider the following maxim: When debugging a problem, always verify and eliminate the simple/obvious possibilities first. Work from simple to complex. This is a pragmatic approach because verifying the simple possibilities usually does not take much time, so little is lost in starting there. It may be boring or tedious, which is why we subconsciously tend to avoid these, but we can save a lot of time if we just follow this simple rule. A general process might be: isolate the problem as much as possible (remove unused functionality, etc.)</li> write down everything you know, possibilities (all of them), and ideas of what you might try.</li> sort your ideas from simple to complex and work on the simple ones first.</li> document all tests and results.</li> meet as a team regularly to discuss results and brainstorm the next steps.</li> </ol> Some specific ideas: verify basic network connectivity any time a network is involved, even if on the same machine.</li> don’t ignore any problem in the system. Often problems are related, and solving one problem sometimes solves another.</li> check that all components involved have good power.</li> compare to a known working reference design (both operation and design).</li> ask for help in relevant communities.</li> review and verify component pin-outs.</li> read the datasheets of the components involved.</li> read the errata for components.</li> work backward from a known working setup.</li> with web apps, open the console, and look for errors.</li> carefully review boot and build logs and diff with logs from working systems.</li> narrow the scope of the problem wherever possible.</li> try multiple systems.</li> carefully review any changes between working and non-working versions. Verify the operation of any new circuits or code.</li> don’t keep doing the same thing and expect different results.</li> </ul> Debugging is an art and a part of the development of any complex system. We must realize that our intuition often fails us in these situations and, with any human weakness, we must have a process to compensate. This debugging process can turn what is often a frustrating and scattered experience into something much more enjoyable and productive. Think horses first, and then zebras! ^{1 https://en.wikipedia.org/wiki/Zebra_(medicine)</a> </div> The Technology Trajectory 2023-01-03T00:00:00+00:00 See also a podcast episode</a> where we discuss this topic. Modern systems require the integration of technology, often from many sources. This can come in the form of processors, integrated circuits, operating systems, open-source software components, databases, messaging systems, cloud infrastructure, programming languages, build systems, frameworks, etc. All technology follows a trajectory – an example is shown below. The timeline may be short or long. Your usability threshold may be low or high, sloped right or left, etc. But the fact remains that most technology will eventually be replaced by something newer. We can debate whether or not newer is better, but the fact remains that time marches on. </figure> Why can’t we use a particular technology forever? With hardware, components may not be available. With software, we potentially have a little more control, but if we want to use a software component beyond the above cycle, then we have to commit to taking over maintenance. This means: if there are bugs, we will fix them</li> if there are security problems, we will fix them</li> if there are build problems with new versions of compilers and other libs, we will fix them</li> </ul> For complex, networked projects, this can be a significant undertaking. But a far bigger issue is that of value. Open source software and corresponding ecosystems are continually improved over their life and value is added. If you use projects in your development that are being improved, then value is being added to your product. The rate at which value is being added correlates to the slope of the curve above. To extract the most value from technology, it is generally best to adopt technology when the trajectory is rising and there is project momentum, not when everyone else has moved on. There are several reasons for this: the rate at which value is being added is greatest during this period. This makes what we are building better.</li> you have an opportunity to participate in and influence the development</li> you have more time to benefit from the technology</li> more support is available as there are more people participating in the project</li> in programming languages, there are more maintained 3rd party packages available</li> </ul> We also need to select technologies that have a reasonably long life cycle as switching out a complex component can be a lot of work. There are many other perspectives by which to evaluate technology – this is just one of them. Technology follows a trajectory from inception, growth, maturity, decline, and finally obsolescence. How do you know where a particular technology is on the curve? It is probably partly gut feeling, part experience, part asking others, and perhaps there are analytical ways to approach this as well (search engine ranking, etc). But the most important thing is to use this concept as a framework for objective thinking and evaluation. Effective technology selection is a long-term effort. We need to have a process. We may have to watch a project for a year or more to get a good feel for it. Most of us can’t significantly change the trajectory of most technologies, but we can choose where on the curve we will participate. Inefficiencies occur early and late on the curve. If we are always too early, we likely won’t get much done. If we are always too late, then our products will be a few steps behind everyone else’s. Go for IIoT Systems 2022-12-09T00:00:00+00:00 As developers, we have many options for programming languages. On one hand, it is great to have choices. On the other hand, it can be a little overwhelming. Many times in life, we need to make decisions before we have the experience to know the best answer. There are many opinions as to what is best, and this article is just another one. There is no one-size-fits-all – a lot depends on your team and the task. In this article, we will examine the challenges we face in developing industrial IoT (IIoT) systems, review desirable attributes in a programming language, and discuss how Go meets these needs. Peter Bourgon defines the industrial programming context</a> as: in a startup or corporate environment;</li> within a team where engineers come and go;</li> on code that outlives any single engineer; and</li> serving highly mutable business requirements.</li> </ul> Additionally, an industrial IoT system can be thought of as a system that gives users access and control of remote systems. There are many variations of this, but let’s focus on a simple case that is typically composed of the following pieces: Edge device (remote equipment, sensors, etc), Cloud, and Browser. IoT Components</figcaption></figure> All of these pieces must be dealt with at some level and will require some programming. The fundamental requirement of this system is networking – data must be moved between the above three instances. IIoT systems are inherently distributed systems. Distributed systems are hard</a>, thus the distributed problem needs to be solved well if you want to have a scalable, reliable IIoT system. It is natural to focus on the end application at the edge and add IIoT as an afterthought. However, this rarely works very well. The context of this article is that of an average programmer working in small teams where engineers do a lot of different things. Many of us are not full-time programmers doing deep theoretical work, but find ourselves building these complex systems. Some of the challenges/requirements we face include: resources are constrained at the edge (CPU, Disk space, etc)</li> networks can be unreliable</li> config/state needs to exist in multiple places (edge & cloud)</li> software updates need to be deployed to remote places (edge & cloud)</li> end-to-end (edge <-> browser) real-time response is often desired</li> many parallel operations are happening (collecting data, running rules, managing the system, etc)</li> physical access to edge systems so debugging problems can be difficult</li> edge systems typically use Embedded Linux, which can be a challenge to develop and maintain</li> IoT systems are inherently concurrent and distributed</li> many of us are doing this part-time, are not computer science gurus, and need pragmatic solutions.</li> </ul> Attributes of Go</h2> Go has several attributes that make it a good fit for this problem: excellent reliability. I’ve written about this before</a>. (our systems are not crashing in the field)</li> it is a compiled language (reasonably fast and small)</li> runtime is small, embedded, and efficient (this means there is very little overhead to distributing the entire runtime with each binary. This makes deployment easy.)</li> statically linked (this makes it much easier to deploy updates as everything you need is embedded in one binary file)</li> easy to embed assets (again, this makes it easy to deploy in one file without containers)</li> is garbage collected, so we don’t have to worry about managing memory (we don’t have memory leaks, invalid pointers, and other problems that can cause field failures)</li> typed language (compiler catches more problems early instead of having crashes in the field)</li> popular and well-supported (will continue to improve and be around for a long time)</li> excellent concurrency support (a lot is going on in these systems, so this helps)</li> popular in cloud and distributed systems (there are plenty of stdlib features and 3rd party packages – metrics, monitoring, logging, communication, etc to help us build an IoT system)</li> language is simple (easy to learn, read, and maintain)</li> tooling is excellent, compile is blazing fast, no Makefiles, etc, built-in support for profiling, and easy to cross-compile for other systems (we spend less time messing around and more time deploying value to our customers)</li> </ul> Go is not the best solution in every case, but if your situation matches what is described above, it might be a good fit. Go removes a lot of complexity</h2> Cloud and edge systems typically run Linux. You may outsource parts of the cloud portion and perhaps even parts of the edge device, but you are generally responsible for the edge device. It must run 24/7. You likely need to deploy updates, maintain security, and add functionality over the product lifecycle to meet customer needs. You will run into issues and need to be able to debug them. You can’t walk over to an edge or cloud system and press the reset button or hook up your JTAG debugger. Most embedded software in the past has been developed for MCUs (microcontrollers) and written in C/C++. This is a fine solution for disconnected control-focused products, but a Linux MPU (microprocessor) solution provides you with more flexibility and functionality for data-focused connected devices that process/store a lot of data and have complex user interfaces. Linux provides unparalleled support for a large number of devices, USB and networking functionality is reliable and mature, and it scales to processing and storing large amounts of data. In an MCU platform, you may write most of the software in the system. In some cases, you may use an RTOS (Real-time operating system), but even these are relatively small as resources are limited. In a Linux system, you are running millions of lines of code you did not write. We can use the iceberg analogy – the part underwater includes Linux and system libraries which you don’t see most of the time. The part you see above water is your application, which is a tiny portion of the total software in the system. </figure> Part of the value of modern systems is the huge number of software components you can reuse in your system as there are software libraries/packages for many tasks. Building modern systems is partly the creative combination of existing/reusable technology and the creation of new technology (the code you write). </figure> So the question arises: How am I going to get up to speed on all this? One solution is to minimize the amount you need to interact with the stuff underwater. If you build applications with C++, the compiler does not give you a lot of help in linking to libraries and cross-compiling to edge system architectures. (Cross compiling is compiling native code on one architecture for another – typically developing on an x86 workstation and then running on an ARM Linux target system). Thus, you need to rely on complex build systems (CMake, Yocto, Buildroot, etc). Every time you add/update a library/package, you are faced with the following questions: How do I integrate this library into my app build?</li> How do I cross-compile to my edge system?</li> </ul> It’s not easy, thus projects such as Yocto and Buildroot exist. All developers need to have a cross tool-chain (compiler and libraries) installed on their development computer that matches the libraries in the edge system. Deploying an application update often means deploying additional libraries, or library updates. So we add container technology to try to manage all this, which is another layer of complexity. Some solutions (like Raspbian for the Raspberry PI) advocate developing directly on the target device to avoid the complexity of cross-compiling. This works fine for small one-off maker projects, but for the development of large industrial programs, it is a poor solution as it is like developing on a PC from the 90s. Sure, cross-compiling is easier, but everything else is much harder. Python or Node.js may look attractive in that they are easier to program than C++ and the program is interpreted on the target device, but many of the underlying packages for these languages depend on C++ code, as interpreted languages are slow, as a result the build problem is worse because it is even more difficult to cross-compile this code in the Python and Node.js build systems. Additionally, a separate run-time is required on the edge system which is relatively large and must be kept in sync with the application. Go sidesteps most of this because it is a mostly pure ecosystem. There are no host library dependencies in a pure Go application – everything it needs is embedded in a single binary file (typically 2-10MB compressed). The Go standard library is rich and has many needed functions. Additionally, there is a rich selection of pure Go 3rd party packages</a> that cover most other needs. Adding packages to a Go application is as simple as go get some/package/name</code>. There are no development tool requirements other than a standard Go installation and an editor. No complex build tools are required to add 3rd party packages or to cross-compile to different target systems (macOS, Windows, Linux). It has built-in support for profiling. If it crashes, you always get a stack trace. It’s so easy, and the learning curve is small. Many software tools today add layer upon layer to solve the problems with the underlying technology. While this helps in some cases, it often adds additional complexity. You can’t gloss over complexity – when things go wrong, you still need to understand the entire stack anyway. Abstraction has its place but can only take you so far. If there are problems in one technology layer, adding additional layers sometimes just makes it worse. Go threw everything out and started over</a>. Simplicity is the only sustainable approach to technology. Go is a relatively new language (released in 2009), so many people have not heard of it. Learning a new language does take some time. Go has a small learning curve, but that is nothing compared to the learning curve of building, deploying, and maintaining C++ code in edge systems. Go allows you to quickly become productive and spend very little time fighting tools over the product lifecycle. Moving your Go application forward over time often does not require any changes to the base Linux system because they are decoupled. Go is familiar and simple</h2> Google originally developed Go</a> as a solution to the pain they were experiencing with C++, and they are still investing in it</a>. Go syntax is similar to C and was developed by some of the same people</a> who were involved in the development of the C language and Unix back in the 1970s. Thus, it feels very familiar to those of us who have spent time developing in C, Java, C#, etc. Go eschews the complexity that has overtaken C++. Language features are added slowly and carefully. Compile errors are generally easy to understand and debug. Backward compatibility has been excellent. Since 2014, I’ve implemented several systems in Go and have not had any significant issues after updating the Go or 3rd party package versions. Go developers take API compatibility seriously, so this helps. Go is a simple language but does not force you to do things correctly (as a language like Rust does). You can still make a concurrency mess with channels. Go has nil</code> values, so you can still crash your program if you try to reference a nil</code>. If you don’t check returned error values, you’ll probably run into problems. But, if you follow a few simple best practices and idioms, it works well and is a nice experience. Simple programming languages are much easier to read and maintain. Go Concurrency Support</h2> IIoT systems have a lot going on, so you need good support for concurrency. Interpreted languages like Python and Node.js are weak in these areas. Go has excellent support for concurrency by providing channels and the go</code> keyword as language features. This allows for running concurrent tasks and communication between them with very little overhead. Go channels are easy to misuse, so it requires a little effort and practice to learn how to use them correctly, but once understood, it is a very simple and reliable way to manage concurrency in a system. Go is an Internet language</h2> While older languages like C/C++ and Python have been excellent tools for solving programming problems in the past (and are still the best in some domains like MCU development, machine learning, etc), the problems we are solving in IIoT systems are different today in several ways: we are dependent on large ecosystems of 3rd party packages to implement all the functionality we need</li> we need better tools for concurrency and distributed systems</li> we need simpler models for network programming</li> we need software that is easier to build and deploy</li> </ul> Communication between systems is the fundamental problem IoT systems solve and Go is good at this. Go is the best-in-class for a new breed of software: an Internet programming language</a>. As evidence of this, most cloud and distributed software infrastructure is written in Go these days (Kubernetes, Docker, NATS, etc). Cloud companies that scale to thousands of systems are very sensitive to cost, thus they need efficient platforms. This efficiency in Go also benefits us at the edge where resources are constrained. An added benefit is that you can write your edge and cloud applications in the same programming language. But Go has no GUI support …</h2> Although there have been many excellent efforts to develop a native GUI library for Go, none of them have emerged as the standard. There is a good reason for this … the browser is the standard UI platform these days. Nothing else can come close to touching its capabilities. It is cross-platform, reasonably performant, and very well-supported. The browser has so many options for programming, packages, etc. The browser has built-in support for video and other forms of media and is highly optimized. Developing applications in the browser is more difficult to get started than in a C++ GUI toolkit. But there is a much richer ecosystem of packages and functionality available in the browser platform. The browser platform is stable and you are ensured of backward compatibility – it will be around and well-supported for a long time. The web platform is built on solid engineering principles that have proven to scale. Many UI technologies designed to replace the browser programming model have come and gone: Java, Macromedia Flash, and Microsoft .net – yet the standard browser programming model of HTML5 + Javascript has displaced all of these and is growing at an astounding rate. Google Flutter</a> is the latest attempt to displace the web platform, but it remains to be seen yet if it will be successful. Most C++ GUI toolkits are highly dependent on a single company that may be here today and gone tomorrow (look at the history of Qt</a> for an example), or licensing requirements may change</a> at any time – there are a lot of unknowns. Plus, with the web platform, you can view the UI remotely just as well as locally. You can also use the same UI building blocks in the cloud as in the edge device. If designed correctly, web applications also function very well on mobile devices, eliminating the need to develop separate mobile apps. With the web platform you can address all of the following: on-device LCD UI (runs a browser full screen)</li> remote access to the device using a browser</li> cloud application UI</li> mobile device access (browser, PWA, Cordova, etc)</li> </ul> The experience of a native phone app might be slightly better than a web application on mobile devices, but how many of us can afford to develop 4 different UI applications for a product (native, web, Android, iOS) when one will do in most cases? The question is not what is possible or technically best, but what is practical with the limited resources we have. Languages like Elm</a> simplify front-end (web) development in many of the same ways that Go has simplified back-end development. But Go is not the best at …</h2> Very true – Go is not as easy as Python, as widespread as C++, as popular as Javascript, as safe as Rust, or as good at concurrency as Elixir. But, Go is almost as good as the best at everything and its tooling is much better than any other language ecosystem – especially for distributed applications that need to be deployed to remote and different architectures. It does not have any of the significant drawbacks other languages have. And especially for networked IoT systems, it is an excellent choice. The below decision matrix likely reflects my biases, but illustrates this point (green is best, red is worst): </figure> But what if I need to use C++ …</h2> There are certainly cases where you may need to use C++ or some other language. You may choose to implement a native UI using Qt or LVGL. Machine learning code is often written in C++. There is no reason your system needs to be written in one language – Go can be used in combination with other languages. There are several ways this can be implemented: call the C++ code directly from Go (CGO)</li> run the C++ and Go code as separate applications and communicate between them using a message bus</li> </ol> Method #2 is preferred in many cases as it keeps your Go code clean, and the C++ code can be built using standard C++ build systems instead of trying to force the Go build system to build C++. Simple IoT</a> is one framework that is designed for exactly this scenario: </figure> This architecture keeps your Go code easy to build/develop/deploy. Your core logic, which manages config/state and communicates with other systems, can be written in Go. This core logic must be reliable and easy to maintain. Specialty code like machine learning models can be dedicated C++ applications. NATS is a modern message bus written in Go that is easy to embed in any Go program and is a great way to connect different applications. </figure> Optimize for where you will spend most of your time</h2> Linux-based IoT edge systems have plenty of memory, storage, and processing power. Thus, features are continually added over the product lifecycle, which can be 5-10 years in an industrial system. The time spent maintaining and improving a product often swamps the initial development. Therefore, it makes sense to optimize for the long term – adding features and deploying updates over the next ten years. Go makes it easy to build and deploy updates with the latest Go and package versions. Friction</h2> Developer productivity is difficult to fully understand. We often think of the features we need in our languages/tools. Commercial software vendors focus on features. Development methodologies are designed to increase productivity by better organizing what needs to be done. But another aspect that is easy to overlook is the amount of time we spend messing around working on stuff that does not directly add value to our product. I call this “friction.” This friction can manifest itself in unforeseen problems that are more common in complex systems: learning curves, build issues, issues with libraries/packages, lack of flexibility, security problems, deployment problems, bugs, race conditions, stability problems, technical debt, lack of control over parts of the system, etc. Go does an excellent job of reducing friction in the development process. You focus on what needs to be done, and the rest is mostly invisible. Summary</h2> In this article, we explored many of the attributes of Go that make it ideal for implementing IoT systems. Go provides a modern programming ecosystem where the build tools are first class, the language is simple, and there is little friction to get things done. Go also provides features such as garbage collection and concurrency support in the language which are very helpful in IoT systems where data is distributed and a lot is going on. Go is an Internet language that is increasingly used to develop distributed systems, which includes IoT systems. It is a stable (almost boring in many respects) technology and is a great option for developers who are familiar with C/C++ and want to expand into Embedded Linux/IoT – the skills you have are very applicable in Go. What is the main constraint in your projects? Application performance?</li> Getting things done and shipped?</li> </ol> If #1, then perhaps C++ or Rust is a good choice. If #2, then take a look at Go. Go is not the best solution for every team/project, but for the scenario described in this article, it has worked incredibly well. With Go, we can develop and deploy remote systems with confidence. Layered, Event, and Data-Centric Architectures in Distributed IoT Systems 2022-06-14T00:00:00+00:00 With the advent of the browser, cloud, embedded Linux systems, and networked microcontrollers, distributed systems are everywhere. There are many models for communication in distributed systems – we will look at the tradeoffs between three of them. There are many perspectives to consider – initial implementation, client libraries, maintenance, adding features now and in the future, client compute and storage requirements, network bandwidth, data structure, etc. This article will discuss a number of these concerns and present several options. The motivation for this discussion is synchronizing data in IoT systems where we have the following requirements: Data (state or configuration) can be changed anywhere, at edge devices or in the cloud, and this data needs to be synchronized seamlessly between instances. Sensors, users, rules, etc. can all change data. Some edge systems have a local display where users can modify the configuration locally as well as in the cloud. Rules can also run in the cloud or on edge devices and modify state.</li> Data bandwidth and allowance is limited in some IoT systems (example Cat-M modems (< 100kb/s) and cellular IoT plans (< 50MB/month)). Additionally, connectivity is not always reliable, and systems need to continue operating if not connected.</li> </ol> The first is the Layered model. In this model, various instances talk directly to other instances. If any information needs to be shared, it is sent directly through an API (application programming interface) call using REST, CoAP, and other protocols – sometimes traveling through multiple layers. The layered, or API model’s advantage is simplicity – almost anything can connect to a REST endpoint. Most networked devices have built-in support for HTTP. Many companies provide services through REST APIs. Polling is typically used in the layered model to determine when things change, which limits the response time to the polling frequency. There are mechanisms that can be used in a layered architecture to obtain real-time response such as HTTP long polling, WebSockets, Server SIde Events (SSE), callbacks/webhooks, CoAP subscribe/notify, etc. However, an event bus is generally a much cleaner implementation if real-time response is needed. Firewall and security concerns can also make callbacks impractical on edge devices. If a device at the end of a layered architecture needs new information, often multiple layers need to be modified to handle this new information, which can make changes expensive. A second model is the Event or Message Bus model. Implementations include NATS, MQTT, Kafka, and many others. In this model, every instance in the system is connected to a message bus. All data communication is routed through the bus. Clients can post or subscribe to topics of interest so that updates happen in real-time. Some advantages of the event bus include real-time response and streaming workflows. All communication and information is on the bus, so if an instance needs to send or receive some information, it simply subscribes or publishes it. Event buses often leverage one connection to the message server so that each message does not need to establish a new connection, which can bring some efficiencies and reduce needed resources</a> (such as number of TCP connections). However, the event bus protocol is considerably more complicated than REST, so a message bus library is typically required to implement the client application. The third model is the Data-Centric model. In this model, each client instance contains a set of data it is interested in. In the below example, the cloud system has an aggregate of all the data in a system, and the browser and edge instances are subscribed to a subset of the data they are interested in or have access to, represented by the different colored blocks. The data models the entire system, so if an action (setting an output, sending a notification, etc) is required, a bit of data is changed, and then the system takes action based on this data. The needed data is replicated to each instance in the system and is automatically synchronized in any direction when it changes. To all you Go programmers out there, this may immediately raise a big red flag. What? Communicate by sharing common data? Don’t communicate by sharing memory; share memory by communicating. R. Pike </blockquote> Pike’s comment is typically used in the context of an application where multiple threads (goroutines) share data in memory, and locks are used to keep multiple threads from writing to it at the same time. This model can be hard to reason about due to race conditions. An analogous model in distributed systems may be a common database that all instances use and all config/state is stored in this remote database. Locks may be analogous to database transactions. A common database is a good solution for many cloud based distributed systems where everything is always connected and, and this model allows data to be modified from multiple locations. However, with IoT systems, we have the following constraints that make it impractical to use a single remote database as our primary configuration/state data store for everything in the system: We often don’t have the network bandwidth or data allowance (ex: LoRaWAN or Cat-M modems and low cost cellular IoT plans)</li> Network connections are unreliable, and edge instances need to continue working (collect data, run rules, etc.), even if the network is down.</li> Network latency can be long – especially with low bandwidth connections.</li> </ol> This leads us to a “local first” data model, where all configuration and state needs to be stored local to an edge device. However, this data is often viewed or edited in the cloud, so it also needs to be present there. We could craft new data structures and messages for every new bit of data and send custom messages over a REST API or event bus every time anything on either end changes. However, a simpler approach is to design a generic data model that can hold any kind of data, and then a mechanism to synchronize this generic data model. Then, adding functionality is much simpler as we can write/read data wherever needed, and synchronization between instances happens automatically with no extra effort. However, we have to learn to work within the constraints of the data model. A new feature might involve tweaking the UI and writing some code on the edge instance. Often, nothing in the middle needs to change: </figure> Each instance has its own local copy of whatever data it needs. But instead of the developer needing to manually manage the synchronization of data between systems using custom messages, changes to the data are automatically synchronized in any direction. The data-centric architecture requires more computing resources and storage to run it – typically an embedded Linux system. The below chart illustrates the trade-offs between client complexity and programming simplicity. For less capable networked devices such as microcontrollers (ex: Arduino), you may be limited to a layered approach, as HTTP may be the only supported protocol. However, as microcontrollers advance, many of them are now supporting event buses such as MQTT. The cost of a data-centric architecture is an up-front cost when building the base system. As time goes on, the incremental costs of adding features is much lower. Since adding features is typically a continuous, long term exercise, the initial cost is paid back many times over. A data-centric architecture also introduces some discipline in how data is structured, stored, and synchronized. This helps avoid the technical debt that rapidly accrues in less structured approaches. All architectures have their place. If the application is fixed in functionality, and you only need to send a temperature reading to the cloud once per hour and this is never going to change, then a microcontroller with a layered architecture is fine. If you need to send data in both directions and real-time response is needed, then moving up to an event bus makes sense. If you have devices that have a decent amount of configuration that can be modified at the device and in the cloud, runs rules locally, and features will continually be added to the system, then a data-centric architecture is beneficial. The data-centric architecture requires a more complex system to start with, but the incremental effort to add features down the road is drastically reduced because the developer does not need to worry about the data storage/synchronization problem. Additionally, the data model is available at each instance, so programming is simpler. It is possible to support multiple architectures in one system. For example, in Simple IoT</a> all data modifications are sent over an event bus which is the primary programming API</a>. A client can connect to the event bus of a local or remote Simple IoT instance. If the local and remote instances are connected using an upstream sync connection</a>, then writes to the common data-set of either instance are automatically synchronized to the other. If a client does not want to implement a local data store, they can connect to a remote instance’s event bus. NATS JetStream</a> is another example of a store built on top of an event bus. Some event bus implementations like NATS allow for easy request/response transactions</a>, which is similar to a layered REST/RPC type interface. Thus clients in a system like Simple IoT can range from simple HTTP clients, to mid-range event-bus clients, to advanced Linux based, data-centric clients with a local data store. In computer science, focusing on data is typically the right approach. I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships. Linus Torvalds </blockquote> Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious. Fred Brooks </blockquote> Distributed systems are no different – the focus needs to be on the data. Synchronizing data efficiently and reliably between distributed instances is a hard problem to solve, and the focus of much research and development effort today (example: the Riffle project</a>). This requires some up-front thought, planning, and constraints, but the payoff is simpler data models that result in systems that are more reliable and cost effective over the product life-cycle. Data-centric architectures free developers from the tedium of synchronizing data and allows focus on the application problem at hand. Data-centric architectures also provide discipline and structure around the data synchronization problem that helps avoid technical dept build-up which is a great impediment in the flow of value to end users. Each distributed system architecture has its place, and with a correct implementation, we can support multiple architectures in one system. This gives us the option to support simple as well as advanced clients, depending on the need. However, as the cost of advanced MPUs systems running Linux such as the Raspberry PI continue to fall, deploying advanced IoT clients to the edge makes a lot of sense. In future articles, we’ll examine in more depth the techniques used to store and synchronize data in data-centric architectures. Thanks to Khem Raj, Bruce Stansell, Gina Brake, and Collin Brake for reading drafts of this. Business Contracts 2022-03-21T00:00:00+00:00 Business contracts are often executed when two companies work together. At the most basic level, a contract should describe a common understanding between the two parties. The contract should assist and aid the relationship. However, many contracts are written by lawyers with the primary goal of covering every legal loophole and taking every legal advantage possible for their clients. This often results in contracts that are very long, filled with legal jargon, benefiting only one party, and in general, a burden to process. For a business relationship to be successful and sustainable, there has to be value for both parties and there has to be trust. If these two key ingredients are lacking, there is little motivation to continue the relationship. We can look at business relationships as a positive-sum game, or a zero-sum game. A positive-sum game is where the relationship is structured as a win-win – both parties find good value in the relationship and are motivated to continue it. There is a belief that the “pie” of opportunity will grow through their combined efforts. The engagement is voluntary and not constrained by force or compulsion. Zero-sum thinking is where one party is always trying to gain the maximum advantage over the other party. This is the “fixed pie” mentality where we think the resources in business are fixed, and if anyone else gains something, it is my loss. This attitude is often found in purchasing departments where they are trying to obtain the lowest possible price for a good or service, net 60 payment terms, etc. I recall in one case a manager of a company exclaiming, as we went into a meeting with a vendor, “Let’s go beat ’em up!” Business relationships are more complex than getting the lowest price or the greatest legal advantage. The people/companies you want to do business with are likely already busy. They likely have plenty of opportunities. They will seek engagements with those who play the positive-sum game. If a company plays the zero-sum game, they will end up doing business with second rate companies who don’t have any other choice. They won’t get good service, and trust in the relationship will be lacking. They will be forced to hop from one vendor or customer to other. There will be few long term relationships. As Bob Metcalfe once wrote</a>, “First, don’t “hire” anyone. B people hire C people – they collect resumes and choose the person they want to honor with a job. A people recruit A people.“ The same applies to business relationships. With this background, let’s examine the topic of contracts a company may present to an independent contractor or consultant providing services to the company. Again, the best developers (the ones you want) are successful and busy. They will not mess around in business relationships where people are playing the zero-sum game. They are successful because they are trustworthy and expect the same in return. They are generous with their time and knowledge because they understand the universal law that the more you give, the more comes back to you. But they also expect the same in return. They are confident in their abilities and do not take measures that lock their customers into their services, but on the contrary, make themselves “easy to fire</a>.” A fundamental problem with trust is that people generally don’t tell you when they don’t trust you. Actions that damage trust are likely unintentional (otherwise you would not do them), but you never get feedback on this. So, I’m giving you a hint now – an onerous contract, with one-sided indemnification clauses, damages trust. It tells the other party that “I might take advantage of you.” It implies that if bad things happen due to a product flaw, “you could be on the hook.” It sows seeds of doubt. Most often, this is likely not the thought of the company presenting the contract, and the contract language is just the result of lawyers trying to do an exceptional job of protecting their client from any possible liability. However, a contract is still a contract. If disaster strikes, the contract will be used in court by various parties trying to recoup damages. Products are often sold over a long time (5-10 years or more). Companies often change owners. Therefore, it is very important that a contract be fair and just. When a developer is presented with an onerous contract, immediately the question arises – can I trust this company? Is it worth it? Even if the project goes forward, it will be with some reserve and guard. It will not be with the wholehearted attitude that is an ingredient for success. As we consider indemnification phrases in a contract, we need to establish who should assume liability for a product: 1) a consultant/contractor who is billing by the hour, or 2) the company selling the product over a span of many years? Risk should be proportional to reward. A hourly contract developer has no reward for a product’s success, other than the baseline costs of doing the development. The company selling the product is the one who will potentially reap the multiplying rewards of a successful product. Therefore, they should be the ones who bear the risk as well. It is not just to expect otherwise. In the situation where a developer is providing consulting or contracting services to a larger company developing a product, it is important that the contract clearly state that: The Company assumes sole responsibility for all testing of the Work, including but not limited to software and hardware, and also assumes sole liability for any use or misuse of the Work developed by Contractor for the Company. An argument may be made to the developer – you are not a lawsuit target because you are not a large company with deep pockets – so just sign the contract. This may be true, but what if the consultant does grow the business to the point where there are more assets? Products live a long time – a lot can change. As mentioned before, a company can change hands. If there is a lawsuit, then anything is possible. An individual developer has limited control of the development of a complex project and the resulting product. (We are not talking about fixed-bid turnkey product design where a larger company designs the entire product, but rather hourly development services). Typically, there is extensive involvement from other developers. The contract developer does not control how the product is manufactured, marketed, sold, supported, or even how other parts of the system are developed. In complex systems there will be bugs, and often there are time to market pressures that prevent extensive testing and verification. These are all factors that are out of the developer’s control and are choices made by the product owner. Again, this further illustrates that those who are ultimately in control of a product (the owner) should be responsible for any liability the product might bring. This brings us to a common concern of the company – they want to be sure the product is developed to professional standards. A contract phrase might be: “Contractor hereby warrants that its work shall be of professional quality and performed consistent with generally accepted industry standards” Who defines these standards? How will they be defined in court? The reality is that few developers have the time and resources to really do the job they want to do. Schedules are always pressing, forcing compromises. Ultimately the product owner makes the decisions concerning these trade-offs – not the developer. Time to market is critical. Therefore it is difficult for a developer to warrant any work will be at any particular industry standard – as the product owner is ultimately in control. The universal laws of reality apply: “Good-Fast-Cheap” – pick two – all three do not exist. In any product development effort, there are compromises. There is a very practical way to handle the concern that a company’s product is being developed to their standards, and that is to encourage collaboration and openness between all developers on the project. Make sure all source code and design files are shared in Git repositories that are easily accessible to all developers – both employees and contractors. Encourage all developers to work on any part of the system needed to get their job done. Google does</a> it</a> – maybe you should, too. Discourage engineering “turf” where developers feel they “own” something in the sense that no-one else should mess with it. This is essentially the process used in Open Source Software, and successful companies have adopted these practices – the results speak for themselves. If this works in a large company, is there any excuse for not doing this in a small company? Open source projects don’t have contracts that specify all contributors have to do a good job, yet open source projects contain some of the highest quality code available. Peer review and doing your work in public (even if only within a company) is a powerful motivator. Because the development is in an open environment, it is quickly evident who is going a good job and who is not. The same dynamic can happen in a company. If there is open collaboration, frequent integration, and early testing, you will soon get a good feel if developers are efficient and are generating work that is of “professional quality.” Share weekly reports with the entire team of what each developer did and how much time they spent on the project. This will produce a level of accountability among peers that will far exceed what any manager can force. If someone is spending excessive amounts of time and not getting anything done, other developers will start to question it. And you will know of problems early in the project at a time when you can do something about it, rather than after a product has been sold in quantity and some horrible defect arises that results in lawsuits. Which is the better option? To summarize: If a company wants to ensure quality work is being done by their developers, then they must foster a collaborative, transparent development environment where: All development is done in the “open” in Git repositories that the entire team has access to.</li> Any developer can contribute to any part of the system required to do their job.</li> Integration is done often.</li> Product testing, by the owner, is started early in the development cycle.</li> </ol> No amount of contract legalese can come close to being as effective as the above. Whether you intend it or not, your legal contracts are both a reflection and driver of your company culture, and its resulting success. They impact the people you will attract – both employees and contractors. Do your contracts reflect a fair and generous mindset where you are willing to take responsibility for the things you should take responsibility for? Do they turn away or create reserve in people you need the most? This is a serious issue because the best people often come through referrals. Will a developer recommend their talented and busy friends if they are required to sign uncomfortable contracts that must be negotiated to even approach an acceptable level? Any friction in this process is slowing you down. Yes, a one-sided legal contract that takes everything you can get may save you some liability in the unlikely event of a product lawsuit. But it also has the effect of introducing friction in relationships that are key to your product being completed in the first place. Which is more important? Which should have priority? In summary, a contract should: clearly represent the concerns of both parties</li> document what is expected</li> not be one-sided</li> be fair</li> avoid complex legal jargon as much as possible</li> aid, not hinder the business relationship</li> include provisions for how the relationship will be terminated, if needed</li> </ul> I’ve been working as a consultant for 20 years. My most successful and longest relationships have been with companies where I have never executed a single contract. Is this a coincidence? Is it cause or effect? I don’t claim to know, but my experiences resonate with the words of Seth Godin: A handshake deal, on the other hand, is about the future. Either side can claim loopholes or wriggle out of a commitment, but the consequence is clear—if you disappoint us, we won’t be back for more. The participant in a handshake deal is investing in the future, doing more now in exchange for the benefits that trust and delight and consistency bring going forward. from: https://seths.blog/2014/10/handshakes-and-contracts-the-future-and-the-past/</a> </blockquote> Somehow an essay about contracts has morphed into a discourse on company culture, development practices, and ultimately success. This is the power of writing – it helps you connect concepts that you did not previously realize were connected. In the end, everything is connected. Your contracts, your company culture, the type of people you attract, etc. It all matters. References: https://www.technologyreview.com/s/400489/invention-is-a-flower-innovation-is-a-weed/</a></li> http://www.unixwiz.net/techtips/be-consultant.html</a></li> http://sethgodin.typepad.com/seths_blog/2014/10/handshakes-and-contracts-the-future-and-the-past.html</a></li> http://badsoftware.com/contract1.htm</a></li> https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext</a></li> https://medium.com/@Jakeherringbone/you-too-can-love-the-monorepo-d95d1d6fcebe</a></li> https://www.clearviewlegal.co.uk/ohlawdy/contracts-are-communication</a></li> </ul> Technology 2020-11-27T00:00:00+00:00 The fire of information technology – will you use it: to create or consume?</li> to comfort or torment?</li> for education or entertainment?</li> as a tool or weapon?</li> to temper or incite?</li> for community or faction?</li> for harmony or rebellion?</li> to encourage or discourage?</li> to spread light or darkness?</li> to bless or curse?</li> to compliment or ridicule?</li> to save time or waste time?</li> to be generous or selfish?</li> to heal or hurt?</li> to share ideas or focus on self?</li> to be open and direct or talk behind backs?</li> to focus on things we can do or complain about things we have no control over?</li> to spread truth or falsehoods?</li> to express thankfulness or discontent?</li> to preserve our autonomy or be manipulated and exploited by others?</li> for productive pursuits or indulgence in vanity?</li> to engage in real life or escape into some virtual universe?</li> </ul> Do we have integrity such that our digital communications match what we would do in person? We live on a great sea of information, and the default path is to be tossed about on the waves of unverifiable communication that wash over our society, and in the end have little to show but discontent and unhappiness. Technology is an excellent tool, but a poor friend, a useful servant, but an oppressive master. Technology is no substitute for the things that bring true joy and satisfaction in life – family, friends, giving, companionship, creativity, craftsmanship, mastery, accomplishment, nature, and faith. Like fire, information technology is only useful when used intentionally and constrained to some useful purpose. Otherwise, it will rob us of our time, relationships, humanity, skills, potential, and ultimately our soul. Adventures with a ATSAMA5D27 board 2020-01-29T00:00:00+00:00 Recently, we brought up a new PCB (printed circuit board) a customer had designed based on the Microchip ATSAMA5D27C-D1G</a> processor. We encountered several interesting challenges. </figure> The first issue was that the design only had serial ports (and possibly a USB Host port that could be used in device mode) for bootstrapping the system. Microchip provides a tool named SAM-BA to do this. There are two implementations of SAM-BA. The older implementation does not support this CPU. The newer implementation</a> does support this CPU, but the documentation and examples are sparse (at least when we were using it). The standard way to use SAM-BA is to directly program the bootstrap and bootloader into SPI flash. However, this design only has eMMC flash. So we came up with a way to bootstrap at91bootstrap and u-boot over the serial port, and then load a Linux kernel with an embedded initramfs from a USB flash disk. This initramfs contains an updater</a> that is used to program the eMMC flash. After the updater runs the first time, the system will then boot from eMMC. We developed and verified this process on a development board, and it is documented in a Github issue</a> (The SAM-BA team has since disabled Github issues on this repository), and as part of the Yoe Distro sama5d27 support</a>. Microchip has also since provided us with a procedure to program eMMC using SAM-BA, but the procedure is complex and we never had a need for it once we figured out the above method. After receiving 10 prototypes of the new board, we learned pretty quickly that VCCBU must have voltage for the CPU to start. After that, the first board loaded u-boot and everything looked good, but things went south from there. U-boot would only load perhaps one out of twenty times. Basic checks of power supply voltage levels all looked good. The custom board used a 12MHz crystal for the main clock, where Microchip’s SOM1 module uses a 24MHz crystal. We decided to try a 24MHz crystal, and after that change, u-boot would reliably load over SAM-BA. We’re not sure why (12MHz is slower than 24MHz, so things should run better), but initializing a clock tree is complex and there is likely something that was not configured correctly in software. Lesson #1: Unless you want to spend a lot of time writing chip initialization software and debugging strange problems, follow a reference design as closely as possible. Working at the level of chip initialization often requires intimate knowledge of the MPU and likely the support of those who designed the chip. This is much more effort than most teams have time for and is best left to the MPU vendor. Now that u-boot was loading reliably, we could turn our attention to the kernel. At this point, a couple boards would start the kernel, but most failed with no kernel messages after the “Starting kernel …” message on the serial console. The boards that would start the kernel would crash partway through the boot process. We started looking at power supplies and observed a 100mV sawtooth on the core power supply. </figure> The SAMA5D27 datasheet specifies a maximum of 15mV of ripple for the core supply. </figure> The ADP5023 power supply used for the core voltage had a mode pin that can be used to force it into PWM-only mode, instead of PWS/PWM mode. After forcing the supply into PWM-only mode, some boards would start Linux. However, there were still stability problems later in the boot cycle. After trying many other things, we finally raised the voltage of the core supply slightly by changing one of the feedback resistors. Presto! Lesson #2: If a CPU has stability problems and everything else looks OK, trying raising the core voltage. The system was now stable. After communicating with Microchip, we learned that the CPU actually required a higher voltage than the documentation specified. Microchip provides a power supply that is designed to be used with the SAMA5D27 – if this supply would have been used, then we would have gotten the right voltage. This bringup experience was harder and longer than most – especially for a board of this complexity. But, it illustrates an important point – full custom MPU design is hard – even with a part like the SAMA5D27-D1G which integrates SDRAM in the CPU package. It can require a significant amount of time. This design was done by a competent designer and was extensively reviewed, but it still contained serious problems that took weeks to resolve. In the end, we were still able to make the first PCB revision work with minor modifications. However, it is very possible that the first revision of a PCB like this might need to be scrapped and chalked up to a learning experience leading to months of project delays. Lesson #3: Use a SOM</a> (System on Module) unless you have a good reason to do a full custom design, and you have the time and resources to pull it off. You are likely not Google or Apple</a>. Yes, you will pay slightly more for a SOM than the raw MPU system components, but the SOM cost may be offset by a lower cost baseboard. A full custom MPU design may require 8 layers and very small routing features where a baseboard for a SOM may only require 4 layers and use larger features, resulting in a less costly PCB. Additionally, a baseboard for a SOM may have a lot less parts on it, and requires less effort to set up for assembly – simpler ones can even be hand built to bypass weeks of assembly lead-time. There are dozens of SOM vendors offering products today that fit about any price/quality/certification price point. Some SOMs contain pre-certified WiFi and Bluetooth radios. Others meet various mil-spec requirements. This situation today is much different than 10 years ago when there were only a few expensive SOM models available. SOM modules allow us to bring advanced technology to a problem, and implement a solution with a small team in a reasonable amount of time. Implementing a SOM based MPU design still requires a fair amount of knowledge, but the unknowns are drastically reduced, and chances of success are far better. Much of the design complexity is already handled, freeing you up to work on the value a product provides. Any time we can simplify our development effort, our chances of success are much greater. Working with Microchip on this project was a good experience. Without their support, we would not likely have resolved the issues we faced in a timely fashion. This brings us to Lesson #4: When developing complex system, you are likely going to need some level of support. This may come from IC vendors, open source communities, consultants, etc. In this case, the MPU documentation was not up to date, and the chip designers were able to point out the problem. We could be critical of Microchip for this oversight, but with a 3000+ pages datasheet, this is a lot of detail, and there will be mistakes and oversights. The fact that most of the datasheet is available and accurate is a commendable accomplishment. Getting this level of documentation from some vendors is like pulling teeth. Before any of us cast the first stone, we must ask the question – is everything we do completely and accurately documented? Overall, the SAMA5D27 documentation is pretty good. The support provided on the SAM-BA Github issues was also essential for us to figure out a way to program the device. The procedure we found for programming this system using only a serial port is relatively simple and easy to implement. Though it may violate some SDRAM refresh timing specs, in practice it works very well – and that is what matters – getting things done. It is unfortunate that Microchip has since closed the Github issues for the SAM-BA project, indicating they are not really interested in building a community around the project, but rather Github is only a way to distribute the tool that is developed in-house. As systems get more complex, Open Source communities are proving to be the only viable mechanism to develop and support all the various needs users might have. Large companies readily acknowledge this as many core technologies are now Open Source, and much effort is put into building communities around them. But, lack of Open Source vision aside, Microchip is a reliable partner – especially for smaller companies wanting to use advanced technology and needing support. Drone for Continuous Integration 2019-11-26T00:00:00+00:00 Recently, we have been evaluating Continuous Integration</a> (CI) systems for a variety of projects (both OSS and customer). There are many OSS options to chose from</a>. Because we already use Docker containers for Yocto/OE builds</a>, Concourse</a> and Drone</a> made the short list. Both communities seem responsive</a> and helpful</a>. https://github.com/YoeDistro/yoe-distro</a></figcaption></figure> Building OE in a container</a> is important, so naturally Drone and Concourse seem like good options as they are focused on containers. Containers make sense for most CI tasks as they allow you to easily control the environment without getting bogged down in setting up the host. Because Concourse does not seem to easily support bind mounting volumes</a> on the host system, we ended up implementing a prototype in Drone</a>. This was improved and merged</a> into the yoedistro master. Ironically, we are not currently building in a container, as the yoedistro CI machine is a 64-core ARM system and we had some problems with the container support. Hopefully we can get containers working again at some point. Since the initial evaluation, Drone has added an exec runner</a>, which allows us to execute a pipeline directly on a machine without a container. Drone is an excellent fit for most of our projects. There are more capable systems (like Concourse, Jenkins, etc), but the simplicity of Drone is a big advantage in that it will more likely be used by developers. A tool is only useful if it gets used. With Drone, a developer can simply cd to a Git workspace that has a .drone.yml</code> file, and then run drone exec</code> and it will run the entire pipeline locally. Because drone typically does everything in containers, this works fairly well – it does not really matter what the host system is. This is powerful because developers can test the entire pipeline on their local machine before enabling it in the drone server/agents for automated testing/deployment. If there are problems, they can be debugged locally rather than the tedious process of commit, run the pipeline on the server, view the logs, repeat. Solutions that are simple to use make a big difference in productivity. Below is a quote from a Drone user: Today we have over 60 engineers interacting with Drone daily, and they really like it, due to its simplicity. We’ve always had a small DevOps team, so we tend to heavily favor solutions that don’t require a lot of maintenance overhead, and Drone fit the bill. Since most developers have at least a basic understanding of Docker, they hardly need any help maintaining their pipelines, and there have been a couple of instances of developers creating their own plugins! This way our team can focus on exciting upgrades to our infrastructure, even as teams regularly update their deployment pipelines and new projects are created every month. https://blog.drone.io/why-quinto-andar-chose-drone/</a> </blockquote> Drone makes some interesting trade-offs in that it trades some flexibility for simplicity and developer ease of use. Initially, Concourse was my first choice as it is a more powerful tool, but because it does not support bind mounting host volumes or exec runners, we ended up using Drone. This tool makes sense for many teams because it is more likely to be used by the average developer without having a dedicated CI/Operations team. This is a trend we are seeing at successful companies where developers are more responsible for everything related to their project, including testing, continuous integration, deployment, and operations. The quote below gives us some insight into how things are done at Amazon: There is another lesson here: Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service. — Werner Vogels (CTO @ Amazon), https://queue.acm.org/detail.cfm?id=1142065</a> </blockquote> Drone fits into this model well because it enables developers to become engaged in CI and deployment. One potential problem with drone is the lack of support</a> for some Git systems like Assembla. One workaround would be to store pipelines in Github/Gitlab/Gitea, etc, and then fetch code from other repos. But it would quickly make sense to add auth/webhook support for whatever Git system is used if it does not already exist. Reflections on KiCad and EDA Tools 2019-11-12T00:00:00+00:00 A recent interview</a> with a KiCad developer prompted some reflection on KiCad and EDA (electronic design automation) tools in general. Below are samples of several PCB (printed circuit board) designs, created with KiCAD, and implemented as part of the SimpleIoT project</a> in the last couple months. </figure> </figure> The experience has been excellent. Above all, the tool is very fast, efficient to use, and stable. Schematic and PCB integration works well enough, and routing and copper pours are easy. Switching between inches and millimeters can be done on the fly. The KiCad library has many parts in it, and other organizations, such as DigiKey</a>, Seeed</a>, SnapEDA</a>, and Ultra Librarian</a> are also providing libraries. If a KiCad symbol/footprint for a part is not already available, it is relatively easy to create new symbols and footprints as needed. There is a good KiCad support forum</a>. KiCad is a pleasure to use and production-ready for standard PCB designs. Choosing an EDA tool is an investment. There are several aspects to consider: purchase/licensing cost of the tool</li> time invested in learning the tool</li> time spent implementing designs</li> time spent maintaining designs</li> </ol> #2, #3, and #4 swamp #1 for many projects – at least when low end EDA tools are used. Thus we want to be confident that the tool we choose will be around for a while and will be well maintained. With all the churn in the EDA industry in the last 10 years, it is hard to have a lot of confidence in the long term prospects of many tools. For KiCad, however, the development statistics indicate that momentum around the project is increasing and that this tool will likely be around for awhile. https://github.com/KiCad/kicad-source-mirror/graphs/contributors</a></figcaption></figure> With 183 contributors, the KiCad project appears to be healthy – it is doing something right. Some may fear that open source projects (like KiCad) are taking away the opportunity for companies to make money and are not sustainable. Has the advent of Linux decreased the amount of money being spent on operating systems? On the contrary, Linux has greatly increased the operating system ecosystem – much more so than if proprietary Unix were the only option. Redhat, Ubuntu, and many other companies seem to be doing well. Linux is used in everything from watches to supercomputers. Unix would never have enabled this breadth of products – it is simply too large a task for any one company or team. Collaboration is the only way to accomplish a project of this magnitude. When the commodity layers are taken care of, then people are free to work on higher level problems. If a commercial EDA tool does about what KiCad does, that product is likely not going to fare well long term. But there is plenty of space to add value higher up the stack. Examples might include tools that work with KiCad such as auto routers, advanced placement tools, library management tools, RF design, SI/EMC simulation, high voltage design rules, etc. There is endless opportunity. The “pie” of opportunity is not fixed – it can grow. Contribute code to KiCad that will enable proprietary tools to run in tandem. If companies build on top of KiCad, then they will not have to duplicate the task of building a basic schematic and PCB layout editor and can instead focus on other features, and as a result, there will overall be more value for society. Companies will already have a large market of people who are more likely to spend incremental amounts of money to add functionality as needed. The alternative is for every company to try to provide the entire EDA stack, which can be a tough sell, as it is expensive and customers will need to change everything they are currently doing. There is no reason EDA software cannot become modular like the software development ecosystem. KiCad may displace a few commercial EDA products, but the value and opportunity KiCad will help create will far outweigh this loss. More companies will build more stuff, with less friction and effort. There will be more innovation. More problems will be solved (and likely a few created). There will be more collaboration. Even now, it is practical for developers, who only design PCBs occasionally, to use production grade tools. EDA tools should not be evaluated merely on technical features and price, but also on the hassle factor, the capability to facilitate collaboration, and the ease of long term product maintenance. Who will maintain your product in two years when a part goes obsolete, the design needs tweaked, and the designer and the obscure tool used on the original design is long gone? If you contract out a PCB design, why not require that the design be implemented using a tool that anyone can use in the future? Part of the value of a product is its maintainability. Companies who can quickly adapt as things change will have an edge. The costs of not being able to quickly respond to some product need, or of having to learn new tools and re-implement a design are significant. Over the life of some products, maintenance costs may be much larger than the initial development costs. In years past, I’ve used a number of different PCB design tools (Mentor Board Station, OrCad, P-Cad, gEDA, Eagle, etc). Some have been good experiences, others not so much. While there are no doubt EDA tools that have more features and are better than KiCad in some ways, KiCad is the low friction solution for your standard PCB design. It can be used easily by any one at any time on any computer. This flexibility is a big part of KiCad’s value. EDA tools are the “build system” for your hardware. They work best if they are open and easy to use by anyone on the team. There are many “free” PCB design tools available, but few of these are open. Some are cloud based, are very convenient to use, and technically a marvel that we can run a PCB design tool in a browser.. But the design is still locked up in a proprietary format. It is not easy to customize or extend these tools. And the sheer number of these commercial options available today suggests that long term they probably won’t all be around. “Open” is more important than “Free” in my opinion – especially for the long term. KiCad is certainly a noteworthy project in the technology landscape, and deserves consideration. The availability of such a tool will change the world. The open source hardware movement will grow. Companies who learn to leverage this movement will benefit in the same way that most companies now benefit from the use of open source software. More wealth will be created for society. The traditional barriers between hardware engineers, software developers, and PCB designers will be reduced. Just as hardware engineers have had access to about any software development tool they want to use, now software developers have access to tools to develop and tweak hardware. The small amount of time required to learn KiCad is a worthwhile investment. The Humble Programmer 2019-10-31T00:00:00+00:00 In 1972, Edsger W. Dijkstra</a> published a paper titled The Humble Programmer</a>. Dijkstra was trained in math and physics and was a university professor for much of his life. This paper is an interesting reflection on the history of computers and contains thoughts for the future. A few quotes are included below: Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with. In other words: both goals point to the same change.</li> To put it in another way: as long as machines were the largest item on the budget, the programming profession could get away with its clumsy techniques, but that umbrella will fold rapidly.</li> The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.</li> Another lesson we should have learned from the recent past is that the development of “richer” or “more powerful” programming languages was a mistake in the sense that these baroque monstrosities, these conglomerations of idiosyncrasies, are really unmanageable, both mechanically and mentally. I see a great future for very systematic and very modest programming languages.</li> The best way to learn to live with our limitations is to know them. By the time that we are sufficiently modest to try factored solutions only, because the other efforts escape our intellectual grip, we shall do our utmost best to avoid all those interfaces impairing our ability to factor the system in a helpful way.</li> We shall do a much better programming job, provided that we approach the task with a full appreciation of its tremendous difficulty, provided that we stick to modest and elegant programming languages, provided that we respect the intrinsic limitations of the human mind and approach the task as Very Humble Programmers.</li> </ul> Dijkstra was 42 years old when we wrote The Humble Programmer. At a similar age of 47 years, I too have come to appreciate the value of simple programming languages and a careful approach to writing software. Two modern languages that embody some of these principles are Go</a> and Elm</a>. While on the surface these languages may seem very different (one is procedural and one is functional), the values of the languages are similar. Both are designed with a great respect for the finite capability of the human brain. Both strive for simplicity first, and features are only added to the language after careful consideration. Both have excellent tooling that helps ensure consistency and correctness. Both are not just a syntax, but rather a culture and mindset. Experience on recent projects indicates that Go and Elm are effective languages, especially when I compare to previous experiences on C++/Javascript projects. There are rarely run-time exceptions. Compiler errors are relatively easy to understand and fix. Tooling is simple and almost invisible. Program maintenance and refactoring is enjoyable. The combined cognitive load of the language and the tooling is small freeing up resources for solving the problem at hand. Today, there is a huge amount of churn in the software development world. Especially in front-end development, new frameworks are being released at an astounding rate. Many have their place, and perhaps each addresses some need. But it seems there is something fundamentally wrong with so many solutions for the same problems. Thus, it is encouraging to see new approaches where the authors have carefully studied the past, tried to learn from everything that has been done, and have created solutions where simplicity is valued. This is key to developer effectiveness, and really to life in general. Why are Go applications so reliable? 2019-10-10T00:00:00+00:00 Go does a lot of things well (good performance, easy to learn, very productive, extensive stdlib, excellent tooling, etc), but after programming with Go for three years (both embedded Linux and cloud applications), stability is the characteristic that really stands out. </figure> In the past, I’ve observed the development of several fairly large C++ applications. Stability problems were more common than they should have been. Memory leaks, null pointers, crashes, etc. On one project (over 500,000 SLOC</a>) we built elaborate mechanisms to capture core dumps when things crashed, at times needed to run Gdb on the target system, and spent a fair amount of time in Valgrind tracking down memory leaks. The worst problems were the rare, intermittent ones which only happened in the field and were very difficult to instrument. It is possible to write reliable C/C++ code (one example is the Linux kernel), but it takes focused effort (code review, testing, coding standards, etc). My experience with Javascript has been similar. What if there was a programming language where basic program stability did not have to be a primary concern? I’ve now participated in four decent-sized Go projects and stability problems are a thing of the past. I only recall one memory leak. Crashes are rare. Several attributes of Go contribute to this: The pattern for returning multiple variables (including error) and aligning the happy path to the left</a> encourages error checking before doing anything with the data a function returns. Therefore you don’t end up trying to do things with nil variables. It is a little verbose, but it works and is a small price to pay for the problems it solves.</li> Garbage collection takes care of memory leaks.</li> The language encourages simplicity, and simple things are generally more reliable.</li> The type system catches many errors during compile time.</li> </ol> I’m sure there are others, but these are the ones that stand out to me. Go strikes a nice balance between safety and simplicity. There are safer languages (Rust, Haskell, etc), but they require more effort to learn, and probably require more coding effort (I don’t have a lot of experience, so I could be wrong about this). Every tool has its use and place. If I was developing an application on a space craft going to Mars, I’d probably use Rust. If I am writing a quick script to process some test data, I typically use Python. When writing MCU code, I still use C/C++, but someday may use Rust. But when I’m writing a reasonably complex cloud or edge (embedded Linux) application, I use Go. In Go, you can get things done quickly, and be reasonably confident that the application you deploy will not crash or leak memory. Go’s type system and simplicity require a little more typing than Python, but likely less effort than more complex languages like Rust or Haskell. Go is a pragmatic language that gets you 98% there without a lot of overhead. In business, you generally don’t want to add a lot of process overhead on the entire company for the 2% of people who may be causing problems, as it imposes drag on the other 98% trying to get things done. Rather, it makes sense to address the 2% at a personal level. Likewise, with Go, the 2% of the time you have to fix a nil bug, or a unhandled case the type system did not catch, is acceptable as 98% of the time you are working in a language that is very productive with very little overhead. Go is ideal for many applications. Auto-formatting/linting Go code 2019-04-22T00:00:00+00:00 Some things in life you just have to experience to truly appreciate the value. One of these is auto-formatting/linting source code. When I started programming in Go in Vim, I naturally looked for editor support, and found the excellent vim-go</a> project. Through this, I learned about gofmt</a>and then goimports</a>. These tools can be configured in your editor to automatically format your code when you save. goimports goes a step beyond and adds missing imports and removes unused ones. Auto-formatting is quickly</a> becoming</a> the</a> norm</a>. The Javascript world also has an excellent formatter available named Prettier</a>. There are formatters for many other languages as well including C/C++</a>, shell</a>, Elm</a>, Rust</a>, etc. The neoformat</a> and ALE</a> plugins add auto-formatting functionality to Vim/Neovim. An example of how to enable these plugins in Neovim is included in my dotfiles</a>. Below are instructions on how to set up Visual Studio Code</a> to auto-format code. Even though I use Neovim</a> for most of my code editing, I still try out VS code and IntelliJ IDEA</a> from time to time to learn what’s new in cutting-edge editors. I also recommend VS Code for most new Go developers – especially those coming from embedded systems development where an IDE is traditionally used. Give it a try – it will only take you a few minutes. The instructions below should work with most systems (Windows, Linux, Mac). First, make sure Go and VS code are installed. Then install the Go VS code extension. </figure> Next, type Ctrl-Shift-P to bring up the command palette and run Go: Install/Update Tools. </figure> This will display a list of tools – select them all. </figure> Next, configure VS Code to format on save. Open the settings by selecting File->Preferences->Settings from the menu. Select the “Format on Save” option and set the timeout to something larger than the default, like 5000ms. </figure> Last, select goimports as the code formatting tool – this will automatically manage the imports in your file, as well as formatting the code. </figure> Now, open a Go source file. Mess up the formatting of some code and then save – the code should be automatically formatted. Similarly, if you remove one of the import statements that is being used, and save, the imports will be corrected. Unused imports will be removed. These tools allow you to focus on writing software and let the tools worry about formatting and imports. Prettier code with less effort – why not? Microcontroller (MCU) or Microprocessor (MPU)? 2019-04-09T00:00:00+00:00 As technology advances, there are two basic processing platforms for implementing embedded systems</a>. The first is the Microcontroller Unit</a> (MCU). These devices have varying amounts of integrated Flash (<= 2MB) and RAM (<= 1MB), and are designed to run bare-metal code or a real-time operating system</a> (RTOS), like FreeRTOS. The second is the Linux-capable Microprocessor Unit</a> (MPU). An example of an MCU based system is most Arduinos, and an example of an MPU based system is the Raspberry PI. An MPU typically does not have embedded Flash and RAM – at least on the same die. The fundamental difference between MCU/RTOS and MPU/Linux systems is the memory architecture and the amount of memory in the system. </figure> There are other differences as well, summarized in the table below. Feature</td> MCU</td> MPU</td></tr> # power supplies</td> 3.3V</td> VCORE, VIO, VDDR, etc.</td></tr> USB Host</td> limited</td> Full support many drivers</td></tr> Networking</td> limited</td> Options for GB and multiple interfaces </td></tr> Cost</td> lower $1-$20</td> higher, starts at $10 (cpu + power/mem) </td></tr> Programming Languages</td> C/C++ Micropython Rust</td> C/C++, Go, Python, Nodejs, Java, Erlang, Rust, about anything</td></tr> Realtime</td> hard</td> soft</td></tr> Data processing</td> limited</td> excellent</td></tr> Processing power</td> less</td> more</td></tr> Display size</td> small</td> any size </td></tr> Startup time</td> fast</td> slow</td></tr> Expandable over time</td> less so</td> more so</td></tr> On-chip Peripherals</td> more </td> less </td></tr> PCB real-estate</td> less</td> more (external flash/RAM)</td></tr></tbody></table> Performance is related to memory architecture. In most MCUs, the memory architecture is fairly simple. Code is executed directly from flash, and on-chip SRAM is accessed directly. Programs do not need to be loaded from flash into RAM before running them. This architecture is very simple, leading to predictable real-time response and code execution. It takes a consistent amount of time to load code from flash into the processor for execution, so the timing for every instruction is fairly predictable. However, with this architecture, clock rate is typically limited to around 190MHz (example STM32F4). Some of the newer STM32 parts, such as the STM32F7 and STM32H7, add a L1 cache, which allows the CPU to run at 216MHz and 400MHz respectively. MPUs have a more complex memory architecture in that they page code from Flash into SDRAM (both external to the MPU), and then from SDRAM into two or more levels of cache memory located on the MPU. The local cache memory is very fast, allowing these processors to run at high clock rates (1GHz or more in some cases). A memory management unit</a> (MMU) implements a virtual memory system in which physical pages from RAM are mapped into a virtual address space. This is very efficient as physical pages are only mapped in as needed and can be discarded if RAM is needed elsewhere. But this memory management introduces delays that are relatively long for some hard real-time systems. The first access to a block of code that has not been run recently takes time, as code needs to be paged from a file in flash into SDRAM and then loaded into the respective caches. But subsequent accesses from cache are blazing fast. An MMU also provides protection against one user space process corrupting memory of another process, or corrupting kernel memory. For complex systems (server/desktop) running multiple processes, this protection increases the system reliability. For many Embedded systems, there is only one main application, and protecting processes from each other is less of a concern. However, this memory protection is still useful in that a bug in the application does not crash the system, and recovery mechanisms can be built in to restart the app, or debug it during development. In an MCU, program flow is controlled through simple loops and state machines (no operating system) or a real-time operating system</a> (RTOS). In an MPU system, a fairly complex operating system like Linux is typically required to manage the multiple levels of memory and storage, schedule the multiple processes that are running, and provide drivers for the complex hardware systems found in an MPU (USB, large displays, networking, etc). There is a great gulf between these two systems. Even though a RTOS may run on an MPU, there are much longer delays in execution due to the memory architecture. You still have to load code from flash to SDRAM and then through several layers of cache memory before the CPU can execute it. Once it is in cache, the code executes much faster and can do more work overall, but the occasional delays are still there. You might be able to play tricks by locking lines of code in cache or by using the small amount of on chip SRAM that may be present, but by the time you do this, you are now back to the smaller memory sizes on an MCU, and may as well just use an MCU. After understanding the differences in memory architectures, we can understand that MCUs and MPUs are optimized for different things. An MCU is optimized for simplicity, cost, and predictable (real-time) response times. An MPU is optimized for getting the maximum amount of work done over time. Cost and real-time response is a secondary concern. There is not really a lot of middle ground between an MCU and MPU, evidenced by the large gap in memory sizes. You can’t fit standard Linux in an MCU, as there is only so much NOR Flash and SRAM that will fit on an MCU die. Once you move to more dense memory technologies such as NAND Flash and SDRAM, these must be implemented as separate dies and typically separate integrated circuit packages. These technologies are so efficient at storing large amounts of data, that you see a huge jump in memory capacity once you switch to the off-chip memories. MCUs and MPUs also have different origins. Today’s 32-bit MCUs have descended from simpler 8-bit MCUs and are scaled-up embedded controller technologies. MPUs have descended from desktop and server computer systems and are scaled-down technologies. The differing priorities of these two paradigms reflect their different uses. Keeping these differences in mind, the most fundamental question to ask in selecting an MCU or an MPU is whether hard real-time performance and reliability is most critical (control-centric application), or whether data processing performance and connectivity is most important (data-centric application). If both of these are important in your application, you should consider having both an MCU and MPU in the system, saving yourself untold pain. Reliability is an interesting topic. I have seen MPU/Linux systems operate very reliably, and have also experienced issues that were very difficult to solve. MPU systems are many times more complex and have more unknowns, which translates into more risk. There are many more physical components and solder connections required to implement the system, all of which can fail over time with environmental stress. There are many millions of lines of code in an MPU system that you did not write, but you are still responsible for all of it – it all has to work for the system to function. Although an MPU system can be made fairly reliable, simple statistics tell us that MCUs will generally be more reliable than an MPU because there is less hardware that can fail and less lines of code running that may contain bugs. One example of a problem we experienced when implementing control in an MPU system is a product where we were collecting data, and the data was being transferred back to a cloud server over a cellular modem. Unfortunately, the cellular modem in the system was not 100% reliable, and the only way to recover in some instances was to reboot the system when we detected a network failure, which cycled power to the modem (this modem module itself ran Linux). At one point, we added some basic control functionality to the system to control plant blowers based on a schedule. The control worked fine until the system had to restart due to modem connectivity issues, then the blower control was inactive during the reboot cycle (perhaps 30s or so). This was not a fatal problem in this application but also not ideal. The obvious solution is to get a more reliable modem or develop a better recovery method, but with units in the field and other development priorities, it is not always so simple. This is a classic example of the tension between complex data/connectivity systems and reliable control. In an MCU system, the hardware system is relatively simple, and you are using a relatively small RTOS, or none at all. You write a greater percentage of the code in the system yourself, so if you have a simple task to do, and write reliable code, there is a potential to have a more reliable system than a comparable MPU system. However, if you are trying to do complex data processing and connectivity tasks on an MCU (such as writing your own database, network, or USB stack), chances are there will be bugs in your code and you will have a less reliable system than if you would have chosen an MPU using proven technologies. If your application is large and complex, then writing it in a safer language like Go or Rust on an MPU may provide a more reliable and maintainable product than trying to implement the same functionality in C++. Again, if you need both reliable real-time control and advanced data processing, then put both an MPU and MCU in the system. Another advantage of an MPU system is they tend to be more general purpose and more functionality can be added over time. There is little danger of running out of code space, which often happens in MCU environments. With interfaces like USB, additional peripherals can be added in the future as requirements change, and with Linux, drivers for a vast array of hardware are available. In some cases these lines between MCUs and MPUs are blurred by technologies like Linux RT extensions, or uCLinux, but you still must keep the above principles in mind when selecting the building blocks for your product. MCUs and MPUs are very different devices designed to handle different tasks. Before choosing, you should understand what you are trying to do, and what is important for your product. Reference: Atmel MCU vs MPU paper</a></li> ST page referencing both MCU and MPU</a></li> </ul> Accepting Constraints in Build Systems 2019-04-01T00:00:00+00:00 As Embedded Systems become more complex, the complexity of the process to build the software for these systems also increases. As humans, our ability to deal with complexity is limited, so we develop tools and processes to manage the complexity. In the end, these tools and processes are about constraints and patterns. A well-designed tool or process encourages you to do things in a way that is consistent and maintainable, which leads to reliable and predictable results. </figure> As an example, consider using CMake vs standard Makefiles for a C++ application build system. This is a trade-off I often encounter, as my customers are using Linux to build products, and they need to develop a relatively complex software application to implement the product functionality. There are usually a handful of library dependencies as well (such as Qt, OpenCV, etc) – after all, the reason you use Linux is it provides a large amount of advanced functionality that you cannot afford to write yourself. For a simple application, CMake is somewhat constraining and annoying. Its syntax is a little clumsy, and it is definitely less flexible than a Makefile. Thus some developers conclude that a Makefile is a better choice for Linux application builds. As the application grows, the Makefile also grows, and the number of dependencies increases. Then comes the time to cross compile the application so it will run on the target system. With a little work, we can convince the Makefile to accomplish this. Tool-chain file names and paths are often hard-coded in the Makefile. As the variations of systems the application needs to build on increase, all these variations are coded in the Makefile. Five years down the road, something major changes. The application developers need some new features in a new version of OpenCV, or the System on Module (SOM) the target system is using just went obsolete. The developer who wrote the original Makefiles is gone. The Linux host distributions have changed. At this point the application build system collapses. So much has changed, and so much is hard coded in the Makefiles that getting them working again is a painful and tedious process. With some effort, they are finally building on one developer’s machine. But on the next developer’s machine, they fail with a different error. The same dynamic happens when developers hack together tools to build an operating system image for an embedded device. This typically involves things like building the application or Linux kernel manually outside the build system, then writing a script to unpack an image file, insert the new binary, then pack the image file back up. The application build process is heavily dependent on the configuration of the developer’s workstation. There is a better way. Use tools like Yocto/OpenEmbedded and CMake as they are were meant to be used. These tools do have a learning curve, and require some investment to learn, but in the end they are well worth it. The reason tools like OpenEmbedded and CMake work is that they are consistent, and use the same patterns over and over. We can illustrate the basic architecture of OpenEmbedded in a simple sketch, as shown below. OpenEmbedded is made up of a relatively small number of primitives:layers, recipes, packages, classes, conf files, and tasks. </figure> Dependencies can be specified at the layer, recipe, package, and task level. Once these basic concepts are understood, then it is easy to find things in the build system and make necessary modifications. The boundaries are precise and well-defined. These patterns are not as flexible as a Makefile or Bash script, but they abstract huge amounts of complexity. This allows our mind to see the big picture and not get bogged down in thousands of details and to reliably execute extremely complex tasks. The same patterns are used over and over and are easy to recognize. Dependencies are well-controlled. The cost is to understand and use the established patterns and to accept the constraints. These constraints may be a little inconvenient at times. But when you compare this to the work these tools do for you, these inconveniences are relatively minor. If you are going to build complex products using complex technologies like Linux, then you need powerful tools to manage this complexity. There is a learning curve. You may need to invest in training or hire someone to help you get started. Most other industries recognize this. Car mechanics undergo continuous training to learn how to work on new vehicles/systems. Nurses and other professionals receive training on new techniques and equipment. Build systems also require a significant investment in learning. We routinely hire experts (doctors, dentists, plumbers, carpenters, etc) in our personal lives to do things we don’t know how or have time to do. But somehow many developers think the build system and associated infrastructure should not need attention, can be ignored, or is trivial and can be implemented from scratch. Never mind the fact that the underlying software in the system (bootloader, kernel, and libraries) are many times more complex than the custom application software written for the system. You must be able to maintain it all. You can’t ignore the stuff you did not write – it all has to work. It has to be reliable and predictable. A build system is largely invisible until it starts causing problems. Getting something working initially is relatively easy; keeping it working for years is another matter. It is similar to product development – building the first prototype that somewhat works often goes fairly quick. Getting something into production is a long, arduous slog. The build system is the foundation of your software project. It must be reliable and predictable because you can’t do much without it. A build system must do its part well without a lot of fuss, so we can focus on the problem being solved. A well implemented build system produces an image ready for installation with a single command and no manual steps. You check source code into git, run the build command, and out pops an image ready to deploy. There is no copying around binaries, etc. The efficiency gained by the effort to implement such a system pays itself back many times over the life of a project – especially those that are produced for 5-10 years. Yocto and CMake are only used as examples in this article and may not be the best solution for every project. If you have something that is working well, keep doing what you are doing. But if what you are doing is not working, then ask yourself some questions: Is my current system the best way to build complex software?</li> Is the process to build your project’s software simple and easy to reproduce by any developer on any machine?</li> Is it well documented?</li> Is it maintainable?</li> Is it consistent across all your products?</li> Is it reliable and predictable?</li> </ul> SD Card Write Speed Tests 2019-03-13T00:00:00+00:00 </figure> During Embedded Linux development, we often need to write large operating system images to SD cards for testing. Recently, I purchased a USB 3.0 SD card reader</a> from Plugable. Before that, I used a USB 2.0 SD card reader from IOGear. </figure> How much better does a USB 3.0 card reader perform compared to a 2.0 card reader? To test, I wrote a large (2.2GB) image to a Transcend v30 card using both readers. IO Gear USB 2.0 Reader</td> Plugable USB 3.0 Reader</td></tr> 15.3 MB/s</td> 33.5 MB/s</td></tr></tbody></table> The USB 3.0 reader is over twice as fast with this same card! I then used the USB 3.0 reader to compare a number of cards. SD Card Brand</td> Size</td> Grade</td> Tested Speed</td></tr> Transcend</td> 8GB</td> C10 HC</td> 8.8 MB/s</td></tr> Sandisk</td> 32GB</td> C10 HC I</td> 16.8 MB/s</td></tr> Transcend</td> 16GB</td> C10 HC I V30</td> 33.5 MB/s</td></tr> Sandisk</td> 64GB</td> XC I V30 A1</td> 44.5 MB/s</td></tr></tbody></table> The markings on the cards mean various things. The Wikipedia SD Card page</a> provides a nice overview. A few details are summarized below: Bus interface I: 50MB/s Half/Full duplex, 104MB/s Half Duplex</li> </ul> </li> Capacity HC: Secure Digital High Capacity (SDHC) format – supports cards 2GB to 32GB in size</li> XC: Secure Digital eXtended Capacity (SDXC) format – supports cards 32GB to 2TB in size</li> </ul> </li> Speed 10 with C around it: Minimal sequential writing speed is 10 MB/s</li> 3 with U around it: UHS Speed Class: 30MB/s</li> V30: Video Speed Class: 30MB/s</li> A1: Application Performance Class 1: Read 1500 IOPS, Write 500 IOPS</li> </ul> </li> </ul> If you have not upgraded to a USB 3.0 card reader, it is probably worth your time to do so. One note of caution – there are counterfeit cards for sale that look like SanDisk or Transcend cards. They are not manufactured by these companies, and use lower quality components. If the price seems too good to be true, it probably is. Always buy from a reputable supplier. Getting started with Embedded Linux 2018-10-04T00:00:00+00:00 </figure> Recently I was asked by a developer, who has done windows development for 10 years, how to get started with Embedded Linux. Embedded Linux covers a lot of ground and includes a broad range of components/skills to put together an entire system. Below are a few suggestions. Do I have the right Mindset?</h3> There are many different types of people; there are those who are who are a good fit for Linux, and those who are not. There are those who think Open Source Software (OSS) and Linux are the craziest thing they ever heard of, and those who think it’s a great idea. There are those who thrive on openness and collaboration, and those who feel threatened by it. A few specific aspects of this mindset are described below. Code Gardening</h4> Much of your time as an embedded Linux developer is spent figuring stuff out (vs writing new code). Figuring out stuff is harder than writing new code (at least initially). It is similar to weeding the garden – often not something you are raring to do, but after you are finished, it feels really good. Much of the truly valuable work in life is like this. It initially appears kind of boring, but once you get started there are many interesting and rewarding aspects. You have to like build systems and automation. Linux systems are complex. You can’t throw millions of lines of kernel and user space code together with a simple Makefile or a fancy IDE. Proficiency with command line tools is required. Builds must be clean and completely automated if you want to keep your sanity. Once you allow one manual step in the build, the floodgate has been opened, and these manual steps will start to accumulate at a frightening pace until you have an unmaintainable mess. Collaboration</h4> Because of the complexity, you have to be able to collaborate with others. This requires a generous mindset. How can I make this easy for someone else to pick up once I am gone? Most likely this someone will be you in a couple months or years when you have forgotten many of the project details. The same really applies to coding. Do I write my code to be clever, do I hack it together, or do I focus on making it maintainable and easily understand by others? Automation and simplicity reduce cognitive load, freeing up resources to think about the problems that actually need solved, and allow multiple people to work on a project without a lot of friction. The flip side of this is humility. With a complex system like Linux, there will be a lot of things you don’t know, and some difficult problems to solve. There is no one “manual” for Linux where you can go look up an answer. So we rely on open source communities for help. There are many people willing to help, so the key is to have the wisdom to know where and when to ask for help, and then the humility to ask. In Linux communities (and OSS in general), you will be surrounded by developers who are much smarter and more knowledgeable than you. At the very least, there will always be someone who knows something you don’t. If you don’t have the humility to acknowledge this and be willing to learn, then you will likely never be comfortable or effective working in an OSS environment. NiH</h4> This can all be a challenge for developers who are endowed with a healthy dose of NiH (Not Invented Here Syndrome). These are people who imagine themselves happiest writing and maintaining their own code, and who are not inclined to share it with others. Some imagine themselves as great software architects and coders (very few of us really are) who can make something better. They don’t want to spend long, arduous hours learning and debugging a complex system written by someone else. If this describes you, then perhaps Embedded Linux is not for you. Learning</h4> Are you a learner? Do you enjoy learning about new technology, tools, techniques, etc? Everything is always changing in the Linux world. New and compelling solutions are coming out every year. The real power of Linux and Open Source is knowing when and how to leverage these new technologies to solve real problems. If you are more comfortable doing things the way they have always been done, then perhaps you should just keep doing what you are doing. The Problem of Abundance</h4> Can you deal with a lot of choices? With Embedded Linux, there are many options for about every aspect of the system. There are multiple init systems. There are many programming languages. There are many libraries available for various tasks. There is no one, true way to implement Embedded Linux – you will have to make a lot of choices. Ideas for Learning Embedded Linux</h3> Books/Presentations/Documentation/Blogs</h4> There are many books published on Linux, Embedded Linux, Yocto, etc. The Yocto documentation</a> is pretty good. Videos from the Embedded Linux Conference</a> are freely available. There are many blogs that cover Embedded Linux topics (such as this one). Subscribing to blog RSS feeds in feedly</a> is a good way to keep up with recent developments. The list of learning resources is endless, but perhaps the key thing to remember is there is no one place you can go to learn everything you need to know, like you might for a commercial RTOS. The good news is there is information available in many different formats, so you can choose what best fits your learning style. Run Linux on your computers</h4> One of the best ways to learn a lot about Linux is to simply use it for your personal and work computers. Ubuntu is very easy to get started with. If you go with something like Arch Linux, the install process will teach you a lot. I am very pleased with the combination of Arch Linux plus the XFCE desktop environment. This is a no-nonsense setup that does its job well, is very customizable, and stays out of my way. Any distribution will work – just pick something. Embedded Linux aside, there is little to lose, and a lot to gain from running Linux. Linux systems are naturally much more immune to viruses, malware, and various other attacks than Windows. Linux systems are much more flexible than Apple computers. Some things might not look as pretty, or might take a little more work to set up. But the stability, flexibility, and consistency of a Linux system will save you time in the long run. As a software developer, most of our time is spent in an editor and browser – these work very well in Linux. Additionally, the terminal is a first class citizen in Linux rather than an afterthought. Learn Git</h4> Git is used in almost all open source projects these days, and enables new levels of collaboration. You need to know Git</a>. Do small projects with Raspberry PI, etc.</h4> There are a number of embedded computer boards out there that are very well supported and quite cheap such as the Raspberry PI. Find something in your home that you want to automate – there’s nothing like a real project to motivate learning. With Linux, you learn best by doing. This will also give you an opportunity to learn application programming under Linux. Build and install OpenEmbedded/Yocto or Buildroot</h4> Take the previously discussed home automation project and create a custom build for the board using Yocto/OE</a> or Buildroot</a>. This will give you practice learning how to put together a custom embedded Linux distribution. Create recipes to package, and include your custom applications in the image at build time (instead of installing them afterwards). These tools have a learning curve and may seem intimidating initially, but Linux is a complex system, and you need powerful tools. We think the BEC OE template</a> (now yoe-distro</a>) is a good way to set up Yocto/OE for a project. Modify the Linux Kernel</h4> Most embedded Linux projects require a small amount of kernel modification. This can be something as simple as changing the GPIO configuration on an ARM SOC (System on chip), or writing a driver for some custom hardware. There are many resources available for learning Linux kernel programming. Perhaps try creating a custom driver for your PC or embedded Linux board. This driver could do something trivial, like monitor several GPIOs, etc. Modify and install U-boot</h4> U-boot is a common bootloader for Embedded Linux systems (but not the only one). Locate the source code for your embedded board and learn how to build, modify, and install a custom version of U-boot. Explore the many Application Programming Options</h4> There are so many interesting application programming options with Linux. Of course you can stick with traditional C/C++. But today, there are many other languages and environments that may be better for various applications (Python, NodeJS, Golang, Qt/QML, etc). Find something that is broken and fix it</h4> There are many opportunities to improve open source code. Find something that does not work quite right, fix it, and get the changes accepted in the upstream project. This will provide valuable experience working with OSS communities. When working with complex systems like Linux, you will run into problems that are very hard to solve. Being comfortable with asking, and getting help from various OSS communities will be key to your success. Again, your mindset is key here. People who don’t have the right mindset will create fake names and email aliases to hide their true identity when asking questions. People with the right mindset are willing to be up front and admit there is something they don’t know. Helping Others</h4> One of the best ways to really learn something is to teach it or help others. Once you reach a certain level of proficiency, there are many opportunities in open source communities (stack overflow, mail lists, IRC, etc) to help others. In doing so, we usually receive back much more than we give. Maintain Yocto or Buildroot support for an embedded board</h4> There are many embedded Linux platforms out there that are not actively maintained or tested in Yocto or Buildroot. Get involved and help keep the platform current and working in the build system. Set up a server in the cloud</h4> It is very cheap (less than $10/mo) to set up a server in Amazon AWS, Digital Ocean, Linode, Vultr, or one of the many other cloud providers. Although this is technically not embedded Linux, the skills required are very similar. It will also open up interesting possibilities of creating systems that are a combination of Embedded Linux devices and cloud servers. Publish an Open Source project on Github</h4> Publishing an open source project is a good experience as it helps you learn to create something that is generally useful. For example, I wrote a little app</a> to test serial ports and published it on Github. Over the past few years, other people have provided nice contributions to the application, making it much more useful, with little effort on my part. However, I had to spend a little more time up front making the app generally useful and documenting how to use it. Summary</h3> Embedded Linux brings a level of functionality to systems that is unmatched by anything else out there, thus its popularity. With this broad scope of functionality comes complexity. Linux has a learning curve, and using it successfully requires some amount of discipline and interaction with various OSS communities. The resources are endless – time and motivation are the only limitations. In life, the rewards are proportional to the effort put forth. If you are committed, working with Embedded Linux is very rewarding, and brings a new level of functionality to products. Git, Versioning, and Branching for Embedded Linux Development 2018-02-14T00:00:00+00:00 When building a product using Linux, versioning and branching of your software is an important consideration. Everyone’s needs are different depending on the size of the team, culture, and testing requirements, so there is no one size that fits all. However, after working on a number of different projects for a dozen or so different companies, there are several practices that are often used. The most fundamental is to use Git to manage your software development. It is surprising the number of small companies developing software who do not use Git, or any version control system yet. The many benefits of Git stand alone</a>, but the minute you enter into the Linux world (or the Open Source Software (OSS) world in general), there are thousands of projects available, and 99.9% of them are developed in Git. Having a good understanding of Git is essential in using OSS. This is not really specific to Linux development, but rather any modern software development (cloud, web, mobile, etc). Most foundational software technologies today are open source. There is no other development methodology that scales to the complexity of modern software systems, and no one company contains the resources needed to create and maintain these systems. Git does have a steep learning curve and usually requires 2-3 weeks of committed use for new users to become comfortable with it. However, it is well worth it. An extension of using Git is to consider hosting your Git repositories in the cloud. There are many companies who provide this service at a reasonable cost (some like Gitlab</a> are free), or it is relatively simple to host your own git server (can be done for $2.50/mo at Vultr</a> using Gitea</a>). Some may balk at putting a company’s valuable IP (intellectual property) assets in the cloud (where hackers might have an easier time getting to them) vs behind the secure company firewall. However, which of the following is the main impediment to your business? The danger of competitors hacking into your cloud server, stealing your IP, and then implementing a product with it.</li> Shipping products in a timely fashion, and then maintaining them.</li> </ol> In my observation, #2 seems to be the challenge for most – especially as systems become more complex. Unless you are a top secret government agency, or a cutting edge chip/algorithm company, your software IP is probably of marginal use to anyone else. The reason for this is that integration and execution is most often the challenge, not coding. There is plenty of free/OSS software out there to do about anything. Integrating this into something that will make a product is the hard part. It is difficult getting software and hardware to work together, debugging problems, communicating with cloud infrastructure, software updates, maintaining a product, adding features, etc. Modern products are typically not static entities where you design it, toss it over the wall to production, produce millions, make lots of money, and never see it again. Rather modern products are complex, dynamic beasts that have continually changing requirements, bugs to fix, features needed for new customers, manufacturing issues to solve, obsolete parts to replace, and are part of a larger complex system. Execution is the hard</a> and valuable</a> part. Increasing developer efficiency and collaboration is important, especially with a distributed team of developers. You can require your developers to use company issued Windows computers and fussy VPNs in an attempt to protect your IP, or you can simply host your Git repositories in the cloud secured by industry proven methods such as https and ssh. As systems become more complex, it is critical to be able to involve domain experts in the project (consultants, contractors, employees at a different office, manufacturing, etc). A Git server in the cloud is the foundation of distributed development collaboration. Emailing zip files may be temping as quick and easy, but it is not a sustainable way to develop products that have so many moving parts. As an extension of this, use a web based Git repository manager (like Gitea, Gitlab, or a hosted solution) to implement your Git server. These repository managers greatly simplify the management of Git repos (teams, users, permissions, creating repositories, issue tracking, etc). Software should be released with a version number. Semantic Versioning</a> is a good place to start. Your needs may vary, but you may as well start with the good ideas of others. Additionally, a changelog</a> should be maintained that lists all changes for each software release. Being able to quickly understand what changed with each release is critical as multiple versions are being used. Keeping a changelog requires a little discipline, but is fairly easy to do if you keep an unreleased section at the top of your changelog file to track changes for the upcoming release, as described in the above link. A discussion on branching only really makes sense if you are using Git. Source code branches greatly improve your development process flow. The ease with which Git allows you to create and merge source code branches enables development flows which were impractical before Git. With an Embedded Linux project you are typically building images that get loaded on some type of hardware device. In this case, it’s useful to have development and production releases. Production releases are sent to customers, and development releases are used internally to test new features under development. Every software component that is under active development (OE build system, custom applications, kernel, etc) should likewise have production and development branches that feed into their respective builds. For a more extensive discussion of this topic, consider this post</a>. Again, start with the good ideas of others, and then use what makes sense. An OE build can be configured to pull from the latest HEAD of your custom components, so that for each new build you get the latest from the respective branches for various components. To do this, you typically have something like the following in a bitbake recipe: SRC_URI = "git://git.mycompany.com/myproject.git;branch=develop;protocol=ssh;user=git PV = "4.1.15+git${SRCPV}" SRCREV = "${AUTOREV}" </code></pre> In some cases, for production builds we’ll manually specify the Git hash or tag in the bitbake recipe so that versions of all components are explicitly locked down. To specify a git version, replace ${AUTOREV} with a Git hash. Either way, if you include the Git version (SRCPV) in the package version (PV) variable, you can always figure out what Git version was used to build a package for a particular release. Below is an example of a package file name: my-app_4.1.15+git0+a05d9b23b9-r0.10_var_som_mx6.ipk </code></pre> In this case, a05d9b23b9 tells you what Git version was used to build my-app. This is often adequate if you don’t spend a lot of time building or patching old software releases. With each release of software, it is beneficial to tag the software components used in the build with a version. This allows a clear view of what changed between releases, and allows us to easily check out old versions of software. With development and production branches, we should consider how to version the different branches. One simple way is to use a sequence like: 1.0.0 (production)</li> 1.0.900 (develop)</li> 1.0.901 (develop)</li> 1.0.1 (production bug fix)</li> 1.0.902 (develop)</li> merge develop to production and do a new release</li> 1.1.0 (production)</li> 1.1.900 (continue development</li> 1.1.901 (develop)</li> </ul> In this case, the development version is always the last production release MAJOR.MINOR</a> and the PATCH version starts at 900 and is incremented with each development release. If a version number ends in 9xx, you know it’s a development build. You can use more complex version schemes like 1.1.0-alpha1 (instead of 1.0.901), but then you might run into sorting problems where 1.1.0-alpha1 is listed after 1.1.0, even though 1.1.0 is a later release than 1.1.0-alpha1. Recent versions</a> of the BEC software updater</a> look at all update files on a USB disk and pick the latest one using a natural sort of version numbers within the text. In the example below, you can see how versions are sorted: [cbrake@mars ver]$ ls -lv total 0 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.0 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.1 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.900 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.901 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.1.0 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.1.0-alpha1 </code></pre> In this case, 1.0.9xx works much better for development builds if you want update systems to automatically pick the latest release from a list of options. You can also use versions like 1.0.0+1.1.0-alpha1 to get around the sorting problem if you want something a little more explicit. Versioning and branching does not have to be complex, and in many cases the above is adequate for small teams. Your needs may vary, but the important thing is to have a source control, versioning, and branching process in place. These practices require some discipline to implement, and may seem unnecessary to those who have not used them, but with a little experience we soon realize that efforts to introduce appropriate amounts of organization and process into our development efforts clearly communicates what we did, what we are doing, and what gets released to production. This then frees us up to focus on what really adds value. Separation of Source and Build Directories 2017-11-16T00:00:00+00:00 As we work with larger and more complex systems (i.e. Linux), more and more of our time is spent on integration and pulling different pieces together. We often need to debug or understand code we did not write – especially in build systems. To work effectively in this scenario you must be able to quickly search through a lot of source code. Therefore, we are always looking for ways to make this more efficient. One pattern that is very useful is separating source and build files into two separate directories at the top level of the project. There are several reasons for this: It is easy to grep the source (or today we ripgrep</a>) if it is not littered with transient build data.</li> Its easy to reset the build by simply deleting the build directory.</li> </ul> This is especially important for large builds like OpenEmbedded, where a build directory can be on the order of 40GiB. If your grep tool has to process 40GiB of data every time it runs, it will be very slow. This separation should occur at the top level of the project. Ideally you want something like this for an OpenEmbedded build: ├── build/ │ ├── sstate-cache/ │ └── tmp/ ├── downloads/ ├── sources/ │ ├── bitbake/ │ ├── meta-bec/ │ ├── meta-browser/ │ ├── meta-freescale/ │ ├── meta-freescale-3rdparty/ │ ├── meta-freescale-distro/ │ ├── meta-openembedded/ │ ├── meta-qt5/ │ ├── meta-variscite-fslc/ │ ├── meta-myproject/ │ ├── meta-mycompany/ │ └── openembedded-core/ ├── conf/ └── etc ... </code></pre> The sources directory contains OpenEmbedded metadata (core, 3rd party, and your own custom meta-myproject layers). The downloads directory might contain 3rd party sources or tar balls that are downloaded by the build tool. And the build directory contains transient build data that can be deleted and re-created at any time. This configuration is used in the Yoe distribution</a>. It should be noted that in this example, the desire is to cleanly separate OE metadata (source) from build data. Source code for individual packages is considered build data from an OE perspective. Most of the development work in this case is creating/customizing OE recipes, images, etc, thus the need is to quickly grep the various OE meta layers. This idea can be contrasted with the Poky reference distribution directory layout: ├── bitbake ├── build │ ├── cache │ ├── conf │ ├── downloads │ ├── sstate-cache │ └── tmp ├── documentation ├── meta ├── meta-poky ├── meta-selftest ├── meta-skeleton ├── meta-yocto-bsp └── scripts </code></pre> In this case, there are multiple meta directories in the top level. If you want to search all of the OE meta data, you need to instruct your search tool to exclude the build directory. Additionally, the build directory is dynamically generated, which includes the conf directory. If you want to completely reset your build, you need to clean several directories in the build directory. For a production build, you need some of the conf files to be in revision control; therefore, they are more difficult to manage when they are dynamically generated, or copied from another location. Why not move conf to the top and use it there? Most decisions are tradeoffs, so there may be good reasons for the Poky directory structure. For a reference (demo) distro, simplicity and ease of setup may be valued over reproducibility. For production builds (builds used in products), rock solid reproducability is important. To get that, you need: all source and configuration is locked down in version control</li> a build should be a minimal number of steps: checkout source, a very simple setup/configuration step, and then a build command. There should be no complex scripts that copy files around that may be modified later, or manual steps.</li> the output is ready to program into the target. There are no manual post processing steps.</li> keep things as simple as possible.</li> </ul> The separation of source and build can also be used/observed with many other build systems. With CMake, if you have a project directory named myproject, you can do something like: mkdir myproject-build cd myproject-build cmake ../myproject make </code></pre> Qt Creator also sets up a separate build directory by default. With the linux kernel, you can specify a separate build directory with the ‘O’ option: cd linux mkdir ../linux-build make O=../linux-build </code></pre> Think about directory structure – it can make a difference. Understanding the NXP i.MX6UL Pin Mux (Part 2) 2017-11-06T00:00:00+00:00 In the previous post</a>, it was noted that bit 30 needs to be set in the i.MX6UL pad config if you want to read the state of a GPIO output. Digging into this a bit more, we find the following text in the Documentation/devicetree/bindings/pinctrl/fsl,imx-pinctrl.txt file: SION(1 << 30): Software Input On Field. Force the selected mux mode input path no matter of MUX_MODE functionality. By default the input path is determined by functionality of the selected mux mode (regular). In the i.MX6UL developers manual, we also find the following: When we look at the SW_MUX_CTL register, we notice the SION bit is located at the bit 4 position. So the question is: How does bit 30 in the device tree file get mapped to bit 4 in the SW_MUX_CTL register? In the Linux kernel source we find the following: drivers/pinctrl/freescale/pinctrl-imx.c: /* The bits in CONFIG cell defined in binding doc/ #define IMX_NO_PAD_CTL 0x80000000 / no pin config need / #define IMX_PAD_SION 0x40000000 / set SION / ... / SION bit is in mux register / config = be32_to_cpu(list++); if (config & IMX_PAD_SION) pin->mux_mode |= IOMUXC_CONFIG_SION; pin->config = config & ~IMX_PAD_SION; </code></pre> drivers/pinctrl/freescale/pinctrl-imx.h #define IOMUXC_CONFIG_SION (0x1 << 4) </code></pre> From this code, we can see that bit 30 is a virtual bit that gets translated into bit 4 in the mux mode register. Hope this helps in understanding how to configure the i.MX6UL gpio. Once you understand the details; the process is very simple. Understanding the NXP i.MX6UL Pin Mux 2017-09-14T00:00:00+00:00 (note, the article is also applicable to the i.MX6ULL as these processors are very similar) The NXP i.MX6UL application processor has a very flexible pin multiplexer, that is somewhat difficult to understand at first glance. Most times when we’re configuring the pin mux in Linux, we modify Device Tree files, so perhaps that is the place to start. The pin mux options for the i.MX6UL are defined in the arch/arm/boot/dts/imx6ul-pinfunc.h</a> file. The arguments to the macros in this file are defined as: /* * The pin function ID is a tuple of * <mux_reg conf_reg input_reg mux_mode input_val> / </code></pre> The first 5 are defined in the pinfunc.h file, where the config value is defined in a user’s DTS file. As an example, consider the following section from imx6ul-pinfunc.h. Column headings are added for clarity. // headings: mux_ conf_ input_ mux_ input // reg reg reg val _val #define MX6UL_PAD_GPIO1_IO09__PWM2_OUT 0x0080 0x030C 0x0000 0 0 #define MX6UL_PAD_GPIO1_IO09__WDOG1_WDOG_ANY 0x0080 0x030C 0x0000 1 0 #define MX6UL_PAD_GPIO1_IO09__SPDIF_IN 0x0080 0x030C 0x0618 2 0 #define MX6UL_PAD_GPIO1_IO09__CSI_HSYNC 0x0080 0x030C 0x0524 3 1 #define MX6UL_PAD_GPIO1_IO09__USDHC2_RESET_B 0x0080 0x030C 0x0000 4 0 #define MX6UL_PAD_GPIO1_IO09__GPIO1_IO09 0x0080 0x030C 0x0000 5 0 #define MX6UL_PAD_GPIO1_IO09__USDHC1_RESET_B 0x0080 0x030C 0x0000 6 0 #define MX6UL_PAD_GPIO1_IO09__UART5_DCE_CTS 0x0080 0x030C 0x0000 8 0 #define MX6UL_PAD_GPIO1_IO09__UART5_DTE_RTS 0x0080 0x030C 0x0640 8 2 </code></pre> From the i.MX6UL processor reference manual (IOMUX Controller chapter), we can determine the mux_reg is indeed located at offset 0x80. We can also see that the mux_val values match MUX_MODE in the datasheet. </a> The mux_val in the imx6ul-pinfunc.h file also match the MUX_MODE field values in the above register. The config register can also be located in the datasheet. Again, we notice that the offset of 0x30C matches config_reg setting. </a> Lets consider the last option listed in the pinfunc file: UART5_DTE_RTS. The input register is used to mux a pin to an input function inside the SOC – in this case, the UART5_RTS_B input. We find the input register at offset 0x640 and can see that the input value of 0x2 matches what we would expect in the reference manual. </a> So, how do we know if the pinmux is configured correctly for a pin? One way is to use devmem2 and dump the register contents directly from user space: root@imx6ul-var-dart:/sys/class/gpio# devmem2 0x20e0080 /dev/mem opened. Memory mapped at address 0x76f02000. Read at address 0x020E0080 (0x76f02080): 0x00000005 </code></pre> The mux value is 0x5, which configures the pin for the default GPIO1_9 function. Next we can look at the config register: root@imx6ul-var-dart:/sys/class/gpio# devmem2 0x20e030C /dev/mem opened. Memory mapped at address 0x76f0b000. Read at address 0x020E030C (0x76f0b30c): 0x000010B0 </code></pre> If we further decode the bits in this register we find: Bit index: 12─╮ 8─╮ 4─╮ 0─╮ Bits: 1 0000 1011 0000 </code></pre> and can conclude that the following options are set: bits 3-5 = 6: DSE_43ohm</li> bits 6-7 = 2: Medium speed (100MHz)</li> bit 12: PKE: Pull/Keeper Enabled</li> </ul> There are additional options that can be configured such as hysteresis, pull up/down, open drain, and slew rate. I/O is very configurable on this device! If we want to use this pin as a GPIO, it appears that everything is configured correctly, so let’s give it a try. GPIO in the i.MX6UL is banked in 32 bit registers, so we need to map this to a sequential GPIO in Linux. The formula for this mapping is: linux gpio number = (gpio_bank - 1) 32 + gpio_bit For GPIO1_9, this would be GPIO9. linux gpio number = (1 - 1) * 32 + 9 = 9 To read the GPIO, execute the following commands: root@imx6ul-var-dart:~# cd /sys/class/gpio/ root@imx6ul-var-dart:~# echo 9 > export root@imx6ul-var-dart:~# cat gpio9/value 0 (current gpio value) </code></pre> Another nuance of configuring the i.MX6UL pin mux in Linux is that for GPIO outputs, you need to set a magical bit 30 in the Device Tree configuration as show below. If you don’t set bit 30, it will always read back as 0. MX6UL_PAD_JTAG_TDO__GPIO1_IO12 0x4001b0b0 /* 2_nSLEEP / </code></pre> The reason for this will be explored in a future post. Why Git? 2016-08-18T00:00:00+00:00 Some time back, I gave a presentation</a> that included an overview of the Git version control system. I still occasionally get asked why Git should be used instead of Subversion, as it seems harder at first. Most developers don’t really understand Git until they have used it for awhile, and then they will have an “aha moment.” There are 3 features of Git that are especially interesting to me: many repositories (vs. one large repository)</li> distributed development</li> cheap branches</li> </ol> The fundamental driver for better tools is increasing system complexity. Developers are required to manage and integrate more third party software, work in distributed teams, and more effectively re-use the software we do have. Git is a tool that helps you accomplish these goals. Git is a fairly new project, but it is already nearly universal among open source software projects. Its uptake and acceptance speaks for itself. Many Repositories</h2> In the past when most companies used Subversion, there was one huge company repository (I’ve seen them in the 10’s of GB in size) that held a hierarchical tree of all the company source code. This worked fairly well, and is comfortable in that it is very similar to a file system directory structure. However, this model is not very flexible in that it does not have a consistent way to re-use components between different projects. Some people simple copy source code. Some have a “common” project that is included in all their other projects. Subversion externals can be used. But with Git, typically a separate repository is created for each software component. There are perhaps several reasons for this, but one reason is that Git simply does not scale to huge multi-GByte repositories. However, this turns out to be a blessing in disguise as I think it is a better model in many cases. What we end up with is more of a catalog of software components rather than a rigid hierarchy. There is much emphasis these days made on modular, re-usable software components (Object Oriented Programming, plugins, etc.). Why not keep things modular at the repository level? Another example of this type of organization is the Internet itself. It is not a hierarchy of information, but rather a flat system that is organized by hyperlinks. One of the benefits of organizing your source code this way is that it encourages clean boundaries between software components. Each software component needs to stand on its own fairly well without being propped up by header files located in an unrelated source tree 3 levels up in the directory hierarchy. This type of organization forces us to make better use of standard build system practices. Distributed Development</h2> The “many repository” paradigm has been partly driven by the distributed development paradigm. Git solves the problem of multiple developers working in multiple repositories very well. Because we want to use and customize projects like the Linux kernel, U-boot, and OpenEmbedded in our projects, then we naturally find ourself in the situation where we need to manage multiple repositories. Yes, you can check the Linux kernel into your company Subversion repository, but you are much better off long term if you bite the bullet and implement your own Git infrastructure. As we consider the product development process, we need to consider the life cycle of a product. Most products live for at least several years, and will go through several software iterations. If we can update the software components we use, then we can add value to the product in the form of new or updated drivers to support new peripherals, new libraries, performance improvements, etc. But we are dealing with millions of lines of source code, so we must have an efficient way to deal with software projects of this size. The below Figure 2, below, illustrates how you might organize a typical project. Notice we can pull updates from the U-boot and Kernel source trees at any time in the development process. We might have an outside team working an application, and we then easily synchronize the repositories when it makes sense. There are many other design flows possible. Once you have the ability to support multiple branches and repositories easily, it becomes trivial to implement a staging/testing repository for QA processes, maintenance repositories for supporting old releases, etc. Cheap Branches</h2> The last feature discussed in this article is that of cheap branches. Just as Git excels in allowing us easily integrate changes from multiple repositories at a team level, Git also allows us to easily work with multiple branches at the personal developer level. This is a revolutionary feature, and perhaps has to be experienced to be fully appreciated. With SVN, branching and merging is painful; therefore branches are not used a lot. In Git, branches are easy, therefore branching is naturally encouraged. Below is a typical flow of development. </a> In this case, a developer is working on a performance improvement that is largely experimental – depending how it works out, it may or may not be used. So he starts a branch and experiments with various changes. As he goes along, he makes nice granular commits that are easy to follow. He may reach a dead-end, delete the branch and start a new one. In this middle of this development, his boss asks him to make a quick bug fix. He checks out master, makes the change, switches back to the branch and keeps working. The switching between branches takes seconds and is very easy. Finally, he has some performance changes he is happy with. He merges master to his perf-improvement branch and tests his changes with the latest master changes. Everything looks good, so he then merges his branch to master. Git makes this type of development flow natural and easy. Because branching is painful in SVN, changes tend to be made in large commits that are difficult to understand. The ability to keep separate tasks isolated is analogous to having a clean, neat desk with only one thing on it, or having a messy desk with a half a dozen different projects going at once. Git is a tool to help maintain different working contexts, and it allows work to be staged in a natural flow. In the end, Git will change how you work. Using Go in place of a Spreadsheet 2016-06-28T00:00:00+00:00 Recently I needed to calculate NAND partition tables for a project where we will be supporting a number of different flash parts from 500MB to 2GB. I first tried this in a spreadsheet, but found it difficult to work easily with hex numbers and do the calculations I needed. I then looked into options for formatting text in columns from a program and found the nice text/tabwriter</a> Go library. With a few lines of code</a>, I was then able to get the below output, which is quite easy to read. The only tricky part was figuring out that for right justified data, you need to: not use tabs for the padding character</li> add a trailing \t in the input data</li> </ol> The Go stdlib is quite extensive, and often has everything you need. This makes “go run” a nice solution for scripting tasks. Old Partitions Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x1c0000 1.75 14 mtd2 uboot env 0x240000 0x40000 0.25 2 mtd3 linux 0x280000 0xa00000 10 80 mtd4 rootfs 0xc80000 0x186a0000 390.625 3125 mtd5 data 0x19320000 0x26ce0000 620.875 4967 total size = 0x40000000, 1024MiB New Partitions 500MB Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x500000 5 40 mtd2 uboot env 0x580000 0x40000 0.25 2 mtd3 linux1 0x5c0000 0x1400000 20 160 mtd4 linux2 0x19c0000 0x1400000 20 160 mtd5 rootfs1 0x2dc0000 0xa000000 160 1280 mtd6 rootfs2 0xcdc0000 0xa000000 160 1280 mtd7 log 0x16dc0000 0xa00000 10 80 mtd8 data 0x177c0000 0x7c40000 124.25 994 total size = 0x1f400000, 500MiB size of boot partitions: 45.75 MiB New Partitions 1G Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x500000 5 40 mtd2 uboot env 0x580000 0x40000 0.25 2 mtd3 linux1 0x5c0000 0x1400000 20 160 mtd4 linux2 0x19c0000 0x1400000 20 160 mtd5 rootfs1 0x2dc0000 0xfa00000 250 2000 mtd6 rootfs2 0x127c0000 0xfa00000 250 2000 mtd7 log 0x221c0000 0x3200000 50 400 mtd8 data 0x253c0000 0x1ac40000 428.25 3426 total size = 0x40000000, 1024MiB size of boot partitions: 45.75 MiB New Partitions 2G Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x500000 5 40 mtd2 uboot env 0x580000 0x40000 0.25 2 mtd3 linux1 0x5c0000 0x1400000 20 160 mtd4 linux2 0x19c0000 0x1400000 20 160 mtd5 rootfs1 0x2dc0000 0x1f400000 500 4000 mtd6 rootfs2 0x221c0000 0x1f400000 500 4000 mtd7 log 0x415c0000 0x6400000 100 800 mtd8 data 0x479c0000 0x38640000 902.25 7218 total size = 0x80000000, 2048MiB size of boot partitions: 45.75 MiB </code></pre> </code> Using Docker for OE/Yocto builds 2016-04-21T00:00:00+00:00 </a> Why Docker? When using OE to build software for products, we often run into the scenario where software needs to be built using the same version of OpenEmbedded over the course of several years. Production builds need to be predictable. We’ve also observed that old versions of OE often break as new Linux distros come out. This is just the result of the complexity of building tool chains. Additionally, for predictable builds you really don’t want to be changing the build OS. This requirement automatically rules out Arch Linux, Debian Unstable, Gentoo, etc as production build machines. Additionally, having developers debug OE build issues on varying workstation distributions is frustrating and time consuming. There are several options: keep a golden build machine around with a fixed OS</li> do builds in cloud on a machine with a fixed OS</li> build in a systemd-nspawn, or chroot</li> build in a virtual machine (Virtualbox, etc)</li> use Docker</li> </ul> I have used all of the above and recently started experimenting with Docker. As an example, I set up a container named cbrake/oe-build</a> and provided an example of how to use a Debian based container for OE builds. Docker requires a bit of a mind-shift from virtual machines. The 1st aha moment with Docker comes when you realize data and build directories should be mapped from the host into the container such that the container does not really have any state. There are no ssh keys, etc in the container. This allows the container to be immutable and easily replicated across any number of machines. Stuff that needs to change goes in your build directory and is managed using Git. Users specific information lives on your host machine. The 2nd aha moment with Docker is running commands with Docker instead of working in the context of Docker. For years, I have run OE builds in a systemd-nspawn. I’d always have two terminal windows open – one for the container context, and one for the host. With Docker, you typically just work in your host context, and spin up a Docker container every time you need to run a command in the container (bitbake, etc). If you define a few shell functions to wrap the docker commands, you hardly even know that commands are executing in the container. This might seem impractical, but it actually works quite well. Of course you can still run a container shell and work in the context of the container if you want, but its not really necessary or desired once you get the container set up. Docker is a neat tool and allows you to have a consistent environment for developer and production build machines. It also encourages a natural separation of immutable containers and transient data that makes it much simpler to manage and distribute containers. Setting up a Go development environment 2016-04-15T00:00:00+00:00 Go has a pretty neat development environment, and its helpful to set up a standard GOPATH on your workstation up front. This is what I do: mkdir ~/go </li> add the following to .bashrc (or some file that configures your env on login) export GOPATH=~/go</li> export PATH=$GOPATH/bin:$PATH</li> </ul> </li> </ul> Now, after you log in, you can do things like: go get -u github.com/monochromegane/the_platinum_searcher/... </code></pre> </code></pre> go get -u github.com/schachmat/wego </code></pre> (set up wego with forcast developer key) </code></pre> </code></pre> There are lots of</a> Go apps</a> out there and if you just set up a global GOPATH, they are trivial to install and use anywhere on your system! Generally your ~/go directory is also used for Go development projects, but that is a topic for another day. </code></pre> Go Language for Embedded ARM Linux Systems 2016-03-08T00:00:00+00:00 In the quest for technologies that work well for embedded Linux systems, I recently gave Go another try. The last time</a>I tried this was very early on and there were some floating point issues on ARM that appear to be fixed now. Having spent a few days porting an existing application to Go, there is a lot to like about Go. Compiles very fast</li> Nice set of default libraries that are nice to use</li> Binaries are reasonable size</li> Memory usage is reasonable</li> Very simple to deploy (run time and libs are bundled with a single executable)</li> many 3rd party libraries available</li> Tooling is good (build, unit testing, etc)</li> </ul> As an example of running a Go app on an embedded Linux system, enter the following in hello.go: package main import ( “fmt” ) func main() { fmt.Println(“Hello”); } To run on x86: go run Hello.go </code></pre> To build for ARM Linux: $ GOARCH=arm GOARM=5 go build hello.go $ file hello hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped </code></pre> As you can see, cross compiling is very easy with Go. Now, to try this exe on a Raspberry PI 3 running Arch Linux: [root@alarmpi ~]# ./hello Hello </code></pre> I did notice the time to build for ARM is a bit longer than building for x86: $ time go build hello.go real 0m0.262s user 0m0.337s sys 0m0.030s </code></pre> $ time GOARCH=arm GOARM=5 go build hello.go </code></pre> real 0m2.533s user 0m4.930s sys 0m0.183s </code></pre> But, still pretty reasonable considering what all it is doing. The simplicity of all this is very attractive. There is no runtime to cross compile and deploy. Just one executable and you’re good to Go. IOT Protocols: MQTT vs CoAP vs HTTP 2015-12-11T00:00:00+00:00 With the explosion of IOT (Internet of things), there are now more technologies we can use to build systems. In reality, we’ve been doing IOT for years. We’ve been networking devices for a long time. We’ve been collecting data from remote nodes. This is nothing new, but what the IOT movement brings to the table is technologies that are much lower cost, and more standardized. Two of these technologies are MQTT and CoAP. Both very interesting, and very useful. Recently, I helped a system manufacturer think through the architecture of a system with the following requirements: They plan to sell a relatively small number (200-1000) of fairly expensive machines ($5,000-$10,000). The point here is we are not dealing with millions of $10 nodes, which is really the vision of IOT.</li> There might be as many as a dozen machines installed in each location.</li> Users of these machines need to monitor them remotely. The manufacturer would also like to monitor them for diagnostics and customer support.</li> The machines can be connected to the Internet at the install sites.</li> These machines are installed at various locations where the manufacturer doesn’t have much control over the network (firewall, filtering, which ports are blocked, etc).</li> The machines don’t need to talk to each other.</li> It should be relatively difficult to hack into a machine and mess up what it is doing.</li> Data will be mostly going from machines to the cloud.</li> </ol> There are several ways to address these requirements, but one of the simplest today is for each machine to talk to a server in the cloud, and then users monitor and control the machines through a web application running in the cloud. The web application can be designed to be viewed on both personal computers and mobile devices. </a> So the question is which protocol do we use to communicate from the installed machines to the cloud? In this case, we really do not need a lot of efficiency, as the number of nodes is relatively small. Bandwidth requirements are not really a huge concern because a) the number of nodes is relatively small, and b) we can use the existing broadband infrastructure at the install sites. While it might be interesting to use one of the new protocols like MQTT or CoAP, we need to carefully consider them in the context of requirement #5 – we don’t have control over what ports are blocked or what type of filtering/proxying happens at the install site. Support costs can run very high debugging networking problems at remote locations. Because MQTT and CoAP typically use non-standard ports, they are less attractive for this application. Even if we did run MQTT or CoAP over port 80, the install site might have internet filters or proxies that would reject non HTTP data. So this puts us back to HTTP, perhaps with a REST implementation. As it turns out, this is not a bad way to go. Even though HTTP is old, it is still a perfectly valid protocol for this application. Some of the advantages include: HTTP is the most compatible with existing network infrastructure, and you are pretty much guaranteed HTTP will work. If a web browser works, then machines in this scenario should be able to send data to the cloud.</li> Stateless HTTP/REST will traverse firewalls and NAT systems easily. Again, web browsers work.</li> Because the Machines initiate the connection to the server (users are not connecting directly to a browser on the machine), the machines are much easier to secure. Thy don’t even need any open ports. Most of the security effort in this system only needs to be focused in one place (cloud server).</li> HTTP provides well understood authentication and encryption mechanisms.</li> </ul> There are other applications where MQTT or CoAP might make more sense. In a system connected via cellular modems where data is very expensive, you might want to consider CoAP, because it can use UDP which eliminates the overhead of TCP/IP, which is significant (both MQTT and HTTP require TCP/IP). Especially if you have 1000’s of nodes, and are trying to keep the monthly usage of each one under 5MB.</li> If the various machines need to talk to each other, then MQTT would be a good choice.</li> </ul> While I’d like to use MQTT or CoAP, in this application I’m going to recommend they stick with HTTP(S). Even though MQTT might be more convenient to work with, the support costs of debugging network issues at install sites will probably swamp any advantages during development. This project can benefit from the host of</a> Wifi enabled</a> microcontroller systems</a>, low cost embedded Linux</a> technologies, or cloud technologies</a> aimed at IOT systems. All of these support HTTP just fine. OpenEmbedded / Yocto Usage Survey 2015-11-18T00:00:00+00:00 Recently, I sent a survey to the OpenEmbedded and Yocto mail lists asking a series of questions about how they use OE. 38 people responded. The average number of years using OE is 4.8 years. Below are the results of the questions. (update 2015-11-23 – the below numbers are absolute numbers, not percentage) </a> </a></a></a></a></a></a></a></a></a> On the question about tooling, 4 people responded they use Toaster (two specified toaster analysis mode). Two use devtool. One uses ADT. I plan to follow up with additional articles discussing some of these results. Thanks to all who responded to this survey. Rebasing a set of changes with Git 2015-11-11T00:00:00+00:00 One of the common things we do during Linux kernel development is move a series of patches from one kernel version to a similar version (say Linux 4.1 to 4.1.12). This is required as new stable versions of particular kernel version are released. One approach is to merge, but then your changes are mixed in with upstream commits and are more difficult to manage. Git rebase offers a convenient way to move a set of patches. In the following example we have a series of changes we made (or patches we applied) on top of the 4.1 kernel. </a> Now we want to move the same patchset to the 4.1.12 stable kernel. To do this, we can use the following commands: git checkout -b 4.1.12-armv7-x0-dugen2 4.1-armv7-x0-dugen2 (create a new branch that will be rebased)</li> git rebase --onto v4.1.12 v4.1 4.1.12-armv7-x0-dugen2</li> </ul> A few details on the git rebase arguments: --onto v4.1.12 (this tells git what version we want to move our changes to)</li> v4.1 (upstream – tells git where our patchset starts)</li> 4.1.12-armv7-x0-dugen2 (branch – is the branch to do the operation on)</li> </ul> After the rebase, our patchset looks very similar, only on top of the 4.1.12 kernel version. </a> It this example (which included many more patches than shown above), I only encountered one conflict which was easy to resolve. This process can also be used to move a patchset to a new kernel version (say 4.2, or 4.3), but you would obviously expect more conflicts. Modifying the BusyBox config in OpenEmbedded 2015-09-30T00:00:00+00:00 Recently, I needed to enable the eject command in BusyBox for an OpenEmbedded (Yocto) based project. Below is a way to do this in a fairly painless way: bitbake -c menuconfig busybox (enable the eject command in the config and save)</li> bitbake -c diffconfig busybox (this generates a config fragment, note the fragment file location)</li> recipetool appendsrcfile -w [path to layer] busybox [path to fragment generated in step #2]</li> </ol> Now, in my project layer, I have the following files: recipes-core/busybox/busybox_%.bbappend: SRC_URI += "file://fragment.cfg" FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:" </code></pre> recipes-core/busybox/busybox/fragment.cfg: # Wed Sep 30 15:23:49 2015 CONFIG_EJECT=y CONFIG_FEATURE_EJECT_SCSI=y </code></pre> Wow, that is pretty neat! The same procedure should work on kernel recipes as well. Notes: If you have trouble running bitbake -c menuconfig in a remote ssh shell, trying installing screen on the build machine.</li> recipetool is located in openembedded-core/scripts/recipetool</li> Chris L.</a> noted the -w option in recipetool would have simplified the append slightly (would avoid the subdir= on the SRC_URI, since it’d install it to workdir instead of the source tree) Great idea – example above modified to include -w.</li> </ul> Best practices for using CMake in OpenEmbedded/Yocto 2015-03-10T00:00:00+00:00 I’ve already written about using autotools</a> and qmake</a> in OE. With recent projects, we’re using CMake to build most C/C++ components. Like any good tool, CMake has a learning curve, but it seems to do its job quite well. Below are examples of a CMake build file, and corresponding OE recipe. CMakeLists.txt cmake_minimum_required(VERSION 2.8) project(MyApp) file(GLOB_RECURSE src/.cpp) add_executable(MyAPP ${src_files}) target_link_libraries(MyApp pthread) install(TARGETS MyApp DESTINATION bin) </code></pre> myapp.bb (OE recipe) DESCRIPTION = “My xyz app” LICENSE = “CLOSED” inherit cmake SRCREV = “${AUTOREV}” PV = “1.1+gitr${SRCPV}” SRC_URI = “git://my-git-server/myapp.git;branch=master;protocol=ssh;user=git” S = “${WORKDIR}/git” BBCLASSEXTEND = “native” Notice the OE recipe above has no build/install logic other than “inherit cmake”. Below are some tips on using CMake. put your source files in a src directory. This allows you to easily build a list by using GLOB patterns. If you put the source files in the root directory of your application source, then the GLOB pattern will also find temporary C files that CMake generates, and you’ll get multiple “main” errors. Some recommend against using GLOB patterns for finding source files because it may miss additions, but in practice I find the benefits of this pattern outweigh the negatives.</li> do all install operations in CMake if possible (vs do_install in the OE recipe). The reason for this is its much easier to debug install in CMake then OE. You can build the app for x86 on your workstation, run “make install” and easily verify the files get installed correctly. Typically, things get installed in /usr/local when building on your workstation, and /usr when built in the context of OE.</li> do most of your development on your workstation. Most of the time, you can test 99% of an application’s functionality on your workstation. Do what you can there with unit tests where development is much faster and easier.</li> change ownership of /usr/local by running “sudo chown -R $USER /usr/local</a>”. This allows you to run “make install” of the CMake project during development without sudo. It also helps ensure you are not accidentally installing stuff outside /usr/local.</li> you don’t need to add header files to the target sources. CMake automatically figures out include dependencies.</li> you can run “make VERBOSE=1” to see the commands make is running during a compile.</li> cmake -DCMAKE_BUILD_TYPE=Debug can be used to add debugging information (essentially -g).</li> When developing on your workstation, do the following to build: mkdir build; cd build; cmake ../; make The advantage of running cmake inside a build directory is you can simply wipe the entire build dir to completely reset the build. Otherwise, the cmake files are mixed up with your source files.</li> The following is the CMake version of printf – very useful for debugging: message(some_var: ${some_var})</li> ccmake and cmake-gui are useful tools for examining the state of various CMake variables.</li> </ol> Most of these tips are general and have nothing to do with OE, but that is the entire point – using CMake with OE rarely requires you to do anything beyond what you normally do with CMake. So, if you use CMake correctly, there is usually not a lot more required to cross compile and package your apps in OE. PCB Schematics Style Guide 2014-08-25T00:00:00+00:00 As I work at the intersection of hardware and software, I spend a fair amount of time reading schematics for printed circuit boards (PCBs). I’m astounded at how sloppy most schematics are, and have come up with a few suggestions to consider. Schematics are not only design entry – schematics are documentation. Unless you are writing all the software, handling all the manufacturing, and debugging all field issues yourself, there will be others who will be using them. Be considerate.</li> Not everyone has a D size plotter (even B size printers are fairly rare). I have a rather large (27“, 2560x1140) monitor, and have seen schematics that I have to zoom in to read. Schematics should be created such that they can be printed on letter size paper (everyone has a letter size printer), and easily read – even by folks in their 60’s. This is fairly simple – just spend some time breaking large symbols into smaller sections, and use a few more pages. Even for viewing on a screen, putting less stuff on each page makes it easier to scroll through pages. Most people print out schematics for review or bench work – so having letter size pages makes this possible.</li> Schematics should be organized to have one function per page, or group similar functions on a page. Don’t intermix power supplies and an audio codec unless they are related. We realize the importance of organizing software in modules, and a large schematic is really no different.</li> Keep all pages the same size. When every page is a different size, you can’t easily scroll through pages, or print them out.</li> Avoid hierarchical schematics. For most PCB designs, hierarchical schematics offer no advantage over a flat schematic logically organized on pages. PCB’s can only be so big, therefore even if you have 20 pages for a schematic, 20 pages is really not all that much, and adding hierarchy to 20 pages just adds complexity and makes them harder to read. In a sense, your schematic page becomes a “hierarchical block.”</li> Off page references are the key to making smaller schematic pages work.</li> Spend a little more time putting a little art into your schematic symbols. If you have a DB-9 connector, and you draw a connector in the shape of a DB-9 (only takes a few more minutes), then you can instantly recognize what it is in the schematic, and it is much quicker to find the circuit you are looking for during debugging sessions. The same can be said for USB connectors, etc. In some cases, it may make sense to draw a connector symbol to match how it looks physically, so you can quickly find a pin on the bench without digging out a datasheet.</li> Schematic text must be easily searchable. I don’t want to install a clunky schematic viewer that only runs on X.XX version of windows just to search for text. I want to use the PDF reader I already have installed. Make sure schematics are cleanly exported to PDF, and are easy to search.</li> </ol> I’m sure there are many other ideas – in a brief search, I turned up the following: http://blog.upverter.com/schematic-style-guide</a></li> http://www.k-state.edu/ksuedl/publications/Technote%208%20-%20Guidelines%20for%20Drawing%20Schematics.pdf</a></li> http://electronics.stackexchange.com/questions/28251/rules-and-guidelines-for-drawing-good-schematics</a></li> </ul> Library sizes for C vs C++ in an embedded Linux system 2013-12-18T00:00:00+00:00 Is the size of the libraries required for C++ (vs C) a concern in Embedded Linux systems? Most Embedded Linux systems likely include some C++ code, so this is probably not even a decision we need to make in many cases. However, often there is a need for a small initramfs that is used as part of the boot process (perhaps for software updates) before switching to the main root file system. In this case, it makes sense to keep the initramfs as small as possible, and we might be concerned here with the size of C++ libraries. To find out what additional libraries are required when compiling with a C++ compiler vs C, a simple hello-world app was created. It was then compiled with gcc and g++. The outputs were then parsed with readelf to determine the required dynamic libraries. The results are shown below: C: 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] C++: 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6] 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] The sizes of packages for these libraries in an OpenEmbedded build are: libc: 1.4KiB (libm is inlcuded with libc)</li> libstdc++6: 256KiB</li> libgcc1: 204KiB</li> </ul> (The above sizes are compressed, so the actually will be some larger, but are useful for comparison.) The installed sizes of libstdc++6 and libgcc1 on a ext3 file system are 1.4MiB and 750KiB. It may also take slightly longer to load a C++ application as there is time required to dynamically load and link additional libraries. Is an extra 2MiB of libraries a big deal? It depends on your situation, but should be considered any time you are trying to create a small image. Setting up an OpenEmbedded Package Feed Server 2013-12-11T00:00:00+00:00 With the BEC OE build template</a>, you can easily set up an opkg feed server that serves up packages from your build directory. This allows you to easily install new packages during development, without generating and reflashing an entire image. To use: On workstation: edit local.sh, and define MACHINE_IP to point to your target machine, and re-run “source envsetup.sh”. Alternatively, you can export the MACHINE_IP variable in your environment.</li> run: oe_setup_feed_server</li> run: oe_feed_server</li> </ul> On target system: opkg update</li> opkg install </li> </ul> Opkg will install dependencies, so if you have a complex package, this is so much easier than copying over an opkg file manually, and figuring out you need 6 other packages as dependencies. Why systemd in Embedded Linux Systems? 2013-12-11T00:00:00+00:00 Recently I was asked why use systemd vs sysvinit in embedded systems? There are many discussions on this, and really most of the reasons people use it for servers and desktops are also valid for embedded systems. Lennart Poettering’s articles</a> explain very well why you might want to consider systemd. A few things that rank high on my list: systemd systems generally boot must faster than sysvinit systems.</li> systemd monitors every process it starts, so you know conclusively if your app is still running.</li> systemd has flexible mechanisms for restarting apps that crash, and rebooting if you get so many restarts in X amount of time. This is very important in embedded systems where you need to keep things running.</li> systemd logging (journald) is very flexible. The stdout of every process that is started by systemd is captured in the journal. You can also capture kernel messages in the journal. The end result is you have one place to look for system logs.</li> systemd has support for watchdog chaining</a> – again, important for many embedded systems.</li> The systemd dependency mechanism is very flexible.</li> systemd has tools to debug the init process: systemd-analyze, systemd-cgls, systemd-cgtop, bootchart, pybootchargui, etc.</li> on demand launch of services can improve boot time and conserve resources.</li> </ol> In summary, with systemd I have more control over the system startup and shutdown, and have better mechanisms to monitor things to make sure they stay running. OpenEmbedded Build Template 2013-06-28T00:00:00+00:00 How does one set an OpenEmbedded/Yocto/Poky/Angstrom build? There are many options. Some include: Angstrom setup scripts</a></li> Poky</a></li> Freescale Community BSP</a></li> OpenEmbedded core standalone</a></li> </ul> (I’m sure there are many others, feel free to add in comments …) Over the past years, we’ve supported a number of customers using OpenEmbedded to develop products using various SOC’s. We also try to keep builds going for a number of different systems so that we can continuously evaluate the state of OpenEmbedded and related Embedded Linux technologies. We also needed a standard way to set up builds for customers (some don’t have a lot of experience with OE and Git) that is simple and consistent. What we ended up with is the BEC OpenEmbedded build template</a>. The goal is to have a quick entry point into OpenEmbedded that includes the necessary layers for a number of different machines, and automates a number of routine tasks such as installing images to a SD card, setting up a development feed server, etc. The build template is only updated when the build is stable and tested on a number of machines, so it provides a series of stable snapshots of OpenEmbedded and associated layers. This build template currently tracks the master branches for all the layers used. This gives us a platform to track the latest OE changes. With most projects, that ability to use the features in the latest versions of software outweighs the stability benefits of OpenEmbedded release branches. There are times when the OpenEmbedded project goes through invasive changes (such as the systemd integration), and using the master branches is not practical, so in this case we simply use the last stable snapshot that builds and works. In most cases if there are issues, simply report or fix the issue and wait a week. Perhaps the most controversial decision is the use of Git submodules for including OpenEmbedded layers. The Internet is full</a> of rants</a> against Git submodules. For heavy developer use, submodules may not be optimal. However, from a user perspective, Git submodules provide a simple mechanism for including external repositories in a project. If most of the submodules (OE layers) won’t be touched (typical OE user scenario), submodules work very well. The fact that Git locks down submodules to a specific commit ensures you are getting exactly what you think you are getting (vs a branch that may have been rebased, modified etc). If the git hash matches, you can be pretty sure it is the same as the last time you built it. This is an important factor in production build systems where you want to be sure of what you are building. Google repo is another option under consideration, but there are still some trade-offs to work through. In the end, build systems are very personal, and must be customized for your product and development team. The number one requirement for a an Embedded Linux build system is that you can get repeatable builds with a single command. There must be no manual steps where human error can be introduced. This is just one way to accomplish this goal. (2023-09-05 update: the BEC OE Build template has been replaced by the Yoe Distribution</a>) OS Containers for Build Systems 2013-05-08T00:00:00+00:00 Since I’ve been running archlinux on some of my systems, one thing I’ve found useful is systemd-nspawn</a>. systemd-nspawn containers (or chroots on non-systemd systems) give you a quick way to install a Linux distribution, that can run inside an existing Linux system. Some cases where systemd-nspawn containers (referred to as containers in this document) are useful: At one point, OpenEmbedded would not build with GCC 4.8 (this is no longer the case with recent versions of OE). So a Debian or Ubuntu OS container was a quick way to get builds going again.</li> For product build systems (may live for many years), typically OE will eventually break as you upgrade the workstation distribution. For projects that need a long-lived OpenEmbedded build system, setting it up in a chroot makes a lot of sense.</li> Someone might be having a compile or build problem with a distribution you don’t currently have installed. With containers, you can quickly set up a test distribution to reproduce problems.</li> I’ve had cases where I need an older version of Qt for a project, but my workstation includes a newer version. Again, setting up a OS container is sometimes simpler than getting two versions of Qt to dwell together peaceably in the same distribution.</li> Backing up or replicating your entire build system is very easy – simply rsync the OS container directory to another machine.</li> </ol> So the solution is to select a relatively stable, long-lived distribution to host your product builds. Debian is good choice. Because the container is simply a directory in the host workstation filesystem, you can use host workstation tools (editors, git, etc) directly in the container filesystem. The only thing you need to use the chroot for is the actual building. If you make sure the user ID is the same between your workstation and nspawn container, then permissions are seamless – you can easily access files in the container from the context of your host workstation. To set up a nspawn-container: Install debootstrap. On arch systems, this needs to be obtained for the AUR.</li> host: sudo debootstrap --arch=amd64 wheezy ~/debian-wheezy/</li> host: sudo systemd-nspawn -D ~/debian-wheezy/</li> container: apt-get update && apt-get install ssh</li> container: edit /etc/ssh/sshd_config, and set port to something other than 22 (23 in this example)</li> container: /etc/init.d/ssh start</li> </ol> (This systemd-nspawn</a> man page gives examples for setting up other distributions.) To set up a user in your container: host: id</li> will return something like: uid=1000(cbrake) gid=100(users) …</li> container: adduser --uid 1000 -gid 100 cbrake</li> host: ssh-copy-id -p 23 localhost (will copy public key to container)</li> </ol> Now, on the host system, you can simply “ssh -p23 localhost” any time you want to log into the container. Soft links between the project workspace on the host system, and the container can also make shifting between the two easier. An alternative way to start the container once its set up is: sudo systemd-nspawn -D ~/debian-wheezy /sbin/init Its also handy to make the shell prompt in the container slightly different than the host OS so that you can easily tell the difference. To accomplish this, add the following to ~/.profile in the container OS: export PS1=“[\u@wheezy \w]\$ “ To create a service that starts your container, put something like the following in /lib/systemd/system/debian-wheezy.service [Unit] Description=Debian Wheezy [Service] ExecStart=/usr/bin/systemd-nspawn -D /scratch/debian-wheezy/ /sbin/init 3 KillMode=process [Install] WantedBy=multi-user.target </code></pre> Hopefully this gives you a quick overview of how OS containers can be set up, and used in your OpenEmbedded build systems. Git submodules can now track branches 2013-04-22T00:00:00+00:00 As of version 1.8.2, Git submodules can now track branches instead of specific commits. This is good news as in many cases, this is exactly the behavior we want. However, Git submodules are still not as flexible as Google repo</a>, but since submodules are built into Git, the submodule command is a good solution in many cases. The “git submodule update --remote” command is the key to tracking branches with submodules. The following is from the Git man pages: --remote This option is only valid for the update command. Instead of using the superproject’s recorded SHA-1 to update the submodule, use the status of the submodule’s remote tracking branch. The remote used is branch’s remote (branch..remote), defaulting to origin. The remote branch used defaults to master, but the branch name may be overridden by setting the submodule..branch option in either .gitmodules or .git/config (with .git/config taking precedence). This works for any of the supported update procedures (--checkout, --rebase, etc.). The only change is the source of the target SHA-1. For example, submodule update --remote --merge will merge upstream submodule changes into the submodules, while submodule update --merge will merge superproject gitlink changes into the submodules. In order to ensure a current tracking branch state, update --remote fetches the submodule’s remote repository before calculating the SHA-1. If you don’t want to fetch, you should use submodule update --remote --no-fetch. So, if you already have a Git submodule set up, its a simple matter to run git submodule update --remote to update the submodule to the latest master branch. If you want a different branch, simple edit .gitconfig. [submodule "meta-bec"] path = meta-bec url = git@github.com:cbrake/meta-bec.git branch = test </code></pre> Now, if you run git submodule update --remote,Git will update the meta-bec submodule to the latest on the test branch. This functionality is purely a convenience feature in the submodule update command. In the actual repository, Git still stores submodules pointed to a particular commit. The same thing could be accomplished with something like git foreach “git fetch && git checkout test”. The branch option in .gitmodules functions more as documentation and convenience. It is very handy to be able to look at .gitmodules and quickly determine that submodule X is tracking branch Y. Normally, this would have to be documented elsewhere, or figured out in some other way. Also, for build systems where you want the build to always track the production branches of various projects, update --remote gives you a convenient way to update the build tree. A quick way to share files from any directory 2013-01-22T00:00:00+00:00 Did you ever need a quick way to share files from a directory on your computer? Or perhaps transfer a large file to another person? With nodejs</a> and express</a>, you can easily set up a temporary web server that allows users to browse and access a list of files in a directory. For convenience, I created a simple github project</a> that can be cloned into any directory, and then a server started in a matter of seconds. Yes, you could upload files to a server, or share them with a file sharing service, but if you can expose a random port on your computer to the person who needs the files, then this is faster, and does not require any intermediate steps. Check out https://github.com/cbrake/http-file-server</a> for more information. OpenEmbedded Source Mirrors 2013-01-07T00:00:00+00:00 When using OpenEmbedded for product development, there are several reasons you may want to consider setting up a source mirror for your OpenEmbedded build: over time, sources disappear from download locations</li> various servers for source packages may be off-line at the time a build is run</li> some servers may be very slow, which slows down your build</li> occasionally the checksums of a source package will change</li> </ul> For a production build system, you want the build to be reliable and consistent, so this means not depending on 3rd party web sites/servers for a clean build to complete. Fortunately, OpenEmbedded makes it easy to set up a source mirror with the PREMIRRORS</a> variable. When bitbake tries to fetch source code, it tries PREMIRRORS, the upstream source, and then MIRRORS. There are several advantages to using a PREMIRROR variable over a MIRROR for your source mirror: your source mirror will be used first, thus slow web sites are not an issue</li> if the checksums of the package change, the build will not fail because its still using the original source package from the mirror. You are guaranteed to be always using the same source package.</li> </ul> Setting up a source mirror is as simple as copying the contents of your downloads directory to a web server, and then populating the following variable in local.conf: PREMIRRORS_prepend = "\ git://./. http://my-server/sources/ \n \ ftp://./. http://my-server/sources/ \n \ http://./. http://my-server/sources/ \n \ https://./. http://my-server/sources/ \n" </code></pre> The Poky reference manual has more details</a>. Perisistent device names for USB serial ports 2012-12-19T00:00:00+00:00 Currently, my workstation has two 8-port USB<->RS232 devices</a>, one dual port USB<->RS422/RS485 adapter</a>, and several single port adapters such as the very useful BUBII</a>. So with around 20 USB->serial devices, figuring out which /dev/ttyUSBx entry corresponds to which port is not really practical. However, with udev in Linux, you can easily give static names to each device. This is especially convenient to do with FTDI devices because each FTDI device has a serial number. In devices such as the 8-port RS232 adapter, there are 4, 2-port FTDI chips. I could not find a serial number available in the adapter with a Prolific IC, so I would avoid those until that is sorted. udevadm can be used to discover the serial number for a FTDI device: udevadm info --attribute-walk -n /dev/ttyUSB0|grep serial </code></pre> After the serial number is known, rules can be created in the /etc/udev/rules.d/99-usb-serial.rules as shown in this example</a>. Now, serial ports can be accessed using convenient names such as /dev/ttyUSB_beagle. A quick way to set up an OpenEmbedded feed server 2012-12-08T00:00:00+00:00 During development with OpenEmbedded (oe-core, meta-oe, meta-angstrom), I often find it useful to set up a feed server so that packages can quickly be installed on the target system without manually copying them over or building a new image. One way to do this is copy your deploy/ipk directory to an existing web server (perhaps Apache running on your workstation), or configure Apache to point at your OE build directory, etc. But, it might be more convenient if your build system could directly create an opkg feed with no extra configuration. In the below example, we use node.js</a> + express.js</a> to create a feed server (basically just a web server that serves up the ipk files). tools/feed-server/app.js: // nodejs script to start a http server with the feed from this directory // default is port 4000 var express = require('express') var app = express() app.use('/', express.static(__dirname + '/../../build/tmp-angstrom_next-eglibc/deploy/ipk/')) app.use('/', express.directory(__dirname + '/../../build/tmp-angstrom_next-eglibc/deploy/ipk/')) console.log("feed server started on port 4000") app.listen(4000) </code></pre> The express.directory function is used to create a directory listing that can be browsed. With most other web servers, at least this much code is required just for configuration. With node.js, this is the code to create an entire server from scratch! This node.js app can be started with a bash function in the environment: function oe_feed_server() { cd $OE_BASE bitbake package-index node tools/feed-server/app.js cd - } </code></pre> The package-index target is used to rebuild the index files that list the available packages. On the target system, the /etc/opkg configuration files must be modified to point to the feed server, and then you can run: opkg update; opkg install <package> </code></pre> An exmaple build environement with this integrated is located: https://github.com/cbrake/oe-build-core</a> https://github.com/cbrake/oe-build-core/commit/23352b9a43c60d67070abe5ac001aba9a9ac5cc4</a> https://github.com/cbrake/oe-build-core/commit/6baceb8b1e4477ccfd03aa553a5fcf501b398196</a> It might be argued that it is just as easy or easier to set up a more conventional web server. However, the benefit of node.js is that it is a full blown programming environment. You can quickly extend it to provide a web interface, perhaps add functionality to automatically push updated opkg configuration files to the target system that point to your feed server, etc. It is much more than just a web server. Setting the root password in an OpenEmbedded image 2012-12-05T00:00:00+00:00 During development, often a blank root password is used for the embedded Linux target system. However, when deploying an embedded Linux system, often there is a requirement to set the root password to something non-obvious. One way to do this is boot the system, and change the password using the passwd command. Then copy the password hash from the /etc/shadow file into the below line in your image recipe. ROOTFS_POSTPROCESS_COMMAND += "\ sed 's%^root:[^:]:%root:password_hash_from_etc_shadow:%' \ < ${IMAGE_ROOTFS}/etc/shadow \ > ${IMAGE_ROOTFS}/etc/shadow.new;\ mv ${IMAGE_ROOTFS}/etc/shadow.new ${IMAGE_ROOTFS}/etc/shadow ;" </code></pre> The ROOTFS_POSTPROCESS_COMMAND is useful for simple modifications like this to the rootfs image. OpenEmbedded: configuring openssh to allow a blank password 2012-12-05T00:00:00+00:00 Noticed the following when browsing around in the OpenEmbedded sources the other day: ROOTFS_POSTPROCESS_COMMAND += "openssh_allow_empty_password ;" </code></pre> This allows a blank password for development, which is conveient for running ssh/scp commands to the device. The above can be placed in an image recipe. The command modifies the PermitEmptyPasswords config option in /etc/ssh/sshd_config or /etc/default/dropbear. Running a reboot cycle test shell script with systemd 2012-08-29T00:00:00+00:00 One of the easiest ways to stress test an embedded Linux system is to continuously reboot the system. Booting is a difficult activity for a Linux system (similar to waking up in the morning). The CPU is maxed out. There are a lot of things happening in parallel. Software is initializing. There is a lot of filesystem activity. Often, if there is an instability in the system (and especially the filesystem), continuously rebooting will expose it. Now that we use systemd for most new systems, we need to run a simple script X seconds after the system boots to increment the boot count, and reboot. This is the classic domain of a shell script. However, for a simple task like this, it’s a little cumbersome to create a seperate shell script and then call this script from a systemd unit. The following is a way to embed the script directly in a systemd unit. Put the following in: /lib/systemd/system/cycletest.service [Unit] Description=Reboots unit after 30s [Service] StandardOutput=syslog+console ExecStart=/bin/sh -c "\ test -f /cycle-count || echo 0 > /cycle-count;\ echo 'starting cycletest';\ sleep 30;\ expr `cat /cycle-count` + 1 > /cycle-count;\ systemctl reboot;\ " [Install] WantedBy=multi-user.target </code></pre> To install and start the script: systemctl daemon-reload</li> systemctl enable cycletest.service (enable the service to start on reboot)</li> systemctl start cycletest.service (start the service, should reboot in 30s)</li> </ul> Bitbake has a new way of displaying build status 2012-08-29T00:00:00+00:00 Now instead of displaying a scrolling log, bitbake will display a simple output that lists which tasks it is working on at the moment: Currently 4 running tasks (185 of 3093): 0: gmp-native-5.0.5-r0 do_configure (pid 22919) 1: lzo-native-2.06-r1 do_configure (pid 27103) 2: expat-native-2.1.0-r0 do_compile (pid 7463) 3: ncurses-native-5.9-r10.1 do_compile (pid 9820) </code></pre> This really allows for a clear view of how the parallel threads option (BB_NUMBER_THEADS) in bitbake works. Mounting a UBIFS partition using systemd 2012-08-16T00:00:00+00:00 Systemd</a> is becoming the defacto system and service manager for Linux, replacing the SysV init scripts. The Angstrom distribution</a> has supported systemd for some time now. Recently, I needed to mount a UBIFS filesystem in one of my projects. The main application is being started with systemd, so it seemed like a good fit to also use systemd to mount a data partition needed by the application. Systemd can use entries from /etc/fstab, but one additional wrinkle in this system is that I also wanted to run the UBI attach in a controlled way. This can be done with a kernel command line argument, but there are times in this system where we will want to format the data partition, so this requires a detach/attach operation. The resulting systemd units are: data.mount [Unit] Description=Mount data partition Requires=data-attach.service After=data-attach.service </code></pre> [Mount] What=ubi1:data Where=/data Type=ubifs </code></pre> data-attach.service [Unit] Description=Attach data ubi partition [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/sbin/ubiattach /dev/ubi_ctrl -m 6 ExecStop=/usr/sbin/ubidetach /dev/ubi_ctrl -m 6 </code></pre> add the following to my-application.service Wants=data.mount </code></pre> The only real problem I ran into is the “After” statement in the data.mount unit. It turns out that the mount will run before the attach operation is finished unless this is included. The RemainAfterExit seems to be required so that ExecStop can be run when the unit is stopped. (There may be a better ways to do all this, so comments are welcome!) One of the benefits of systemd is that everything is very controlled. If the dependencies are specified properly, there are no race conditions. Additionally, if you need to manage units (services, mounts, etc) from a program, it is much easier to check the states as everything is very consistent. For example, if you want to query the state of a unit, the systemctl show command will return easily parsed output (as shown below): ... Before=umount.target After=data-attach.service systemd-journald.socket -.mount Description=Mount data partition LoadState=loaded ActiveState=active SubState=mounted FragmentPath=/lib/systemd/system/data.mount UnitFileState=static InactiveExitTimestamp=Thu, 01 Jan 1970 00:00:17 +0000 ... </code></pre> If the data-attach service is stopped, systemd automatically unmounts the data partition first – very nice! Hopefully this example illustrates how to do simple tasks in systemd. It appears that instead of having a complex script or program to initialize a system, the systemd “way” is to create a number of small units that get connected with dependencies. This seems like a much more controlled and flexible approach. A Review of Graphical Application Solutions for Embedded Linux Systems 2012-08-07T00:00:00+00:00 One of the decisions we face when building Embedded Linux systems is what components to use. With Open Source software, there is often more than one good option. Graphical libraries are no exception. In this article, we’ll examine GTK+, Qt, EFL, Android, and HTML/Javascript. There are many factors that go into a choice like this, but some of them are: Does the application need to run on Windows or MacOS?</li> Does the GUI need to be viewed remotely over a network?</li> Are dynamic effects (think iPhone) desired?</li> Does the application need to run on low end CPU’s (ones without a GPU)?</li> </ul> Putting some effort into selecting the right GUI technology is important for the following reasons: Most of the development effort for the product will likely be in the graphical application, so it makes sense to maximize the productivity of the application developers. Typically there might be 3-10 application developers on a project for every system software developer.</li> You want a technology that will scale with new revisions and improvements of the product.</li> You want a technology that will be supported and improved long term.</li> </ul> With Open Source software, things are always changing. Leading technologies change. This can be illustrated by following Intel’s support for their open source software platforms. While some of this may be driven by politics, we can also see clear technical and licensing reasons why these shifts were made. Moblin (GTK+), became public around 2007, merged into MeeGo in 2010</li> MeeGo (Qt), announced in 2010-02, cancelled 2011-09</li> Tizen (HTML/Javascript, EFL), announced in 2011-09</li> </ul> We can see similar shifts in other companies: Nokia started with GTK+ as the GUI technology for their tablet computers, and then shifted to Qt</li> Samsung has been using GTK+ on top of DirectFB, and now is moving toward EFL/HTML/Javascript for some phones</li> Many phone manufactures are producing Android products</li> Palm moved from a proprietary GUI to HTML in webOS</li> </ul> GTK+</h2> GTK+ is part of the GNOME desktop project and is perhaps the most used graphical library for desktop Linux applications, and in the past has been very popular in Embedded systems. Nokia has invested heavily in GTK+ in the past with its early tablet products (N770, N800, N900, etc). However, with the advent of the iPhone and faster processors with GPUs, everything has changed. The standard is now dynamic GUI’s with sliding effects, etc. The Clutter project is a library that can be used to build dynamic GUIs and fits in with the GNOME stack. GTK+ supports Windows and MacOS, but probably not as well as Qt. Qt</h2> Qt is a very mature project that is also extensively used for desktop projects, and recently is being used on some of Nokia’s phones. Qt was originally developed by the Norwegian company Trolltech. Originally, Qt was only offered with either a proprietary license or the GPL license. This meant if you wanted to write a proprietary application using Qt, you had to purchase Trolltech’s commercial license. This factor alone probably made GTK+ a much more popular solution for many years for Embedded Linux GUI’s, including most cell phone stacks. In 2008, Trolltech was acquired by Nokia, and shortly after that, Qt was offered under the LPGL license, giving it the same license as GTK+. One of Qt’s compelling features introduced in Qt 4.7 is its QML (or Qt Quick) technology. This allows you to write declarative GUI’s in a Javascript like syntax with many automatic bindings. There is good support for dynamic operations like sliding effects, and the performance is reasonable, even on low-end systems without a GPU. In the future, Qt 5.0 will require OpenGL, and hence be relegated to high end ARM CPU’s with a GPU, or desktop systems. Qt’s cross platform support is excellent, and provides good native support for Linux, MacOS, and Windows. Recently, Nokia has made efforts to set up Qt as more of a community project, instead of retaining exclusive control over it. EFL</h2> EFL (Enlightenment Foundation Libraries) is a project that started out as the Enlightenment window manager, and grew into a set of general purpose libraries. It claims to be to be more efficient than GTK+ and Qt, and work with or without hardware acceleration. Recently, EFL seems to have garnered the commercial interest of Samsung, Intel, and others involved in the Tizen project. According to a presentation by Carsten Haitzler (one of EFL’s founders and developers), Samsung was using GTK+ and DirectFB, but switched to EFL after seeing the performance. Perhaps the most compelling story for EFL is the high performance over a range of hardware capabilities (from simple phones with low end processors, to high end smart-phones running OpenGL). Parts of EFL are also used in the commercial GUI FancyPants. Android</h2> Android is an interesting GUI solution, especially as many developers have experience working on Android applications. Android now seems to be used in many applications where Windows CE was used in the past, probably due to its polished application development tool-set. HTML/Javascript</h2> Application development with HTML and Javascript is one of the more interesting developments because many embedded systems are headless (don’t have a local display). Couple this with the fact that many users now have smartphones or tablets readily available, and it may make sense to simply use an external device for displaying the UI. There is often a requirement for accessing the UI of a device remotely, and in this case, HTML/Javascript works very well. If the UI needs to be displayed locally, then a fairly powerful CPU is required (an ARM Cortex-A8, etc) to run a modern web browser. If there is no physical display on the device, and the UI is accessed remotely on a computer or mobile device, then a less powerful CPU is required because the embedded device does not actually have to do any of the rendering. HTML/Javascript also has the benefit that it is a very popular technology, thus there are many experienced developers. Summary</h2> Each of the above technologies has benefits and drawbacks. Understanding your project’s requirements, and what each solution offers is key to making the best decision. A Linux Kernel Tracing Tutorial 2012-01-31T00:00:00+00:00 The Linux kernel has a fairly extensive tracing infrastructure that is quite useful for debugging. There are a number of things you can do with tracing, but the focus of this article will be the traditional printk type debugging we often end up doing to trace initialization issues with a driver. The following links provide additional information on the linux kernel tracing infrastructure: </a>http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=tree;f=Documentation/trace</a></li> Debugging the kernel using Ftrace - part 1: http://lwn.net/Articles/365835/</a></li> Debugging the kernel using Ftrace - part 2: http://lwn.net/Articles/366796/</a></li> Secrets of the Ftrace function tracer: http://lwn.net/Articles/370423/</a></li> http://elinux.org/Kernel_Trace_Systems</a></li> </ul> In this example, I am working on a new audio driver. The typical experience with a new driver is that you install it and nothing happens because something is not registered correctly with the Linux driver model. So, the first thing I do is start with with the platform_device_add() function in my drivers init function. To observe the kernel activity around the kernel platform code, I can do the following: cd /sys/kernel/debug/tracing/ echo 0 > tracing_on (keep trace from filling up until we set filter) echo function_graph > current_tracer echo platform > set_ftrace_filter echo 1 > tracing_on cat trace_pipe (leave running in a different shell) After executing the above, we see the following. For this example, trace_pipe is preferred because the trace is then emptied and only new information is shown. 0) + 30.518 us | platform_device_alloc(); 0) | platform_device_add() { 0) 0.000 us | platform_uevent(); 0) + 30.518 us | platform_uevent(); 0) 0.000 us | platform_uevent(); 0) + 30.518 us | platform_match(); 0) + 30.518 us | platform_match(); 0) 0.000 us | platform_match(); 0) 0.000 us | platform_match(); ... 0) + 30.518 us | platform_match(); 0) 0.000 us | platform_match(); 0) 0.000 us | platform_match(); 0) 0.000 us | platform_match(); 0) 0.000 us | platform_match(); 0) ! 3936.767 us | } 0) + 30.518 us | platform_uevent(); 0) + 30.518 us | platform_device_alloc(); </code></pre> From the above, I can conclude that the platform_match() is not succeeding, because I would expect some more activity. At this point I chose to add a printk: diff --git a/drivers/base/platform.c b/drivers/base/platform.c index 7a24895..f9ce0c7 100644 --- a/drivers/base/platform.c +++ b/drivers/base/platform.c @@ -662,6 +662,8 @@ static int platform_match(struct device dev, struct device_driver drv) struct platform_device pdev = to_platform_device(dev); struct platform_driver pdrv = to_platform_driver(drv); + trace_printk("pdev->name = %s, drv->name = %s", pdev->name, drv->name); + /* Attempt an OF style match first / if (of_driver_match_device(dev, drv)) return 1; </code></pre> Now, if I re-run the trace, I see the following: 0) | / pdev->name = soc_audio, drv->name = davinci_emac / 0) 0.000 us | } 0) | platform_match() { 0) | / pdev->name = soc_audio, drv->name = snd-soc-dummy / 0) 0.000 us | } 0) | platform_match() { 0) | / pdev->name = soc_audio, drv->name = soc-audio / 0) 0.000 us | } 0) | platform_match() { 0) | / pdev->name = soc_audio, drv->name = omap-pcm-audio / 0) 0.000 us | } 0) ! 4241.943 us | } / platform_device_add / </code></pre> From the above, it looks like we have a simple mismatch between “soc_audio” and “soc-audio.” Fixing this problem, and re-installing the module, we now have: 0) | platform_match() { 0) | / pdev->name = soc-audio, drv->name = snd-soc-dummy / 0) 0.000 us | } 0) | platform_match() { 0) | / pdev->name = soc-audio, drv->name = soc-audio / 0) 0.000 us | } 0) + 91.553 us | platform_drv_probe(); 0) ! 4241.943 us | } / platform_device_add / </code></pre> Now we can see that the names match, and the probe function is now being called. At this point, we may want to turn on tracing of some additional functions to try to determine what is happening next. echo "platform snd* mydriver" > set_ftrace_filter </code></pre> And the result: 0) | / pdev->name = soc-audio, drv->name = snd-soc-dummy / 0) 0.000 us | } 0) | platform_match() { 0) | / pdev->name = soc-audio, drv->name = soc-audio / 0) + 30.517 us | } 0) | platform_drv_probe() { 0) | snd_soc_register_card() { 0) + 30.518 us | snd_soc_instantiate_cards(); 0) ! 17852.78 us | } 0) ! 17883.30 us | } 0) ! 22125.24 us | } / platform_device_add / </code></pre> With the above additional information, we can continue to learn more about the flow through the kernel. While all of the above could have been done with printk’s, it would have been more time consuming. The kernel function tracing capabilities allow us to quickly get a high level view of the flow through the kernel without manually adding a bunch of printk statements. The kernel tracing features are completely contained in the kernel without requiring additional user space utilities which makes it very convenient to use in embedded systems. The low overhead is also important in resource constrained embedded systems. The easy way to get serial terminal in Linux 2012-01-26T00:00:00+00:00 When doing embedded Linux development, most of us spend out time tethered to a target system with a serial cable, which is used for a serial console. Minicom is the defacto serial terminal software for Linux. However, Minicom is a little fussy in that you typically have to set it up for each port you want to use. This is no big deal, but is generally difficult for new users to understand, and yet another hurdle. And with 8-port USB->serial adapters, I have a lot of ports to set up. Just recently, I discovered that screen can be used as a serial terminal program: screen /dev/ttyUSB0 115200 </code></pre> A few notes on using: to exit screen: Ctrl-a k</li> to write a hardcopy of the screen image: Ctrl-a h</li> to get help: Ctrl-?</li> </ul> All the neat features of screen are two numerous to list here, but one more that I’ll point out is the scrollback/copy feature (activated by Ctrl-a [ ). This allows you to scroll back and the navigation works much like VI – what could be nicer? Verizon UML290 and Sprint U600 USB Modems in Embedded Systems 2011-03-31T00:00:00+00:00 Recently I tested support for the Verizon UML290 and Sprint U600 USB Cellular modems in an embedded Linux system. Both modems support 3G and 4G networks, but only the 3G modes were tested due to lack of 4G coverage at the testing location. </a> Fortunately, both modems function very similar to previous modems, so with the drivers available in the Linux kernel, and standard pppd support in OpenEmbedded</a>, they worked fine. The Verizon UML290 modem provides a challenge in that it must be manually switched between 4G and 3G modes. Typically this is done automatically by the vzaccess program Verizon supplies with the modem that runs on Windows. The solution for this system was to manually set the modem to 3G mode as detailed on the following page: http://www.evdoinfo.com/content/view/3492/64/</a> It appears that some embedded systems such as the Cradlepoint routers have implemented automatic 3G/4G switching support for the UML290, so this is no doubt possible with a little effort. The Sprint U600 modem appears to default to 3G, or automatically switch inside the modem. The same pppd scripts can be used with both modems: # /etc/ppp/peers/verizon_um290 user a password v connect "/usr/sbin/chat -v -f /etc/ppp/peers/verizon_um290_chat" defaultroute usepeerdns ttyACM0 921600 local usepeerdns debug -detach </code></pre> # /etc/ppp/peers/verizon_um290_chat '' 'ATZ' 'OK' 'ATDT#777' 'CONNECT' '' </code></pre> To initiate a connection: pppd call verizon_um290 </code></pre> With Verizon cellular modems, it appears that port 22 is often blocked, so if you need to access a remote device via ssh, you may need to run ssh on a higher port number. With 4G networks, it appears that the networking setup may be different in that a public IP address may not be assigned. From the above evdoinfo.com page, we find the following text: This fix will also work for users looking to use their device for remote based applications because it assigns a public facing IP address (3G ONLY). With eHRPD you’re assigned a private IP in either 3G or 4G mode, which has prevents UML290 users from accessing remote applications. Perhaps the Rev A HDR Service mode will also work in 4G mode, but its seems as cellular networks become more complicated, there will be more issues to deal with in using USB Cellular modems for remote access in embedded systems. Past articles on USB Cellular modems: /353/sprint-598u-usb-broadband-modem-in-embedded-systems/</a></li> /203/using-a-verizon-usb720-modem-in-an-embedded-linux-system/</a></li> </ul> Git and Distributed Development 2011-03-29T00:00:00+00:00 This is part of an ongoing series of articles</a> on the Git version control system. The “many repository</a>” paradigm has been partly driven by the distributed development paradigm. Git solves the problem of multiple developers working in multiple repositories very well. Because we want to use and customize projects like the Linux kernel, U-boot, and OpenEmbedded in our projects, then we naturally find ourself in the situation where we need to manage multiple repositories. Yes, you can check the Linux kernel into your company Subversion repository, but you are much better off long term if you bite the bullet and implement your own Git infrastructure. As we consider the product development process, we need to consider the life cycle of a product. Most products live for at least several years, and will go through several software iterations. If we can update the software components we use, then we can add value to the product in the form of new or updated drivers to support new peripherals, new libraries, performance improvements, etc. But, we are dealing with millions of lines of source code, so we must have an efficient way to deal with software projects of this size. The below Figure 2 illustrates how you might organize a typical project. Notice we can pull updates from the U-boot and Kernel source trees at any time in the development process, and merge the changes with our own modifications. We might have an outside team working an application, and we then easily synchronize the repositories when it makes sense. </a> There are many other design flows possible. Once you have the ability to support multiple branches and repositories easily, it becomes trivial to implement a staging/testing repository for a QA processes, maintenance repositories for supporting old releases, etc. Even at a personal developer level, Git’s distributed capabilities offers many advantages. Each Git workspace is actually a full Git repository. This means you can check changes in locally, re-organize your changes, be able to track changes when off-line, etc. For this reason, many developers are now using git-svn</a> when they need to work with Subversion repositories. With git-svn, you have all the benefits of using git locally, and yet you can easily synchronize with Subversion repositories. And this leads us to our next topic: cheap branches (coming soon). How do modern USB chargers work 2011-03-19T00:00:00+00:00 As we help customers design products, we often try to leverage the latest cell phone practices and technologies. One of these is USB charging. There has been a push in recent years to standardize on USB chargers for cell-phones. There are a number of organizations involved including the USB-IF</a> and OMTP</a>. Two of the specification available include: [http://internal.omtp.org/Lists/ReqPublications/Attachments/59/OMTP CCLDC V1.1 Final.pdf](http://internal.omtp.org/Lists/ReqPublications/Attachments/59/OMTP CCLDC V1.1 Final.pdf)</li> http://www.usb.org/developers/devclass_docs/Battery_Charging_V1_2.zip</a></li> </ul> There is also a Chinese standard titled the “Telecommunications Industry Standard of the PRC“ that has been instrumental in influencing these standards. The fundamental problem is a battery powered device needs to know if it is plugged into a computer USB port, or a dedicated charger. A standard USB port on a computer is only specified to provide 500mA. Typically a dedicated charger that plugs into a wall outlet provides more current than 500mA so that the battery can be charged quicker. To differentiate between a computer USB port, and a dedicated charger, the dedicated charger shorts the D+ and D- USB signals together with a resistance of less than 200 ohms (specified by USB-IF). One of the goals of these standards is that any charger can be used with any battery powered device. The next question is how do we handle the case where every USB charger has a different rated current? How can a device that charges its battery at 1.7A be compatible with a charger that only outputs 0.7A? One theoretical solution would be for the device to query the current capacity of the charger and then only use that much current. It turns out a much simpler approach is used. The USB-IF provides the following chart in the above specification: </figcaption></a></figure> As long as the charger outputs power in the Required Operating Range, the battery powered device must be able to use whatever power is available to charge its battery. If the device uses more current than the charger can supply, the charger simply goes into a constant current mode and the battery charges slower than it would with a higher capacity charger. Thus we have a very simple scheme where we can theoretically use any charger with any device. As a simple test, I charged a Nokia N900 phone (which came with a 1.2A charger) with a smaller LG charger that is rated for 0.7A. I monitored the LG charger a couple times during charging to make sure it was not getting hot. It seemed to charge the N900 battery just fine. When purchasing after-market USB chargers, it is sometimes difficult to determine if they have the USB D+ and D- lines shorted together. In once case I purchased a car USB charger that did not work with my phones, but was able to “fix” it by disassembling it, and putting a solder blob between the D+ and D- signals on the USB connector. </a> </a> </a> While shorting the D+ and D- pins usually makes a charger work, it is nice to know if the charger is compliant with the USB-IF specification and is designed to work in the constant current mode without shutting down, catching fire, etc. Git Overview Screencast 2011-03-17T00:00:00+00:00 This screencast</a> (use Firefox to view the screencast) provides an overview of the Git version control system. There are 3 features of Git that are especially interesting: many repositories (vs. one large repository)</li> distributed development</li> cheap branches</li> </ol> The fundamental driver for better tools is increasing system complexity. More and more we are required to manage and integrate more third party software, work in distributed teams, and more effectively re-use the software we do have. Git is a tool that helps you accomplish these goals. Git and Why Multiple Repositories 2011-03-17T00:00:00+00:00 This is part of an ongoing series of articles</a> on the Git version control system. This article discusses the trend in software configuration management toward multiple repositories, rather than one large repository. In the past when many companies used Subversion or comparable systems, there was typically one huge company repository (I’ve seen them in the 10’s of GB in size) that held a hierarchical tree of all the company source code. This worked fairly well, and is comfortable in that the organization is very similar to a file system directory structure. However, this model is not very flexible in that it does not have a consistent way to re-use components between different projects. Some people simply copy source code. Some have a “common” project that is included in all their other projects. Subversion externals can be used. With Git, typically a separate repository is created for each software component. There are perhaps several reasons for this, but one reason is that Git simply does not scale to huge multi-GByte repositories. However, this turns out to be a blessing in disguise as I think it is a better model in many cases. What we end up with is more of a catalog of software components rather than a rigid hierarchy. </a> There is much emphasis these days on modular, re-usable software components (Object Oriented Programming, plugins, etc.). Why not keep things modular at the repository level? Another example of this type of organization is the Internet itself. It is not a hierarchy of information, but rather a flat system that is organized by hyperlinks. One of the benefits of organizing your source code this way is that it encourages clean boundaries between software components. Each software component needs to stand on its own without being propped up by header files located in an unrelated source tree 3 levels up in the directory hierarchy. This type of organization forces us to make better use of standard build system practices. How do you implement this type of repository infrastructure? Build systems such as OpenEmbedded, Gentoo, and the Android build system manage this fairly well. However, Git also includes a feature named “submodules” that provides a mechanism for one repository to reference source code from another. What you end up using really depends on your needs, and what you are trying to accomplish. A screencast</a> is also available that covers this topic. C++ callbacks to member functions 2010-12-16T00:00:00+00:00 How to properly do callbacks to C++ member functions is something that has intrigued me for some time now. There are a number of solutions, none of which I really liked. But now with the std::tr1::function and std::tr1::bind functions, there appears to be a clean solution. In the past, if you wanted a callback to a C++ member function a typical approach was to create a static function wrapper as described here</a>. However, this is ugly, and a step backwards from the capabilities of plain old C. std::tr1::function is a function wrapper that can be used to create function objects for a number of scenarios. We are interested in representing class member functions. #include <tr1/functional> struct X { int foo(int); }; std::tr1::function<int (X, int)> f; f = &X::foo; X x; f(&x, 6); </code></pre> f() is now a simple function representation that can be passed around, etc. However, what we don’t like is the reference to the struct X type, which makes it much less generic. This is where the std::tr1::bind function comes in. With bind, we can dynamically bind one function to another, and re-arrange the arguments, etc. So to make a generic portable function that points to X::foo(), we can do something like the following. #include <tr1/functional> struct X { int foo(int); }; std::tr1::function<int (int)> f; X x; f = std::tr1::bind(&X::foo, &x, _1); f(6); </code></pre> Now, f() is a very generic function that could be used in various interfaces without specifically referencing struct X. A more complete example of how callbacks might work is given below: #include <iostream> #include <tr1/functional> struct B { std::tr1::function <void ()> _callback; void reg_callback(std::tr1::function <void ()> callback) { _callback = callback; } void unreg_callback() { _callback = NULL; } void process() { // simply calls callback if (_callback) _callback(); } }; struct A { B * _b; A(B * b) : _b(b) { std::tr1::function<void()> callback; callback = std::tr1::bind(&A::callback, this); b->reg_callback(callback); } ~A() { _b->unreg_callback(); } void callback() { std::cout << "A::callback called\n"; } void process() { _b->process(); } }; int main() { B * b = new B(); A * a = new A(b); // the following is example where object b calls // back into a method // in object a a->process(); // now delete object a, and verify the callback // in B does not crash or still get called delete a; // now b should not execute the callback b->process(); } </code></pre> In the above example, the callback in the object a will be called once. Yocto and OpenEmbedded 2010-12-10T00:00:00+00:00 Recently, I attended an Embedded Linux summit hosted by the Linux Foundation to discuss the Yocto project. Overall, I thought the meeting was very positive and constructive. Having met and discussed at length the methods and goals of the Linux Foundation with some of their people, I’m impressed with their approach. They are there to help Linux succeed in the Embedded space in any way they can. It is my understanding that the Yocto project has the goal of making Linux much more accessible and usable in embedded systems, and improve efficiencies. While the OpenEmbedded project has served many of us well for many years, we all can readily understand there are some deficiencies. These can be overcome by experienced developers, but there are a number of areas that can obviously be improved. Not only are we concerned with making it easier for new users to use Embedded Linux, I think there are areas where we can drastically improve the efficiency of implementing embedded Linux systems for those who are experienced. It was stated once that tools implemented must be useful to both the new developer as well as the experienced developer. It should be noted that building an embedded Linux system is an inherently complex undertaking, and although we can improve tools and processes to make it more efficient, and somewhat easier, in the end it is still a very complex problem, and will require considerable skill to be an effective embedded Linux developer. There is no substitute for experience, and developer skill. Just as we would not slap a fancy GUI on top of a robotic surgery instrument, and tell a novice to have at it, likewise it is still going to require considerable engineering skill to be effective in developing Embedded Linux systems. But, if we improve the base technologies, and tools, we will spend less time messing with these and doing the same things over and over, and will have more resources toward implementing new things. One example of the pain experienced in a typical OpenEmbedded system is getting Oprofile to run. Oprofile requires that ARM system be built with frame pointers (OE defaults to not), and that you have symbols in your binaries. Figuring out how to build a reasonably sized image with the needed symbols might be a 1-2 day process for me. Then there is the issue of kernel symbols, etc. I’m sure many other developers go through the same steps, but as many of us are paid to develop products on tight schedules, we don’t have the time to polish the Oprofile integration and tools. As an extension of this, the application development story with OpenEmbedded is not all that great. Yes, we can generate a binary SDK, but again it may take a couple days of messing around to figure out the process, and get all the necessary libraries, etc. Then you still have the problem that application developers want Eclipse/Qt Creator integration, etc. Again, this can all be done, but takes time, and many people are doing the same things over and over again. The Yocto project seems to have two primary thrusts: stabilize a subset of the OpenEmbedded (called Poky) metadata and make it very high quality.</li> improve tools associated with the build process, and general embedded Linux development.</li> </ol> One thing the OpenEmbedded project has historically lacked in the past is industry funding and ongoing developer commitment. The OE project is characterized by a rotating group of developers. When someone needs OE for a project, they jump in and do a lot of work on OE, then they disappear again once their need is met. We have very few developers who work on the project consistently for long periods of time. This has been positive in some ways in that we have a very large number of contributors, and a very open commit policy. There are so many patches coming in right now, we can’t even keep up with processing them. Week after week, we have a healthy number of committers and changesets. The community behind OE is rather astounding, and it is amazing how OE succeeds almost in spite of itself as a self organizing project without much organization. In the past the OpenEmbedded and Poky projects have existed as two independent trees, and things were shared back and forth manually. This works fairly well as recipes are self contained, and generally can simply be copied from one tree to another. However, some of the core recipes are fairly complex, and if they start to diverge, then sharing information can get more difficult. It seems the vision that is emerging is that Poky could become the core component of OpenEmbedded. Poky would contain a very controlled subset of core recipes that must remain at a very high quality level. OpenEmbedded could then expand on this (through the use of layers) to provide the breadth and scope it has traditionally provided. We may bemoan the lack of quality in the OpenEmbedded project as it has 1000’s of recipes, and many of them have rotted, etc. But as a consultant helping people implement products, I still find considerable value in this. For example, one of the products I support uses the HPLIP drivers from HP. Yes, the recipe is broke every time I go to use the latest version, but with a little work I can get it working again. Having something to start with provides value. The same is true for the geos mapping libraries. Very few people are going to use geos, so it will never be in a Poky type repository, but some of us do use geos, so having a common place like OpenEmbedded to put recipes like this is very important. Using Poky as the core of OpenEmbedded seems like a win-win. We are relieved of some of the burden of maintaining core components (like compilers, C library, tool integration, sdk generation, etc), but we can still have a very open culture, and provide a wide scope of platform, library, and application support that we have historically provided. Richard Purdie is poised to become the Linus Torvalds of Yocto, and if OpenEmbedded choses to base on Poky, then the Torvalds of the OpenEmbedded core. I am personally fine with this as I don’t know anyone else who has contributed so much time and has the same level of skill in the OE/Poky area. He has proven over time to be levelheaded and able to deal with difficult situations. Also, as a Linux Foundation fellow, he is positioned to be company neutral, which is very important for someone in this position. Yocto is using a “pull” model similar to the Linux kernel for accepting changes. It is planned to put maintainers in place for various subsystems. With the goal of providing a very high quality subset of embedded Linux technologies, it seems this makes a lot of sense. If OpenEmbedded choses to base on Poky, there is no reason OpenEmbedded can’t continue to still use the push model for its layer that has worked well in the past. But, as we see patches languishing in the patchwork, perhaps a pull model might actually be more efficient at processing incoming patches so they don’t get lost. This is also an area where layers might actually help in that we have dedicated people committed to processing patches for each layer. One of the problems with the OpenEmbedded project with its 1000’s of recipes, the structure is very flat, and its fairly difficult to divide it up into areas of responsibility. So going forward, it seems that if OpenEmbedded can somehow use Poky as a core technology (just like it uses bitbake, etc), then we could have the best of both worlds. With Poky, we get strong industry commitment and dedicated people working on making the core pieces better, and the tools better (50 engineers are already dedicated). This is something those of us developing products don’t really have time for and we are not currently funded to do this. With OpenEmbedded we can still keep our broad involvement, and vast recipe and target collection. Yes, the quality of OpenEmbedded will be lower, but it still provides a different type of value. Poky provides the known stable core, and OpenEmbedded is just that–it is “open”, and provides the broad menu of components that can be used. Over time this could evolve into something different, but for now it seems a good place to start. The last thing to consider is the OpenEmbedded brand. It is recognizable from the name what it is (unlike Yocto, and Poky). It has had broad exposure for many years. It has historically been very vendor neutral with very little corporate direction or influence. From this perspective, it seems the Yocto project can benefit greatly from having an association with the OpenEmbedded project. This topic was discussed at the summit, and there was general consensus on the strong awareness of the OpenEmbedded brand, as well as an openness to how branding for the Yocto core build system might look. Qt Developer needed 2010-11-05T00:00:00+00:00 We are currently in need of a Qt developer to help on a project. If you are interested, please contact us</a>. Thanks. OpenEmbedded srctree and gitver 2010-10-11T00:00:00+00:00 Recently an OpenEmbedded</a> class name srctree became usable. The srctree.bbclass</a> enables operation inside of an existing source tree for a project, rather than using the fetch/unpack/patch idiom. The srctree.bbclass in combination with the OpenEmbedded gitver.bbclass</a> and git submodules provides a very interesting way to build custom software with OpenEmbedded. One of the classic problems with OpenEmbedded is how do application and kernel developers use it. While OpenEmbedded excels at automating image builds, it is less friendly when used as a a cross development tool. Historically there are several options for iterative development: develop in the working directory: cd /work/arm…/; ../temp/run.do_compile …</li> a variation of #1: bitbake -c devshell </li> manually set up an environment that uses the toolchain generated by OE. As example see this script</a> in the BEC OE template</a>.</li> a variation of #3: export a SDK that includes a toolchain and libs</li> </ol> While the above solutions work OK, the process can be a little cumbersome. Unless your OE recipe pulls software directly from the TIP of a SVN repository, you may have to manually update the recipe after you make changes, create patch files, etc. There is also the problem that if your recipe fetches the latest from SVN, it drastically slows down the recipe parsing as it has to check the repository for a new version every time the recipe is processed. The optimal solution would to be to simply check a software component out of a version control system, and build it directly using OpenEmbedded. Icing on the cake would be if the package generated would automatically extract version information from the version control system. This would facilitate iterative development for software components that need to be cross compiled. Although srctree can be used with any directory of source code, it really works best with a git repository. The gitver.bbclass provides a GITVER variable which is a (fairly) sane version, for use in ${PV}, extracted from the ${S} git checkout, assuming it is one (text from recipe). gitver uses the ‘git describe’ command to extract the last tag and uses that for the version. The best way to illustrate the use of these tools is an example: https://github.com/cbrake/autotools-demo</a></li> </ul> The easiest way to try this is clone the above project in your openembedded/recipes directory: $ cd openembedded/recipes $ git clone https://github.com/cbrake/autotools-demo.git $ cd autotools-demo $ git describe 1.1 $ git tag -l 1.0 1.1 </code></pre> Notice that git describe simple returns the latest tag. The recipe</a> can be located in the same directory as the source code and has the following contents: # recipe to build the autotools-demo in the working tree inherit srctree autotools gitver PV = "${GITVER}" </code></pre> Can’t get much easier than that! If you build the recipe, you end up with a package: $ bitbake autotools-demo $ ls tmp/deploy/glibc/ipk/armv5te/autotools-demo_1.1-r0.6_armv5te.ipk </code></pre> Now, what happens if you make changes and commit them? $ cd .../autotools-demo $ (make a change and commit) $ git describe 1.1-1-gfbc1ecc (notice the count and hash automatically appended) $ (make another change and commit) $ git describe 1.1-2-g7ad3715 (notice the count is now 2) </code></pre> If we bitbake the recipe now, we end up with a packaged named: tmp/deploy/glibc/ipk/armv5te/autotools-demo_1.1-2-g7ad3715-r0.6_armv5te.ipk </code></pre> The gitver class in OpenEmbedded automatically takes care of creating a usable PV (package version) that always increments. So in summary, srctree and gitver give developers a convenient way to handle custom components that change often in an Embedded Linux build without increasing parse times, requiring manual tweaks to version numbers, or creating a separate workspace for each version of the application that is built. As practices such as continuous integration become more common, OpenEmbedded features like this are increasingly needed. An added benefit is that the OpenEmbedded recipe can now be stored in the same location as the source code. Perhaps in the future, most applications will include an OpenEmbedded recipe as part of their source code and git submodules could be used to simple populate the components you want to use. 2017-11-08 update: A tool named devtool</a> is now the preferred way to do much of the above. Qt Creator for C/C++ development 2010-08-07T00:00:00+00:00 Recently, I’ve been evaluating Qt Creator</a> for general C/C++ development. I’m currently involved in the development of a rather large C++ application that is approaching 200,000 lines of code and 1000 source modules. In the past, I’ve typically used Vim for editing, and Eclipse as a gdb front-end when needed. Qt Creator is a rather new tool developed by Nokia’s Qt team. What initially attracted my attention was that one of the reasons the project was started was no existing tools effectively handled the Qt codebase, which is quite large. Things I like about Qt Creator: it works fairly well with any C++ Make based project. This includes projects built with autotools as well as the Qt Qmake build system.</li> easy to import existing projects</li> it is very fast. Indexing 200,000 lines of code happens in around a minute or so.</li> Provides a Vim compatibility mode that actually works</li> provides fast methods for finding files, symbols, and seeing where symbols are used</li> did I mention yet it is fast?</li> </ul> I also recorded a screencast that demos Qt Creator with a large project</a> (can be viewed in firefox). As always, I’m interested in what others find interesting in this or other tools. Future efforts will be to use Qt Creator to build and remotely debug ARM binaries – I am interested in what others have done in this regard. If you do try Qt Creator, I recommend the latest pre-release snapshot</a>. Embedded Linux Presentation in Cleveland 2010-07-26T00:00:00+00:00 Chris Cole</a> and I will be giving a presentation</a> in Cleveland Tues evening (2010-07-26). Stop by if you are interested. Git submodules: what to do when you commit to (no branch) 2010-06-17T00:00:00+00:00 When using git submodules, eventually you’ll run into a situation where a workspace has checked out a submodule as (no branch), and you unknownly make modifications and commits to (no branch). Now what? It turns out Git has a very useful feature named reflog. Reflog keeps a local time based log of all activity. That means that as long as you do not garbage collect your workspace, you can go back to any point in time and get any commit that was made. This is useful in a number of scenarios, but lets consider the case where we made two commits to a submodule in the (no branch) state. We are then ready to push our changes to the upstream repo and we get: cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git push bec HEAD error: unable to push to unqualified destination: HEAD The destination refspec neither matches an existing ref on the remote nor begins with refs/, and we are unable to guess a prefix based on the source ref. error: failed to push some refs to 'git@git.bec-systems.com:openembedded.custom' </code></pre> If we do a git log, we find we are on the infamous (no branch). cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git branch * (no branch) master </code></pre> At this point we may wonder what to do. How do we get these changes onto the master branch? Using git reflog, we can view all commits and changes to our local repository based on time. cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git reflog 6d195b3 HEAD@{0}: commit: add recipe for qt-embedded image 4cf1bd6 HEAD@{1}: commit: add touchscreen utilities bcc673c HEAD@{2}: checkout: moving from master to bcc673c9824e916a7ec6c04f60a17116af539423 56d9def HEAD@{3}: commit: add more features 5286f87 HEAD@{4}: commit: rename debug image 90896de HEAD@{5}: pull : Fast-forward 5cc3f3b HEAD@{6}: checkout: moving from 90896deb2a25f0d40b1e6305a0ad994f932f22d8 to master 90896de HEAD@{7}: checkout: moving from master to 90896deb2a25f0d40b1e6305a0ad994f932f22d8 </code></pre> Git will keep our keeps commits to (no branch) even though we might switch to another name branch. And using git reflog, we can easily find these commits and merge them to the master branch. cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git checkout master Previous HEAD position was 6d195b3... add recipe for qt-embedded image Switched to branch 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git reflog 56d9def HEAD@{0}: checkout: moving from 6d195b3909553b04cba3d3ac182076a2c6b866b2 to master 6d195b3 HEAD@{1}: commit: add recipe for qt-embedded image 4cf1bd6 HEAD@{2}: commit: add touchscreen utilities bcc673c HEAD@{3}: checkout: moving from master to bcc673c9824e916a7ec6c04f60a17116af539423 56d9def HEAD@{4}: commit: add more features 5286f87 HEAD@{5}: commit: rename debug image 90896de HEAD@{6}: pull : Fast-forward 5cc3f3b HEAD@{7}: checkout: moving from 90896deb2a25f0d40b1e6305a0ad994f932f22d8 to master 90896de HEAD@{8}: checkout: moving from master to 90896deb2a25f0d40b1e6305a0ad994f932f22d8 cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git merge 6d195b3 Updating 56d9def..6d195b3 Fast-forward recipes/images/bec-debug-image.bb | 38 +++++++++++++++++++----------- recipes/images/bec-qt-embedded-image.bb | 8 ++++++ 2 files changed, 32 insertions(+), 14 deletions(-) create mode 100644 recipes/images/bec-qt-embedded-image.bb </code></pre> At this point, the changes have been merged to the master branch, and we can now push to our public repository: cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git push bec HEAD Counting objects: 14, done. Delta compression using up to 8 threads. Compressing objects: 100% (9/9), done. Writing objects: 100% (10/10), 1.13 KiB, done. Total 10 (delta 3), reused 0 (delta 0) To git@git.bec-systems.com:openembedded.custom bcc673c..6d195b3 HEAD -> master </code></pre> Git reflog can also be used to restore commits that you decided at one point to throw away with “git reset --hard HEAD~2”, etc. As long as you don’t garbage collect the repo, they are simply dangling state and always still exist. This is just another example of the ways git can be very confusing, but also very powerful if you understand how it works. This screencast</a> covers the operations in this article. Installing OMAP3 images on a SD card 2010-06-14T00:00:00+00:00 This article and screen-cast is a continuation of that last couple posts describing the BEC OE build template</a>. The purpose again for a build system is to automate tedious manual tasks, and in doing so, we end up documenting how the build system works. Having a good build system is important during product development so that you have an easy, repeatable process for building and deploying images. One of these tasks is setting up a SD card and installing images to the SD card that an OMAP system can boot from. This screencast</a> goes over the script and makefile targets that are used to automate this process. In summary, the steps to set up a SD card and install an image using the build template are: cd </li> determine the SD card device name (/dev/sdX)</li> unmount any file systems mounted on the SD card</li> sudo ./scripts/omap-sd-partitions.sh /dev/sdX (make sure you have the right device!!!)</li> make omap-install-boot</li> make omap-install--rootfs</li> </ol> Previous articles in this series: OpenEmbedded build template</a></li> Creating a Custom OpenEmbedded Image</a></li> </ul> OpenEmbedded Build Template posted 2010-06-08T00:00:00+00:00 A password protected version of this article accidently got pushed into the feed before it was finished, so this post is simply to indicate the following article is now available. Sorry for the confusion. /637/oe-build-template/</a> Creating a Custom OpenEmbedded Image 2010-06-08T00:00:00+00:00 In this article screencast, we’ll demonstrate how to create a custom Linux OS image using the OpenEmbedded build system. This demonstration builds on the earlier article about using the BEC OE build template</a>. The OpenEmbedded build system is similar to Linux distributions in that you can select from a wide array of components to install. One of the big differences is you can select these components when building the OS image, instead of after you have installed the “standard” image. This screencast</a> demonstrates how to set up a custom image (can be viewed with Firefox). A related article</a> also covers this procedure. In summary, the steps include: Create an image recipe (typically in your meta data overlay). An example is located here</a>. An existing image can be referenced for simplicity.</li> Figure out what packages need to be added. This can be done by browsing the recipe tree</a> to see what is available.</li> Bitbake the recipe.</li> Look at what packages were generated. In this example they are found in the build/angstrom-2008.1/tmp/deploy/glibc/ipk/armv7a directory.</li> Add the relevant package names to your image recipe.</li> </ol> As the OpenEmbedded project includes over 6000 recipes, it gives you a big head start in including standard components in your Embedded Linux build. Leveraging these existing components is why we use Linux, and having a way to easily build and deploy these components to your target hardware is what OpenEmbedded is all about. OpenEmbedded Build Template 2010-06-05T00:00:00+00:00 Setting up an OpenEmbedded build is a fairly simple process if you carefully follow the instructions</a>. There are also a number of scripts available that automate the setup such as the OpenEmbedded Tools for Eclipse</a>, the Angstrom setup scripts</a>, the KaeilOS Openembedded Manager</a>, and I’m sure there are many more (feel free to add in comments section). As we have helped a number of clients use OpenEmbedded in commercial projects, we have the following requirements for a build template: The build needs to be completely automated such that only one step is required to build images.</li> We need to be able to lock down versions of various components.</li> Needs to be as simple as possible.</li> </ol> As we continue to think about creating a build template, the primary reason for creating one is to get repeatable builds with very little effort. The build template serves as documentation; how did we build the 1.10 release. A good build system allows any developer to create any release on about any machine at any time. A good build system does not depend on the “golden build machine.” We have been using makefile wrappers around OpenEmbedded for several years now to accomplish much of this. These wrappers were typically stored in a Subversion repository with the rest of the project files. However, now that most open source projects have moved to Git repositories, there are significant advantages to using Git for the build system. Git has a feature called submodules which allows you to embed a git repository inside a source tree, always pointed at a particular commit. In concept, this is perfect. In practice, Git submodules are sometimes difficult to use, but considering the functionality provided, I think it is a reasonable trade-off. The use of Git encourages a different development paradigm. Typically when people use Subversion within a company, there is one repository that stores everything. Subversion is actually quite good at this. I’ve run into a number of companies who have repositories many GB in size that have been running for many years. The stability is remarkable. Git does not scale in this way, so people tend to set up a separate repository for each component, application, etc. While there may be some disadvantages to this model, there are also significant advantages, and we can learn some of these by observing Open Source Software (OSS) development. Just as we strive for modularity in software development where each class can stand on its own and be generally useful, OSS maintains this dynamic at the project level. Each OSS project moves at its own pace, largely decoupled from other projects. This dynamic forces a level of organization that might be neglected otherwise. Likewise, in any project, if each major component is stored in its own source code repository, there will also tend to be less coupling, willy-nilly including of header files from other components, etc. It helps people think in a more modular manner at the component level. A component may be the Linux kernel, bootloader, each application in the system, etc. A similar concept may be thinking about storing documents in hierarchical filesystem directory structure versus documents in a Wiki, which are stored in a flat structure and then organized by cross linking. Yes, there are times to use a directory structure, but flexibility of a linked organization system (the essence of the Internet) is hard to beat. Git submodules (or similar concepts) provides a similar linking method of organization. With this in mind, we look at a build system as a way to collect a number of independent components and build them in a repeatable fashion. While this may seem obvious to anyone somewhat familiar with OpenEmbedded, our vision is to extend this concept to easily facilitate product development so that we can move quickly and efficiently, but yet in a controlled and reliable way. The first step is a new build template that is available here</a>. The following video</a> provides a short overview of how to use the build template (can be viewed in Firefox). </video> The functionality provided so far is fairly simple and provides a way to quickly set up a build environment. With Git submodules, the versions of Bitbake and the OpenEmbedded are locked down. Submodules are also used to provide an array of components to choose from, and the user initializes only the submodules they are interested in using. The first example of this is providing both the recipe meta data from the OpenEmbedded project or the modified version from Gumstix. In the future, examples will be provided for doing custom application and kernel development using the OE srctree and gitver classes. Feedback is welcome; please let me know if you have questions or suggestions. MeeGo Review 2010-06-02T00:00:00+00:00 As we evaluate various technologies that might be applicable in embedded systems, MeeGo</a> is the subject of this article. MeeGo is a collaboration between Intel and Nokia, and is replacing the Moblin and Maemo efforts. For this review, MeeGo was installed to a USB flash disk and booted on a Asus EEPC. This was quite trivial to do, and the instructions on the MeeGo website are very good</a>. The following video</a> (can be viewed in Firefox) provides a quick overview of the MeeGo Netbook UI. </video> Overall, MeeGo looks interesting. Hopefully with the collaboration between Intel and Nokia, there will emerge a number of components that become “standards” for embedded systems such as ways to manage wireless networks, cell phone radios, etc. It would also be nice to have better options for implementing GUI’s on embedded devices with small screens. While the MeeGo user interface is nice, it is not a radical departure from desktop applications. Maemo (used on the Nokia N900) is more advanced in that it has optimized widgets for a small screen. With Maemo, all applications run in a true full screen mode, which is generally desired for devices with smaller screens, or industrial devices where we want to keep user confusion to a minimum. MeeGo applications seem to run full screen, but you can still drag them around, and some applications like the terminal do not start full screen. It seems there is considerable interest in providing distributions based on MeeGo. Novell has announced</a> it plans to provide MeeGo based distributions for netbooks, and Intel claims</a> there are a number of other companies planning to build on top of MeeGo. Gumstix Overo review 2010-02-08T00:00:00+00:00 Based on the interest and number of embedded modules currently available, it appears that the OMAP3 CPU from TI will be very popular in the general purpose embedded Linux market. One of the OMAP3 modules available is the Overo from Gumstix</a>. As the company name suggests, this module looks about like a stick of gum, but smaller, as shown in the photo below. The Gumstix module provides a lot of functionality in a very small package. It is reasonably priced, and is an excellent way to quickly create a product that has advanced functionality such as a high powered CPU (OMAP3), high speed USB, DSP, 3-D graphics acceleration, etc. For an example of the OMAP3 performance compared to previous generations of ARM processors, see this article</a>. The module contains all the core components, and can then be attached to a custom baseboard using the two high density connectors shown in the above photo. There are 70 signals on each of the two connectors. BEC provides a spreadsheet</a> of the Overo signals to assist in designing a custom baseboard. Due to the high level of component integration, the Overo has very few components on the module. As shown in the below photo, there are really only 3 major visible components. The flash and RAM is stacked on top of the CPU. The PMIC contains all the required power managment circuitry, as well as Audio and USB interfacing functionality. The Overo module must be used with a baseboard. Gumstix provides several development baseboards. For production, a custom baseboard is typically created. The Gumstix baseboards are reasonable cost, so they could be used for prototyping or low volume production if they have the required functionality. Below is a photo of the Summit baseboard that provides DVI video out, audio, USB host and OTG, and an expansion connector with a number of signals. Below is the Palo baseboard from Gumstix that can be used to interface with a LCD display. The Palo board contains a resistive touch screen controller, and circuitry required to interface with a LCD. The Overo does not include an Ethernet controller, so you typically use a USB-Ethernet adapter for development. Below shows a typical development setup. One of the things about the OMAP3 that makes it very attractive for embedded development is the amount of software support, and the number of development and production solutions available. The OMAP3 is very well supported by the OpenEmbedded project, and TI is very active in making contributions to various open source projects. This greatly increases the quality and availability of advanced software functionality needed to support a complex system like the OMAP3. There are many options for software development with the OMAP3 systems. Below is a photo of a very simple Qt application. GTK+ and Enlightenment are other popular GUI toolkits. With 256MB of both flash and RAM, this is more than enough memory for most embedded applications, and provides plenty of headroom for adding features for future product revisions. The Overo has the CPU processing capability to run advanced GUI applications with 3-D affects, as well as advanced web application frameworks like web2py that previous generations of ARM CPUs could not effectively run. The Gumstix Overo provides an excellent value for developing advanced products. Especially for low volume products, when you compare the effort and time required to develop a full custom OMAP3 solution versus a simple Overo baseboard, using a module can provide significant advantages in terms of time to market, and development cost. The Go language for embedded systems 2010-01-06T00:00:00+00:00 As one of the things I do is evaluate new technologies for embedded systems, the Go language</a> from Google is an interesting development. I have been looking for a better “system” language for some time. What I mean by a better system language is one that is nearly as efficient as C, does not require a large runtime, has a fast start-up time, and yet supports modern language features. It is interesting that the constraints for very high performance server applications are often similar to those found in embedded systems in that applications must make very efficient use of memory and CPU cycles. This is why C++ is still used a lot by Google and is also popular in embedded systems, but is not as common in applications where CPU power is plentiful compared to the resources to implement the application such as business applications. One area of divergence is the focus on multi-core for high end server applications, but in the next few years, multi-core CPU’s such as the Atom and ARM Cortex-A9 will likely be common in low power embedded systems. There are other options. Vala</a> is very nice to use, is efficient, offers advanced language features, and extensive language bindings. Vala is being used to implement some of the http://freesmartphone.org</a> software stack. C/C++ are the old standby languages, but are rather tedious to program in compared to some of the newer languages in that source and header files are separate, and memory management is manual. My interest in Go at this time is for ARM systems, so I ran a few experiments. From what I can tell, there does exist an ARM Go compiler that is able to produce ARM code, but it currently lacks support for EABI soft floating point</a>. The lack of floating point support is pretty much a show stopper for now, but it is at least a start. The following is an example: export GOARCH=arm cd $GOROOT/src/ ./make-arm.bash cd (create the following source file: hello.go) ================== package main import "fmt" func main() { fmt.Printf("Hello, world\n"); } ================= 5g hello.go (5g is the ARM compiler) 5l hello.5 scp 5.out root@n900: (copy to ARM system) ssh root@n900 ?:~# ./5.out Hello, world (modify with floating point code) ================== package main import "fmt" func main() { var f = 1.5; fmt.Printf("Hello, world\n"); fmt.Printf("float f = %f\n", f); } ================= (now run on ARM system) ?:~# ./5.out Segmentation fault </code></pre> As, you can see, floating point does not work as expected. I’m not sure yet if gogcc can be configured to compile ARM binaries. The other area where Go appears to need a lot of work is bindings to various libraries. Though it does not appear to be ready for embedded ARM systems at this time, Go will be an interesting language to watch over the next few years. OpenEmbedded development activity 2009-12-29T00:00:00+00:00 Ever since I have been sending out weekly change logs</a>, I have been impressed by the consistent amount of development activity in the OpenEmbedded project. Every week there are consistently over a dozen developers making changes. Developers come and go, but the contribution level always seems healthy. While this amount of development leads to some amount of churn and issues, this amount of development is also required to keep pace with all the new developments in the OSS space. Not having to wait 2 years for the next big release is one of the big advantages of using Linux and OpenEmbedded. Having access to the latest technology provides a competitive advantage for many products, and makes dealing with the occasional issues that come up an acceptable trade-off. Coupled with a flexible and consistent build system, OpenEmbedded is the vehicle to build advanced products. Yes, its hard. Yes, you better budget for some system software development time. Yes, you should get some help if you’re not an embedded Linux expert. Yes, you can start with some canned vendor “BSP” that seems to work and is seemingly the low-cost way to go, but eventually you’ll hit a brick wall where you need some bit of functionality that is not there, and then you’ll start up the exponential curve of dumping lots of time into trying to make something work with little progress forward. It is much better to count the cost up front, and do things right. OMAP3 Resume Timing 2009-12-07T00:00:00+00:00 One of the most common power management modes for ARM processors is the suspend mode. In this mode, peripherals are shut down when possible, the SDRAM is put into self-refresh, and the CPU is placed in a low power mode. A useful bit of information is to know how soon the system can respond to a resume (wake) event. It turns out the answer to this question depends on where you want to do the work. In the first test, I created a simple application that continuously toggled a GPIO. This is an easy way to tell determine with a scope when the application was running. I then measured the time between the wake event, and when this application signal started toggling. It was a consistent 180ms. For the next test I simply toggled a gpio in the omap3_pm_suspend() first thing after resume. This tells me roughly how fast I can execute code in the kernel after a wake event. This turned out to be 250uS. So the good news is we can run kernel code very quickly after a resume event (250uS), but it currently takes a long time until user space processes start running again (180mS). The 180ms can likely be reduced with some work, but this at least gives us a baseline. 180ms is fairly quick from a human perspective (seems pretty much instant), but for industrial devices that are responding to real-world events, then 180mS can be a long time. Linux kernel stats, and long term advantages 2009-12-01T00:00:00+00:00 I just read an interesting interview</a> with Greg Kroah Hartman. According to Greg, we add 11,000 lines, remove 5500 lines, and modify 2200 lines every single day. This rate of change is something that few organizations have the resources to match. It is interesting that Google chose to use the Linux kernel in the Android project even though they implemented most other parts of the system. Another interesting thing to consider is the consequences of the Linux kernel license, which requires many drivers to be released as GPL. While initially, this may be viewed as a disadvantage, it leads to some interesting long-term implications. The obvious effect is that most Linux driver code is maintained in the mainline kernel source. This leads to other effects. One is that things are allowed to change, and get better. The USB stack in both Linux and Windows has been re-written several times, but the difference is that in Linux the old software does not need to be maintained, as most drivers are maintained in the kernel, and can simply be modified along with core changes. Another advantage is that drivers tend to be much simpler as common code is merged. As a result, a Linux driver tends to be about 1/3 the size of a driver in other operating systems. Yes, getting driver code accepted in the mainline kernel can be difficult at times, but it is clearly the best long-term option. Linux PM: OMAP3 Suspend Support 2009-11-23T00:00:00+00:00 This article provides an overview of the Linux kernel support for the suspend state in the TI OMAP3. Power management has always been one of the more difficult parts of a system to get right. The OMAP3 power management is quite extensive. There are many levels of very granular control over the entire system. Initially, we are going to focus on a simple power state where we want to put the CPU in a low power mode (basically not doing anything), but resume very quickly once an event occurs and continue running applications where they left off. This is typically called “Suspend”. Power States</h2> We must first match up terminology between the Linux kernel and the OMAP documentation. The Linux kernel supports several suspend power states: PM_SUSPEND_ON: system is running</li> PM_SUSPEND_STANDBY: system is in a standby state. This is typically implemented where the CPU is in a somewhat static state. After all the peripherals have been shut down, and the memory put into self refresh, the CPU executes an instruction that puts the CPU into a low power state. After the next wake-event occurs, the CPU resumes operating at the next instruction. The CPU state is static in this case.</li> PM_SUSPEND_MEM: This goes one step beyond STANDBY in that the CPU is completely powered down and loses its state. Any CPU state that needs to be saved is stored to SDRAM (which is in self refresh), or some other static memory. When the system resumes, it has to boot through the reset vector. The bootloader determines that the system was sleeping, and re-initializes the system, enables the SDRAM, restores whatever state is needed, and then jumps to a vector in the kernel to finish the resume sequence. This mode is fairly difficult to implement as it involves the bootloader, and a lot of complex initialization code that is difficult to debug.</li> PM_SUSPEND_MAX: From browsing the kernel source code, it appears this state is used for “Off” or perhaps suspend to disk.</li> </ul> The OMAP3 documentation uses the following terms: SLM: Static Leakage Management. Includes “suspend” functionality.</li> Standby: OMAP3x retains internal memory and logic. (uses 7mW)</li> Device Off: System state is saved to external memory(0.590mW). This could theoretically map to the PM_SUSPEND_MEM state.</li> WFI: Wait for Interrupt. This is an ARM instruction that puts the CPU in the standby state.</li> </ul> Based on the source code in the Linux kernel, there is currently only support for the Standby state: static int omap3_pm_enter(suspend_state_t unused) { int ret = 0; switch (suspend_state) { case PM_SUSPEND_STANDBY: case PM_SUSPEND_MEM: ret = omap3_pm_suspend(); break; default: ret = -EINVAL; } return ret; } </code></pre> What kernel branches/versions support PM</h2> Kevin Hilman has been maintaining an OMAP3 power management branch at http://git.kernel.org/?p=linux/kernel/git/khilman/linux-omap-pm.git</a>. Bits are being merged upstream. Some of the things that don’t seem to be merged yet are SmartReflex support, and a number of other details. But, it seems that basic suspend/resume support is available in the upcoming 2.6.32 kernel. More Details on the Standby Mechanism</h2> Due to the granularity and complexity of the the OMAP3 power management, relevant pieces of documentation are scattered throughout the OMAP35x Technical Reference Manual. The following were helpful: Table 3-17: details the MPU subsystem operation power modes</li> Table 3-18: Power Mode Allowable Transitions. Standby mode can enter from active mode only by executing the wait for interruption (WFI) instruction.</li> SDRC, “Self Refresh Management”. The OMAP3 can automatically put the SDRAM in self refresh mode when the CPU enters the idle state.</li> MPU Power Mode Transistions: The ARM core initiates entering into standby via software only (CP15 - WFI).</li> </ul> The omap34xx_cpu_suspend function is called to put the CPU into standby mode. The actual instruction that stops the CPU is the ARM WFI (wait for interrupt) instruction. This is a little different than previous generations of ARM CPUs that used coprocessor registers. Testing</h2> With the 2.6.32-rcX kernels, suspend/resume appears to work at least from the console: root@overo:~# echo mem > /sys/power/state PM: Syncing filesystems ... done. Freezing user space processes ... (elapsed 0.00 seconds) done. Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done. Suspending console(s) (use no_console_suspend to debug) <after key press on serial console ...> omapfb omapfb: timeout waiting for FRAME DONE CLIFF: going into suspend Wake up daisy chain activation failed. CLIFF: returning from suspend Successfully put all powerdomains to target state Restarting tasks ... done. root@overo:~# </code></pre> The CLIFF messages are debug statements placed in the low-level suspend code to make sure we were getting to that point. The console is disabled early in the suspend process, so we don’t get the debug messages until after resume. The no_console_suspend kernel option can be used to leave the console enabled during suspend, but that feature does not seem to work. Summary/Future work</h2> This covers the basics of Linux Suspend/Resume support for the OMAP3 CPU. There are many more options and details such as configuring wake sources, dynamic power management while running, etc. Integrated CAN solutions for Linux 2009-11-19T00:00:00+00:00 I just received an email notification from EMS (http://ems-wuensche.com/</a>) that support for their CAN controllers is now in mainline Linux kernels. The EMS PCI products are supported in 2.6.31, and the CPC-USB product will be supported in 2.6.32. I’ve used various Linux CAN stacks in the past, but none were as well integrated as the SocketCAN</a> solution that has been merged into the mainline Linux kernel. Though I’ve not used the EMS products yet, the fact that support is in the mainline kernel source provides a good indication that they know how to do things correctly, and that the software is of good quality. The Microchip MCP2515 is another option for interfacing CAN with a Linux system, but uses the SPI bus so it typically requires a custom hardware design. The availability of a USB adapter provides an easy way for CAN to be used with about any existing Linux system including small embedded systems. From looking at the kernel source code, it appears that EMS solutions use the SJA1000 CAN controller, which has been an industry standard controller for many years. Message buffering is an important consideration when selecting a Linux CAN solution. Often in a system, the Linux computer is responsible for data processing, user interface, and other CPU intensive operations. The real-time control is often relegated to a separate microcontroller. The CAN bus is then just a convenient and robust way to get data from the control part of the system to the data processing part. The amount of hardware buffering in the CAN controller determines how quickly the system needs to respond to process the CAN data, so the CAN controller has space to receive new data. The SJA1000 provides a 64-byte receive fifo, or roughly space for 4 CAN messages. This provides a little more buffering than the 2 messages in the MCP2515. Dealing with full bandwidth CAN data in a Linux system can be a challenge with small amounts of buffering as you typically have to implement the Linux RT patch, and spend some time tuning it. Another upcoming CAN solution that looks nice is the TI High End CAN Controller (HECC) found in the new Sitara CPUs. This controller provides 32 hardware mailboxes which should provide even more buffering, which makes it even better suited for operating systems like Linux. As with any technology, there are tradeoffs, so its nice to have options. One area the EMS solution may be very useful is for prototyping and development. The ideal development flow when developing an Embedded Linux product is to development as much as possible on a PC. The EMS product allows you to easily implement SocketCAN functionality on a PC so you can talk to the rest of your system directly from a PC. The final embedded solution may be based on a more cost effective solution like the MCP2515 or the TI HECC, but being able to interface to the rest of the system directly from a PC has the potential to improve development efficiency. This is part of the “Big Win” with Embedded Linux where the same technology scales from the embedded system, to the PC, to the server. Notification at the end of builds 2009-11-05T00:00:00+00:00 I do quite a few OpenEmbedded project builds during the course of a week. This process usually takes 3-5 minutes. That is just enough time to get distracted doing something else and forget about the build until an hour later when you realize – oops, I was supposed to send out a release email once the build was finished and uploaded. It occured to me that it would be nice if my computer played a distinct sound at the end of the build. With Linux this is incredibly easy: wget http://upload.wikimedia.org/wikipedia/commons/7/76/Ding_Dong_Bell.ogg</li> sudo mv Ding_Dong_Bell.ogg /usr/local/</li> sudo aptitude install cplay</li> sudo echo “cplay /usr/local/Ding_Dong_Bell.ogg” > /usr/local/bin/bell</li> sudo chmod 775 /usr/local/bin/bell</li> </ul> Now, when I have a long build, I simply do something like: bitbake my-image; bell </code></pre> There is probably something better than cplay that does not start a UI – any suggestions? A quick serial logging application 2009-10-21T00:00:00+00:00 Recently when debugging a device connected to a rs485 bus, I needed a simple application to dump the raw data coming over the rs485 bus. Minicom does all kinds of weird terminal stuff, plus it will not display binary data, so that was out. While looking for serial analyzer programs for Linux, and pondering writing my own simple C application, it occurred to me to use pyserial. With a few lines of code, I got exactly what I needed: import serial import sys import struct if len(sys.argv) != 3: print "Usage: serial-dump.py <serial port> <baud rate>" sys.exit(-1) print "Dumping data on %s" % (sys.argv[1]) ser = serial.Serial(sys.argv[1], sys.argv[2], timeout=0.3) while(1): buf = ser.read(500) if len(buf) > 0: print "Read %i bytes:" % (len(buf)), for i in buf: value = struct.unpack('B', i)[0] print "%02x" % (value), print "" </code></pre> The full source is available here</a>. This could obviously be improved to do true blocking reads, automatically parse protocols, implement a gui, etc. Ideas for efficiently reading data using pyserial is available in a previous post</a>. Another example of using a scripting language for quick tasks is /226/a-really-nice-hex-calculator/</a>. Apache and how to correctly use NameVirtualHost 2009-10-16T00:00:00+00:00 As I often get involved in server administration (SVN, git, redmine, etc setup), I deal with Apache on a regular basis. For simple configurations, the default Ubuntu/debian config works well. However, for a more complex setup with virtual hosts, multiple IP addresses, and SSL support, it is common to run into the following message: mixing * ports and non-* ports with a NameVirtualHost address is not supported, proceeding with undefined results Googling does not return a clear explanation to this problem, so that is why I’m writing this article. The impulsive approach of randomly changing things does not get you too far – especially with the Debian/Ubuntu Apache configuration that is composed of seperate files for each virtual host. After reading the documentation, things become clearer: Note that the argument to the <VirtualHost></code> directive must exactly match the argument to the NameVirtualHost</code> directive. And, by considering the fact that the seperate virtual host config files really just get concatenated into one big file, it makes sense that you can’t have more than one NameVirtualHost entry. Ubuntu by default puts the NameVirtualHost in the default virtual host file, which gets soft linked to sites-enabled/000-default. I presume the ‘000’ means it gets loaded first. I find it much more intuitive to put the NameVitualHost directives in the /etc/apache2/httpd.conf file. This way it is clear that these entries are common to all virtual host entries. So what I end up with is: /etc/apache2/httpd.conf NameVirtualHost 74.208.184.45:80 NameVirtualHost 74.208.184.46:80 NameVirtualHost 74.208.184.45:443 </code></pre> /etc/apache2/sites-available/bec-systems.com <VirtualHost 74.208.184.45:80> ServerName bec-systems.com ... </code></pre> /etc/apach2/sites-available/bec-systems.com.ssl <VirtualHost 74.208.184.45:443> ServerName bec-systems.com ... </code></pre> /etc/apach2/sites-available/svn.bec-systems.com <VirtualHost 74.208.184.45:80> ServerName svn.bec-systems.com ... </code></pre> </code></pre> /etc/apach2/sites-available/site2.com (with different IP address) <VirtualHost 74.208.184.46:80> ServerName site2.com ... </code></pre> Again, the points to remember are: The argument to VirtualHost must match a NameVirtualHost directive</li> the NameVirtualHost directive is common, so there should only be one entry</li> </ol> Best practices for kernel development with OpenEmbedded 2009-10-13T00:00:00+00:00 A common question is how do you do kernel development with OpenEmbedded? Typically, OpenEmbedded builds a kernel by checking the source out of a git repository, or by applying patch files to a released version of the kernel. See the many recipes</a> for examples. This works very well for a Linux distribution build system, but is less than ideal for ongoing kernel development where you will have many iterations over a period of time. The OE work directory is considered transient, and should not be used for any ongoing development. Fortunately, the kernel build system is self contained, so it is a very simple matter to reference the toolchain built by OpenEmbedded and work outside the OpenEmbedded build system. Use Git</h2> Trust me on this one, just use it! It will make your life easier. There is a learning curve to any powerful tool. Another way to state this is if a tool does not do much, then it would be easy to learn. So expect to spend a little time learning to use git. With modern Linux kernel development, much of your time is spent integrating work that others are doing. For example, if you are using the OMAP3 CPU, you might want to start with linux-omap branch, integrate some patches from the OE beagleboard build, merge the OMAP PM tree, and then merge some bits from the Linux-wireless tree. This is the way embedded Linux development is done today. Git makes it possible to have many parallel developments going at their own pace, including your own. Even in your own development, you will find significant advantages to being able to quickly create and merge branches, revise commits, etc. There are several articles about git</a> on this site. Importing an OE kernel into git</h2> If you are starting with an existing OE kernel tree, then you typically need to set up an equivalent tree in your local git workspace. Typically you just look at the recipe to see how to manually reconstruct it. For example, lets consider the SRC_URI from the palm-omap1</a> recipe: SRC_URI = "${KERNELORG_MIRROR}/pub/linux/kernel/v2.6/linux-2.6.22.tar.bz2 \ http://www.muru.com/linux/omap/patches/patch-2.6.22-omap1.bz2;patch=1 \ file://defconfig" </code></pre> To set this kernel up in git, we might do the following: git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git</li> cd linux-2.6</li> git checkout -b my_branch v2.6.22</li> wget http://www.muru.com/linux/omap/patches/patch-2.6.22-omap1.bz2</code></li> bzip2 -d patch-2.6.22-omap1.bz2</li> git am patch-2.6.22-omap1</li> </ol> If the git am fails, then you might need to add a subject and author line to the patch in the following form and then retry git am. From: Name <name@company.com> Subject: omap1 patch </code></pre> Now when you do a git log, you will see the above patch applied. git apply can also be used to apply patches, but I prefer git-am as it automatically handles new files and fills in the commit message and author. Setting up the toolchain</h2> Now that you have a kernel, you must set up a toolchain so you can build it. I typically set up my kernel directory next to my OE build directory in my project space, so I can then source the following script</a> to set up the toolchain: CROSS_COMPILER_PATH=`cd ../oe/build/angstrom-2008.1/tmp/cross/armv7a/bin; pwd` BUILD_ARCH=`uname -m` OE_STAGING_PATH=`cd ../oe/build/angstrom-2008.1/tmp/staging/${BUILD_ARCH}-linux/usr/bin; pwd` STAGING_KERNEL_DIR=`cd linux-2.6; pwd` export PATH=$PATH:$CROSS_COMPILER_PATH:$OE_STAGING_PATH export ARCH=arm export CROSS_COMPILE=arm-angstrom-linux-gnueabi- </code></pre> Adjust to whatever you are building for. Now you can simply type make. The reason the OE_STAGING_PATH is added to the PATH in the above example is so that the uboot mkimage utility can be used directly from the OE build if your build target is make uImage. Installing modules</h2> Sometimes you need to re-install all modules on the target because you changed kernel version, etc. Typically OE packages up all modules and adds the one you specify to the rootfs. Because we are building the kernel outside the OE directory, OE can no longer do this. However, its still very easy to install modules from your kernel development directory: rm -rf modules_install; INSTALL_MOD_PATH=modules_install make modules_install rsync -av modules_install/lib/modules/2.6.22 root@192.168.1.115:/lib/modules/ </code></pre> The first command installs the modules to a temporary directory, and the second command rsync’s the modules to your target system. Creating a kernel recipe with your changes</h2> Once you get to the point where you want to create a recipe with your changes, you can easily export a patch, or series of patches using git diff or git format-patch. Simply add the patch files to your kernel recipe SRC_URI. You can also teach bitbake to fetch your kernel source directly from a git repository, but I’ve generally found the patch mechanism to be adequate and easier in most cases. As an example, you can create one large patch with all your changes that can be applied on the 2.6.22 released kernel: git diff HEAD..v2.6.22 > my_kernel_changes.patch </code></pre> Other ways …</h2> This is just one flow and there are many variations. There is also a new mechanism in OpenEmbedded called srctree that allows you to use OpenEmbedded in an external source tree. srctree has the potential to further streamline this type of development. As with any task, you want to use the tools that make sense for the task. OpenEmbedded makes sense for building Linux distributions, but it is not a kernel development tool. Git and the kernel build system make sense for kernel development. converting digital camera movies to ogg format 2009-09-09T00:00:00+00:00 Now that firefox (v3.5) has built in support for ogg videos, ogg is now the most convenient format for “easy to view” video files (assuming you don’t want to use a service like youtube). Some digital cameras (like my Canon) records movies in AVI format. I created a script that uses gstreamer to convert the AVI file to an ogg file for easy viewing. http://dev.bec-systems.com/svn/pub/avi_to_ogg/avi_to_ogg</a> And some of the results: /video/</a> (hint, download firefox 3.5 to view these videos) Best practices for building Gtk+ applications with OpenEmbedded 2009-09-08T00:00:00+00:00 I recentlywrote an article about best practices building Qt applications with OpenEmbedded</a>, and it occured to me that I should write an equivalent article for Gtk+ applications. The same points apply – put your application source in a SCM system, and put the install logic in the application source (read the above article). The difference is that Gtk applications typically use autotools where Qt applications use qmake to build the application. This article details how a minimal GTK+ application should be set up and built using OpenEmbedded. Application Source</h2> I created a sample GTK hello application located at: http://dev.bec-systems.com/svn/pub/gtk_hello</a>. This is a SVN repository, so you can simply “svn co” the above URI to check out the code. If you are running Ubuntu, you can install the necessary tools to build a native Gtk+ application by: sudo apt-get install libgtk2.0-dev build-essential autoconf automake pkg-config</li> </ul> To build on your x86 Linux PC, run the following steps: svn co http://dev.bec-systems.com/svn/pub/gtk_hello</a></li> cd gtk_hello</li> autoreconf -i</li> ./configure --prefix=`pwd`/install</li> make install</li> </ul> This compiles and installs the application in to the “install” directory. If you look in this directory, you will notice the application binary is installed in the “install/bin” directory. Typically, the install directory is set to /usr/bin, but in this example we set it to install so we don’t need to run “make install” as root, but yet we can verify the install logic works properly. OpenEmbedded Recipe</h2> Now that you have verified the application builds and installs properly on a x86 PC, it is trivial to build the application in OpenEmbedded. Create a recipe in your OE recipes directory named gtk-hello_svn.bb with the following contents: DESCRIPTION = "Sample Gtk+ Hello application, used to demonstrate build system" AUTHOR = "Cliff Brake <cbrake@bec-systems.com>" SRCREV = "17" PV = "0.0+svn${SRCREV}" PR = "r0" DEPENDS = "gtk+" SRC_URI = "svn://dev.bec-systems.com/svn/pub;module=gtk_hello;proto=http;rev=17" S = "${WORKDIR}/gtk_hello/" inherit autotools </code></pre> Now, run: bitbake gtk-hello. That is it! Building Linux applications is easy if you simply use the tools, whether it be autotools, qmake, etc. Too often there is the tendency to set up your own compile steps with ${CC} variables, etc. While this seems to be the simple approach at first glance (autotools is too hard), it quickly becomes unmaintainable and in the end is a lot more work than simply learning the basics of the industry standard tools. See previous autotools article</a> for more information. Best practices for building Qt applications with OpenEmbedded 2009-08-06T00:00:00+00:00 This article describes how to cross compile a Qt application (named qt_tutorial) with OpenEmbedded, and several best practices you should consider. OpenEmbedded currently includes fairly good support for building Qt</a> – both Qt Embedded and Qt X11. OE also includes a number of qt classes</a> that make building Qt applications easy. One of the main considerations with embedded Linux application development is that you keep the build system flexible so that you can easily build on a PC or for your embedded hardware. This means that hand crafted Makefiles with hardcoded paths to cross compilers do not quality. Put Your Application in a SCM system</h2> No matter what type of application you are building, it is a good idea to put your source in a SCM or revision control system and fetch it directly using OE. OpenEmbedded supports fetching sources from a number of revision control systems including Subversion, Git, Bazaar, Mercurial, CVS, etc. The not so obvious reason we do this is so you can easily check out the source code and build it on a PC as well as your target system in OE. In this example we fetch the application source from a SVN repository: SRC_URI = "svn://dev.bec-systems.com/svn/pub;module=qt_tutorial;proto=http" </code></pre> For the above repository, the direct URI to the source would be: http://dev.bec-systems.com/svn/pub/qt_tutorial/</a> Put the Install logic in the Application Source</h2> Most Linux applications support “make install”, and this is the case with autotools, and qmake (Qt’s build tool). We could put logic in the OE recipe to install the application something like the following: do_install() { install -d ${D}/${bindir} install -m 0755 ${S}/qt_tutorial ${D}/${bindir} } </code></pre> But, the problem with this approach is you can’t install the application in other environments (like a native x86 PC build) unless you are building with OE. So a better approach is to put the logic to install the application in the application source, so that in can be installed in both your PC environment, and your OpenEmbedded build. To accomplish this, you can set up the project as follows application qmake project file (qt_tutorial.pro): TEMPLATE = app TARGET = qt_tutorial DEPENDPATH += . INCLUDEPATH += . # Input HEADERS += lcdrange.h SOURCES += lcdrange.cpp main.cpp target.path = /usr/bin INSTALLS += target </code></pre> OpenEmbedded recipe (qt-tutorial_svn.bb): inherit qt4e SRCREV = "${AUTOREV}" PV = "1.0+svnr${SRCREV}" PR = "r1" SRC_URI = "svn://dev.bec-systems.com/svn/pub;module=qt_tutorial;proto=http" S = ${WORKDIR}/qt_tutorial FILES_${PN}-dbg += "${bindir}/.debug" do_install() { export INSTALL_ROOT=${D} make install } </code></pre> Now, the same mechanism is used to install the application on both a PC native build, as well an OpenEmbedded build. If you look in the Makefile generated by qmake, you see the following: install_target: first FORCE @$(CHK_DIR_EXISTS) $(INSTALL_ROOT)/usr/bin/ || $(MKDIR) $(INSTALL_ROOT)/usr/bin/ -$(INSTALL_PROGRAM) "$(QMAKE_TARGET)" "$(INSTALL_ROOT)/usr/bin/$(QMAKE_TARGET)" install: install_target FORCE </code></pre> INSTALL_ROOT can be set to force the entire system to be installed in a subdirectory. This is required for build systems that generate packages, like OpenEmbedded. To build this example, put the qt-tutorial_svn.bb file in your OE recipes tree, and run: bitbake qt-tutorial. This will fetch the source code and build a package. To run the tutorial, install the package on a system that includes Qt Embedded, and then run: qt_tutorial -qws. Gumstix Overo Pinout Spreadsheet updated 2009-07-30T00:00:00+00:00 The Gumstix Overo Pinout Spreadsheet</a> has been updated with the Palo board connector pinouts, and a few mistakes have been fixed. How to capture source changes to an OpenEmbedded package 2009-07-14T00:00:00+00:00 One task that is a often confusing to new OpenEmbedded users is how to capture changes to the source code for a package/recipe. First, lets review the progression in tools use to capture source code changes. Figure 1: Progression of tools</figcaption></figure> In days of yore, developers often used gnu diff to capture modifications to a source tree. The typical practice was to create two source trees: a pristine, and a dirty tree. Modifications were made in the dirty tree, and periodically gnu diff was run between the two trees to capture the changes. Then came quilt which allowed you to capture changes without a pristine tree – kind of a cheap version control system. The typical flow was: cd </li> quilt new my_changes.patch</li> quilt add file1.c</li> edit file1.c</li> quilt refresh (now my_changes.patch records the edits to file1.c)</li> debug</li> edit file2.c</li> (oops forget to quilt add file2.c – now what ….)</li> restore file2.c from pristine source</li> quilt add file2.c</li> edit file2.c</li> quilt refresh</li> (now my_changes.patch records edits to file1.c and file2.c)</li> </ul> As you can see, things are much better than plain diff, but you still have to keep track of the files you edit. With git, we can do essentially the same things, only git keeps track of the files you edit, plus it is easier to organize patch files. The flow with git is something like this: cd </li> git init</li> git add </li> git commit (now the entire source tree is recorded by git)</li> edit file1.c</li> edit file2.c ….</li> git status (shows you files that were modified)</li> git add file1.c file2.c</li> git commit</li> git log (shows you changes you have made)</li> git format-patch HEAD~1 (outputs the patch created by the last commit)</li> </ul> Once your source code is in git, then you can do all kinds of fancy things like merging, re-ordering, and re-naming patch files with the git-rebase --interactive</a> command. One of the uses of this technique is when you need to record a small change to a package source in a build system such as OpenEmbedded, and apply that change for subsequent builds. In this case you might do something like the following: bitbake -c devshell <recipe_to_patch> </li> in devshell capture changes in git as shown above to create my_changes.patch </li> cp my_changes.patch <oe_tree>/recipes/<recipe_to_change>/files/ </li> edit <oe_tree>/recipes/<recipe_to_change>/<recipe_to_change>.bb tell bitbake to apply the patch when building the recipe: SRC_URI += “file://my_changes.patch;patch=1”</li> increment the PR variable so that bitbake knows to rebuild the recipe.</li> </ul> </li> </ul> Figure 2 illustrates the above steps. Figure 2: Flow to capture package source changes in OpenEmbedded</figcaption></figure> While the above steps might seem overly complex, you must consider what you are trying to do. Building a Linux distribution is not a simple task. We are dealing with large amounts of source code from many different projects. Putting all this source code in one tree is not an option, because then we would be defeating the very reasons open source is attractive in the first place: flexibility, largly decoupled projects, advanced technology, fast bug fixes, many options to choose from, developer support, etc. Each open source project must remain a self contained project that is largly decoupled from other projects so it can develop at its own pace. It is for these reasons that open source technologies are used to give products a competitive edge. So, like any complex process or task, there must be some understanding of how it works, an appreciation for what it can do for you, and an investment in tools and techniques required to be effective. update 2023-09-23: recipetool</code> and devtool</code> are updated tools that can be used to make and capture changes. Embedded Linux versus Windows CE 2009-06-20T00:00:00+00:00 Occasionally I am asked how Embedded Linux compares with Windows CE. I have spent the past 5 years doing mostly embedded Linux development, and the previous 5 years doing mostly WinCE development with a few exceptions, so my thoughts are no doubt a little biased toward what I understand best. So take this with a grain of salt :-) In my experience, the choice is often made largely on perception and culture, rather than concrete data. And, making a choice based on concrete data is difficult when you consider the complexity of a modern OS, all the issues associated with porting it to custom hardware, and unknown future requirements. Even from an application perspective, things change over the life of a project. Requirements come and go. You find yourself doing things you never thought you would, especially if they are possible. The ubiquitous USB and network ports open a lot of possibilities – for example adding Cell modem support</a> or printer support</a>. Flash based storage makes in-field software updates</a> the standard mode of operation. And in the end, each solution has its strengths and weaknesses – there is no magic bullet that is the best in all cases. When considering Embedded Linux development, I often use the iceberg analogy; what you see going into a project is the part above the water. These are the pieces your application interacts with, drivers you need to customize, the part you understand. The other 90% is under water, and herein lies a great deal of variability. Quality issues with drivers or not being able to find a driver for something you may want to support in the future can easily swamp known parts of the project. There are very few people who have a lot of experience with both WinCE and Linux solutions, hence the tendency to go with what is comfortable (or what managers are comfortable with), or what we have experience with. Below are thoughts on a number of aspects to consider: SYSTEM SOFTWARE DEVELOPMENT</h2> Questions in this realm include CPU support, driver quality, in field software updates, filesystem support, driver availability, etc. One of the changes that has happened in the past two years, is CPU vendors are now porting Linux to their new chips as the first OS. Before, the OS porting was typically done by Linux software companies such as MontaVista, or community efforts. As a result, the Linux kernel now supports most mainstream embedded cpus with few additional patches. This is radically different than the situation 5 years ago. Because many people are using the same source code, issues get fixed, and often are contributed back to the mainstream source. With WinCE, the BSP/driver support tends to be more of a reference implementation, and then OEM/users take it, fix any issues, and that is where the fixes tend to stay. From a system perspective, it is very important to consider flexibility for future needs. Just because it is not a requirement now does not mean it will not be a requirement in the future. Obtaining driver support for a peripheral may be nearly impossible, or be too large an effort to make it practical. Most people give very little thought to the build system, or never look much beyond the thought that “if there is a nice gui wrapped around the tool, it must be easy”. OpenEmbedded is very popular way to build embedded Linux products, and has recently been endorsed as the technology base of MontaVista’s Linux 6 product, and is generally considered “hard to use” by new users. While WinCE build tools look simpler on the surface (the 10% above water), you still have the problem of what happens when I need to customize something, implement complex features such as software updates, etc. To build a production system with production grade features, you still need someone on your team who understands the OS and can work at the detail level of both the operating system, and the build system. With either WinCE or Embedded Linux, this generally means companies either need to have experienced developers in house, or hire experts to do portions of the system software development. System software development is not the same as application development, and is generally not something you want to take on with no experience unless you have a lot of time. It is quite common for companies to hire expert help for the first couple projects, and then do follow-on projects in-house. Another feature to consider is parallel build support. With quad core workstations becoming the standard, is it a big deal that a full build can be done in 1.2 hours versus 8? How flexible is the build system at pulling and building source code from various sources such as diverse revision control systems, etc. Embedded processors are becoming increasingly complex. It is no longer good enough to just have the cpu running. If you consider the OMAP3 cpu family from TI, then you have to ask the following questions: are there libraries available for the 3D acceleration engine, and can I even get them without committing to millions of units per year? Is there support for the DSP bridge? What is the cost of all this? On a recent project I was involved in, a basic WinCE BSP for the Atmel AT91SAM9260 cost $7000. In terms of developer time, this is not much, but you have to also consider the on-going costs of maintenance, upgrading to new versions of the operating system, etc. APPLICATION DEVELOPMENT</h2> Both Embedded Linux and WinCE support a range of application libraries and programming languages. C and C++ are well supported. Most business type applications are moving to C# in the WinCE world. Linux has Mono, which provides extensive support for .NET technologies and runs very well in embedded Linux systems. There are numerous Java development environments available for Embedded Linux. One area where you do run into differences is graphics libraries. Generally the Microsoft graphical APIs are not well supported on Linux, so if you have a large application team that are die-hard windows GUI programmers, then perhaps WinCE makes sense. However, there are many options for GUI toolkits that run on both Windows PCs and Embedded Linux devices. Some examples include GTK+, Qt, wxWidgets, etc. The Gimp is an example of a GTK+ application that runs on windows, plus there are many others. The are C# bindings to GTK+ and Qt. Another feature that seems to be coming on strong in the WinCE space is the Windows Communication Foundation (WCF). But again, there are projects to bring WCF to Mono, depending what portions you need. Embedded Linux support for scripting languages like Python is very good, and Python runs very well on 200MHz ARM processors. There is often the perception that WinCE is realtime, and Linux is not. Linux realtime support is decent in the stock kernels with the CONFIG_PREEMPT option, and real-time support is excellent with the addition of a relatively small real-time patch</a>. You can easily attain sub millisecond timing with Linux. This is something that has changed in the past couple years with the merging of real-time functionality into the stock kernel. DEVELOPMENT FLOW</h2> In a productive environment, most advanced embedded applications are developed and debugged on a PC, not the target hardware. Even in setups where remote debugging on a target system works well, debugging an application on a workstation works better. So the fact that one solution has nice on-target debugging, where the other does not is not really relevant. For data centric systems, it is common to have simulation modes where the application can be tested without connection to real I/O. With both Linux and WinCE applications, application programing for an embedded device is similar to programming for a PC. Embedded Linux takes this a step further. Because embedded Linux technology is the same as desktop, and server Linux technology, almost everything developed for desktop/server (including system software) is available for embedded for free. This means very complete driver support (see USB cell modem and printer examples above), robust file system support, memory management, etc. The breadth of options for Linux is astounding, but some may consider this a negative point, and would prefer a more integrated solution like Windows CE where everything comes from one place. There is a loss of flexibility, but in some cases, the tradeoff might be worth it. For an example of the number of packages that can be build for Embedded Linux systems using Openembedded, see http://cgit.openembedded.org/cgit.cgi/openembedded/tree/recipes</a>. GUI TRENDS</h2> It is important to consider trends for embedded devices with small displays being driven by Cell Phones (iPhone, Palm Pre, etc). Standard GUI widgets that are common in desktop systems (dialog boxes, check boxes, pull down lists, etc) do not cut it for modern embedded systems. So, it will be important to consider support for 3D effects, and widget libraries designed to be used by touch screen devices. The Clutter library</a> is an example of this. REMOTE SUPPORT</h2> Going back to the issue of debugging tools, most people stop at the scenario where the device is setting next to a workstation in the lab. But what about when you need to troubleshoot a device that is being beta-tested half-way around the world? That is where a command-line debugger like Gdb is an advantage, and not a disadvantage. And how do you connect to the device if you don’t have support for cell modems in New Zealand, or an efficient connection mechanism like ssh for shell access and transferring files? SUMMARY</h2> Selecting any advanced technology is not a simple task, and is fairly difficult to do even with experience. So it is important to be asking the right questions, and looking at the decision from many angles. Hopefully this article can help in that. For additional assistance, please do not hesitate to contact BEC Systems</a> – we’re here to help. Dealing with large data structures efficiently in embedded systems 2009-05-26T00:00:00+00:00 I’m currently dealing with a programming problem where I need access to several 64MB, file-backed data structures concurrently on an Embedded Linux system that only has 64MB of RAM. The data structures are fairly sparse (mostly zero data), and I typically only need to access small portions of them at any particular time. There is always the brute-force approach where you write code to manually load sections of the file as you need them. But with a little thought, the realisation hits “this is what operating systems do.” This article explores using memory mapping, and the sparse file</a> support in file systems to solve this problem in a very efficient manner. Sparse File Support</h2> Most Unix file systems support sparse files</a>. This means that sections of data that is zeros is not stored on the disk. Consider the following example where we create a 200MB file that is all zeros: root@cm-x270:/media/mmcblk0p1# df Filesystem 1k-blocks Used Available Use% Mounted on /dev/mmcblk0p1 3917212 202448 3515776 5% /media/mmcblk0p1 root@cm-x270:/media/mmcblk0p1# dd if=/dev/zero of=sparse-file bs=1 \ count=1 seek=200M root@cm-x270:/media/mmcblk0p1# df Filesystem 1k-blocks Used Available Use% Mounted on /dev/mmcblk0p1 3917212 202456 3515768 5% /media/mmcblk0p1 root@cm-x270:/media/mmcblk0p1# ls -l -rw-r--r-- 1 root root 209715201 May 22 08:54 sparse-file root@cm-x270:/media/mmcblk0p1# time od sparse-file 0000000 000000 000000 000000 000000 000000 000000 000000 000000 1440000000 real 0m 17.23s user 0m 14.21s sys 0m 2.66s root@cm-x270:/media/mmcblk0p1# du sparse-file 68 sparse-file </code></pre> It this case we created a 200MiB file on a SD card formatted as ext3. Even though the file is 200M in size, it only uses a few KiB of disk space. This same test also worked fine with a JFFS2 filesystem. With sparse file support, we get a cheap form of run length compression (at least for zeros) with no effort. mmap()</h2> The mmap() system call is used to map a file, or portions of a file into memory. The data in the file can then be accessed directly in memory. Because Linux is a demand paged system, portions of the file are paged in as needed, so the entire file does not need to be present in RAM at one time. There are several advantages to using mmap(): mmap() avoids the extraneous copy that occurs when using read() or write() as the data does not need to be copied to a user space buffer.</li> there is very little overhead</li> you can directly access any part of the file without doing a lseek() and keeping track of where you are.</li> the operating system takes care of paging in sections of the file you are using, and discarding sections that are not in use when memory is low. This includes flushing dirty portions to disk.</li> </ol> What this means, is mmap gives you an easy way to work on large, file backed data structures, and the OS takes care of loading the portions you need, and saving the modified portions back to disk. As most embedded systems are 32-bit, there is a limit to the file size you can use as virtual memory space is limited. Test Application</h2> Next I wrote a small application that is used to create, and modify files using mmap(). I wanted to experiment with creating files of various sizes, writing non-zero data at various intervals, and test the performance of this. The test application source is located here</a>. Usage: sparse_file_mmap_test [OPTION] -s, --filesize File size to allocate -o, --offset Will write a few bytes every offset bytes -m, --modify Modify existing file -d, --data Data to write to file at offset location (0-255) </code></pre> The application creates a file of size filesize, and writes the value of data to the file every offset bytes. There is also an option to modify existing files, so we can test the performance of opening a large file, making a small change, and then closing it. Test Results</h2> The results are pretty amazing, and the performance is beyond my expectations. This type of problem is where you learn to appreciate the performance of an advanced operating system, and file system. There is a reason for the complexity! There are 3 things I wanted to test: 1) does the sparse file support work as expected? 2) can mmap be used to easily modify large files? 3) can mmap be used to work on data structures that are larger than physical RAM? 1. Sparse File Support</h2> The basic tests above confirm that sparse file support works for an empty file, but what about a file that has some data every X bytes? Below are the test results for creating a 10MB file, and writing data at various offset intervals. Offset (bytes)</td> File Size (KB)</td> </tr> 1024</td> 9784</td> </tr> 2048</td> 9784</td> </tr> 4096</td> 9784</td> </tr> 8192</td> 4904</td> </tr> 16384</td> 2464</td> </tr> 32768</td> 1244</td> </tr> 65536</td> 632</td> </tr> 1048576</td> 60</td> </tr> 2097152</td> 40</td> </tr> 4194304</td> 32</td> </tr> </tbody></table> As soon as the offset was greater than the MMU page size (4KB), then the sparse file effect started to kick in and worked as expected. 2. Can mmap() be used to efficiently modify large files?</h2> The test here was to open a large file, make a small change in the middle, and then close it. In this case a 100MB file was created with a data value of 1 written every 1MB, and then re-opened the same file and wrote a data value of 2 every 2MB. The od utility was used to examine the file to verify the contents were correct. root@cm-x270:/media/mmcblk0p1# time sparse_file_mmap_test_arm -s104857600 \ -o1048576 -d1 Sparse File mmap() test Filesize = 104857600, offset = 1048576, data = 1 size = 102400 KB size on disk (KB): 508 sparse-file real 0m 1.10s user 0m 0.00s sys 0m 0.43s root@cm-x270:/media/mmcblk0p1# time sparse_file_mmap_test_arm -s104857600 \ -o2097152 -d2 -m Sparse File mmap() test Filesize = 104857600, offset = 2097152, data = 2 size = 102400 KB size on disk (KB): 508 sparse-file real 0m 0.37s user 0m 0.01s sys 0m 0.05s root@cm-x270:/media/mmcblk0p1# time od -x sparse-file 0000000 0002 0000 0000 0000 0000 0000 0000 0000 0000020 0000 0000 0000 0000 0000 0000 0000 0000 * 4000000 0001 0000 0000 0000 0000 0000 0000 0000 4000020 0000 0000 0000 0000 0000 0000 0000 0000 .... * 614000000 0001 0000 0000 0000 0000 0000 0000 0000 614000020 0000 0000 0000 0000 0000 0000 0000 0000 * 620000000 real 0m 12.93s user 0m 7.33s sys 0m 1.70s </code></pre> Creating and modifying large file-backed data structures is very fast and efficient, and takes on the order of 1 second for a 100MB file that contains data every 1MB. Dumping the data with od took considerably longer (13 seconds) as 100MB of data needed to be processed. Possible Issues</h2> There are several things to watch out for when using sparse files and mmap() With sparse files, there is the potential to run out of disk space as the files are using less space on disk than the file size. It is a good idea to monitor disk space when working with sparse files, so you don’t use up all of the disk space.</li> mmap() requires virtual memory space for the size file it maps. With embedded systems, this is less of an issue, because the physical RAM size tends to be much less than the 4GB virtual address space. With a system that only has 64MB of RAM, mmap()’ing files in the 10’s of MB in size makes a lot of sense because it insures that you will not run the system out of memory, and yet there should be plenty of virtual address space to map in files of this size.</li> </ol> Conclusion</h2> mmap() and sparse file support provide a very convenient solution for dealing with large, file-backed data structures. One example of this type of data structure is any type of large matrix such as maps used in mapping applications. Writing a “from scratch” solution to solve this problem would be a fairly large and difficult task. Processing large amounts of data efficiently is becoming more and more important in many embedded systems. This example provides another compelling reason why implementing modern, data-centric embedded systems using Linux makes a lot of sense. MontaVista Linux 6 is based on OpenEmbedded Technologies 2009-05-13T00:00:00+00:00 Very interesting news from MontaVista yesterday as they announced MontaVista Linux 6</a>. It turns out they are using bitbake, which is the core of the OpenEmbedded</a> build system. Along with the adoption of OpenEmbedded in many leading Embedded Linux efforts (Beagleboard, Gumstix, Bug Labs, etc), this is a resounding affirmation that the OpenEmbedded way of building distributions is worthy of consideration. It is a well known fact that OpenEmbedded has some deficiencies: a steep learning curve, the testing/quality efforts could be improved, and the organisational aspects of the project are somewhat lacking. That said, OpenEmbedded is still the most effective tool for building Embedded Linux distributions that I’ve found yet, and is a very viable solution if you are willing to spend some time to learn it, or hire a consultant to get your project set up and work through the rough spots. MontaVista is attempting to address these difficulties with their MVL6 commercial offering. Below are few a notes from watching a few videos on the MV web site: Discussion with Jim Ready</a> around 2006 things changed</li> Semiconductor vendors now porting Linux to all their CPUs, so MV has moved up the stack</li> MVL6 first embedded linux distro produced in post-victory phase (Linux has won)</li> very friendly with other open source pieces (I assume OpenEmbedded)</li> enable open source</li> The decision the use Linux is the right decision technically. What is the right business decision to accompany the technical decision?</li> </ul> </li> MVL6 Demonstration</a> toolchains pre-packaged</li> sources are all downloaded from MV servers</li> </ul> </li> Interview with Joe Green</a> manager of developer tools team </li> MLV6 is new approach </li> previous editions were consistent for all targets </li> MLV6 more flexible model. Market specific distros depending on market. </li> source driven product. Previous products based on binary RPMs. MLV6 is a lot easier to build the whole environment. Very customized/fine-tuned distro. </li> key problems MLV6 solves Complete starting point for target hardware</li> complete build system, so you have total control vs pre-canned distribution that sets all the rules</li> </ul> </li> Integration platform based on Bitbake. Working closely with community. Enhanced in a number of ways. Customers have the option of bringing in software developed by the OpenEmbedded community and bringing it into their project. </li> Gives you a system that you can configure at almost any level. </li> Enhancements to make it easy to make changes, as well as provide a pre-built starting point. </li> Features to control sources, and environment so the host environment does not contaminate your build. </li> Essentially hardening, and commercializing the bitbake and openembedded projects. </li> Why MVL6 vs roll-your-own. Qualified starting point</li> Support. Community or Semi-conductor vendor support is hit or miss.</li> Consistent environment accross multiple systems.</li> Includes devrocket graphical tools</li> </ul> </li> </ul> </li> </ul> Overall, this seems like a very good approach. I’m glad they are not re-inventing the wheel, but instead have chosen to build on top of a proven open source solution, and leverage the work that is being done by a very active community of OpenEmbedded developers. The OpenEmbedded project currently has 61 developers (not all are active), and weekly contribution rate is very good. This type of effort would be very difficult to match by any but the largest commercial organizations. This should be a win-win situation for all involved. As a common base technology, the Bitbake and OpenEmbedded projects should improve due to MV’s involvement. Through the MVL6 product, the OpenEmbedded project and way of doing things will get a lot more exposure and there will be a lot more developers who understand how it works. Developers who are familiar with the OpenEmbedded build system will be able to apply their knowlege to a greater number of projects. And developers building products will have more options for getting the support and techology they need. How to set up a NFS root filesystem for embedded Linux development 2009-05-11T00:00:00+00:00 Although ssh and friends work really well for embedded systems, occasionally you want to set up a NFS root for development. One of the scenarios where nfsroot is useful is if you are making a lot of rootfs changes, and you don’t want to spend the time to reprogram the flash on the target system. Fortunately, the Linux kernel includes complete support for NFS root, and does not require any userspace changes. This setup assumes the following: the target system has Ethernet support built into the kernel</li> the target system is on the same network as a Linux workstation</li> said network includes a DHCP server</li> </ol> This configuration emphasises “simple” and does not require you to spend 4 hours trying to get a bootp server configured on a test network. Kernel Support</h2> Make sure you have the following options turned on in the kernel: CONFIG_IP_PNP_DHCP=y</li> CONFIG_ROOT_NFS=y</li> </ul> Then, add the following to your kernel CMDLINE: ip=dhcp root=/dev/nfs nfsroot=<nfs server IP>:/path/to/nfsroot </code></pre> </li> </ul> Workstation Setup</h2> The following setup is for Ubuntu. apt-get install nfs-user-server (edit /etc/exports to contain something like the following) /path/to/nfsroot 192.168.1.0/255.255.255.0(no_root_squash,sync,rw) /etc/init.d/nfs-user-server restart If you are using OpenEmbedded, then instruct OE to generate a rootfs tar image, and extract to your nfsroot directory: cd /path/to/nfsroot sudo tar -xvf <path to OE images dir>/Angstrom-image...rootfs.tar </code></pre> And the result</h2> eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1 Sending DHCP requests ., OK IP-Config: Got DHCP answer from 192.168.1.1, my address is 192.168.1.112 IP-Config: Complete: device=eth0, addr=192.168.1.112, mask=255.255.255.0, gw=192.168.1.1, host=192.168.1.112, domain=hq.bec-systems.com, nis-domain=(none), bootserver=192.168.1.1, rootserver=192.168.1.11, rootpath= Looking up port of RPC 100003/2 on 192.168.1.11 Looking up port of RPC 100005/1 on 192.168.1.11 VFS: Mounted root (nfs filesystem) readonly. Freeing unused kernel memory: 220k freed Write protecting the kernel text: 2792k Write protecting the kernel read-only data: 788k INIT: version 2.86 booting </code></pre> Compulab cm-x270 kernel updated to 2.6.29 in OE 2009-04-24T00:00:00+00:00 The cm-x270 kernel support in OpenEmbedded has just been updated</a> to version 2.6.29. Wi2Wi W2CBW003 Wifi/Bluetooth module review 2009-03-19T00:00:00+00:00 The Wi2Wi W2CBW003</a> is a highly integrated module that provides both Wifi and Bluetooth radios for embedded designs. This module is ideal for embedded designs, as it provides a lot of functionality in a small package and includes standard interfaces like SPI, SDIO and serial that connect with most embedded CPUs. With the availability of modules like the W2CBW003 and standard drivers in the Linux kernel, including radio functionality in an embedded device is very doable, even for low volume products. Wi2Wi provides an evaluation board for the W2CBW003 with a SDIO connector, UART connector, and BT Audio Connectors. For this review, the eval board was connected to a Marvel PXA270 ARM processor, and evaluated with current Linux and associated software. W2CBW003 Overview</h2> The W2CBW003 module integrates both WiFi and Bluetooth functionality in a 12mm x 12mm x 1.6mm package. The WiFi portion is based on the Marvell 88W8686, and the Bluetooth on the CSR BC04. Both of these components are well supported by Open Source software. Some other features include: Separate Antennas for WiFi and BT.</li> ROHS compliant</li> Single Supply at 3.3V</li> 802.11g support (54Mbps)</li> Both SPI and SDIO interfaces for WiFi</li> UART interface for BT</li> PCM audio interface for BT</li> </ul> Pictures of the module and the demo board are shown below. Evaluation System Configuration</h2> The test software included with the Wi2Wi eval board is for a Windows PC, is provided in binary format only, and was not used in this review. For this review, I plugged the W2CBW003 demo board into a Compulab cm-x270 board (PXA270 cpu) running a 2.6.29-rc7 Linux kernel and a recent OpenEmbedded Angstrom distribution. With the exception of the Marvell Wifi Firmware, all software in this setup is Open Source and is available in the Linux kernel, and as packages in the OpenEmbedded Project. When booting the kernel, you will see the following messages: mmc0: new SDIO card at address 0001 libertas_sdio mmc0:0001:1: firmware: requesting sd8686_helper.bin libertas_sdio mmc0:0001:1: firmware: requesting sd8686.bin libertas: 00:19:88:06:0b:2e, fw 9.70.3p25, cap 0x00000303 eth2 (libertas_sdio): not using net_device_ops yet libertas: PREP_CMD: command 0x00a3 failed: 2 libertas: PREP_CMD: command 0x00a3 failed: 2 libertas: eth2: Marvell WLAN 802.11 adapter </code></pre> The “command 0x00a3 failed” messages are harmless, and have to do with features that are not supported. After the system boots, you will now see a new ethX network device: root@cm-x270:~# ifconfig -a ... eth2 Link encap:Ethernet HWaddr 00:19:88:06:0B:2E BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:65902 errors:0 dropped:0 overruns:0 frame:0 TX packets:1758 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:13550002 (12.9 MiB) TX bytes:251627 (245.7 KiB) </code></pre> The “iwlist eth2 scanning” command will list available access points. Connecting to Open WiFi Networks</h2> A connection to an open wifi network can be accomplished by placing the following in /etc/network/interfaces: /etc/network/interfaces: iface eth1 inet dhcp wireless_mode managed wireless_essid any </code></pre> And now execute “ifup eth2”: root@cm-x270:~# ifup eth2 udhcpc (v1.13.2) started run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 Sending discover... Sending select for 192.168.1.115... Lease of 192.168.1.115 obtained, lease time 86400 run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 adding dns 208.67.222.222 adding dns 208.67.220.220 root@cm-x270:~# iwlist eth2 iwlist: unknown command `eth2' (check 'iwlist --help'). root@cm-x270:~# iwconfig eth2 eth2 IEEE 802.11b/g ESSID:"bec3" Mode:Managed Frequency:2.437 GHz Access Point: 00:18:39:C1:AD:4A Bit Rate:1 Mb/s Tx-Power=13 dBm Retry short limit:8 RTS thr=2347 B Fragment thr=2346 B Encryption key:off Power Management:off Link Quality=84/100 Signal level=-37 dBm Noise level=-87 dBm Rx invalid nwid:0 Rx invalid crypt:14707457 Rx invalid frag:0 Tx excessive retries:58 Invalid misc:3 Missed beacon:0 </code></pre> WPA Secured WiFi Networks</h2> The OpenEmbedded console image includes the WPA Supplicant packages which is used to manage wireless connections to secured networks. To set up the system for WPA encryption, modify the following files: /etc/network/interfaces: iface eth2 inet dhcp wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf wpa-driver wext </code></pre> /etc/wpa_supplicant/wpa_supplicant.conf: ctrl_interface=/var/run/wpa_supplicant ctrl_interface_group=0 eapol_version=1 ap_scan=1 fast_reauth=1 network={ ssid="bec" proto=WPA2 key_mgmt=WPA-PSK pairwise=CCMP TKIP group=CCMP TKIP scan_ssid=1 psk="ascii passphrase" priority=10 } </code></pre> Then, execute “ifup eth2”, and you should see something like: root@cm-x270:~# ifup eth2 WPA: Configuring Interface udhcpc (v1.13.2) started run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 Sending discover... Sending select for 192.168.1.115... Lease of 192.168.1.115 obtained, lease time 86400 run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 adding dns 208.67.222.222 adding dns 208.67.220.220 </code></pre> root@cm-x270:~# iwconfig eth2 eth2 IEEE 802.11b/g ESSID:"bec" Mode:Managed Frequency:2.412 GHz Access Point: 00:40:10:10:00:03 Bit Rate:1 Mb/s Tx-Power=13 dBm Retry short limit:8 RTS thr=2347 B Fragment thr=2346 B Encryption key:<too big> Security mode:open Power Management:off Link Quality=64/100 Signal level=-68 dBm Noise level=-89 dBm Rx invalid nwid:0 Rx invalid crypt:-1463809279 Rx invalid frag:0 Tx excessive retries:22524 Invalid misc:3 Missed beacon:0 </code></pre> Other Observations</h2> With the above networks, the bec access point was much further away than the bec3AP, so you will notice the difference in link quality. “iwlist eth2 rate” can be used to list the current connection rate. When the network is idle, it sits at 1Mb/s. When downloading a large file, it will climb to 36 or 54Mb/s, depending on link quality. Production Issues</h2> The review demonstrates that it is fairly simple to set up a demo quality Embedded Linux system with WiFi. Some of the issues that would likely need addressed for a production system include link management, test software for certification, and power management. There are several ways to programatically control WPA Supplicant including linking to the wpa supplicant control interface, or using DBus. There are several WiFi management applications available including Gnome NetworkManager</a> (used in desktop systems), and connman</a> which seems a little better suited for embedded systems. Unless you use a completely pre-certified module + antenna solution, you will likely need to do some level of agency certification. As many products use a custom antenna, this is an important issue to consider and plan for. While Marvell provides test software and firmware, it will likely require some work, as their software is designed to be used with their in-house drivers rather than the libertas driver which is available with modern kernels. Also, if you want to minimize the power usage, some work will be required to figure out the power modes supported, and how to implement/control them. With this module, the Wifi and BT portions run off the same crystal, so if you only want the BT active, you will need to actively power manage the Wifi portion to a low power state instead of completely disabling it. Conclusion</h2> TheW2CBW003 module is an attractive solution for products that need WiFi functionality. With the availability of modules like this, and mainstream open source software, the technology is available to about anyone, including low volume manufacturers. Standard interfaces such as SDIO make it possible to interface this module with about any modern ARM processor that can run Linux. Software support in the Linux kernel, wpa supplicant, and the Linux wireless tools provide the needed software support to implement a very complex system with relatively little effort. Memory Performance on various Embedded Systems 2009-03-04T00:00:00+00:00 Marcin just published an interesting article</a> about memory performance on various embedded systems using the hdparm -T as a simple benchmarq. This test gives a pretty good indicator of memory performance in the system. From the hdparm man page: Perform timings of cache reads for benchmark and comparison purposes. For meaningful results, this operation should be repeated 2-3 times on an otherwise inactive system (no other active processes) with at least a couple of megabytes of free memory. This displays the speed of reading directly from the Linux buffer cache without disk access. This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test. A few results I find interesting: modern desktop systems have an order of magnitude more memory bandwidth than ARM systems.</li> the i.MX31 is the highest performing ARM device tested</li> the i.MX31 performs better than the OMAP3 in this test – why is this? As the ratio is 2, I’m guessing the bus is twice as wide?</li> </ul> Who is using OpenEmbedded? 2009-02-20T00:00:00+00:00 Having used OpenEmbedded for a number of Embedded Linux projects over the past 5 years, it is interesting watching new users come on board. The most recent announcement came from Koan</a> that the KaeilOS</a> distribution will be joining the OpenEmbedded project. In the announcement, they summed up my thoughts well: We at KOAN wanted to avoid waste of energy converging KOAN efforts into a stable and solid Open Source project like OE is. “Avoid waste of energy” – isn’t this what OSS is all about? Life is way too short to re-invent every wheel. Considering OpenEmbedded has dozens of active contributors, it would be very difficult for any one company to replicate the functionality of OpenEmbedded. Although there are many ways to build embedded Linux distributions, and some tools do some things better than OE, there still does not seem to be any other embedded Linux build tool that fulfills the following requirements as well as OE: build from scratch on any x86 Linux system</li> large number of contributors</li> large number of packages</li> support a large number of architectures</li> Can modify images and integrate custom applications in a consistent way as part of the build system, and not a post processing hack.</li> Open Source</li> </ol> Some other companies using OpenEmbedded include: TI</li> AMD</li> Intel (was OpenedHand)</li> Gumstix</li> Bug Labs</li> OpenMoko</li> Atmel</li> Techsol</li> Compulab</li> Toradex</li> 4G-Systems</li> Dream Multimedia TV</li> M&N Solutions</li> emQbit</li> Siemens</li> Embedded Alley</li> and the list goes on</a> …. (If you are using OpenEmbedded, please add your name to this list</a>.)</li> </ul> I’m looking forward to seeing the KaeilOS improvements such as mdev, and boot time improvements. Sprint 598U USB Broadband Modem in Embedded Systems 2009-02-07T00:00:00+00:00 One thing that works really well in Linux is support for USB Broadband modems. While it usually takes a little fiddling with PPP scripts, it is generally not too difficult to get working. One of the reasons for this is most modems implement a USB serial interface, and then the modem is controlled with standard AT commands. A customer who currently supports a Verizon USB modem</a> in their product wanted to add support for a Sprint modem. After a little research, we settled on the 598U from Sierra Wireless</a>. Below is a picture of the 598U plugged into an industrial terminal. Why are USB Broadband modems useful in Embedded Systems?</h2> As systems become more complex, and time to market is shrinking (thus less testing), it is becoming increasingly important to be able to remotely diagnose problems. With a broadband modem like the Sprint 598U, developers can remotely log into a device using SSH, and debug problems. This is a huge benefit. The cost of the modem and a 1 year data plan becomes fairly insignificant when you consider the cost to fly a developer on-site to debug issues. While there are other ways to remotely access an embedded system, such as reverse ssh</a>, the simplicity of the broadband modem is attractive as it bypasses the firewall issues. Even with reverse ssh, I’ve run into paranoid sys admins who block the outgoing port 22 on their networks, which makes reverse ssh difficult. Once you have the IP address of the broadband connection, you can connect to it directly with ssh from anywhere in the world. I’ve accessed systems that were being field tested in New Zealand from the United States, and it worked great. Increasingly, the value of complex embedded systems is shifting from control to data. While the control aspect is as important as it ever was, access to data is what is giving many products and businesses the competitive edge. Being able to collect data and visualize how the system is performing is becoming increasingly important to maximize performance. And sometimes this data must be collected in real-time. Scenarios where USB broadband modems might make sense: remote sites that do not have internet connectivity</li> mobile applications such as vehicle terminals</li> temporary installations where the system is only located in one place for a limited time</li> situations where it is very difficult to get administrators of the local network to cooperate in adding the embedded system to their network</li> </ul> Finding PPP Config information</h2> I usually develop as much as I can on x86 systems (vs. an embedded target) as it is a lot more convenient, so I plugged the modem into a Ubuntu 8.10 laptop. While Ubuntu 8.10 recognizes and configures a UM150 Verizon modem, it does not automatically configure the 598U. Typically, the next step is to google around for information on various support forums on how to set up the PPP config files, as most manufactures don’t provide support for Linux. As I could not find any existing information for the 598U, I decided to see if Sierra Wireless published any information. Was I ever suprised when I found this page</a>. Sierra Wireless provides fairly comprehensive information on using the modem under Linux, and their PPP scripts actually worked – great job! But in the end, when I plugged the 598U into the embedded device that already supported the Verizon modem, it just worked with my existing PPP scripts. This is another indication of the value of using common code and interfaces in Linux. The Linux Advantage</h2> When you run Linux in an embedded system, using a USB broadband modem becomes fairly simple. It is actually a little bit deceiving as to what all has to happen. Different modems use different chipsets, and thus different drivers. While they all eventually expose a USB serial interface, there are enough differences to require separate drivers. As an example, the 598U uses the “sierra” USB-serial driver. This driver is 782 lines, plus it utilizes a common USB serial framework that is many times this size. On the application side, the pppd framework is used to manage the modem and the network connection. Starting the connection is as simple as running “pppd call cdma” from your application. The pppd application manages getting an IP address, setting up nameservers in resolve.conf, etc. Because all broadband modems are supported by a common application interface and very similiar drivers, supporting multiple modem vendors is actually quite easy. Contrast this to the Windows driver scenario where every vendor provides their “own” driver, and their “own” application user interface, and their “own” way to initiate an internet connection, and it is not obvious to me how you could easily control all this in a consistent, automated fashion in an embedded device. The way we set it up is you plug the modem in, and a few seconds later the system is automatically connected to the internet. There is nothing for the user to configure, initiate, etc. Printing from Linux in embedded applications</a> is similiar. This is another example of how it is possible to leverage the work of many to add advanced functionality and features to a device with very little development cost. Getting more done with less should be an attractive (and perhaps critical) way to operate for any company seeking to remain competitive. Socketcan utils and test apps added to OpenEmbedded 2009-01-28T00:00:00+00:00 The Socketcan test applications and utilities are now available in OpenEmbedded</a>. The socketcan kernel modules already exist in OE</a>. To build and deploy: get the latest version of OE metadata</li> bitbake socketcan-utils-test</li> scp /build/angstrom-2008.1/tmp/deploy/glibc/ipk//socketcan-utils-test_0.0+svnr917-r0_armv5te.ipk root@:</li> </ul> And then on the target system: opkg install socketcan-utils-test_0.0+svnr917-r0_armv5te.ipk</li> opkg files socketcan-utils-test</li> </ul> /usr/bin/tst_bcm_rx_sendto /usr/bin/tst_bcm_tx_sendto /usr/bin/canlogserver /usr/bin/tst_raw_filter /usr/bin/tst_raw /usr/bin/isotpsniffer /usr/bin/isotpsend /usr/bin/tst_bcm_throttle /usr/bin/tst_packet /usr/bin/vcan /usr/bin/isotprecv /usr/bin/cansniffer /usr/bin/tst_bcm_tx_read /usr/bin/tst_err /usr/bin/log2asc /usr/bin/tst_bcm_single /usr/bin/tst_filter_master /usr/bin/canplayer /usr/bin/cangen /usr/bin/slcanpty /usr/bin/log2long /usr/bin/tst_bcm_rtr /usr/bin/asc2log /usr/bin/slcan_attach /usr/bin/tst_bcm_filter /usr/bin/cansend /usr/bin/canecho /usr/bin/tst_filter_server /usr/bin/tst_bcm_cycle /usr/bin/tst_bcm_dump /usr/bin/isotpdump /usr/bin/candump /usr/bin/tst_raw_sendto /usr/bin/isotptun /usr/bin/tst_bcm_server /usr/bin/tst_proc /usr/bin/canbusload </code></pre> An impressive array of utilities! Socketcan CAN-bus drivers added to OpenEmbedded 2009-01-26T00:00:00+00:00 I just added a recipe</a> to OpenEmbedded to build the Socketcan</a> kernel modules from the socketcan SVN. So if you are using the latest OpenEmbedded metadata, you can: bitbake socketcan-modules</li> scp /build/angstrom-2008.1/tmp/deploy/glibc/ipk//socketcan-modules_0.0+svnr917-r0_cm-x270.ipk root@:</li> </ul> and then on the target system: opkg install socketcan-modules_0.0+svnr917-r0_cm-x270.ipk</li> opkg files socketcan-modules</li> </ul> Package socketcan-modules (0.0+svnr917-r0) is installed on root and has the following files: /lib/modules/2.6.27/extra/net/can/can-bcm.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/ems_pci.ko /lib/modules/2.6.27/extra/net/can/can-raw.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/sja1000.ko /lib/modules/2.6.27/extra/drivers/net/can/softing/softing.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/ems_pcmcia.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/sja1000_platform.ko /lib/modules/2.6.27/extra/drivers/net/can/can-dev.ko /lib/modules/2.6.27/extra/drivers/net/can/softing/softing_cs.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/pipcan.ko /lib/modules/2.6.27/extra/drivers/net/can/mcp251x.ko /lib/modules/2.6.27/extra/net/can/can.ko /lib/modules/2.6.27/extra/net/can/can-isotp.ko /lib/modules/2.6.27/extra/drivers/net/can/vcan.ko </code></pre> Then to use the modules: depmod</li> modprobe mcp251x</li> modprobe can-bcm</li> </ul> What is CAN? Can stands for Controller-area network and is popular in industrial and automotive applications. A convenient way to add CAN to your Embedded Linux system is with the Microchip MCP2515</a>. This device connects to a SPI bus which means it can be interfaced with a number of popular SOC’s such as the PXA270, OMAP3, 91SAM9xxx, etc. More on Socketcan later … BTW, this recipe really illustrates how easy it is to compile kernel modules outside a kernel in OpenEmbedded: DESCRIPTION = "Socketcan kernel modules" HOMEPAGE = "http://developer.berlios.de/projects/socketcan/" SECTION = "kernel/modules" LICENSE = "GPL" DEPENDS = "virtual/kernel" PV = "0.0+svnr${SRCREV}" SRC_URI = "svn://svn.berlios.de/socketcan/trunk;module=kernel;proto=svn" S = "${WORKDIR}/kernel/2.6" inherit module </code></pre> This recipe tells OE to download the source, cross-compile it against your target kernel build dir, and then package it for easy install on the device. This is the way things should work – no messing around figuring out make options, kernel source paths, compiler env variables, etc. Intel Atom vs TI OMAP3 2009-01-22T00:00:00+00:00 As we look at new projects, both the Intel Atom</a> and the TI OMAP3</a> processors generate considerable interest. As we have already shown</a>, the OMAP3 does offer a considerable performance improvement over earlier generations of ARM CPUs. The following video</a> I found on YouTube shows a similar comparison of a OMAP3 and Atom systems rendering web pages: As one would expect, the Atom does perform better (about 14%), but considering the power differences, the OMAP does surprisingly well. It is also unknown in this demo if the screen size would make a significant difference in the results. Like most things, the choice depends on the application, and no two applications are the same, and each solution has advantages. Some things to think about: Power: OMAP3 platform consumes on the order of <1-2W while the Atom is more in the range of 2-5W. </li> High Speed I/O Interfaces: Atom supports PCI and PCI expansion interfaces where OMAP3 is limited to more special purpose user interfaces such as SD, Camera, Asynchronous bus, etc. Both Atom and OMAP support High Speed USB. </li> Packaging: OMAP3 packaging is very aggressive with the stacked Package-on-Package. To get an idea how much space an OMAP3 solution takes, check out the module from Gumstix</a>. There are basically only two chips in the system: the OMAP3+stacked RAM/Flash and a power management+I/O chip. This is very high integration! </li> Module availability: for many embedded systems with volumes in the 1000’s of units per year, a module solution is very attractive compared to a full custom design. This drastically reduces the engineering effort and time to market. A sampling of the modules available include: OMAP3: Gumstix Overo</a> ($117 for 3000 units)</li> LogicPD OMAP35x SOM-LV</a> (Cost unknown)</li> </ul> </li> Atom: Eurotech Catalyst</a> (Cost unknown)</li> </ul> </li> </ul> </li> Software support: TI and the open source community have done a remarkable job of supporting the OMAP3 with the BeagleBoard</a> effort. Gumstix maintains open source software for their devices, and has a very active development community. Intel also has invested significantly in software with their Moblin</a> project. Other factors to consider is the boot software (bootloader vs BIOS, is it open?), are there 3D graphics libraries available, etc. </li> Multimedia processing: The OMAP3 is available with an on-chip DSP. Intel has traditionally offered extensions for multimedia processing such as SIMD. </li> </ol> Tradeoffs! GTK performance on PXA270 vs. OMAP3 2009-01-21T00:00:00+00:00 Several of my customers have built applications using the GTK+</a> tookit. While GTK+ works fairly well for what we have done, I have been wondering how the performance compares on the new Omap3</a> processors from TI. As we are evaluating the OMAP3 for several projects, I did a simple comparison with an existing application. Below is a video</a> that shows a fairly complex application running on both a PXA270, and a OMAP3530. While the PXA270 gets the job done, the result on the OMAP3 is much more pleasing. With that advent of a OMAP3 module available for $117 in volume</a>, it seems like the OMAP3 will be a popular solution for upcoming Embedded Linux projects. Fix for PXA270 MMC/SD Controller Write Corruption 2009-01-17T00:00:00+00:00 During system verification for a customer, we noticed occasional failures when writing to a SD card using the Marvell PXA270 MMC/SD controller. The failure is a 4KB block of data is shifted on byte, where the first byte is duplicated and the last byte is dropped. The test app that found this problem is available here</a>. This test app simulates an application that writes data to a SD card at a certain rate. We found that typically in about 2000 - 10000 hours of simulated testing, we would see a failure. Its very interesting that the field failure rates matched this rate as well. This failure had all the signs of a hardware (likely DMA) problem, and sure enough there is a Marvell Eratta for this issue. A patch has been developed and tested by several people and is available in this mail thread</a>. Installing Ubuntu on a Core i7 DX58SO motherboard 2009-01-15T00:00:00+00:00 UPDATE: 2009-03-17 – It appears that hard drive issues I encountered are likely due to the HW RAID formatting on the drive I tried. I just tried a brand new hard drive, and Ubuntu 8.10 installed flawlessly, and everything works. I’m not sure why computer upgrades are always such an epic struggle for me, but here I go again … The goal is to upgrade my aging AMD X2 system to a new Intel Core i7 system. So after researching it a bit, I decided on the Intel DX58SO</a> motherboard. I liked the architecture, and have had good luck with Intel motherboards. And since I don’t upgrade all that often, I try to get something good when I do. As I learned later, I should have researched Linux compatibility a little more … The plan was to simply replace the CPU/MB/RAM in my existing box as I already have a raid controller, decent case, etc. However, the first challenge was the existing PS does not supply a EPS/ATX12V8 pin connector – it only had the 4-pin version. So, I got to upgrade my PS to a Corsair CMPSU-650TX</a> which seems very nice, and quiet. After I put everything together, the raid array (3ware 9500S-4LP PCI) was detected, but the system would not boot. So I booted from a Ubuntu install disk and selected boot from 1st hard disk, and then it seemed to boot fine. However, the Ethernet controller and CDROM don’t work under Ubuntu 8.04 (64-bit) which was currently installed on the raid array. However, everything else works pretty well, and it is quite fast. Because the core-i7 brought back hyperthreading, Linux now shows 8 cores. Several days later in an attempt to get the system booting from the hard drive, I upgraded the BIOS to the latest version. However, this broke booting from the raid array; even with a Ubuntu install disk. At this point, I decided to just rebuild the system by re-configuring the 4 hard drives currently in a hardware raid-5 array to two raid-1 software raid arrays. As the raid card is only PCI, I think this would provide better performance and be simpler to maintain in the future. So, I copy all my data off to a USB HD, and then try to install Ubuntu to a HD attached to the motherboard. To my suprise, I can’t get MB attached hard drives to work. Lots of messages like: [ 361.943329] ata1.01: status: { DRDY ERR } [ 361.943332] ata1.01: error: { ABRT } [ 362.043760] ata1.01: configured for UDMA/133 [ 362.043771] ata1: EH complete [ 362.218194] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 [ 362.218200] ata1.01: BMDMA stat 0x45 [ 362.218207] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in [ 362.218209] res 51/04:08:00:00:00/04:00:1d:00:00/f0 Emask 0x1 (device error) </code></pre> I tried booting a 32-bit Ubuntu 8.10 live CD and I get the same thing when trying to fdisk a hard drive attached to the MB. So for now, the only way to get this running is to go back to the old RAID controller. Fortunately, Intel keeps old BIOS versions on their web site and after two tries, I was able to find one that works again with the HW RAID controller. The Ethernet and CD ROM did work with the Ubuntu 8.10 32-bit live CD, so once I upgrade to 8.10, hopefully that will all work. So for now, I’ll limp along with my raid controller and a USB->Ethernet adapter for networking. Anyone else have success with Linux on the DX58SO? I’ve filed a bug on launchpad</a>. Switching to Wordpress 2009-01-13T00:00:00+00:00 After using Joomla for several years, I’m switching my web site to wordpress. As I mostly use my site for writing technical articles, it makes sense to use something that works very well for just that. So far, I’m finding Wordpress very polished, and it has all the functionality I need built-in, so I don’t have to rely on 3rd party modules for very basic operations like tagging, RSS feeds, comments, etc. Now I have no excuse for now writing more as Wordpress works so well. Mono 2.2 RC1 added to Openembedded 2009-01-10T00:00:00+00:00 I recently addedMono 2.2 RC1 to OpenEmbedded</a>. The last usable version of Mono in Openembedded was 1.2.6, so this is a big step up. Lots of interesting things have been happening since then (like Full Static Compilation</a>) that continue to make Mono interesting for embedded. There is still plenty to do yet such as clean up the packaging, and update the other related recipes such as mono-xsp, and gtk-sharp. Using the Vala Programming Language in Embedded Systems 2009-01-10T00:00:00+00:00 Recently I’ve been following the Vala programming language</a> and using it some in embedded systems. Vala is a new programming language that aims to bring modern programming features to GNOME developers without imposing additional runtime requirements and without using a different ABI compared to applications and libraries written in C. A few notes and observations about Vala: language syntax that resembles C#, so you can write code faster with less mistakes</li> Vala compiles to C, so it starts fast and runs fast like native applications</li> binaries are not platform independent (like Mono or Java)</li> lots of bindings already exist, as it is very easy to write bindings to existing C libraries</li> easy to write programs that are mixed C and Vala</li> documentation is still a bit sparse, so you end up reading the binding files to figure out how to use the libraries</li> </ul> Much of the core Vala functionality is built on top of Glib. Having programmed extensively with Glib, I can say programming in Vala is a lot more fun, and a lot less tedious. Vala also provides dynamic D-Bus bindings which makes if very nice for writing system daemons or other bits of software that need to implement a D-Bus server. As far as real world experience, in a recent application, we ported a system monitoring application from C# to Vala. There were no real serious problems with the original application, but it started slow (7 seconds or so), and the customer wanted to reduce the boot time of the system. Once you get past the core language syntax, the libraries for Vala are all different than C#, so most of the library function calls needed tweaked a little. Most of the functionality was implemented with Vala, but there were a few minor functions that were implemented in C. It now starts fast, which is what we needed. We are also using the fsod</a> application from the OpenMoko project in a customer project. This is an excellent example of a well written, advanced Vala project that uses features such as plugins and D-Bus. Watching the releases from the Vala project, it is amazing the progress this language is making. I’m sure we’ll be hearing more about Vala in the future. How to implement an interrupt driven GPIO input in Linux 2009-01-10T00:00:00+00:00 With Linux, some of the things that seem like they should be easy are not – at least at first glance. For example, how do you read an interrupt driven GPIO input in a Linux application? With simpler microcontroller systems, this is straightforward, but with a system like Linux, you have to navigate through several layers of software (and for very good reasons). You can’t handle interrupts directly in a Linux application, so this means you need a kernel component involved. This operation of reading a GPIO resembles a key press, so the Linux input subsystem might be a good place to start looking. Once we take that route, we discover the gpio_keys</a> driver. Configuring the gpio_keys driver</h2> The gpio_keys driver is configured with a few lines of code in your Linux board configuration file. An example is below: static struct gpio_keys_button svs_button_table[] = { { .code = KEY_RECORD, .gpio = PP_GPIO_MIC_EN, .active_low = 1, .desc = "MIC_EN", .type = EV_KEY, .wakeup = 0, }, }; </code></pre> In this application, we are reading button presses on a cell phone style headset. Once the gpio_keys driver is configured, a new entry will show up in /dev/input/eventX. An application can then do a blocking read on this device. Reading the GPIO in an application</h2> To read the GPIO, we simply do a blocking read on the new /dev/input/eventX device. The read will block until there is a change in GPIO state. An example is show below: #define MIC_INPUT_DEV "/dev/input/event0" static gboolean mic_button_callback(GIOChannel source, GIOCondition condition, gpointer data) { struct input_event ev; int bytes_read; g_io_channel_read_chars(source, (gchar )&ev, sizeof(ev), &bytes_read, NULL); if (bytes_read > 0) { if (bytes_read != sizeof(ev)) { s_debug(1, "warning, only read %i bytes from mic input"); return TRUE; } } else { return TRUE; } if (ev.type != EV_SYN && ev.value == 1) { /* button pressed, do something ... / } return TRUE; } void mic_button_init() { GIOChannel micbutton = g_io_channel_new_file(MIC_INPUT_DEV, "r", NULL); if (micbutton == NULL) { s_debug(TRUE, "Error initializing mic button"); return; } g_io_channel_set_encoding(micbutton, NULL, NULL); guint id = g_io_add_watch(micbutton, G_IO_IN, mic_button_callback, NULL); } </code></pre> The above example also shows how to incorporate the GPIO read into a GLib mainloop so that you don’t need to create a separate thread. ( As a side, GLib mainloop programming is worth learning!) Using this method, reading a GPIO interrupt is easy and requires very few lines of code. This is typical of complex systems like Linux – if you know how to do something, it is relatively easy, but getting started down the right path is sometimes the challenge. Tips for reading a serial data stream in Python 2009-01-09T00:00:00+00:00 Interfacing with a RS232 serial device is a common task when using Python in embedded applications. The easiest way to get python talking to serial ports is use the pyserial project found at http://pyserial.sourceforge.net/</a>. This module works on most platforms and is straightforward to use (see examples on project web site). However, getting the read function in this module to operate in an optimal way takes a little study and thought. This article investigates how the pyserial module works, possible issues you might encounter, and how to optimize serial reads. We start out with several goals as to how we want the application to behave in relation to the serial port: application must block while waiting for data.</li> for performance reasons, we want to read decent size chunks of data at a time if possible. Python function calls are expensive, so performance will be best if we can read more than one byte at a time.</li> We want any data received returned in a timely fashion.</li> </ul> A key parameter in the pyserial Serial class is the timeout parameter. This parameter is defined as: timeout=None, #set a timeout value, None for waiting forever </code></pre> The Serial class read function also accepts a size parameter that indicates how many characters should be read. Below is the source for the read function on Posix systems (Linux, etc): def read(self, size=1): """Read size bytes from the serial port. If a timeout is set it may return less characters as requested. With no timeout it will block until the requested number of bytes is read.""" if not self.fd: raise portNotOpenError read = '' inp = None if size > 0: while len(read) < size: #print "\tread(): size",size, "have", len(read) #debug ready,_,_ = select.select([self.fd],[],[], self.timeout) if not ready: break #timeout buf = os.read(self.fd, size-len(read)) read = read + buf if self.timeout >= 0 and not buf: break #early abort on timeout return read </code></pre> The easy way to use this module is to simply set the timeout to None, and read size to 1. This will return any data received immediately. But, this setup is very inefficient when transferring large amounts of data due to the Python processing overhead. To meet our goal of reading multi-byte blocks of data at a time, we need to pass the read function a size greater than 1. However, if timeout is set to None, the read will block until size bytes have been read, which does not meet the goal of returning any data read in a timely fashion. The solution then is to: set the read size high enough to get good performance</li> set the timeout low enough so that any data received is returned in a reasonable timeframe, but yet the application spends most of its time blocked if there is no data.</li> </ul> As an example, a size of 1000 and a timeout of 1 second seems to perform well. When used this way, the pyserial module performs well and returns all data read quickly. How to Optimize Python reads for general data 2009-01-09T00:00:00+00:00 The Python file object read() function acts a little differently than the standard read() found in C. This article describes some of these differences and how to optimize reads for general continuous data streams such as reading data from a collection device through a pipe. Python read()</h2> The Python read function seems to be optimized for reading files and text oriented streams. By default, a read() call will block until a EOF is encountered. This is very handy for reading files from a disk. You can just slurp them up with one read() statement. If you pass a size parameter to read(), it will block until size bytes have been received. This is less than ideal for reading a continuous data stream where some of thye data may be stuck until the size threshhold is reached. Non-blocking read()</h2> The way you get read to return with whatever data is available even if it is less than the read size parameter is to set the file object up in non-blocking mode. This can be done using the fcntl module: flags = fcntl.fcntl(fp, fcntl.F_GETFL) fcntl.fcntl(fp, fcntl.F_SETFL, flags | os.O_NONBLOCK) </code></pre> However, now the application does not block while waiting for data and spins using CPU resources. select()</h2> Enter the select call. The Python select module does much the same thing as the C select() function. In this case, it can be used to block waiting for data from a non-blocking file object with the added benefit of a timeout. So the resulting code might look like: fp = os.popen(<application that returns data to stdout>, 'r') flags = fcntl.fcntl(fp, fcntl.F_GETFL) fcntl.fcntl(fp, fcntl.F_SETFL, flags | os.O_NONBLOCK) while 1: [i, o, e] = select.select([fp], [], [], 5) if i: s_ = fp.read(1000) else: s_ = '' if s_: logging.debug("received %i bytes of data, total = %i" % (len(s_), total)) <do something with s_> </code></pre> The above select statement blocks until data is available from the fp object or times out after 5 seconds. Reading continuous data streams in Python is very possible, but usually requires the file object to be set in non-blocking mode and a select used to block while waiting for data. Implementing Asterisk 2009-01-09T00:00:00+00:00 Over the past month, we have been working on implementing Asterisk in our office. Asterisk (http://www.asterisk.org/</a> ) is a complete software PBX (Private Branch Exchange) that runs on top of Linux. Asterisk supports a number of features commonly found in a PBX including voicemail, hosted conferencing, call queuing, and many other features. Having a real PBX provides many benefits for a business and has the potential to improve our service and lower our costs. Over the next couple months we will be writing about our experiences implementing Asterisk, tips for setting it up etc. One of the most amazing things about Asterisk is its flexibility. You can mix and match IP phones, traditional analog phones, soft phones, traditional PSTN lines, and VOIP services in about any way you want. Our setup is fairly simple – IP phones and connections to the outside world using our PSTN land line. We are also evaluating several VOIP providers and trying to determine if VOIP is viable. Stay tuned … How to Implement a Web Application Framework in an Embedded Linux System 2009-01-09T00:00:00+00:00 As devices are increasingly more networked, an embedded web server is becoming a standard way for users to interact with and configure an embedded device using a standard web browser. As an example, most pieces of networking equipment (such as routers and wireless access points) are configured this way. There are many ways to implement a web server. One way is to just write a monolithic program that handles the requests and outputs HTML using print statements. Without a lot of planning, you usually end up with something where logic is intermixed with presentation. The disadvantage of such an approach is that it quickly becomes very difficult to maintain your web application as it grows and changes. The current best practice with web application frameworks (such as Ruby on Rails) splits a web application into 3 distinct components (Model, View, Controller – MVC) so that changes to one component can be made with minimal impact to others. This article details a solution used in a recent project to implement such an architecture using the following open source components: SQLite, Clearsilver, and Python. The challenge was to find a solution that performed acceptably on a resource constrained 130MHz ARM-Linux system. Components Used</h2> The requirements for the web application framework for this system are: Footprint must be fairly small – less than 10MiB.</li> Must enable us to implement a clean MVC type architecture.</li> Must support a high level language like python for rapid development.</li> Includes a database.</li> Reasonable performance – less than 1 second to render a typical page.</li> </ul> After researching and testing several options, the following components were chosen to implement the framework: Web Server: Cherokee (1MiB)</li> Model: SQLite (290KiB)</li> View: Clearsilver (170KiB)</li> Controller: Python (2-3MiB)</li> </ul> Total size for the above components is about 4MiB. The following sequence occurs during a typical web transaction: An HTTP request is received by Cherokee</li> If the request matches the URI for the web application, Cherokee forwards the request to the web application framework via CGI.</li> Clearsilver parses the HTTP headers (including CGI GET and POST parameters) and provides the data to python in a easy to access HDF format.</li> Python looks at the URI and dispatches the request to the appropriate function. Data is extracted from the SQLite database and an HDF datastructure is created.</li> Clearsilver is called to render the appropriate template. Data from the HDF datastructure is used in the template to provide the dynamic content in the page.</li> The rendered HTML is passed back to Cherokee, and is then returned to the user’s web browser.</li> </ul> Each component is discussed in more detail in the following sections. Performance</h2> One of the issues with modern web frameworks is that most require a lot of processing power. Many frameworks are written in an interpreted language which tend to be not very efficient on embedded systems such as the 130MHz ARM-Linux system used in this project. As most of the application development for this project is done in Python, I tried several other Python based solutions with the following results: Django: takes 10 seconds to render a page</li> webpy: takes 5 seconds to render a page</li> Clearsilver/Python: takes about 1 second to render a page</li> </ul> Anything over 1 second is very slow for a web interface. With fastcgi, webpy could probably have been sped up to be acceptably fast, but I don’t think there is much hope for frameworks like django on this type of system. I have read of similar experiences trying to run Ruby-on-Rails on ARM-Linux systems. As I’m currently just using a basic CGI interface, the Clearsilver solution could also be sped up significantly as well, if the Python portion could be kept running between requests with fastcgi or a similar mechanism. Cherokee Web Server</h2> A web application typically consists of a web server (like Apache) and a program that provides the dynamic web content. Some of the functions of a web server are: receives HTTP requests</li> handles the request by serving static files/images or routes the request to a program that outputs dynamic content</li> authentication</li> encryption (SSL, TLS)</li> </ul> There are many web servers that can be used in embedded devices. Some of them are: Lighttpd (http://www.lighttpd.net</a> )</li> thttpd (http://www.acme.com/software/thttpd/</a> )</li> Appweb (http://www.appwebserver.org/</a> )</li> Boa (http://www.boa.org</a> )</li> Cherokee (http://www.cherokee-project.com</a> )</li> SHTTPD (http://shttpd.sourceforge.net/</a> )</li> </ul> Cherokee was chosen for this application because it provides a good balance between the size and functionality I am looking for. It is also included in the Openembedded (http://openembedded.org</a> ) build system I am using. Other systems will have different requirements – use the one that fits your application best. Clearsilver</h2> Clearsilver (http://www.clearsilver.net/</a> ) is the real gem that was discovered during this exercise. Clearsilver is a language-neutral HTML template system written in C. It is used as the templating system for many high volume sites such as Google Groups 2. Clearsilver also provides CGI handling functions and bindings to several languages including Python. The fact that it is written in C and is fast for high volume sites also makes it fast enough on slower embedded systems. Because Clearsilver is written in C, the templating system is already fast. Parts or all of the application can also be moved to C as needed to get the required performance. Having a performance upgrade path is nice. Clearsilver also forces a strict separation of application logic and presentation templates, which keeps things in line with the MVC architecture. SQLite</h2> SQLite (http://www.sqlite.org/</a> ) is a small C library that implements a self-contained, embeddable, zero-configuration SQL database engine. SQLite works as expected and also has bindings to a number of languages including Python. A SQLite database is just a single file that requires no configuration, making it very easy to use. The database engine runs in the same process as the application using it, which increases performance because there is no context switch when running database operations. Because SQLite does not have its own process, it does not handle concurrency as well as other databases such as MySQL, but in an embedded system you seldom have more than one process using the database. SQLite provides some support for concurrency with reader/writer locks for the entire database. If your application is not real busy, this is often adequate if you need two processes to share some data. Python</h2> Python is the glue that ties things together in this web application framework. It sits between Clearsilver and the database. The advantages of using a language like Python are rapid development and an extensive library. Python is considerably slower than C, but there is always the option to move parts of the application into C as needed. Openembedded Build System</h2> How does one put together a system with all these components? The answer is the Openembedded Build System (OE) (http://openembedded.org</a>). OE includes support for all of the components discussed in this article. Summary</h2> The combination of Python, Clearsilver, and SQLite provides a compelling solution for a web application framework in an Embedded Linux System. The solution is reasonably small, performs well, and provides a path to improve performance if needed. We also get a clean architecture to create clean, maintainable web applications. About the author – Cliff Brake owns BEC Systems, a consulting operation that helps customers utilize modern computer technologies in their products. BEC offers a range of services to make your embedded project a success including technology selection, development, troubleshooting, and training. Please visit our website (http://bec-systems.com</a> ) for more information and other free resources. Do you have an Exit Strategy? 2009-01-09T00:00:00+00:00 Some time ago, eWeek published in interesting article by Scott McNealy named “Technology’s Barriers to Exit</a>”. In this article, Scott makes an interesting point about considering the cost of technology. There are three costs to consider: cost of acquisition</li> ongoing cost</li> cost to exit</li> </ul> Scott states that most people analyze the first two costs fairly well. What is not thought about as much is the the cost to exit – how do I move on to the next thing. I think these basic questions apply very much to embedded technology decisions. Most of the time we carefully consider our requirements, think about ongoing manufacturing costs, etc. There may be some thought given to upgrading (the exit), but we encounter several difficulties when planning for the future with complex embedded systems: we often can’t predict what technologies will be available in the future</li> if may be difficult to predict our product requirements in the future</li> </ul> As an example, we may have a product with an ethernet networking interface, and two years after product launch we have an opportunity that requires a wireless network interface. How much is it going to cost to add wireless support to our product? If the product is based on technologies that do not fundamentally support wireless networking, we are looking at very costly exit strategy. Another example is application-specific, off-the-shelf solutions. While these are great for many problems and provide a quick way to get something working, what happens when we want to upgrade? Do we have to re-implement the entire system? How does one hedge against the unknowns in the future? One way is to try to anticipate every possible requirement and design it in. I think this is bad strategy because you usually end up spending a lot of time working on things you don’t really need and the quality of the product suffers. A better approach is make sure your product does what it needs to do well and is based on a flexible platform where significant parts of your design and software can be re-used in future versions. This approach has many parallels to recent trends in software development and manufacturing such as Agile Software Development, and Lean Manufacturing. Follow Scott’s advice and make technology selection decisions with the following questions in mind: Is the technology based on open standards?</li> Are programming interfaces and protocols open, published, and been adopted by industry?</li> Is there a community process behind the technology (not just one company)?</li> Can the same technology be sourced from multiple companies?</li> </ul> The use of Linux in embedded systems is an excellent example of a technology that meets the above criteria. You can develop your prototype on a x86 PC. Move it to an embedded ARM single board computer (SBC) for the first production run. Switch to a different SBC or design your own board when you run into new product requirements. Sure, it will cost you more to develop the product initially due to the complexity of a system like Linux, but you now have options to expand and change your product while directly re-using the work you have already done. But wait, as long as I write my application in C, it is portable – right? Most embedded systems support development in C so I will just port my application to a new platform when needed. C is a great language and the language is more or less portable between systems. But as systems get more and more complex, embedded engineers find themselves spending less time writing original code and spending more time putting together pieces that already exist. A modern embedded application is often little more than some glue between various libraries and system functions. The fact that you can develop in C for a particular system does not really mean a lot. What about the threading model, driver model, library APIs, etc. I just spent a significant amount of time recently porting a large C application from a proprietary platform to new hardware platform as the original system was going obsolete. It was a huge amount of work. The driver model was completely different, APIs were different. Changing platforms can be very expensive. It rarely pays to be short sighted when choosing technologies for complex embedded systems. The best strategy is to stay flexible and choose a platform with a future. Gumstix Overo Connector Spreadsheet 2008-12-15T00:00:00+00:00 One of the first things I do when designing a system based on a processor module is create a spreadsheet listing all the connector I/O. This is the easiest way I’ve found to make sure nothing gets missed. My first pass at a I/O spreadsheet for the Gumstix Overo is available at: http://redmine.bec-systems.com/bec/documents/1</a> How to disable SSH host key checking 2008-12-12T00:00:00+00:00 This article</a> presents a very nice overview of options to disable ssh host key checking. When working with embedded systems where you are constantly reloading the rootfs, it is really annoying to have to edit your ~/.ssh/known_hostsevery time you update the rootfs on the target device. The solution is to simply disable host key checking for your local subnet by adding something like the following to the beginning of /etc/ssh/ssh_config: Host 192.168.1.* StrictHostKeyChecking no UserKnownHostsFile=/dev/null </code></pre> Anything not on the 192.168.1.* subnet is still checked. Update 2009-09-26: Another place to put these config settings is: ~/.ssh/config Printing from Embedded Systems 2008-12-02T00:00:00+00:00 How does one implement support for printing in embedded systems? I recently had the opportunity to add printing support to an embedded Linux system. The device is an industrial touch screen powered by a Compulab cm-x270 module (PXA270 CPU), and runs a GTK+ application. The customer is implementing a device calibration system where customers bring their equipment in to get calibrated, and the system prints out a report on a local printer. This article describes how components of Hewlett Packard’s HPLIP solution along with Cairo can be used to implement printer support in a non-desktop Linux system. Requirements</h3> The requirements for this project were fairly basic – we needed to print a single page report that contained text and some elementary graphics. We initially wanted to support several low-end Inkjet and Laser USB printers. As the system was powered by a fairly slow ARM processor (slow compared to modern desktop systems), the solution needed to be efficient, and not require excessive amounts of memory. Desktop Linux Print Flow</h3> With desktop Linux systems, the standard printing flow looks something like this: Application -> PS output -> Ghostscript -> Rasterized output -> Printer Driver (filter) -> I/O Backend -> Printer CUPS is typically used to manage this flow, provide queuing, etc. Print Architecture</h3> As the application is written in GTK+, we decided to generate the report using Cairo, which is a 2D graphics library. Cairo is easy to use, and is well suited for this application. With a Cairo generated report, we already had a raster image of the report, so it seemed that the PS output/Ghostscript steps were not really needed and only added more processing to the data flow. Also the PXA270 CPU does not have a FPU, and PostScript processing can be floating point intensive. So the need was now to figure out how to get a Cairo generated raster image to a USB printer. Hewlett Packard offers lots of interesting software for their printers. Their APDK is a OS independent library in source code form. However, implementing this would have required quite a bit of integration, and writing the I/O layer. It seemed like there should be something available that would work with a little less effort. The HPLIP project</a> is a comprehensive set of software for Linux printing, and is used by desktop Linux systems. It is not obvious at a glance how all the components of HPLIP fit together, but after spending some time digging through source code, and asking questions on several maillists/forums, we were able to figure out that the basic flow is: Ghostscript -> HPIJS (driver) -> HP backend -> Printer In this application, we simply launched the HPIJS driver directly from our application instead of Ghostscript. HPLIP</h3> There were several challenges using HPLIP in our ARM system. The HPLIP is not cross-compilation friendly out of the box, so we had to fix a few issues, and ended up disabling all the Python pieces as we only needed the HPIJS and backend components. We also had to figure out the data flow from the application to HPIJS, and then to HP backend. In a nutshell, the process is: start the hp backend process by forking, and obtain a file handle to STDIN for the hp backend process</li> start the HPIJS process, send various parameters to it including the file handle for the backend STDIN</li> send HPIJS the print raster and tell it to print</li> </ul> There are some additional details such as handling margins, error conditions, cancel support, etc. Some of the details can be gleaned from the Ghostscript source code, and the IJS reference implementation</a> provides some very useful library code for implementing the IJS client functionality in the application that is doing the printing. When finished, our printing module was 403 source lines of code (SLOC) – not bad considering the functionality. How well does it work?</h3> Overall, we are very pleased with the result. With a simple industrial terminal, we can now support just about every USB printer made by HP with the exception of some of their very low end LaserJet printers that require a binary plugin. The HP backend can be used to detect what printer is attached at run time, so everything is plug-n-play with no user configuration. Let me repeat as this is significant – on a simple Industrial terminal, we can support about every HP USB printer available with no user configuration. This is easier for users than their desktop as they simply need to buy a printer and plug it in! Kudos to HP for their excellent open source software. Because the source code is available, we were able to customize it for our application with very little support from HP. Performance wise, the printing process is quite fast; the printer starts almost immediatly after the user initiates the print. Now if I could just get HP to build a ARM version of their binary plugin for the few low end lasers we can’t support … but this is not a critical issue for this vertical application. Future Direction</h3> Hopefully, this type of solution can evolve into a standard printing solution for Embedded Linux systems. Epson also supports the IJS driver model, so adding support for their printers should be possible. It may eventually make sense to integrate portions of CUPS for queing, and other management tasks. Some of the tasks I hope to implement in the future: Clean up the HPLIP build for inclusion in OpenEmbedded.</li> Clean up the IJS reference code library build and packaging in OpenEmbedded.</li> Figure out portions of CUPS that may make sense.</li> Keep conversations going with the Linux printing group so we can someday have a “standard” printing solution for embedded Linux systems.</li> </ul> Thanks to Matt Gessner for helping implement this solution, and for providing feedback on this article. Things that work! 2008-11-19T00:00:00+00:00 Running a small business has a special set of challenges, and if you are not careful, the overhead can quickly consume you. This article is a list of things that I use in my business that work well for me, and help me to be efficient when there is so much to do … Entire article at: /web/content/view/101/9/</a> Asterisk</h3> While I can’t say the same for VOIP proiders, the Asterisk PXB is a very solid product. I run it on an ancient 1GHz celeron system, and use it for my office PBX. In combination with a VOIP service, I can: handle multiple calls at once</li> easily conference multiple callers</li> forward calls to my cell phone or other numbers</li> provide extensions for associates who are working with me</li> </ul> and all at a very low cost. To mitigate some of the reliability problems I’ve had with VOIP providers, I set the VOIP provider to use my cell phone as a backup if the VOIP link is down. I also use multiple VOIP providers for outgoing calls so that if one is not working, I can usually use the backup. Overall, I really like the features, and for my business the occasional hassle is worth it. SQL-Ledger</h3> SQL-Ledger is a web based accounting program that is open source. It provides all the features I need to run my business, and offers the following advantages over more traditional programs such as Quickbooks: web based so I can access it anywhere, even remotely.</li> subcontractors and associates can log time remotely directly in my accounting system.</li> granular permissions for different people who need to use it.</li> upgrades just work.</li> consistent, reliable operation that I expect from an accounting system.</li> </ul> OpenOCD</h3> We finally have a low cost, open source JTAG solution that works reasonably well. While there are still issues, it gets the job done for most of my projects, and offers my customers a low cost way to program their systems using JTAG. Often customers are not doing extensive development, and it is nice to be able to provide a solution for the occasional need. One customer fought with a very expensive commercial unit for weeks. I finally shipped them a $50 Olimex unit that did everything they needed. There can’t be enough said for everyone on a team using the same tools, and they only way this can happen easily is if the tools are fairly low cost. Tracks</h3> Tracks is a nice web-based task managment tool to help implement the GTD methodology. The user interface is supurb and better than most native applications I’ve used. I use the “starred” items for my “next action” on each project. Very nice! Printable CEO Emergent Time Tracker</h3> I use David Seah’s handy form</a> for tracking my time during the day. It forces me to use a pen and paper every day which is a pretty rare thing, and is a nice compromise between simplicity and automation. OpenEmbedded</h3> I’ve written a lot about the benefits</a> of OpenEmbedded. It makes the huge task of creating a build system for Embedded Linux systems very manageable and repeatable. When build systems are fully automated, things are so much easier. As an example, I’ve been working on one project for over two years, and have built 313 releases. We have had almost no image failures due to forgetting to put something in the image, or some other manual step you tend to find in build systems. This is the power of a fully automated build system with _no_ manual steps. Ubuntu</h3> Once I become convinced that Ubuntu could do distro upgrades, I finally quit using Gentoo. Between servers, workstations, and home computers, I now maintain about 10 Ubuntu systems. Over the past several years, I’ve had very few problems using the distro upgrade mechanism. Also, the fact that Ubuntu includes fairly recent Linux technology is a big plus for experimenting with components I use in embedded Linux systems. DHL/UPS Pickup</h3> Working from a small office where I do not regularly make shipments, it is really convenient to be able to schedule a pickup with shipping companies like DHL and UPS. This can be done over the web in 15 minutes or so, and a shipping label then prints out on your printer. The DHL website will also provide commercial invoices for overseas shipping. This sure beats driving to the local UPS shipping place, filling out paper work with a pen, waiting in line, etc. Just another thing that makes it easier for a small business to run efficiently. Intel Bare Bones Server</h3> Last year, I was looking for a cost effective 1U rack server. I selected a Intel SR1530 Bare Bones server. I installed it over a year ago and have not touched it since. Last time I checked, the uptime was 254 days. 3-ware Raid Controllers</h3> (I guess it is AMCC now …) They just work with Linux – currently running in two systems. Sandusky Cabinets</h3> Doing embedded system development means I get a lot of equipment from customers. As some of the development stretches over several years with occasional ongoing support, its nice to be able to store this equipment and then find it efficiently. I’ve found the “Sandusky Standard Storage Cabinets” from Staples to work very well. They are fairly cheap, yet durable enough for what I need, and keeps everything enclosed which looks neat, and keeps the dust problem under control. And, Staples offers free delivery. Bob Pease organization system</h3> Bob Pease had an article</a> in Electronic Design some time ago about organizing things in his refrigerator. He used small boxes to group things in his refrigerator. I use the same method in my Sandusky cabinets and it makes it very easy to find something when I need it. GTD Labeling System</h3> I’ve slowing been adopting some of David Allen’s organization ideas from “Getting Things Done”. One of his best ideas is to get a labeller. I used to use a Brother handheld labeller, but lately, I’ve been using a Dymo LabelWriter 400 left over from a project. The LabelWriter connects to your computer via USB, works well with Linux, and allows me to use my standard keyboard, and OpenOffice to create labels. Note, the labels on the boxes in the above picture. The high contrast black on white makes it much easier to find things versus hand writing notes on cardboard boxes, file folders, etc. GTD Filing System</h3> I used to use hanging file folders, but no more. David Allen’s suggestion of simply using plain manila file folders with nice labels works very well. Once again the large, high contrast labels makes it easy to find things. Snaketray Cable Management</h3> I’ve already written about my cable management system</a>. A year later, I still like it and no longer dread digging through cables. Steelcase Leap Chair</h3> I’ve sat in various chairs over the years – mostly the cheap office store variety that most companies buy. I must be getting old enough now that they no longer cut it. I tried a Herman Miller Aeron for a week as that is what everyone recommends, but I found it too hard. I then tried a Steelcase Leap chair and it works great for me. I no longer even think about the chair I’m sitting in, even after a long day. Expensive, but well worth it if you spend as much time sitting as I do. Tomboy Notes + gnome-do</h3> Tomboy notes is a Linux desktop note-taking application that is essentially a personal wiki. It automatically tracks links to other pages, links to file locations, etc and organization is done mostly through links instead of hierarchy. It is simply the best note taking system I have ever used. Coupled with gnome-do, I can instantly bring up any note with only a few keystrokes. In this day and age where information abounds, being able to take and recall notes efficiently is critical to being effective. Konsole</h3> Being able to easily name the tabs in the KDE console application is very handy as I’ll typically have 5-10 named consoles going at one time. 8-port USB serial expander</h3> As I work with a lot of different devices, it is very handy to have lots of serial ports available. The SerialGear device</a> available from http://www.usbgear.com works out of the box with modern Linux distributions. Git</h3> I’m starting to really appreciate the benefits of distributed source control systems. For commercial embedded development, it allows me to easily track upstream projects, yet have a local source control system for tracking project changes. Having everything in one version control system makes it much easier to extract patches to push back upstream, etc. Redmine</h3> I’ve been a happy trac</a> user for years, but have recently switched to using Redmine</a> for project management. Redmine offers some addition features such as multiple project support, and the user interface is a little more polished. As all of my projects are accomplished with distributed teams, it is very important that there be clear visibility as to what is going on in the project. It also helps to give customers the “Warm Fuzzy Feeling</a> TM” as they have clear visibility into the project, and they can easily verify there is progress being made on the project. Benefits of OpenEmbedded switching to Git 2008-10-22T00:00:00+00:00 Recently the OpenEmbedded project has switched to the Git version control system. This is good news for many reasons. The obvious reasons are Git is faster than Monotone, handles branching better, has lots of nice features, larger user base, etc. Monotone has served us well, but as new tools become available, it is time to change. For those of us helping customers use OE for embedded projects, Git will make using OE much easier. Companies want to be able to track OE with minimal effort and typically maintain a few minor tweaks to OE. As most Embedded Linux projects now implement Git infrastructure for kernel development, the OE source tree can be maintained using the same infrastructure. Branching is very natural in Git, so it is fairly easy to create a OE “topic branch” and periodically merge with upstream changes as needed. The ability for easy repo hosting and branching allows OE users to lock down a version of OE for periods of project development, and still maintain a connection to the upstream OE repository for easily updating to new versions, or cherrypicking changes. So a big thank you to all who helped make this change happen. more fun with git branches 2008-08-26T00:00:00+00:00 One thing you learn after using git for awhile is you get used to trying random things, and it will often just work. For example, if I want to know the differences between Linus’s kernel tree, and the Wolfson dev branch for a particular directory: cbrake@happy:/build/linux-2.6$ git diff --stat origin/master..wolfson/dev sound/soc/codecs sound/soc/codecs/Kconfig | 125 +++ sound/soc/codecs/Makefile | 34 + sound/soc/codecs/ad1939.c | 690 +++++++++++++++ sound/soc/codecs/ad1939.h | 70 ++ sound/soc/codecs/ad1980.c | 309 +++++++ sound/soc/codecs/ad1980.h | 23 + sound/soc/codecs/cs4251x.c | 771 +++++++++++++++++ </code></pre> Another way to get similar information is: git log origin/master..wolfson/dev sound/soc/codecs </code></pre> If I’m working on a topic branch, and I want to see a summary of all the changes in my Topic branch: git log --stat origin/master..origin/my_topic_branch > my_topic_branch_changelog.txt </code></pre> One of the neat things about git branches is you don’t have to be on a branch to interact with it. You can diff, log, and checkout from branches other than the one you are currently in, and it is all very fast. For instance, if I want to grab the latest copy of a file in the Wolfson dev branch, I can do something like: git log checkout wolfson/dev sound/soc/codecs/wm9713.c </code></pre> Not that this is a very good idea, but just to illustrate a point. Git branches are amazing and make development work so much less tedious. Linux Input Testing and Debugging 2008-08-22T00:00:00+00:00 The Linux input layer has made a lot of progress in recent years. When writing a new input driver (such as keyboard, trackball, etc), it is useful to be able to monitor input events using a test application. This article describes two ways to accomplish this using kernel input debugging, and the evtest utility. kernel input debugging</h3> The Linux kernel includes some support for printing input events as debug messages. To use, set the following in the kernel config: CONFIG_INPUT_EVBUG=y </code></pre> Because the print messages are at the KERN_DEBUG level (7), we typically need to change the debug level: echo 8 > /proc/sys/kernel/printk </code></pre> At this point, you will see messages like the following on the console: evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 108, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 106, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 1 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 1 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 </code></pre> evtest</h3> The linux-input project (http://linuxconsole.sourceforge.net/input/input.html</a>) includes a neat little tool named evtest that is very handy for testing input devices. If you are using OpenEmbedded, simply: bitbake linux-input </code></pre> If you are not using OpenEmbedded, you can glean enough details about how to get and build the this package from the OE recipe: http://gitweb.openembedded.net/?p=org.openembedded.dev.git;a=blob;f=packages/linux-input/linux-input_cvs.bb</a> Install the resulting package on your device. To use, run evtest with a /dev/input/eventx argument: root@machine1:~$ evtest /dev/input/event0 Input driver version is 1.0.0 Input device ID: bus 0x19 vendor 0x1 product 0x1 version 0x100 Input device name: "gpio-keys" Supported events: Event type 0 (Sync) Event type 1 (Key) Event code 103 (Up) Event code 105 (Left) Event code 106 (Right) Event code 108 (Down) Event code 116 (Power) Testing ... (interrupt to exit) Event: time 1009.433605, type 1 (Key), code 108 (Down), value 0 Event: time 1009.433737, -------------- Report Sync ------------ Event: time 1010.735605, type 1 (Key), code 108 (Down), value 1 Event: time 1010.735740, -------------- Report Sync ------------ Event: time 1011.075586, type 1 (Key), code 106 (Right), value 1 Event: time 1011.075720, -------------- Report Sync ------------ Event: time 1011.103379, type 1 (Key), code 108 (Down), value 0 Event: time 1011.103550, -------------- Report Sync ------------ Event: time 1011.333924, type 1 (Key), code 108 (Down), value 1 Event: time 1011.334058, -------------- Report Sync ------------ Event: time 1011.339880, type 1 (Key), code 106 (Right), value 0 </code></pre> Notice that evtest lists the keymap when it is initially run which is very handy, and then lists key events as they occur. The key descriptions are also included with events so you don’t have to manually decode the keycodes. Using a Verizon USB720 modem in an Embedded Linux system 2008-06-02T00:00:00+00:00 The Verizon USB720 Cellular modem is a quick and easy way to add wireless internet connectivity to your embedded system. Why might you want to add cellular connectivity to your embedded device? Remote access and diagnostics is a big reason. One customer I have is planning to supply all their distributors with a USB720 so they can easily upload log files from devices in the field. The alternative is copying the log files to a flash drive, finding a PC connected to the internet, and then uploading the files – too much hassle, therefore it does not get done. Also, think about the cost of sending a field support person on-site to troubleshoot problems versus shipping them a USB720 and accessing the device remotely. The USB720 is a fairly small device that plugs into the ubiquitous USB port. This articles covers some of the details on how to configure an embedded Linux system to work with this device. Why is the USB720 a good fit for wireless connectivity in Embedded Systems?</h3> Some reasons the USB720 (or other USB based cellular modems) is an attractive option: fairly fast: provides typical download speeds of 600-1400kbps, and upload speeds of 500-800kbps</li> convenience: it plugs into a USB port. No taking apart the box and installing an embedded module, etc.</li> affordable: with plans around $60/mo, its a lot cheaper than sending field support people on-site to debug every problem.</li> uses standard PPP protocols for connection. That means it works with Linux.</li> supported by the mainstream Linux kernel</li> </ul> Kernel Configuration</h3> The USB720 looks like a USB-serial device to the host system. If you plug it into your Ubuntu desktop Linux system, all the needed kernel pieces are there and it will just work. With your embedded system, you need to make sure a number of kernel options are enabled including: CONFIG_PPP=m CONFIG_PPP_ASYNC=m CONFIG_PPP_DEFLATE=m CONFIG_SLHC=m CONFIG_USB_SERIAL=m CONFIG_USB_SERIAL_OPTION=m </code></pre> The CONFIG_USB_SERIAL_OPTION selection enables a driver that is specifically optimized to work with the USB chip found in many cellular modems. Once you enable the above modules, and if you are running udev, simply plugging in the device will cause the appropriate modules to load and a /dev/ttyUSB0 device will now appear. PPP Scripts</h3> The most difficult part of this exercise is figuring out how to configure the PPP scripts. Verizon of course does not tell you how to configure their device and only supplies software for Windows and MAC. There are a number of howtos available on the internet, but most of them are tailored to using kppp or wvdial. Neither are very practical for embedded systems. Fortunately, the standard pppd daemon is fairly easy to use, is easy to build for embedded systems (it is already included in OpenEmbedded), and seems to work very well. The following scripts are needed: /etc/ppp/peers/verizon noauth connect "/usr/sbin/chat -v -f /etc/ppp/peers/verizon_chat" defaultroute usepeerdns ttyUSB0 921600 local usepeerdns debug -detach </code></pre> /etc/ppp/peers/verizon_chat '' 'ATZ' 'OK' 'ATDT#777' 'CONNECT' '' </code></pre> /etc/ppp/chap-secrets /etc/ppp/pap-secrets "<phone number>@vzw3g.com" * "vzw" </code></pre> At this point, you can run pppd call verizon and you should see something like: root@cm-x270:~$ pppd call verizon Serial connection established. using channel 9 Using interface ppp0 Connect: ppp0 <--> /dev/ttyUSB0 Warning - secret file /etc/ppp/pap-secrets has world and/or group access sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x539657b7> <pcomp> <accomp>] rcvd [LCP ConfReq id=0x0 <mru 1500> <asyncmap 0x0> <magic 0xb9b652e5> <pcomp> <accomp>] sent [LCP ConfAck id=0x0 <mru 1500> <asyncmap 0x0> <magic 0xb9b652e5> <pcomp> <accomp>] rcvd [LCP ConfAck id=0x1 <asyncmap 0x0> <magic 0x539657b7> <pcomp> <accomp>] sent [CCP ConfReq id=0x1 <deflate 15> <deflate(old#) 15>] sent [IPCP ConfReq id=0x1 <compress VJ 0f 01> <addr 192.168.1.119> <ms-dns1 0.0.0.0> <ms-dns3 0.0.0.0>] rcvd [LCP DiscReq id=0x1 magic=0xb9b652e5] rcvd [LCP ProtRej id=0x2 80 fd 01 01 00 0c 1a 04 78 00 18 04 78 00] rcvd [IPCP ConfReq id=0x0 <addr 66.174.61.4>] sent [IPCP ConfAck id=0x0 <addr 66.174.61.4>] rcvd [IPCP ConfRej id=0x1 <compress VJ 0f 01>] sent [IPCP ConfReq id=0x2 <addr 192.168.1.119> <ms-dns1 0.0.0.0> <ms-dns3 0.0.0.0>] rcvd [IPCP ConfNak id=0x2 <addr 70.210.82.191> <ms-dns1 66.174.95.44> <ms-dns3 66.174.92.14>] sent [IPCP ConfReq id=0x3 <addr 70.210.82.191> <ms-dns1 66.174.95.44> <ms-dns3 66.174.92.14>] rcvd [IPCP ConfAck id=0x3 <addr 70.210.82.191> <ms-dns1 66.174.95.44> <ms-dns3 66.174.92.14>] not replacing default route to eth0 [192.168.1.1] local IP address 70.210.82.191 remote IP address 66.174.61.4 primary DNS address 66.174.95.44 secondary DNS address 66.174.92.14 Script /etc/ppp/ip-up started (pid 2303) Script /etc/ppp/ip-up finished (pid 2303), status = 0x0 </code></pre> root@cm-x270:~$ ifconfig ppp0 Link encap:Point-to-Point Protocol inet addr:70.210.82.191 P-t-P:66.174.61.4 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:4 errors:0 dropped:0 overruns:0 frame:0 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:3 RX bytes:64 (64.0 B) TX bytes:94 (94.0 B) </code></pre> You are connected :-) Using the OE ppp package</h3> Add ppp support to your image using OE is fairly straightfoward: add the kernel modules to your image</li> addppp to your image</li> add a custom package that includes the configuration scripts listed above</li> </ul> One thing that may trip you up is the ppp scripts are expecting resolv.conf to be in the following location: /var/run/resolv.conf. The reason for this is we want resolv.conf in a RAM file system to prevent unneeded flash wear. Therefore, /etc/resolv.conf should be configured in your rootfs as a soft link to /var/run/resolv.conf. Summary</h3> The Linux kernel support for USB720 and the pppd package offer a quick, robust way to add cellular network connectivity to about any embedded Linux system that has a USB host port. As systems become more complex, it will be increasingly important to be able to remotely access systems in the field to troubleshoot problems, help users with configuration, etc. The USB720 offers a quick way to provide this type of connection without going through the expense of designing cellular connectivity into your product. One thing I usually encourage all my customers to do when designing new products is expose a USB host port, even though they may not see any immediate use for it. This is one of those reasons. Below is a picture of a USB720 connected to a Compulab cm-x270 development system. Autotools quick reference 2008-05-03T00:00:00+00:00 At some point, most Linux developers need to master Autotools. For most of us, this is a fairly painful process, but like any good tool, Autotools is extremely useful and well worth learning. For example, if your program is set up correctly using Autotools, it will cross compile with almost zero effort in OpenEmbedded, or with slight effort manually. Compare this to the tedious effort of manually referencing toolchain locations, cross toolchain library and header file locations, etc. Despite first impressions, Autotools will make your life easier. This article contains a collection of tips on how to do standard operations using Autotools. Minimal Autotools files required for a simple C application</h2> configure.ac AC_INIT([fooprog], [1.0], [email@company.com]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_PROG_CC AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([Makefile]) AC_OUTPUT </code></pre> Makefile.am bin_PROGRAMS = foo foo_SOURCES = foo.c </code></pre> The above files will create a binary name foo (compiled from foo.c) when you run the following steps: touch NEWS AUTHORS ChangeLog</li> autoreconf -i</li> ./configure</li> make</li> </ul> How to add libraries to the link step</h2> For example, if you want to link in the pthread library to your application, add the following line to Makefile.am: foo_LDADD = -lpthread </code></pre> How to add compiler options</h2> You can easily add compiler options using the foo_CFLAGS or foo_CPPFLAGS variables. foo_CFLAGS = -DMY_DEFINE How to use pkg-config in Autotools</h2> pkg-config is a helper tool that gives you library and include file information for various libraries installed. For example, to link to alsa and glib in your project: configure.ac PKG_CHECK_MODULES(PACKAGE, alsa glib-2.0) AC_SUBST(PACKAGE_CFLAGS) AC_SUBST(PACKAG_LIBS) </code></pre> Makefile.am foo_CFLAGS = @PACKAGE_CFLAGS@ foo_LDADD = @PACKAGE_LIBS@ </code></pre> How to add a command line option to the configure script</h2> AC_ARG_ENABLE(option-foo, [ --enable-option-foo help for option foo], [CFLAGS="$CFLAGS -DOPTION_FOO_DEFINE"], []) </code></pre> The above example sets a define on the compiler command line if configure is run with the --enable-option-foo command line option. Disable shared libraries in your project</h2> Add the following to configure.ac: AC_DISABLE_SHARED </code></pre> Pass a variable from configure.ac to Makefile.am</h2> Often there is a need to pass a variable in configure.ac to a Makefile. This is done with the AC_SUBST macro: configure.ac AC_SUBST(VAR_FOO) </code></pre> Makefile.am foo2 = @VAR_FOO@ Installing data files</h2> By default, Autotools will install executables built when you run “make install”. To install other non-executable files, you need to tell autotools explicitly about those files. The standard place to install data files for an application is in $(datadir)/appname. Makefile.am fooextradir = $(datadir)/@PACKAGE@ </code></pre> fooextra_DATA = \ foo_data1.txt \ foo_data2.txt </code></pre> The above will install the foo_data1.txt and foo_data2.txt files in the $(datadir)/fooprog directory. $(datadir) is typically /usr/share/ or /usr/local/share, but it can be anything. How does my program know where $(datadir) is located</h2> Because $(datadir) can be anywhere in the system, we need some way to tell the program where it was specified to be at time of compilation so that it can locate foo_data1.txt and foo_data2.txt. This can be done by passing a define at compile time: Makefile.am AM_CFLAGS = -DDATADIR=\"$(datadir)\" </code></pre> How to install data files during development without cluttering up my system directories</h2> If you use the above method to install data files needed by your project, you may run into a case during development where you want to run the program from the build directory, but the program needs the data files at run time. Since, the program is looking for them in the location specified by the DATADIR define, they need to be installed. You could just run “make install”, but this would install them into /usr/local/share, require you to run as root, and clutter up your system directories. A much more elegant solution is to specify the install prefix at configure time to be a temporary “install” directory in your source tree. The --prefix option for configure requires an absolute directory, so the `pwd` trick is used to figure out what the current directory is. ./configure --prefix=`pwd`/install make install </code></pre> Your project will now be installed in a directory in your source tree named ./install. This is also a nice way to test that the install for the project is working correctly as you can easily see everything that gets installed. Now, the program has been compiled to reference extra files in ./install and will run just fine from the build directory. The Linux kernel container_of macro 2008-04-02T00:00:00+00:00 The Linux kernel contains a very useful macro named “container_of” that is used extensively in back-casting a data structure to its containing data structure. This article includes a simple program that illustrates how this macro is used, and explains why it is so useful. If you do a lot of C programming, this program is worth figuring out :-). The program source can also be downloaded from the following location: /linux/linux_container_of_example.c</a> /* test code to illustrate use of Linux kernel container_of macro * * Copyright (c) 2008 Cliff Brake, BEC Systems LLC * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. * / #include <stdio.h> #include <string.h> #include <stdlib.h> / This program illustrates how the container_of macro works. * The container of macro is very useful in multi layered * software systems where you have progressivly more detailed * software layers. Below is an example of a bus layer, * and then a device layer where a number of different * devices might register with the bus. * The device registers itself with the bus subsystem, and * then the bus subsystem makes a callback into the device. * Normally if there are multiple devices registered, the * bus subsystem must store and pass a device structure * when making callbacks. With the container_of macro, this is * no longer necessary, and the bus subsystem only has to * know about one generic device structure, and does not need visibility * into lots of different device structures, or do tricks * by casting void pointers, etc. With the container_of macro * we can backcast from the generic data structure, to the containing * datastructure. This forces good separation of code in that * that bus layer cannot modifiy data structures that are specific * to the device layer. * */ /** * (from Linux kernel source) * container_of - cast a member of a structure out to the containing structure * @ptr: the pointer to the member. * @type: the type of the container struct this is embedded in. * @member: the name of the member within the struct. * / #define container_of(ptr, type, member) ({ \ const typeof( ((type )0)->member ) mptr = (ptr); \ (type )( (char )mptr - offsetof(type,member) );}) #define offsetof(TYPE, MEMBER) ((size_t) &((TYPE )0)->MEMBER) /========================================================== BUS layer code ==========================================================/ /* generic bus device structure / struct bus_device { int general_device_param_a; int general_device_param_b; void (device_callback)(struct bus_device * bd); }; /* the following is a global list of * devices that have registered with the * bus subsystem. Normally this would * be something like a dynamic linked list. / struct bus_device bd_list[5]; /* function to register a device with the bus / void register_with_bus(struct bus_device bd) { /* since this example only deals with one * device, will put it in slot 0 / bd_list[0] = bd; } void start_bus() { int i; struct bus_device bd; /* make callbacks to all devices on bus / for (i=0;i<sizeof(bd_list)/sizeof(bd_list[0]);i++) { bd = bd_list[i]; if (!bd) continue; / call device callback with generic * bus device structure / bd->device_callback(bd); } } /========================================================== * device X specific code * this would normally be in a different module ==========================================================/ /* structure that holds device X specific stuff, as well as * generic bus_device structure / struct device_x { int device_x_specific_param_a; int device_x_specific_param_b; struct bus_device bd; }; void device_x_callback(struct bus_device bd) { /* if we know the structure type that contains the bus_device structure, * we can extract a pointer to the containing structure using the container_of * macro / / ptr type member / struct device_x devx = container_of(bd, struct device_x, bd); /* the above statement expands to * struct device_x * devx = ( * { * const typeof( ((struct device_x )0)->bd ) __mptr = (bd); * (struct device_x )( (char )__mptr - ((size_t) &((struct device_x )0)->bd) ); } * ); / printf("device_x_callback called!, device_x_specific_param_a = %i\n", devx->device_x_specific_param_a); } void device_x_init() { / dynamically allocate structures / struct device_x devx = malloc(sizeof(devx)); memset(devx, 0, sizeof(devx)); /* set a parameter in the device_x structure so * we can test for this in the callback / devx->device_x_specific_param_a = 1001; / set up callback function / devx->bd.device_callback = device_x_callback; / we register the generic bus device structure * as the bus layer does not need to know * about the device_x stucture. Note, the * devx structure is not stored anywhere, yet * its location is being preserved without * specifically passing it to the bus * layer. / register_with_bus(&devx->bd); } int main() { / test the above system / / first, initialize device_x / device_x_init(); / now, start the bus. This should make * a callback into the device_x / start_bus(); } / when run, this program returns: * device_x_callback called!, device_x_specific_param_a = 1001 / </code></pre> Should you be using monotonic timers? 2008-03-19T00:00:00+00:00 In a previous article</a>, I covered some of the basics of Linux timers. Any time you are doing any type of fixed time delay in a program, you should really be using monotonic times, so the delay will not be affected by system time changes. In an effort to save cost, some embedded systems today do not have a battery backed up RTC, and instead get the time via GPS, NTP servers, or other clever means. What this means is your applications had better be able to handle the system time changing as the system time may not be set until well after the unit boots. This article describes how you can quickly test your system for timer problems. There are two cases where delays may fail if you are using non-monotonic timers. The first is if the time advances forward by a large amount. Delays will expire immediately in this case. The other case is if the time advances backward by a large amount. Delays will never expire in this case. To test for these situations, write a simple test application (sample included below) that rapidly changes the time and then test your application while the time is changing. #include <stdio.h> #include <time.h> int main() { time_t system_time; static int count = 0; printf("Starting test application...\n"); while(1) { // Sleep for 0.01 seconds usleep(10 1000); system_time = time(NULL); system_time += 30; stime(&system_time); printf("Cycle %d - system date/time set to %s", ++count, ctime(&system_time)); } } </code></pre> Compulab cm-x270 kernel update to 2.6.24 2008-02-29T00:00:00+00:00 The kernel support</a> in OpenEmbedded for the Compulab cm-x270 has been updated to version 2.6.24. The 2.6.24 kernel opens up several possibilities including better real time and high resolution timer</a> functionality, an improved SD Card driver, and a full SDIO stack with support for SDIO Wifi devices</a>. Also included is a patch to set the default framebuffer depth to 16 for several commonly used displays. Xfbev really does not like when you give it a 8-bit framebuffer when it is expecting 16, and typically returns a very unhelpful error message like: “error: Invalid argument”. GESBC-9302E kernel update to 2.6.24, and reboot fix 2008-02-18T00:00:00+00:00 I recently updated the GESBC-9302 machine support in OpenEmbedded to include the 2.6.24 released kernel. Also, Glomation has kindly provided me with a patch for the software reboot problem which I have integrated into the OpenEmbedded build for the GESBC. The patch files are part of the OpenEmbedded dev branch or can be obtained here: http://cgit.openembedded.net/cgit.cgi?url=openembedded/tree/recipes/linux/linux-2.6.24/gesbc-9302</a> Other related OpenEmbedded bits: http://cgit.openembedded.net/cgit.cgi?url=openembedded/tree/recipes/linux/linux_2.6.24.bb</a></li> http://cgit.openembedded.net/cgit.cgi?url=openembedded/tree/conf/machine/gesbc-9302.conf</a></li> </ul> Future features to be integrated include RTC driver configuration, and support for loading Initramfs images from NOR flash. Glomation GESBC-9302E Review 2008-01-29T00:00:00+00:00 Recently, I’ve had the opportunity to work with a Glomation GESBC-9302E single board computer. This SBC contains a Cirrus Logic EP9302 ARM processor and options for plenty of RAM and Flash memory which means you have many options for writing your applications including high level languages like C# and Python. In this review, I’ll cover some of basic features of this board and where this board might be useful. Why the GESBC-9302?</h3> One of the most attractive features of the GESBC-9302E is cost. The base model with 32MB of RAM, and no NAND flash costs $95 at quantities of 1 unit. For a model with 64MB of RAM, and 128MB of flash, the cost is $120. Looking at the board, its simplicity is very evident. There is basically a CPU, flash, ram, a few linear power supplies, and a very limited amount of support circuitry. There really is not much to it. We are helping a customer use the GESBC-9302E in a project where it is functioning as a gateway between a local Zigbee wireless network, and a server located on the internet. The customer had originally considered using a Linksys NSLU2 for the gateway device during the alpha testing, but upon consideration it seemed like the GESBC-9302E is a better choice in this application for the following reasons: the GESBC does not cost a lot more than a NSLU2</li> you can buy the GESBC with plenty of NAND flash.</li> the GESBC provides two serial ports</li> the GESBC provides many signals on headers than can be used with cables or a daughter card.</li> </ul> Having raw NAND flash on the board is a significant advantage in my opinion. I’ve run into considerable hassles with consumer grade flash devices in the past (SD, USB Flash, etc). Although there are industrial grade flash devices, they tend to be more expensive. My preference at this point for reliable storage in an embedded Linux system is JFFS2 on a 64-128MB raw NAND flash device. With 128MB of NAND flash, there is plenty of space</a> for a full featured Linux distribution. With 2 USB ports, plenty of general pupose I/O pins (GPIO), and lots of memory, there is not a lot you can’t do with the GESBC in your typical headless control or gateway type application. Being able to interface to a number of USB peripherals, and support for high level languages makes this SBC+Linux ideal for implementing a device that may need to talk web services or other complex network protocols. Software Support</h3> Most of the GESBC functionality is fairly well supported in the mainstream Linux kernel. Simply configure the kernel for the EDB9302 machine and most things will just work. There are several pieces missing from the mainstream kernel for the GESBC that are useful: NAND flash support, and a change needed to make the Ethernet work. For the NAND flash, I extracted a MTD driver from the Glomation kernel source code. The Ethernet change is a small tweak to the platform files. These changes are available for the 2.6.24 kernel in a public git repository at: http://dev.bec-systems.com/linux-2.6-gesbc-9302.git. The GESBC uses Redboot for the bootloader. No complaints with the bootloader so far – it does everything I need including loading images from a TFTP server. The bootloader does not support writing images to NAND flash, so to load an OS image into NAND flash you typically have to boot a Linux OS from NOR flash, USB flash, or NFS root, and then use the userspace mtdutils to write an image into NAND flash. Glomation provides a sample Linux image, but I prefer to use images generated by OpenEmbedded so that I have support for things like C#</a> , Python, and thousands of other packages. Support for the GESBC has been added to OpenEmbedded and I’m in the process of updating the kernel support to 2.6.24. Summary</h3> Overall I’m very pleased with this SBC. Glomation has been very responsive to requests and has been a good company to work with. I still need to apply a fix for a known soft reboot issue with the 2.6 kernel and enable a driver for the RTC on the SBC. Future articles will detail how to load Linux images built with OpenEmbedded on this device. Marvell Embedded SDIO Wifi Success 2008-01-09T00:00:00+00:00 As detailed in the article I wrote back in September of 2007 (/web/content/view/75/9/</a>), getting embedded wifi modules functioning is not a simple task. However, due to recent advances in the Linux kernel, it looks like a viable solution for low-mid volume products is emerging. This article provides a few details on how to get a Zcomax (Zcom) XG-180MU module working with a PXA270 processor. System Setup</h3> The setup used includes: Zcomax XG-180MU SDIO Wifi module</li> Compulab cm-x270 + sb-x270 system (includes the PXA270 processor)</li> Linux kernel 2.6.24-rc5</li> OpenEmbedded</li> </ul> Support for the Marvell 8385 and 8686 based Wifi chipsets is now included in the mainstream Linux kernel 2.6.24 release candidate source code. To use this driver, enable the CONFIG_LIBERTAS_SDIO option. I also had to enable the CONFIG_WIRELESS_EXT option to prevent a compile error, but this is probably required anyway to build a functioning wireless system. Firmware</h3> The firmware for the Marvell SDIO wifi solutions is downloaded at runtime, so the next challenge is to find firmware files. I am working on a project that has access to a driver from Marvell, so I was able to extract this firmware from their source code header files. I’m sure there are other places where this firmware is available. The firmware files must be placed in the following location: root@cm-x270:~$ ls /lib/firmware/sd* /lib/firmware/sd8385.bin /lib/firmware/sd8385_helper.bin </code></pre> The version of udev used in OpenEmbedded determines the location of the firmware files. The Libertas Linux driver requests these firmware files, and a userspace component of udev provides these files to the kernel. Success</h3> The following kernel modules are required: root@cm-x270:~$ lsmod Module Size Used by pxamci 7008 0 libertas_sdio 8488 0 mmc_core 46868 2 pxamci,libertas_sdio libertas 85288 1 libertas_sdio ieee80211 29956 1 libertas ieee80211_crypt 4768 1 ieee80211 root@cm-x270:~$ </code></pre> And the result: root@cm-x270:~$ modprobe pxamci root@cm-x270:~$ mmc0: new SDIO card at address 0001 libertas: eth1: Marvell WLAN 802.11 adapter </code></pre> ifconfig -a </code></pre> ... </code></pre> eth1 Link encap:Ethernet HWaddr 00:60:B3:34:77:66 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) </code></pre> Very nice! So it looks like there is finally an embedded Wifi solution for products that sell less than 500,000 units per year. I also briefly tried this on a AT91SAM9260 system, but it did not work. So there is still some work to do on the Linux SDHC driver for the AT91SAM9260. C# on embedded ARM Linux systems is now practical 2007-12-15T00:00:00+00:00 Update: Michael Dominic K. has kindly provided feedback on this article and has some additional points</a> about Mono and embedded systems. Mono has included support for ARM systems for some time, but with the release of Mono 1.2.6, and its inclusion in OpenEmbedded, it is now practical and easy to build and run C# applications on your everyday ARM Linux system. Several ARM issues in 1.2.5 have been fixed, and thus far it seems the average program runs quite well. In this article, we’ll explore what advantages C# offers over more traditional languages such as C or Python, and how OpenEmbedded simplifies adding C# support to your embedded system. Why C#</h3> Why is C# an attractive language for embedded Linux systems? There are many reasons including memory protection and exception handling, native types, good threading support, extensive library, and developer productivity. In the past, I’ve typically used C or Python for implementing large applications on embedded Linux systems. This section will compare C# to my experiences with both languages. A typical experience when implementing a large application in an embedded system is to deploy the system for field testing (before it is ready) and start getting reports that the application is crashing. Somewhere in the mass of C code your team just wrote there is a stray pointer clobbering something it should not. Of course this does not happen in the lab where you have access to the machine, but in some piece of industrial equipment many miles away. And, you cannot reproduce the problem in the lab because with embedded systems, simulating a real world situation is challenging because it is difficult to simulate real I/O. This is less of a problem with desktop applications where you are less dependent on external I/O. Sound familiar? Even if you can reproduce the problem on a machine you have access to (perhaps even remotely</a>), the problem can be caused by a stray pointer corrupting data in a completely unrelated piece of code. So while the debugger may tell you what got clobbered, you still don’t know exactly who did it. If you have lots of time and a team to do lots of testing, this is less of an issue, but available resources and time to market sometimes don’t give you all you need to do the job right before it gets to the field. One solution to this problem is use a language that has exception handling (I’ve used Python a lot in the past</a>), and log all exceptions. If something bad happens in the field, I ask for the log. With the stack trace in the log, and some focused testing I can usually find the problem relatively quickly. Add in the memory protection and things get a lot easier. For small applications this is not so much of an issue, but lets face it, now that we have access to very capable hardware at a low cost</a>, applications in embedded systems tend to get large and complex. C# gives you exception handling and memory protection that makes problems much easier to find and debug; especially during field testing. While its hard to beat Python or Ruby for developer productively and ease of use, there are performance issues like threading support, and support for static types that must be weighed in the balance. With Python, everything is an object or a reference. The checking is done at run time to make sure a variable is of the correct type. With C#, this is done at compile time (much like C). While it is generally more work to write code in C# than Python, the compiler tends to catch more errors for you. There is probably also a performance advantage to being able to implement simple variables statically. Threading support is another consideration and is often critical in embedded systems because you are usually tending to I/O, processing data, running a user interface, etc. This type of problem is well suited to a threaded application. Threading in Python is implemented using a global interpreter lock mechanism that is inefficient in some situations. C# threading support is more similar to C. While it may require you to do more manual locking than Python, it gives you more control. When it comes to developer productivity, in my experience C# is more work to write than Python or Ruby, but less work than C. So it seems to fall somewhere in the middle of the spectrum. So, C# seems to fall nicely into the middle ground between C and very high level languages such as Python and Ruby. It offers many of the advantages at both ends of the spectrum and seems to be a nice balance for programming embedded systems. Mono</h3> What is Mono? From the Mono web site</a> : Mono provides the necessary software to develop and run .NET client and server applications on Linux, Solaris, Mac OS X, Windows, and Unix. Mono includes all the necessary components to build and run C# applications on Linux systems including embedded ARM Linux systems. The Mono source tree is fairly clean and is nicely segmented between native and managed code. This is critical when it comes to cross compiling. The Mono team has also stuck with traditional build tools such as autotools which makes the project a lot easier to cross compile for an embedded target. Mono includes a very extensive C# library that is comparable to Microsoft’s implementation. Most of what you might ever need is built right into the Mono class library; much like Python or Java. With C or C++, you typically need to add a number of additional libraries to the system. Mono Support in OpenEmbedded</h3> OpenEmbedded includes comprehensive support for Mono including crosscompiling support, granular packaging of the Mono class library, and automatic run-time dependency generation for Mono applications. In this section, we’ll see how OpenEmbedded can make your life a lot easier. Mono is typically cross compiled in two steps</a> : The managed portions are generated by compiling Mono for the host machine architecture. A very nice feature of the Mono build process is that it can bootstrap itself.</li> The native only portion of Mono is then cross compiled over top of the above output and the native host binaries will then get replaced with the cross compiled target binaries.</li> </ul> To implement this, there are two recipes implemented in OpenEmbedded. The mono-mcs-intermediate is used to generate all the managed binaries, while the mono recipe builds only the native portion of Mono. The first step when building the mono cross recipe is to extract the results of the mono-mcs-intermediate in the install directory and then proceed to overlay the native binaries produced by the mono recipe in that same directory. As mentioned before, the use of standard tools like autotools makes all this work pretty well. All of this is automated in OpenEmbedded and happens when you run “bitbake mono”. One issue with embedded systems is you typically have limited flash/disk space. So you only want to include that pieces of Mono in your target image that your application needs. The OE build of Mono makes this very easy by packaging up the Mono class library into a number of different packages. The dependencies between packages is automatically tracked so that if you add a Mono package to an OE build, its dependencies will automatically be included with no extra work on your part. This mechanism uses the monodis tool to figure out what each Mono library provides and the dependencies of each library. This saves a huge amount of very tedious manual work to have all this automated. Building and Running your Mono application</h3> Now that we have Mono running on our target system, there are a number of ways to build Mono applications to run on the target. The quick-n-dirty method is to simply build the mono binary on any system, copy it to your embedded system and run it. Building and running your applications is easy as Mono binaries are platform independent and will run on any system that supports Mono. So, you don’t need to worry about cross compiling, toolchains, etc. The quick-n-dirty method works well for getting started, experimenting, etc, but there are several advantages to integrating your Mono application into an OpenEmbedded build, even though you technically don’t need to worry about cross compiling. The first reason is packaging. A very simple OpenEmbedded recipe provides an easy way to automatically fetch your project from a separate version control system, build it, and package it properly into the OS image for your target. It also gives you the ability to deploy updates to target systems using the ipkg package manager. The second reason is automatic run-time dependencies. If you inherit the mono OpenEmbedded class, all of the Mono components your application needs are automatically added to the image. An example OpenEmbedded recipe for a Mono application is shown below: DESCRIPTION = "My Application" DEPENDS = "mono" SRCREV = "${AUTOREV}" PV = "1.0+svnr${SRCREV}" PR = "r1" SRC_URI = "svn://svn.mycompany.com/svn/myproject/;module=myapplication;proto=https" S = ${WORKDIR}/myapplication </code></pre> inherit autotools mono </code></pre> The above recipe assume you have set up your application to be built with autotools, which is a good idea. This page</a> provides lots of good information on building Mono applications using autotools. You can also look at a number of open source applications such as Tomboy for additional clues about how to use autotools with Mono. For a simple application that I built, the following run time dependencies where automatically determined by OpenEmbedded: mono, libmono0, libmono2.0-cil, libmono-corlib2.0-cil.Each of these dependencies may have additional dependencies, and in this case a dozen or so mono packages ended up being installed. Summary</h3> As embedded systems become larger and more capable, we can expect that higher level languages such as C# and Java will start to become more common. In the past, the focus on embedded systems programming was often efficiency, minimizing resource utilization, etc. Now that very capable hardware is becoming cheaper</a>, the focus in some systems is shifting to features and time to market. Operating systems like Linux and languages like C# help us reach those goals and develop competitive products quickly with small teams. C# provides the features needed to rapidly develop, deploy, and debug complex applications that still perform reasonably well. OpenEmbedded provides a way to manage builds, so that you can focus on real work instead of tedious build issues. Mono 1.2.6 added to OpenEmbedded 2007-12-14T00:00:00+00:00 I recently updated the Mono recipes in OpenEmbedded to include the Mono 1.2.6 version. Several critical bugs that were present in 1.2.5 have been fixed, and its looking very good. More on this later … Embedded Linux development on Windows??? 2007-11-30T00:00:00+00:00 From http://www.embedded.com/design/opensource/204300381</a> : Host platform Although everything I will describe can be also done on a Windows host, I recommend using Linux. It’s more convenient, and more tools and utilities are available. And if you rely on a few Windows applications such as Word and Outlook, you can still run them on Linux in emulation using VirtualBox, Wine, or other commercial package. If you’re new to Linux, using a Linux host will also force you to learn the new platform faster. </blockquote> How true. Those who think they can do embedded Linux development without learning the basics of using a Linux desktop computer are in for a long hard road, and they are missing out on a valuable opportunity to learn a lot about embedded Linux development. This is the primary advantage of embedded Linux – there are many things in common with desktop and server systems. There is a tremendous amount of reuse. Things can be easily prototyped on your desktop Linux system before deploying to an embedded system. Instead of viewing the suggested that a Linux workstation be used for embedded Linux development as stumbling block, it should be viewed as a huge advantage. Compulab EM-X270 Review 2007-11-07T00:00:00+00:00 The EM-X270 is a full featured computer board from Compulab (http://www.compulab.co.il/x270em/html/x270-em-datasheet.htm</a>) designed for handheld/mobile applications. The board includes options for about anything you might need in a portable system including a PXA270 processor, GSM radio, GPS, Wifi, Bluetooth, Audio, SD/MMC, USB, battery circuitry, and the list goes on. The EM-X270 is designed to give you a big head start in designing a handheld computing device, and can significantly reduce time to market and design costs. This article provides a high level overview of this product, how it might be used, and lots of photos. EM-X270 Overview</h3> The EM-X270 is a computer board about the size of a display you might find in a typical PDA. While the size is probably a little large to be used in consumer devices like phones and PDAs, it seems like a nice size for slightly larger products used in vertical industries. As already mentioned, the EM-X270 includes about all the base features you might need in a mobile computing device. An expansion connector is also included that allows you to easily add custom circuitry to the system. The stack-up between the EM-X270 and an expansion board is shown below: The EM-X270 differers from most other embedded computer modules in that it is a little more application specified, and quite a bit more circuitry is included in the design such as power supplies, battery charging circuitry, radios, etc. Why EM-X270?</h3> Why are we interested in the EM-X270? If all you need is a generic handheld computer, there are plenty of options from PDAs to off-the-shelf industrial handhelds. However, if you need a handheld that provides some additional features not typically found in existing devices, then you can use the EM-X270 plus a custom baseboard and packaging to add your high value features. Depending what you need to add, the expansion board can be a simple 4 layer board which is very easy to design and manufacture. All of the complex PCB design and assembly is already done for you on the EM-X270 module. This allows low to moderate volume manufactures design products where they would not otherwise have the time and budget for designing a full custom processor board. Some example applications might be handheld products for the medical, industrial, and test and measurement industry. Compulab’s pricing model is also very interesting. Compulab will custom build EM-X270 modules with just the options you need. The base price without any extra features (such as radios) is $122 for 1000 pieces. The EB-X270 Evaluation Kit</h3> Compulab provides a EB-X270 Evaluation kit for getting started with EM-X270 development. Like their other Evaluation klits, is is very full featured and very well done. An extender board is provided that adds Ethernet, JTAG, and several other useful connectors for development. The eval kit also includes all necessary cables, a display, battery, etc. Software for the EM-X270</h3> Compulab provides software support for both Windows CE and Linux. This includes very up-to-date support for kernel version 2.6.23. EM-X270 machine support has also been included in the OpenEmbedded project for building full featured Linux images. This combination will drastically reduce development costs compared to solutions provided by many other vendors. All too often, single board computer vendors provide an ancient kernel, a hacked up root file system and call it a “Linux BSP”. Actually, any time the term “Linux BSP” is used, you should probably be a bit wary. A much more attractive solution for those of us building products using Linux is for processor and module vendors to say “our products are supported in the mainstream kernel and OpenEmbedded sources”. Compulab is doing it right. OpenEmbedded provides many interesting options for devices like the EM-X270. One of the most interesting might be the ability to run the OpenMoko software on industrial devices. Compulab provides a Linux demo image built with OpenEmbedded that runs the GPE Palmtop Environment. The method for loading this image is interesting, and similar to the method I have been using for some time with systems based on the cm-x270 module (/web/content/view/82/9/</a>) in that it used mtd-utils from a running Linux system to erase and write to the NAND flash. The bootloader in the EM-X270 is able to boot a kernel and rootfs directly from a USB Flash disk. This “LiveDisk” image, as Compulab calls it, contains a small utility that programs an image from the USB flash disk directly into NAND flash. As with any modern Linux based embedded system, there is a lot going on in a very small package. Managing this complexity effectively requires some amount of experience. Stay tuned for future articles that discuss using OpenEmbedded to build software for the EM-X270. Additional Photos</h3> Linux 2.6.23 for the Compulab cm-x270 2007-10-22T00:00:00+00:00 Updated 2.6.23 kernel patches for the Compulab cm-x270 are now available in OpenEmbedded: http://www.openembedded.org/filebrowser/org.openembedded.dev/packages/linux/linux-2.6.23/cm-x270</a> The cm-x270 support in OpenEmbedded has undergone a lot of clean-up in recent weeks. Now is a good time to give OpenEmbedded a try if you need a full featured Linux distribution for the cm-x270. Compulab cm-x270 NAND flash eraseblock sizes 2007-10-08T00:00:00+00:00 If you are having trouble mounting a JFFS2 file system on a Compulab cm-x270 module, you may have a device that has a NAND flash with 16KiB eraseblocks. All of the devices I have personally used to date have had 128KiB eraseblocks, but I just helped a cm-x270 user through some flash issues and after much pain discovered the eraseblock size was 16KiB on his module. A variable has been added to the cm-x270.conf file in OE to set the eraseblock size that can be overridden in your local.conf file if you are using OpenEmbedded to generate jffs2 images: http://git.openembedded.net/?p=openembedded.git;a=blob;f=conf/machine/cm-x270.conf;h=3c21c546b1320914c9b5d2c44e7332782fc58748</a> Howto load Openembedded on the Compulab cm-x270 computer module 2007-10-08T00:00:00+00:00 There are quite a few people using (or would like to use) OpenEmbedded on the Compulab cm-x270. The cm-x270 is a high performance, low cost computer module that can be used with a custom baseboard in embedded systems (see /web/content/view/62/9/</a> for a review). I get a lot of questions on how to load OpenEmbedded on this system, so this howto attempts to detail a procedure that can be used to load OpenEmbedded. The NAND Flash wrinkle</h3> The Linux solution provided by compulab includes a proprietary NAND flash driver. Most developers are interested in using standard Linux MTD drivers with the JFFS2 filesystem, but its not readily obvious how to flash a JFFS2 NAND image using the cm-x270 bootloader. Fortunately, you can boot a small Linux image stored in NOR flash, and then use standard MTD utilities to then flash the NAND jffs2 image. Creating the needed images</h3> For this method, you need to generate two images: a small image to run from NOR flash, and a larger full featured image that will run from NAND flash. Because you only have about 2.3MB of NOR flash, you want to use uclibc in the NOR image as it is much smaller. A base Linux image that will boot into a shell using uclibc and contains a few utilities is about 1MB in size, which leaves plenty of space for custom apps should you need them in this partition. The local.conf settings to build this image are: MACHINE = "cm-x270" DISTRO = "angstrom-2008.1" ANGSTROM_MODE = "uclibc" IMAGE_FSTYPES = "jffs2 tar cpio.gz" </code></pre> At this point, “bitbake angstrom-minimal-image-with-mtd-utils”, and OE will generate a “minimalist-image-mtdutils-cm-x270.cpio.gz” image that can be programmed into the NOR ramdisk partition using the cm-x270 bootloader, and tftp download. A kernel will also be built that should be programmed into the kernel partition. To build the NAND image, simply remove the ANGSTROM_MODE line, and “bitbake angstrom-console-image”. This will generate a “console-image-cm-x270.rootfs-summary.jffs2” that can be programmed into NAND flash. Programming Images</h3> Loading the software onto the cm-x270 is a multi-step process. The first step is to the load the kernel image and the minimalist-image into NOR flash using the cm-x270 bootloader and tftp. After this is accomplished, run the following commands to boot into the NOR flash rootfs: ramdisk on</li> bootos</li> </ul> If successful, you will be presented with a prompt – log in as user root, and no password. You now need to download the jffs2 NAND image into RAM on the cm-x270. One way to do this is to use the tftp download feature in busybox: tftp -g -r console-image-cm-x270.rootfs-summary.jffs2 <tftp server IP> </code></pre> At this point you can program the jffs2 image into NAND flash: flash_eraseall -j /dev/mtd3</li> nandwrite /dev/mtd3 console-image-cm-x270.rootfs-summary.jffs2</li> </ul> After programming completes, reboot the system and enter the following in the cm-x270 bootloader: ramdisk off (this will disable the initramfs in NOR flash)</li> bootos</li> </ul> At this point, your system should boot into the main NAND rootfs. With a little imagination, it should be obvious that you can do a lot of neat things with this type of setup by extending the functionality in the NOR rootfs including field upgrades and system recovery. Mono support in OpenEmbedded and Openmoko packages 2007-10-04T00:00:00+00:00 Original article published2007-10-04 As of today, the Mono build in OpenEmbedded is working pretty well. Mono support for ARM and other embedded architectures should add some interesting capabilities for embedded Linux devices. Having written a number of large embedded Linux applications, I can really appreciate the advantages of high level languages for exception handling. It is also interesting to note that Java support in OpenEmbedded is making progress as well (http://jalimo.org/wiki/doku.php</a>). Windows CE developers have already taken to the .net compact framework in a big way, so its nice to see similar capabilities for embedded Linux systems. A feed has been set up for installing mono packages on your Openmoko phone. Add the following line to a /etc/ipkg/-feed.conf file: src/gz mono-armv4t http://dev.bec-systems.com/feed/openmoko/mono </code></pre> And then: ipkg update </code></pre> ipkg install mono </code></pre> The packaging is fairly granular, so look at the listing in the above URL to see a complete list of what is available. How to implement realtime periodic tasks in Linux applications 2007-09-25T00:00:00+00:00 (see also should you be using monotonic timers?</a>) Have you ever wondered what is the best way to implement periodic tasks in Linux applications – something better than usleep()? This article covers a number of issues related to this subject including real-time tasks, the different timers available, timer resolution, and how to implement periodic tasks accurately so that error is not accumulated. The recent inclusion of the high-resolution timers in the mainstream kernel makes this a very interesting subject. High Resolution Timers</h3> Historically Linux has done all timing off the OS timer tick which is usually between 1ms and 10ms. While this is adequate for many tasks, it is really nice to have a high resolution timer for timing tasks. With the integration of the high resolution timers into the mainstream Linux source tree, this is now possible. From a user space perspective, there are no API changes. The only difference you will notice is that now you can sleep for less than OS timer tick period. clock_getres() can be used to check the timer resolution and will tell you instantly if you have high resolution timer support. If the clock resolution is 1ns, you have high res timer support. Realistically, you can’t delay for 1ns in a Linux application, but delays in the range of 100us should be possible, and depending on the configuration, much better performance is possible. The kernel config entry for high resolution timers is CONFIG_HIGH_RES_TIMERS. The difference between CLOCK_REALTIME and CLOCK_MONOTONIC</h3> Some of the Linux timer functions take a clockid_t argument that can be specified as CLOCK_REALTIME, or CLOCK_MONOTONIC. The big difference between these two is that changing the system time will have an affect on CLOCK_REALTIME, thus affecting your timers. Changing the system time will have no affect on CLOCK_MONOTONIC – it will always count upward. For periodic tasks, CLOCK_MONOTONIC may be more applicable. The best way to get burned using CLOCK_REALTIME is when your application takes a timestamp, does something, takes another time stamp and then compares them. If you are using CLOCK_REALTIME, and the system time gets changed between the two timestamps, your comparison will not be valid. For most timeouts and relative timekeeping in Linux, use CLOCK_MONOTONIC. This issue becomes more important as many systems now have a process that periodically sets the time automatically from network time servers, and you have no idea when this might happen. Preemption</h3> Kernel preemption (/web/content/view/69/9/</a> ) makes a huge difference in the performance of real time applications by allowing the kernel to be preempted by higher priority application processes. Historically, any kernel code that was runnable ran before the kernel returned control to applications. This all changes with kernel preemption. Kernel preemption has been available in mainline kernels for some time now (I think since 2.6.16). The worst offender I’ve found for locking the kernel for long periods of time have been flash drivers – especially proprietary ones. But, even jffs2 can cause problems in realtime applications without kernel preemption. The PREEMPT_RT patch</h3> Much of the realtime work being done for Linux is maintained in the PREEMPT_RT patch. Bits of this patch have already been merged into the mainline kernel, but there is still a lot of very useful functionality in the patch and it should be considered if you are doing any type of realtime work. More details will be presented in a future article. Absolute vs Relative timekeeping</h3> When implementing a periodic task, you really want to base your timekeeping off an absolute time, versus relative delays such as usleep(). It is not possible to achieve precise periodic activation with a relative sleep such as usleep(). The reason for this is you must first get the current time, make a calculation to determine how long to sleep, and then call the relative sleep function. If your process gets preempted between the time you acquire the timestamp, and the sleep, your relative sleep time will probably be wrong. This problem is solved by using the clock_nanosleep() function. clock_nanosleep() can be called with the TIMER_ABSTIME value in the flags argument. If the TIMER_ABSTIME flag is set, then clock_nanosleep() will sleep until the absolute timer value is reached. It does not matter if you get preempted between the time you take a timestamp, and the sleep function. Summary</h3> In summary, if you want to implement accurate, realtime, periodic tasks in a Linux application, consider the following: Use high resolution timers. Although Linux is still limited in its response time, at least the scheduling resolution is now quite high.</li> Enable kernel preemption. This makes sure long kernel processes don’t get in the way of your real-time application process.</li> Use the CLOCK_MONOTONIC for relative timekeeping. You don’t want your application to lock up due to the system time changing.</li> use the clock_nanosleep() function instead of relative delays like usleep(). This is the only way to accurately schedule periodic tasks.</li> if needed, apply the PREEMPT_RT patch.</li> </ul> The clock_nanosleep man page also includes a lot of useful information about timer functions. The Embedded WiFi module Quest 2007-09-19T00:00:00+00:00 original article written in Sept, 2007 How does one implement WiFi functionality in vertical, low volume portable products? This is a good question, and one I’ve been struggling with for the past 3 months. I have a customer who is designing a portable data acquisition system based on a AT91SAM9260 processor, and needs WiFi functionality. The fundamental problem is that no one has time to deal with low volume customers, and the task of implementing WiFi functionality is obviously complex. At volumes of 5000 units a year or less, it makes a lot of sense to go with a WiFi module rather than trying to integrate the WiFi chipset on the board. This article covers some of the options we have looked at and some of the possible solutions. How is WiFi done on x86 systems?</h3> With desktop Windows, most devices are well supported by device manufacturers. With x86 Linux some of the devices are well supported by OSS drivers, but these tend to be PCI or USB devices. Some of the devices that are not supported can still be used by running the Windows NDIS drivers inside of ndiswrapper, which allows you to run the Windows driver on a x86 Linux system, but this obviously does not work very well on non-x86 processors such as ARM. WiFi solutions for portable, embedded systems</h3> While there are many off-the-shelf solutions for PCs, Notebooks, etc, the current solutions are generally in the USB formfactor. The ideal solution for portable systems running non-x86 processors are the small modules that connect to the host processor using the SDIO or SPI bus – like the modules you would find in newer cell phones and PDAs. Older modules tend to use the Compact Flash interface, but these modules tend to be larger and are not packaged as nicely for deeply embedded applications. So Ideally we want: a modules with a SDIO or SPI interface</li> packaging options such as solder down for robust packaging</li> </ul> The scope of this article is limited to modules that implement a SPI or SDIO interface. Why are we interested in modules? Putting a WiFi chip down on a custom PCB is a lot of work. Because the WiFi silicon includes a processor, if things do not work, it may be very difficult to debug. There is also the issue of factory calibration which usually requires proprietary PC based tools that can communicate directly with the WiFi silicon. If the WiFi silicon is buried in your product, getting all this to work can be a challenge. The test equipment required for calibration is expensive. Add to this the difficulty in getting support from anyone who makes WiFi silicon … Complexity</h3> It should be noted that WiFi modules are fairly complex devices. They are typically based on a highly integrated IC that includes an ARM processor, Flash memory, and radio circuitry. The fact that these modules include a significant amount of firmware on the module contributes significantly to the complexity of getting WiFi solutions to work properly. Hopefully no bugs are encountered in the module firmware, because I can only imagine how difficult it would be to get these fixed for a small customer. The fact that the module firmware is involved in many of the WiFi functions like authentication further complicates the problem with the abundance of authentication and encryption options available for WiFi. The worst possible problem is the case where you must design an embedded WiFi system that must operate at the enterprise level in every environment. It is one thing to design a system that works most of the time in most environments where WiFi is more of a convenience (like a Cell phone or PDA). It is another thing to design an industrial grade system where it must work all the time and in all environments. The Fundamental Problems with low volume embedded WiFi</h3> I have never worked with a technology with so many dead ends as embedded WiFi modules. Many of the companies I email or call never even bother to return emails and phone calls. This is certainly true of the chipset manufactures. From what I can gather, the fundamental problems are: WiFi implementation is difficult; therefore, a manufacturer’s scarce resources are dedicated to high volume customers. Low volume customers are a significant distraction.</li> WiFi companies are very busy right now – they are not hungry for business from small customers.</li> WiFi solutions are very competitive and highly proprietary. Many companies are very secretive and will not release driver source code. This is certainly true at the module firmware level.</li> </ul> There are many WiFi module manufacturers and resellers out there, but very few of them offer any type of driver solution for Windows CE or Linux other than binary modules that are supposed to work on perhaps one reference platform. Sorry, this is not going to cut it. So, with plenty of hardware available, the gating item is the availability of software drivers and knowledge of how to use the software. It would seem to me that this is the perfect opportunity for chipset vendors to open source some drivers and develop vibrant support communities so that their modules can be used without a lot of hand-holding by the chipset vendors. So What are the options?</h3> Atheros AR6001</h4> The Atheros AR6001 seems like a nice solution. The driver situation for the AR6001 is progressing. For Windows CE, there is an opensource WinCE 6.0 driver available at: http://www.codeplex.com/CEWifiDriverAR6000/</a>. Unfortunately, this project appears to be a snapshot of code and does not have any significant amount of community activity or development. Perhaps once more AR6001 modules are available, this will change. For Linux, there are several options: Atheros has released a driver that is available: http://sourceforge.net/project/showfiles.php?group_id=186068</a>. There are several issues with this driver in that it is written for a SDIO stack that will likely never be part of the mainstream kernel.</li> A driver is being developed as part of the OpenMoko project.</li> </ul> Embedded Works supplies Atheros based modules and development boards: http://www.embeddedworks.net/wlan/oem_sdio_80211g.html</a>. AR6001 development boards are also available from Cardaccess: http://www.cardaccess-inc.com/products/index.php?a=wlan_sdio</a>. Marvell</h4> The Marvell 88W8385 seems to be a very popular IC in WiFi modules available from a number of different companies. OSS Linux drivers are in progress and are reportedly somewhat functional at this point, so it is probably just a matter of time before these devices are well supported in Linux. This chipset/driver is often referred to as “libertas”. The SDIO stack being developed for the Linux mainline (http://lwn.net/Articles/242747/</a>) is where support for this device is being developed. For Windows CE, it seems the only option is to get the drivers from Marvell, which is a very difficult and time consuming process – at least for low volume customers. First you have to find a module reseller that can get you the source code from Marvell, and then the process takes about 3 months. Other Options</h4> There are a number of other WiFi Silicon manufacturers and module vendors, but the driver options from them seem to be very limited in the form of binary only drivers that will only work with certain processors/operating systems. As noted before there is almost no hope of getting anyone to even talk with you if your volumes are low. So your only hope is finding a module vendor that can do any driver work for a NRE fee, obtain the source from the silicon manufacturer, or use OSS drivers. Summary</h3> The Embedded WiFi situation is changing fast, so by the time you read this, it is probably already out of date. We are currently at the phase where it is very difficult to deeply embedded WiFi in low volume products. I expect this will change during the next year. Eventually, WiFi modules re-sellers will figure out that supplying hardware is just one side of the equation. As things get more and more complex, the software availability and support is becoming the gating item. Many thanks to James Nahra, Dave Anders, and Erik Strack for sharing information and their WiFi experiences with me over the past few months. The correct way to add packages to an OpenEmbedded Image 2007-08-30T00:00:00+00:00 Update 2007-10-14: use IMAGE_INSTALL in image recipe Update 2007-10-22: a few corrections, added full path for include and comments Update 2010-06-07: use recipes instead of packages directory As more and more OpenMoko developers are coming on-line, it is becoming obvious that my previous post (/web/content/view/59/9/</a>) about adding packages to OpenEmbedded could use some improvements. This post suggested using the DISTRO_EXTRA_RDEPENDS variable, which was intended to only be used in distro.conf files. A better approach is to create a custom image recipe that includes the packages you want. This article covers how to create a custom image recipe. What are the problems with the DISTRO_EXTRA_RDEPENDS approach?</h3> As detailed by Marcin Juszkiewicz in his blog post “Why using of DISTRO/MACHINE variables in local.conf is wrong</a>”, and from discussions with OE developers, the following issues surfaced: You have to remember to rebuild task-base every time you update the DISTRO_EXTRA_RDEPENDS variable.</li> The image is no longer angstrom-console-image or whatever you have extended, so when asking for support, other developers do not realize exactly what you are building. If it is a custom image, then that is obvious by the image name.</li> Changing DISTRO_EXTRA_RDEPENDS in local.conf requires that you bitbake reparse the entire recipe tree (which takes minutes). Changing a custom image recipe only requires one file to be reparsed.</li> </ul> Creating a custom image file</h3> Creating a custom image file is quite easy, and works much better than modifying DISTRO_EXTRA_RDEPENDS in your local.conf file. A custom image is created by simply copying or including an image file closest to what we want to use, and then adding additional packages to IMAGE_INSTALL variable. In the following example, we require angstrom-console-image.bb and extend it with a few lines: require recipes/images/console-image.bb IMAGE_INSTALL += "\ xset xserver-kdrive-fbdev xinit \ kernel \ kernel-module-mmc-block \ kernel-module-mmc-core \ kernel-module-pxamci \ kernel-module-ac97-bus \ " export IMAGE_BASENAME = "my-custom-image" </code></pre> Note, this method requires your custom image file to be in the same location as the original image recipe. If you use a bbcollections overlay, then you may want to copy the original recipe. Note the use of a full path specified in the require statement. This allows you to have your custom image recipe in a bbcollections overlay, and bitbake will still find the required recipe in the main OE tree. Thats it! With the many advantages to this method, there is no reason not to create your own image recipes – it works better. Howto use a Git topic branch to maintain a patchset, and isolate problems 2007-08-16T00:00:00+00:00 The more I use Git, the more I am really impressed with some of its capabilities. What is Git? Git is a very advanced distributed source code control system that can do some very neat things. For the average developer using open source software, we are mainly concerned about maintaining patch sets (often called a topic branch). Git includes a rebase capability that is very useful for a number of different operations related to maintaining a branch of code including moving a branch forward, moving a branch around on an upstream branch to look for breakage, and merging changesets to create patch files. This article provides a brief overview of some of these operations and several tips. When in doubt, make a branch</h3> The most important concept to understand when working with Git is to get used to creating and destroying branches. This is different than most other SCMs where all branches are stored on a central repository, and you don’t want to clutter the branch namespace. With Git, get used to branching without even thinking about it. For example, the following are the branches in a Git repository I use to maintain patch sets for the Compulab cm-x270: git branch cmx270 cmx270_2.6.20 cmx270_2.6.21 cmx270_2.6.22 master svs test v2.6.20_test </code></pre> Most of the names are self explanatory. The cmx270 branch is where I track the latest kernel version. The svs branch is for a customer platform based off the Compulab cm-x270. To view the changes on any particular branch: git-log cmx270_2.6.22 </code></pre> an Git will start showing you a list of changes on that branch. This works even if you are not working on that branch, so it is very handy for seeing what changes are in other branches. With Git is also very easy to move changes between branches using git-cherry-pick. For example, if we are working on the cmx270 branch, and we want to pull the 3rd changeset from the HEAD of the cmx270_2.6.22 branch into the cmx270 branch, we would do something like: git-cherry-pick cmx270_2.6.22~2 </code></pre> Git is really good at maintaining patch sets</h3> Many developers use quilt to maintain patch sets. Once you understand how Git works, Git is even better at maintaining patch sets. Although there are some add-on tools available for maintaining patch files in a git repository like Patchy GIT and Stacked GIT, I’ve found that core Git works well enough. Actually the Patchy GIT maintainer has quit working on Patchy GIT because core GIT now meets his needs. The fundamentals are you simply export each Git commit as a patch using the git-format-patch command as shown below: git-format-patch -o tmp/ master </code></pre> Produces the following files (all of the commits since the branch has left the master): </code></pre> tmp/0001-cm-x270-base2.patch tmp/0002-cm-x270-match-type.patch tmp/0003-cm-x270-ide.patch tmp/0004-cm-x270-it8152.patch tmp/0005-cm-x270-pcmcia.patch tmp/0006-ramdisk_load.patch tmp/0007-mmcsd_large_cards-r0.patch tmp/0008-cm-x270-nand-simplify-name.patch </code></pre> Pretty slick! HEAD~8 means to back up 8 revisions from the current HEAD and then start outputting changes as patch files. The name of the patch file is the commit comment. Moving your patch set forward in time</h3> If you are maintaining a patch set for the Linux kernel, you might want to move your patchset up to the latest version for testing, and also maintain patchsets for each stable version of the kernel. Lets say we already have a patch set for the 2.6.20 version. The first thing we need to do is create a branch. In this example, I am maintaining a patch set for the Compulab cm-x270 computer module. git-branch cmx270_2.6.20 v2.6.20 (branch name, tag name in repository) </code></pre> git-checkout cmx270_2.6.20 </code></pre> At this point, I would manually apply each patch, and then commit each patch as its own changeset. Now, I want to move the changeset forward to kernel version 2.6.21. The easiest way to do this is create a new branch, and simply rebase your patchset to the new kernel version. Make sure you create a new branch first, or your existing branch will be moved! git-checkout -b cmx270_2.6.21 cmx270_2.6.20 </code></pre> git-rebase --onto v2.6.21 master (tag point to relocate branch, upstream branch) </code></pre> The upstream branch is required to be specified so that Git knows where our branch changes start – specifically where the cmx270_2.6.21 branch diverges from the master branch. There may be a few things to clean up along the way if changes don’t apply cleanly. The most critical thing to know is that if you want to skip a patch, you must run “git-reset --hard” before running “git-rebase --skip”. This is often handy if you have an especially troublesome patch that you don’t want to apply now. You can skip it and then cherry-pick it later. Otherwise the process is fairly straightforward. After you are finished, you can now export a set of patch files that can be applied to the 2.6.21 kernel. Bisecting the master branch to determine when things broke</h3> You may run into a situation where something broke on the main branch, and you don’t know where. Most of the time, you would just run git-bisect to do a binary search to find where things broke. This does not work so well if the platform you are testing is not supported in the main branch – the changes in your topic branch are required to make the system run. Once again, git-rebase to the rescue. The first thing we must do is find the mid-point between the known good and bad points. git-rev-list --bisect v2.6.20 v2.6.21 13f7e5acc8b329080672c13f05f252ace5b79825 </code></pre> Now, rebase out changes to the above rev and test. git rebase --onto 13f7e5acc8b329080672c13f05f252ace5b79825 master </code></pre> The build failed, so now we have a new “bad” point and can continue our binary search. git-rev-list --bisect v2.6.20 13f7e5acc8b329080672c13f05f252ace5b79825 </code></pre> If you can build working kernels from the main tree, then you can directly use git-bisect – another good reason to get your changes into the mainstream source, but this is not always possible. Another option for doing this type of binary search is to use git-bisect, and then cherry pick your changes at each bisect point. It may also be possible to use git-merge, but I have not figured out how yet. Summary</h3> That is it for now. Git is very useful for maintaining multiple patch sets for different versions of an open source project, and is just another thing that makes working with open source projects a real pleasure. Future articles will cover how to further manipulate patches such as how to combine two patch files. Linux 2.6.22 for the Compulab cm-x270 2007-08-13T00:00:00+00:00 I just completed porting the cm-x270 Linux kernel patches to the 2.6.22 kernel. Kernel build is available in OpenEmbedded. Patches are available in the OE tree: http://www.openembedded.org/filebrowser/org.openembedded.dev/packages/linux/compulab-pxa270-2.6.22</a> Shortly I will be making 2.6.22 the default kernel for the cm-x270 in OpenEmbedded and posting instructions on how to use JFFS2 with the NAND flash on the cm-x270. Design tips for your embedded Linux product 2007-07-20T00:00:00+00:00 When designing a system that runs embedded Linux, there are number of things you can do to make your life a lot easier and provide for future expansion to cover cases you did not anticipate. Following these simple guidelines can drastically improve the efficiency of developing the system, and troubleshooting problems during field testing, and provide the flexibility needed in todays products. #1 Have a dedicated serial console</h3> This is the single most important interface to provide. A serial console is your gateway into the system, and allows you at any time to poke around at the internals of the system, debug problems, run applications, change configuration, etc. A serial console does not have to be a big DB9 connector accessible outside the system. It can can be a simple 3 pin header and you make a special cable to connect to a PC. Also, do put the RS232 level translators on the board. This saves so much hassle during the development and testing cycle. You can always not populate them later, but you will probably find that the debug console is so useful, you will leave it in – even in production. Another consideration is to make sure you use a serial port provided by the processor if possible. The advantage of this is very little hardware has to be working for the serial console to work. If you use a serial port off an external UART, then the bus interface between the processor and the UART, the UART and the processor need to be working. If something is wrong with the system, you are less likely to get debug messages out the serial console which will help you debug the system. #2 Implement a USB Host port</h3> A USB host port is so useful it should be considered for about any system. Some of the things USB can be used for include: keyboard and mouse if you are designing a display based system</li> USB networking</li> Wifi networking</li> extra serial ports</li> WAN modems</li> </ul> Many embedded processors that can run Linux include a USB Host interface, so assuming you have one of these, it is very cheap to provide this interface as it only requires two wires, and a 5V power supply. Once again, you don’t have to put a full size USB connector in your device – a simple header wired to a connector might work in most situations. I can’t begin to describe how useful a USB keyboard was on a recent touchscreen based product. The product is used in agricultural vehicles, so anytime in the field we can connect a USB keyboard and look at logs, run commands, and other operations that are very difficult to do though a touch interface. #3 Implement a Network Interface</h3> Working on any Linux system is very painful without a networking interface. Even if you don’t plan to use Ethernet in the final product, make sure a networking interface is available for development. Some reasons why a network interface is critical: can easily log into the system remotely for development and debugging</li> easily copy files to and from the device quickly</li> install software updates easily during the development process</li> </ul> Trust me on this one. The neat thing about Linux, is if you have a USB Host interface, and a reasonably modern kernel, you can use off-the-shelf USB->Network devices for Ethernet. #4 Provide a fast way for the bootloader to download SW updates</h3> This is important. Don’t depend on JTAG or a serial port for programming large OS images. Some options for BL image download include: network (preferred)</li> MMC/SD</li> USB Client</li> USB Host (using a USB Mass storage device)</li> PCMCIA/CF</li> </ul> The good news is if you did steps #2 & #3, this might be taken care of – just make sure you check what the bootloader actually supports. One project I worked on recently only had a serial interface and a USB client interface that the BL did not support. We ended up creating a mini rootfs that included just enough smarts to download the main rootfs over the USB Client interface. So, there are ways to work around some of these issues, but its a lot easier if you do a little planning up front. Summary</h3> There is a lot to be said for using industry standard interfaces and components like Linux, USB, and Ethernet. When you design a system using standard building blocks, your product will have the flexibility to implement features you never thought of when the product was originally designed. And, your development and testing will go so much smoother. Open JTAG debugging tool for the PXA270 2007-07-10T00:00:00+00:00 Looking for a low cost, high functionality JTAG solution for the PXA270? As of several weeks ago, the openocd project now works with the Intel/Marvell PXA270 processor. There are a number of USB based JTAG debuggers available for under $100, so the cost is minimal. This article provides a brief overview of JTAG, suggestions for your hardware design, and how to use OpenOCD (Open On-chip-debugger) with the PXA270. OpenOCD</h3> Over the past 8 years, I’ve used a number of JTAG solutions for programming and debugging embedded processor systems. At the low end, there are JTAG solutions for flash programming only that just wiggle pins on a PC parallel port which in turn wiggle pins on the scan chain of the processor. This way of doing things works, but is very slow because you have to fabricate the bus cycles in software and clock the entire jtag chain just to change the state of one pin. The jtag tools from the openwince project on Sourceforge fall into this category. More advanced JTAG tools use the debugging capability of a processor to speed up flash programming and add debugging features. This typically involves downloading a tiny bit of code into the cache or internal SRAM in the CPU. This bit of code then can receive blocks of data or instructions and run bus cycles directly through the memory controller in the device. This makes flash programming very fast as most of the work is being done by the CPU, instead of manually controlling CPU pins using a jtag chain. In past years, there have been various attempts at creating an open source JTAG solution, but none of them have gotten very far. When I had a project that needed to get done quickly, I simply bought or upgraded a BDI2000 debugger – which is an excellent tool. The OpenOCD project now provides much of the same functionality as a BDI2000, but as in the case of the PXA270, it sometimes takes some work to get it all working. OpenOCD was created by Dominic Rath as part of a diploma thesis (http://openocd.berlios.de/web/</a>). It provides support for a wide range of target processors and JTAG debuggers. OpenOCD includes functionality for On-Chip Debugging, In-System Programming, and Boundary-Scan Testing. Hardware design considerations</h3> Although openocd has worked with other Xscale parts for some time, functionality for the PXA270 has not worked until recently. The primary reason for this is the way the power supplies are sequenced in most PXA270 designs. There is one supply to the PXA270 that is always on (VCC_BATT). All of the other supplies are often kept off until reset to the processor is de-asserted and the SYS_EN and PWR_EN signals go active. Because of this, some special considerations need to be taken during the sequencing of the reset signals, and the setting of the hold_rst bit in the PXA270 DCSR register. Make sure you power the JTAG connection (Pin #1 of your JTAG connector) with VCC_BATT or some power that is always active. If you power the JTAG connector with a power supply that is controlled by the SYS_EN or PWR_EN signal, then you will loose JTAG power during the connection process and things obviously will not work. JTAG Debugger Hardware</h3> Most of the JTAG debugger debuggers used with OpenOCD are based on the FTDI FT2232 USB chip. The debugger I used is the arm-usb-ocd device from Olimex. At a cost of $80, you can’t afford not to have one. OpenOCD Installation</h3> OpenOCD can use two different libraries that provide access to the FTDI chip: an open source library named libftdi, and a closed source library from FTDI named libftd2xx. I could not get the opensource libftdi library to work, so I suggest you use the FTDI library for now. Otherwise, installation is pretty straightforward. Find yourself a reasonably modern Linux computer and follow instructions on the following page: http://openfacts.berlios.de/index-en.phtml?title=Building_OpenOCD</a> . Make sure you use the --enable-ft2232_ftd2xx option when running configure. On Ubuntu 7.04: Install the FTDI library: wget http://www.ftdichip.com/Drivers/D2XX/Linux/libftd2xx0.4.13.tar.gz</li> mkdir libftd2xx; cd libftd2xx; tar -xzf ../libftd2xx0.4.13.tar.gz</li> sudo cp *.h /usr/local/include/</li> sudo cp libftd2xx.so.0.4.13 /usr/local/lib</li> cd /usr/local/lib/; ln -s libftd2xx.so.0.4.13 libftd2xx.so; sudo ln -s libftd2xx.so.0.4.13 libftd2xx.so.0</li> cd /usr/lib; ln -s /usr/local/lib/libftd2xx.so.0.4.13 libftd2xx.so.0</li> </ul> Install OpenOCD: sudo apt-get install build-essential</li> sudo apt-get install automake</li> sudo apt-get install autoconf</li> svn checkout svn://svn.berlios.de/openocd/trunk openocd</li> cd openocd</li> ./configure --enable-ft2232_ftd2xx</li> make</li> sudo make install</li> </ul> OpenOCD Configuration</h3> OpenOCD is configured using a configuration file. An example is: # config for Intel PXA270 # not, as of 2007-06-22, openocd only works with the # libftd2xx library from ftdi. libftdi does not work. telnet_port 3333 gdb_port 4444 interface ft2232 ft2232_layout olimex-jtag ft2232_vid_pid 0x15BA 0x0003 ft2232_device_desc "Olimex OpenOCD JTAG" jtag_speed 0 # set jtag_nsrst_delay to the delay introduced by your reset circuit # the rest of the needed delays are built into the openocd program jtag_nsrst_delay 260 # set the jtag_ntrst_delay to the delay introduced by a reset circuit # the rest of the needed delays are built into the openocd program jtag_ntrst_delay 0 #use combined on interfaces or targets that can't set TRST/SRST separately reset_config trst_and_srst separate #jtag scan chain #format L IRC IRCM IDCODE (Length, IR Capture, IR Capture Mask, IDCODE) jtag_device 7 0x1 0x7f 0x7e #target configuration daemon_startup reset target xscale little reset_halt 0 pxa27x # maps to PXA internal RAM. If you are using a PXA255 # you must initialize SDRAM or leave this option off working_area 0 0x5c000000 0x10000 nobackup run_and_halt_time 0 30 #flash bank <driver> <base> <size> <chip_width> <bus_width> # works for P30 flash flash bank cfi 0x00000000 0x1000000 2 4 0 </code></pre> Programming Flash</h3> After OpenOCD is installed, copy the above config file contents to a file name openocd.cfg, and run openocd in the same directory as your configuration file. Currently, there is an issue in that the openocd is expecting a debug handler file at a specific relative path from the openocd binary. The easiest way to get around this for now is to to run openocd from the src directory in the openocd source code. If you get the following message: Error: ft2232.c:1338 ft2232_init_ftd2xx(): unable to open ftdi device: 2, there is a permissions problem with the USB device. To get around this, run openocd using sudo, or add the appropriate rules to udev to allow the usb device to be used by any user. OpenOCD runs as a daemon waiting for telnet or GDB connections. After openocd starts, you can connect to it by running the following command: telnet localhost 3333 At this point you should be presented with a OpenOCD prompt. The following is an example of how to program a bootloader into flash memory: > flash probe 0 flash 'cfi' found at 0x00000000 > flash protect 0 0 5 off cleared protection for sectors 0 through 5 on flash bank 0 > flash erase 0 0 5 off erased sectors 0 through 5 on flash bank 0 in 4s 615249us > flash write 0 /build/tmp/u-boot-holon.bin 0 wrote 121832 byte from file /build/tmp/u-boot-holon.bin to flash bank 0 at offset 0x00000000 in 6s 138592us (19.381735 kb/s) > </code></pre> Debugging</h3> In this example, OpenOCD waits for GDB connections on port 4444. To debug code running on the target, you would use gdb in the same way you would use a BDI2000, or most any other hardware debugger with GDB. Summary</h3> OpenOCD is a robust, full-featured JTAG debug solution that supports many processors and low cost JTAG debuggers. There are still a few rough edges to clean up with PXA270 support, but OpenOCD is already a very useful tool for embedded software development. How to control the cable nightmare 2007-06-20T00:00:00+00:00 Do you ever wonder if there is a better way to manage cables in your lab? I typically have have 5-10 embedded systems on the bench powered up at any give time. When you start thinking of all the cables required to connect these systems (every one requires at least power, serial console, and network), it starts to add up. I recently ran across a product called snaketray (http://snaketray.com/products/wall.html</a> ) and decided it might be the solution to my rats-nest of cables. This article includes a few pictures of my implementation, and some ideas. I’m quite pleased with the result and have high hopes that I’ll keep my sanity for a while longer. I’m sure there are many solutions to cable management. The IT folks managing racks of computers no doubt have lots of good solutions. There are lots of different cable ties available, etc. But, the needs of a developer are a little different in that you need a solution that can easily be changed. You need something that is not permanent where you can quickly add, change, and re-route cables as needed. Most cable management solutions are more permanent solutions not designed for flexibility. Then I read the following article: http://www.joelonsoftware.com/articles/BionicOffice.html</a>. It occurred to me that snaketray might be the perfect solution for an embedded systems development lab. The idea of snaketray is very simple as shown in the picture below: I installed two runs about 1.5 feet above my lab work space. There are power strips along the back of each lab surface. All cables now run in the snaketray. The idea is cables go straight into the snaketray and exit exactly where the are needed. If the cable is too long, I loop it back in the snaketray. The result is pretty amazing when you compare it to what it was before. My wife is not real impressed with how it looks – apparently she has not spend enough time under tables untangling cables :-) Some of the benefits I’ve discovered so far: even though it still looks pretty messy, I’m finding it very easy to pull cables out of the tray and re-route them – even network cables which are notorious for getting hung up due to the little plastic clip on the connector. I think the reason is the cables are always running orthogonal – in the tray and the come out at a 90 degree angle where they are needed. So if cables are not crossing each other they do not get tangled.</li> bench space is not used for cables, so I have lots of space for more systems</li> I’ll be more likely to try builds routinely on more targets. One of my goals is to routinely do OpenEmbedded builds for a variety of embedded systems to keep my finger on the OE pulse, and help with QA. Obviously crawling around on the floor untangling cables to connect various systems is a negative motivator for such activities.</li> I have a place to put cables when I’m not using them – just put them in the tray. Before, they were left on my lab bench taking space, or falling on the floor. Now I have a few extra network and serial cables waiting in the tray for when I need them.</li> </ul> The bottom line is this makes me more efficient at what I do, so that I can better serve my customers and the projects I work on. I imagine snaketray would also be very useful in factory setups and other situation where you routinely need to change cabling. What you you found that works in your lab? Atmel AT32AP7000 development board review 2007-06-09T00:00:00+00:00 I recently purchased an Atmel NGW100 development board which includes an Atmel AT32AP7000 CPU. This CPU is based on the new Atmel AVR32 architecture, and is capable of running full Linux. This review covers the basics of this development board, the AVR32 architecture, options for building Linux for this system, as well as opinions on how a new architecture can succeed in a field of well established architectures like ARM and MIPS. What is AVR32?</h3> The AVR32 architecture defined by Atmel is a 32-bit RISC load/store architecture. There are several different versions of the core. Atmel currently supplies two classes of AVR32 cpus: UC3 Flash MCUs - low end devices with integrated RAM and flash, no MMU, and look similar in capability to ARM7 microcontroller type devices.</li> AP7 Application Processors - faster devices, include MMU, seem similar in capability and speed to ARM9 application processors such as Atmel’s AT92SAM devices.</li> </ul> As I am most interested in CPUs that can run Linux, this review will focus on the AP7 (AVR32 AP) class devices. It seems that AVR32 is Atmel’s answer to ARM Cortex. Some features of the AP7 devices: good code density</li> DSP instructions that execute in a single cycle</li> Full MMU</li> Instruction and Data caches</li> SIMD extensions</li> dynamic branch prediction</li> </ul> The MMU in the AP7 devices is interesting in that page tables are handled in software similar to a MIPS device. In a ARM CPU, the page tables are located in physical RAM and are strictly defined by the ARM architecture and page faults are handled in hardware. With MIPS (at least some devices) and AVR32, the MMU basically consists of a TLB (Translation Lookaside Buffer) that maps virtual addresses to physical address. As the TLB is limited in size, any misses are handled in software. NGW100 Development board</h3> The NGW100 development board is available from a number of distributors (including Digikey) for about $70. The board includes the following major components: Atmel AT32AP7000 CPU</li> 32MB of SDRAM (MT48LC16M16A2)</li> 8MB Parallel NOR Flash (AT49BV642D-70TU)</li> 8MB Serial NOR Flash (AT45DB642D)</li> 2 Ethernet ports</li> SD/MMC slot</li> RS232 port</li> USB Client port</li> Microcontroller for board control (ATtiny24-20SSU)</li> </ul> Linux Support</h3> Atmel is directly supporting the development effort for the AP7 processors. They seem to be doing it right in that they are pushing support directly into the mainstream kernel. Currently, Atmel provide patches for GCC and uClibc. There is already efforts underway to provide AVR32 support in Openembedded, which will enable developers to quickly build full featured Linux distributions for AVR32 based products. Summary</h3> It will be interesting to see how the AVR32 based products work out. On one hand, one could argue that a custom architecture like AVR32 can’t succeed as it does not have an extensive ecosystem of tool vendors in place like ARM and MIPS. But, the availability of open source software such as GCC and Linux is rapidly lowering the barrier to entry for creating tools for new architectures. No longer do you need to wait for commercial tool vendors to provide software support. The model for software development is changing. In comparison to Atmel’s AT91SAM parts, its difficult initially to see what the AVR32 parts offer over the SAM parts. For one, they are lacking a USB Host port which is becoming increasingly important to have in embedded Linux systems. Perhaps they offer other advantages such as lower cost and power. As we gain experience with these parts, I’m sure we’ll find applications where they are an excellent fit. The Embedded Industry Transformation 2007-06-09T00:00:00+00:00 I just ran across an interesting article written by Doug Gaff titled “Is the embedded industry dead?</a>”. The article points out the transformation that is happening in the “embedded” industry. For many systems, we are no longer constrained by minimal resources (/web/content/view/67/9/). Embedded Systems are no longer isolated systems, but parts of larger systems. A few of my own observations: There is more to modern embedded systems than control and logic. We now have to worry about data. In the past the task might be to get a microcontroller to run a state machine, control motors, etc. Today, we need to present a calendar to the user through a web interface, collect information about the location of the Sun, upload statistics to a remote server, support a rich GUI application on a VGA color display. The control aspect is suddenly the small part of the problem, and managing data is often the primary concern. Embedded system development now encompasses many disciplines including OS development, web services, advanced application development in high level languages, and the traditional hardware debugging and low level coding tasks. It is becoming less about writing code and more about integration of the right pieces. As Embedded Systems become more complex, access to specialized knowledge is becoming more important. There is just too much for one person (or even one company) to know. Companies developing competitive products in the future will be the ones who have access to experts. Is your company taking advantage of this transformation? Can you afford not to use Linux in your product? 2007-05-29T00:00:00+00:00 How much does hardware capable of running Linux cost? Recently, I priced components for a fairly minimal system. CPU: Atmel AT91SAM9260 (< $10)</li> 32MBytes of SDRAM ($6)</li> 256MBytes of NAND flash ($6)</li> </ul> These are prices for relatively low volumes on the order of 1000 pieces. So, you can get a processor, RAM, and flash that is more than capable of running Linux for about $20. Think about the possibilities. The AT91SAM9260 is a rather impressive part and includes interfaces for about anything including USB, SD/MMC, Ethernet, and the list goes on. It also includes that capability to boot from NAND flash which eliminates a dedicated NOR boot flash and reduces cost. If you don’t want to do a full processor design, processor modules can be obtained for less than $100</a> . So, at this price you have to ask: can I afford not to embed Linux in my product? Can I afford not to take advantage of standard interfaces like Ethernet and USB? Can I afford not to take advantage of 1000’s of high quality applications and drivers? Can I afford to not implement a system with virtually unlimited storage</a> and capability for expansion? Think about this carefully because your competitors are. Have questions or need help getting started? Contact us</a> – we can help. Is Linux ready for real-time applications? 2007-05-12T00:00:00+00:00 Have you ever wondered about the real-time response of Linux and how well it performs? I recently developed a system where I had the requirement to send a packet of data out a serial port every 30ms. This article describes how stock Linux can be used to accomplish this and solutions to several problems I encountered. There are several concepts that need to be understood such as how to set a Linux process to real-time, and kernel preemption. All of the work referenced in this article was done on a Compulab cm-x270</a> system which is based on the PXA270 ARM processor. Before utilizing any real-time optimizations, the packets sent out the serial port where anything but consistent. There was a significant amount of jitter in the packet timing and occasionally we would see delays of up to several seconds. Obviously this was not good enough. Real Time Process Priority</h3> The first step was to partition the application into several threads, and give the serial communication thread a priority of real time. Linux supports several scheduling policies: SCHED_OTHER: default universal time-sharing scheduler policy used by most processes.</li> SCHED_BATCH: intended for “batch” style execution</li> SCHED_FIFO: First In-First Out scheduling. Can only be used with static priorities higher than 0. A SCHED_FIFO process will always run before a SCHED_OTHER or SCHED_BATCH process. A SCHED_FIFO process runs until it is blocked by an I/O request, or is preempted by a higher priority process.</li> SCHED_RR: Round Robin scheduling. SCHED_RR is similar to SCHED_FIFO, except the process will only run for a maximum time quantum if there is a process of equal priority waiting to be run.</li> </ul> Practically, in a simple system there is really not much difference between SCHED_FIFO and SCHED_RR as you will typically only have at most a couple real time processes and you will probably assign them different priorities. The sched_setscheduler() system call can be used to set a process or thread priority – see the Linux man pages for more details. After setting the communication thread to real time, there was a noticeable improvement in the packet scheduling, but there was still way too much jitter and the occasional several seconds of silence with no packets. Preemptible Kernel</h3> After some experimentation, and reading, it was suspected that the NAND flash driver in the system was locking the kernel for long periods of time. With older Linux kernels, the kernel was not preemptible. Any process running in the kernel had to finish before something else could run. Recent 2.6 Linux kernel include an option to enable Kernel Preemption (CONFIG_PREEMPT). This allows a real-time process to preempt the system even if it is running in kernel space. After enabling the CONFIG_PREEMPT option in the kernel, the scheduling of packets was very consistent most of the time with almost no perceptible jitter. Be careful what you put in your real-time thread</h3> Now the system was running fairly well, but we still noticed at system startup that there were still some long delays between packets. It turns out the application was writing log messages to a file on NAND flash in the real-time thread using the standard glib logging functions. At system startup, the NAND driver blocks for long periods of time (up to several seconds). After the system has been running for a short time, this went away. The solution in this case was to send any logging messages from the communication thread to a separate normal priority logging thread. This fixed any remaining issues and the packet scheduling is now rock solid all the time. The leads to an fundamental real-time concept which seems very obvious in hindsight: Do not access any I/O (such as storage or network interfaces) that may block for long periods of time****in a real time process. Although we had designed all the primary I/O to be handled this way, we had not given any consideration to log file writes, which are an important part of any program. The glib library provides several very nice mechanisms for sending data between threads. Linux Real-time</h3> With improvements in the 2.6 Linux kernel, Linux works very well for some real-time applications. While I would still be very careful in using Linux for critical, hard-real time requirements due to its overall complexity, Linux works very well for multi-media, communication, and data acquisition systems. NAND Flash is almost "free" and what it means for Embedded Systems 2007-02-26T00:00:00+00:00 original article published in March, 2007 EETimes recently ran an article titled “Plummeting prices make NAND appear almost free” (http://www.eetimes.com/news/latest/showArticle.jhtml?articleID=197002923</a>). 1GBytes parts cost around $55 in Nov 2005. As of 2007-03-28, a 1GBytes part costs around $8. This is almost an order of magnitude reduction in a little over a year. This article examines the impact this price drop will have on embedded systems and the changes we will likely see including: storage is now at post-scarcity, will “embedded” distributions become obsolete, will uclibc become obsolete, and challenges with large NAND flash parts. 3 major components in an embedded computer</h3> The 3 major components in an embedded computer are: processor</li> RAM</li> Flash Storage</li> </ul> All of these components have been gradually decreasing in price over the years, and there has been a corresponding gradual increase in the use of complex embedded systems. However, the drop in NAND flash in the last year represents more of a step change, and will likely result in some more drastic changes. We are to the point of post-scarcity for Flash storage</h3> Up to this point, most full featured embedded Linux systems contained 16-64MB of NOR flash. A full featured graphical Linux terminal requires about 20-30MB of flash disk space just for applications and libraries. So, in the past we were still concerned about application footprint. The was a lot of incentive to not cross the 32MB to 64MB boundary, etc. With 1GiB of flash now costing less than $10, we essentially have unlimited flash storage in the context of an embedded Linux distribution. We will probably see the following: faster time to market as we need to spend less time optimizing</li> increased use of GTK+ as the standard GUI toolkit. We no longer care how much space GTK+ takes.</li> more debugging features can be left in the production system. Why not leave VIM, Minicom, and SSH in the build if we have plenty of space. These non-critical features will lower the cost of maintaining a product.</li> </ul> Will “embedded” distributions become obsolete?</h3> There has always been a great deal of interest in running Debian and other “standard” distributions in embedded systems. With 1GiB of storage, this is going to be a lot more practical. However, with a little experience, you soon realize that distribution size is only one of the many factors. Another equally important factor is the ability to easily automate the build process for the entire project, including custom applications and image generation for programming. With complex systems like Linux, it is easy to get into the hack-n-slash mode where it takes a lot of manual steps to build the system. While this may seem OK when you building the system, it ends up causing a lot of pain and errors once you get in a hurry, of down the road when you are trying to maintain the system. Ability to easily customize the distribution is another requirement. OpenEmbedded (http://openembedded.org), and other build systems provide these features. I think we will see a big increase in Debian and other standard Linux distributions in Embedded systems, but OpenEmbedded and other embedded build systems will continue to grow in use as well because there are many factors in the equation. This is the nice thing about open source – use what works best for you. Building and maintaining an embedded system is a lot different than setting up and maintaining a desktop or server system. Other articles related to this topic include: /web/content/view/63/9/</a> and /web/content/view/35/37/</a>. Will uclibc become obsolete?</h3> Because flash is now cheap, why do we need to use uclibc (small footprint libc) – can’t we just use glibc. For full featured embedded Linux systems, this argument makes sense. However, there is push toward higher integration at the low end. Many companies are offering 32-bit “microcontrollers” with integrated flash and RAM. The processors in these devices are more than capable of running Linux. The limitation to date has been the RAM and Flash size. The available memory in 32-bit microcontrollers will increase over time enabling them to run a low end uclibc based system. The idea of running Linux on a system that is composed of a single chip is very interesting. Challenges with large NAND flash parts</h3> What challenges will using large NAND parts present? In general, dealing with NAND is a lot more complex than with NOR flash. For reading, NOR parts present a simple memory mapped interface. A NAND interface is not memory mapped and looks more like a FIFO or a Hard Drive interface. This makes booting from NAND flash much more difficult. Some parts present a memory mapped interface for an initial loader. Some processors build NAND flash bootloading into the processor. Other systems boot from a small NOR flash and use NAND after the system has booted. Filesystems are also a challenge with NAND flash. The standard method is to use JFFS2 which was designed for smaller NOR parts. JFFS2 works well, but is a log based filesystem. This means it has to scan the entire file system on mount and store the entire inode chain in memory. This takes lots of time and uses lots of RAM. For small devices (64MB), this is tolerable, but for large devices the mount time can take 10’s of seconds. YAFFS2 is another log based flash filesystem designed specifically for NAND flash which includes a feature called checkpointing that bypasses the normal mount scanning. Another development is LogFS which looks promising, but will probably not be ready until mid/late 2007. Programming NAND flash can also be a challenge. Because NAND flash can contain bad blocks, programming is a lot more complex. Traditionally, flash programming in a NOR system was done by the bootloader. For a system that boots from NOR flash, programming the NAND flash can be done from a small Linux system (see /web/content/view/66/9/</a>). If you are booting from NAND flash, programming may be done with a JTAG programmer that supports NAND flash, or by downloading a small bit of bootstrap code into the processors internal memory using JTAG, and this bootstrap code then has enough functionality to program the NAND flash. Summary</h3> The lower NAND prices will bring about change in the way embedded systems are built. Please comment if you have addtional ideas or observations. Do you need "software update" functionality in your Embedded Linux system? 2007-02-21T00:00:00+00:00 In this day and age, most embedded systems include a way for users to easily update software once the device has been deployed. This article discusses the requirements for a field update mechanism along with pointers for how to implement. Update Mechanism Requirements</h3> Requirements for a field update mechanism might be: easy for users to perform updates.</li> little chance of “bricking” a unit if reset occurs during update.</li> ability to update all of the software in the unit</li> in this case, update from files on a USB Storage device</li> ability to program NOR and NAND flash</li> </ul> Other systems may have requirements to update from a network, Compact Flash card, etc. The USB wrinkle</h3> System update is typically done by the bootloader. In this case, we are using a Compulab cm-x270 module (/77/compulab-cm-x270-pxa270-module-review/</a>) that has a proprietary boot loader and does not support updates from a USB storage device. Therefore, we need to run the updates from the context of the Linux operating system which has drivers for USB and the NAND flash. We decided that the best way in this case is to have a small Linux OS image that contains enough functionality to update the main filesystem. This also provides us with the capability to recover if the main filesystem ever gets corrupted. The flash layout for the system is: Partition</td> Flash Device</td> Size</td> File System</td> Description</td> </tr> Kernel</td> NOR</td> 1536KiB</td> NA</td> Linux kernel</td> </tr> Update rootfs</td> NOR</td> 2304KiB</td> Initramfs (RO)</td> Small rootfs that contains just enough functionality to update the system</td> </tr> Main rootfs</td> NAND</td> 512MiB</td> JFFS2 (RW)</td> Main rootfs including application</td> </tr> </tbody> </table> There are other schemes that could work as well. One way might be to have two identical application partitions and only update one at a time. A reset while updating the kernel or the update rootfs does have the potential of “bricking” the system, but most of the updates will be for the Main rootfs, so we think this risk is low enough to be acceptable in this system. The update rootfs is loaded into RAM before it is used, so there is little chance of it being corrupted during normal operation. Running the update rootfs from ram is also convenient in that we can easily re-program the update rootfs in flash as we are not directly mounting the filesystem in flash. Building the Update rootfs</h3> The base update rootfs is built with OpenEmbedded as described in /77/compulab-cm-x270-pxa270-module-review/</a>. Once we had the system booting into the update rootfs, we had to add the following functionality: Boot the main rootfs from NAND flash</li> look for update files on a USB Storage device and flash the kernel, update rootfs, and main rootfs as needed.</li> </ul> Booting the NAND flash image is accomplished using the busybox switch_root application. This utility allows you to easily switch to another rootfs from an initramfs. There are a few gotchas to be aware of when working with switch_root: switch_root must be “exec’d” from the init process with PID=“1”.</li> switch_root requires a “/init” file to be present</li> </ul> The above functionality was accomplished by writing a shell script called /init. If an error occurs, the /init shell script then launches the standard init process in /sbin/init which provides a terminal and allows for easy debugging. The mtd-utils package provides a handy utility called nandwrite which can be used to write jffs2 images to NAND flash (handles bad blocks, etc). At some point, we may remove all the extra functionality from the update rootfs like the standard sysvinit, terminal login, etc, but for now it is handy to have for debugging. To give you an idea how easy it is to write to flash from a shell script, consider the following snippets: # write Image into update rootfs parition </code></pre> cat /usb/nor_rootfs.img > /dev/mtd2 </code></pre> # write Image to NAND rootfs partition </code></pre> flash_eraseall /dev/mtd3 </code></pre> nandwrite /dev/mtd3 /usb/nand_jffs2.img </code></pre> # and to launch rootfs in nand flash </code></pre> exec switch_root -c /dev/console /jffs2 /sbin/init 5 </code></pre> Summary</h3> Having typically used bootloaders for system update in the past, I’m very pleased with how this mechanism worked out. Thanks to the flexibility of Linux and OpenEmbedded, this only took several days to implement and debug. Writing update programs with shell scripts at the application level gives you a lot of flexibility and allows you to use standard Linux drivers for USB and Flash access which are very robust. At some point if there is interest, we may look at cleaning up our code and contributing functionality to OpenEmbedded to generate a more generic update initramfs. Please contact us or leave a comment if you are interested in collaborating on something like this. Building a tiny "safe" boot image using OpenEmbedded 2007-02-16T00:00:00+00:00 We are currently working a project that contains cm-x270 module (/web/content/view/62/9/</a> ) which contains 4MB of NOR flash and 128MB of NAND flash. The Linux kernel and a small root file system (rootfs) will be stored in NOR flash, and the main rootfs in the NAND flash. The rootfs in the NOR flash will be used to update the system from a USB Flash drive in the field. This article describes how to build a small boot rootfs using OpenEmbedded. We also look at various ways to store the image in flash. Requirments</h3> In many embedded systems, field upgrades is a requirement. There are many ways to organize this, but one of the most reliable is to have two rootfs partitions. The first partition is a “safe” rootfs that is only used to update the second partition. The second partition contains the main application and can be updated in the field. If a programming error, or power loss occurs while updating the 2nd partition, you can always boot into the first rootfs partition and restart the process. The kernel and the bootloader in this system take up about 1.5MB (out of 4MB total) of the NOR flash. This leaves us with about 2.5MB for the update rootfs. Because we are building the rest of the system using OpenEmbedded (OE), it may make sense for us to also build this update rootfs using OE to keep everything in one build environment. uclibc, glibc, or klibc?</h3> There are several libc libraries that are commonly used to build Embedded Linux systems. glibc is the standard that is used in desktop and sever systems and is what is being used in the main NAND rootfs partition. uclibc is smaller than glibc is often used in space constrained systems. klibc is a very minimal libc subset that supports operations typically used in an initramfs. Because we are already using glibc, we decided to try this first as we could then use the same toolchain and build tree and just generate two images – one for NAND, and one for NOR. OpenEmbedded support for small images</h3> OE includes a task-base-minimal meta target that can be used as a basis for a small images, or you can create your own based on task-base. You will then need to create a custom image recipe that will use this meta task. An example is shown below: PR = "r0" export IMAGE_BASENAME = "svs_nor" export PACKAGE_INSTALL = "task-min" # keep extra language files from being installed export IMAGE_LINGUAS = "" RDEPENDS = "task-min" IMAGE_FSTYPES = "tar jffs2 ext2 cramfs squashfs squashfs-lzma cpio.gz" IMAGE_ROOTFS_SIZE_ext2="10240" #EXTRA_IMAGECMD_jffs2="--pad=0x1000000 --eraseblock=0x40000" EXTRA_IMAGECMD_jffs2="" inherit image </code></pre> In this case, I had defined a custom recipe for task-min based on task-base, which installs the following packages: root@compulab-pxa270:~$ ipkg list_installed base-files - 3.0.14-r58 - base-passwd - 3.5.9-r2 - busybox - 1.2.1-r11 - initscripts - 1.0-r86 - ipkg - 0.99.163-r1 - ipkg-collateral - 1.0-r5 - libc6 - 2.5-r4 - libgcc1 - 4.1.1-r10 - libipkg0 - 0.99.163-r1 - makedevs - 1.0.0-r2 - sysvinit - 2.86-r32 - sysvinit-inittab - 2.86-r32 - sysvinit-pidof - 2.86-r32 - task-min - 1.0-r1 - tinylogin - 1.4-r3 - update-rc.d - 0.7-r0 - </code></pre> Storing the NOR filesystem in flash</h3> OE can generate images for many different flash filesystems and initrd mechanisms. These are specified in the IMAGE_FSTYPES variable in the above recipe. In this case, I chose to generate many different filesystems so I could see evaluate how much compression I would get with each filesystem type. The results: Rootfs Type</td> Size (bytes)</td> </tr> squashfs-lzma</td> 1785856</td> </tr> tar.bz2</td> 1918243</td> </tr> cpio.gz</td> 2024558</td> </tr> squashfs</td> 2109440</td> </tr> cramfs</td> 2265088</td> </tr> jffs2</td> 2455396</td> </tr> ext2 (uncompressed)</td> 5481000</td> </tr> </tbody></table> I ended up choosing the cpio.gz format as it can be loaded directly into an initramfs filesystem by the kernel. There are several advantages to using an initramfs for this task: You don’t have to mess around setting up tmpfs filesystems for directories that need to be writable.</li> You can update the NOR flash partition while you are running out of the initramfs filesystem.</li> It is fairly small with only squashfs-lzma being smaller.</li> </ul> Future Optimizations</h3> For now, the glibc based solution is good enough because it fits in the flash space we have and it is easy to build with our existing build environment. In the future, we may move to a uclibc or klibc based solution and try to reduce our flash size (OpenEmbedded supports both klibc and uclibc). The above image sizes can also be reduced by removing ipkg and other components that are not needed. This exercise illustrates many of the advantages of Linux and OpenEmbedded. With very little work, I can produce exactly what I need. The ability of the 2.6 Linux kernel to load a cpio archive into an initramfs is a very elegant solution for small boot images. Compulab cm-x270 PXA270 module review 2007-01-30T00:00:00+00:00 We are in the process of supporting a customer who is designing a product that uses the Compulab cm-x270 “Computer-on-module”. The cm-x270 is a small computer module 66x44x7mm that contains an Intel PXA270 ARM processor similar to those found in many PDAs or smart phones. This review provides an overview of the cm-x270, why it was selected, and how we are using it. Project Needs</h3> For this project, we needed a low cost embedded computer that could drive a VGA (640x480) color display. The graphical user interface for the device was fairly advanced, so we were also looking at a system that could run a modern OS with an advanced graphical toolkit. After discussing the display requirements with the customer, we concluded that we needed a 32-bit processor with a LCD display controller. As there was some custom circuitry needed, a configuration where the customer could design a custom baseboard and use an off-the-shelf computer module seemed optimal. The schedule was also very aggressive and we did not have the time or budget to develop a full custom processor board. CM-X270</h3> After researching a number of computer modules, we chose the cm-x270 from Compulab. The cm-x270 is a module that includes the following components: Intel PXA270 ARM processor (up to 520Mhz)</li> 802.11b Interface</li> up to 128MB SDRAM</li> 128-512MB NAND flash</li> up to 4MB NOR flash</li> PCMCIA</li> PCI</li> Serial Ports</li> Touch panel controller (UCB1400)</li> Host and Slave USB</li> and other features …</li> </ul> For more details visit the compulab web site (http://http://www.compulab.co.il</a> ). The cm-x270 provides a very impressive number of features in a very small space. Below is a picture of the cm-x270 mounted on the Compulab SBC-X270 baseboard with a VGA display. The system is running the GPE Palmtop Environment built with OpenEmbedded. Low Cost</h3> Perhaps the most impressive feature of the cm-x270 is the cost. When ordering in volume, Compulab will build the module with only the components you need. For prototyping, you can order a standard offering with all components populated. Starting a price of $50, it is the lowest cost module we found. The configuration we will be using costs around $98 in 2K quantities (64MB DRAM, 4MB NOR Flash, 512MHz + 128MB NAND Flash + Audio/Touchscreen + RTC). Eventually, we may be able to reduce some of the features and get the price down around $70 per module. At this price, the cm-x270 provides a very cost effective way to develop a low volume product (a few thousand per year) with advanced features provided by the PXA270 processor. Documentation</h3> The bootloader and hardware design of the cm-x270 is proprietary. Compulab does not supply schematics for the design or source code to the bootloader. The documentation provided is adequate and provided all the information we needed to complete the design. Compulab also provides schematics for their development baseboard, which is helpful. Bootloader</h3> The cm-x270 bootloader provides typical features found in an embedded processor bootloader, including: a serial console, NAND and NOR flash programming, download images from a tftp server, downloading images over a USB connection to a PC, and a number of other configuration commands. The tftp download works very well and is fast. The USB download is much slower, but is convenient for systems that do not have a network connection. The USB download requires a windows PC with a specific version of Microsoft ActiveSync installed, which is unfortunate. A more general solution that uses something like libusb (http://libusb.sourceforge.net/</a>) instead of ActiveSync would be preferred as it could then run on both Windows and Linux computers. Linux Support</h3> Compulab provides a fairly complete port of the 2.6.16 Linux kernel for the cm-x270 in the form of a kernel patch. Most of the peripherals we tested seem to work with the following exceptions: The SD card driver is not preempt safe. In this application, we need kernel preemption turned on to meet certain real-time requirements.</li> The Touch driver contains a small bug that does not allow applications using it to exit properly</li> </ul> Overall, support is pretty good and enabled us to get started with development quickly. NAND Flash Driver</h3> As the Compulab module only provides 4MB of NOR flash, using the NAND flash is a requirement when implementing a full featured Linux system with an advanced graphical user interface. Compulab provides a binary module that is linked into a kernel driver to support the NAND flash. This driver provides a block interface on which standard filesystems like ext2 can be used. The NAND driver has worked fairly well, although we have seen several cases where we get I/O errors if we remove power to the device without shutting down the system properly. The way to recover is to do a low level format the NAND flash. This should not be an issue with this product as we will have a backup battery that will be used to power the system during shutdown if power is removed. We may eventually move to an open source JFFS2 flash driver as time permits. The proprietary NAND driver presents an additional wrinkle in that the driver cannot be linked into the kernel due to GPL licensing requirements. One solution is to boot a small root file system from NOR flash that includes the NAND driver module, and then proceed to boot from NAND flash. GNU/Linux Root File System</h3> Compulab provides an example root file system (I think based on Debian) and suggests using scratchbox to build applications. We looked at it briefly, but then decided to build our own rootfs using OpenEmbedded for the following reasons: We can easily add additional packages from the extensive OpenEmbedded catalog as needed.</li> Allows us to have an rootfs independent of the module – we can more easily move to other processor modules or architectures in the future.</li> OpenEmbedded provides us with mechanisms for package management and building complete images that include our applications and tweaks to the OS.</li> </ul> For more information on OpenEmbedded, read our “Introduction to OpenEmbedded</a> “. With OpenEmbedded, we were able to quickly build a toolchain and a root file system that included the kdrive version of Xwindows and GTK+ libraries. This rootfs fits in less than 32MB of flash disk space, which is much smaller than Compulab’s OS image. We have also added machine configuration files to the OpenEmbedded project to support the cm-x270: http://www.openembedded.org/repo/org.openembedded.dev/conf/machine/compulab-pxa270.conf</a> Design Support</h3> Compulab seems mainly interested in customers who are interested in purchasing modules in volume. They do not sell low cost development boards such as LogicPD’s $500 board. Compulab requires you to purchase a fairly high cost development system ($1900) in order to get started. Part of this cost is to pay for Compulab’s support costs. For companies who do not have experience designing systems with PXA270 class processors, we highly recommend getting some up-front help from someone who has experience with these systems. Even though the module takes care of a lot of the design complexity, it helps to have a detailed understanding of the PXA270 and associated components, as you still need to interact with them. Consulting companies like BEC Systems can provide assistance in the form of hardware design reviews and Linux OS support to accelerate projects and help avoid costly mistakes and delays. As Compulab designs only the module, there is only so much they can do when you run into problems – as shown in the next section. Issues Encountered</h3> Overall, the project using the cm-x270 has gone well. The only real challenge we have faced is a latch-up issue with the ITE PCI controller on the cm-x270 module. It appears that this part is very sensitive to the characteristics of the 3.3V supply to the module. After a considerable amount of debugging and testing, we were able to solve this problem with some filtering and slowing down the rise time of the 3.3V supply. This is a good example of where a project can get difficult very quickly when things don’t work as expected. Summary</h3> So far, the cm-x270 has worked very well in this project. With its low cost, it is a very attractive option for companies wanting to implement advanced functionality in their products, but don’t have the time, sales volumes, or experience to justify a full processor board design. Feel free to post comments about your experiences using embedded computer modules. Introduction to OpenEmbedded part 1 2007-01-12T00:00:00+00:00 This is the first in a multi-part series on the OpenEmbedded development environment. This and subsequent articles will provide an introduction to OpenEmbedded, why to use it, how it works, and will provide examples of how BEC has used OE in various projects. Part 1 – Why do we need OpenEmbedded? This article discusses some of the problems developers face when building embedded Linux systems, such as collecting the pieces needed, setting up a repeatable build environment, cross-compiling, customizing a system, and licensing issues. Future articles will cover how OE addresses some of these issues. Embedded Linux = Complex System</h3> The fundamental problem with building Embedded Linux systems is the sheer complexity. Lets face it, there is a lot going on in a typical Embedded Linux system. Millions of lines of code are being compiled. Typically there are tens to hundreds of applications and libraries that need to be built. Linux supports many different processor architectures. The size of Linux leads to the conclusion that a good deal of automation is required to manage all this. In my paper “Tips for Planning your Embedded Linux Project” (/web/content/view/35/37/</a> ), the analogy of the Iceberg is presented. The application specific software in the project is the part of the Iceberg that is above water – the part you see. Most of the system (Linux and system applications) is below water – you don’t see it. The part under water is the big unknown and has the potential to add a lot of variance to a project. Note: this problem is not specific to Linux – any system of this complexity that is targeted to so many different applications faces the same problem. As a side, one might ask why use Linux if it is so complex? The answer is simple – features. Linux has the features and device support required for many Embedded products to be competitive. Collecting and building all the pieces</h3> There are a lot of pieces of software that need to come together to build an Embedded Linux system. This may include a toolchain, kernel, system libraries, and many system applications and configuration scripts required to make a system work. The is not something you want to do manually, or building an embedded Linux system quickly becomes impractical for most developers. Repeatable Builds</h3> With any software project, it is highly desirable to have a build system that is easily repeatable – which means it can be reproduced at any time on a wide variety of development workstations. Some of the of the situations I have run into that are less than desirable are: Being dependent on a vendor to provide a pre-compiled root file system image for a system. What happens when you need to add something two years down the road (such as foreign language support) and you don’t have a build system where you can easily reproduce the current build?</li> Being dependent on a golden build machine that takes a lot of work to set up and is very difficult to reproduce. If you have a build environment where all the developers use one machine to build software, you may want to investigate if you are in this situation.</li> </ul> What is really needed is a build system where you can check the build system out of a source control system onto about any Linux workstation at any point in time, type make, and a few hours later you have an image. GPL Compliance</h3> If you distribute binaries in your product compiled from GPL software, you must offer to provide the source code for these binaries. Do you have easy access to the source for everything that is in your product? Do you even know what all is in your product? It is not good enough to simply point to someone else’s distribution or web site and say “I use that”. As the founder of the MEPIS distribution discovered</a>, the GPL license explicitly states the distributor of the GPL code is obligated to provide source code for up to 3 years. So if you build a distribution by copying binaries from a pre-build distribution, you may have a problem. In one case I am familiar with, a company was in the last stages of a multi-year product development cycle. They then discovered that they did not have the source code used to build the GPL binaries in the system. The company that built the distribution for them was out of business and the open source project the binaries where originally obtained from no longer maintained the sources. Although it may be possible to find all of the old sources, it is a huge amount of work and would probably be easier to just start over and build new distribution. Customizing the system</h3> For most embedded systems, the distribution needs to be customized. There are often requirements such as custom software update schemes. Ideally, a good build system should easily allow you to integrate changes into the build system in a automated way, so that you don’t have any manual steps in the process of creating an image for the embedded system. Cross Compiling</h3> The above “repeatable build” requirement that you be able to build an image for your embedded target on about any Linux workstation leads to a requirement for cross-compiling. Cross-compiling is the process of creating executable code for a platform different than the one on which the cross compiler is run (http://en.wikipedia.org/wiki/Cross_compiler</a>). Cross compiling many standard open source applications and libraries can be difficult on a good day, and near impossible on a bad day, as many of them were not written with cross compiling in mind. Therefore, a good build environment should have tools that automate the process of cross-compiling as well as provide the ability to easily cross compile the many common programs and libraries that may be needed in an embedded Linux system. Summary</h3> As we examine some of the challenges, it should be obvious that putting together a production embedded Linux system is not something to be done in an ad-hoc fashion. There are several ways to approach this problem. Stay tuned for future articles where we explore how the OpenEmbedded build system solves these problems (subscribing to our blog or newsletter is a good way to do this). Also, please post comments about your experiences – we like to hear and learn from you. We are also interested in what problems you face when building Embedded systems. This input will be used when writing future articles. Thanks! Remote access for embedded systems behind a firewall 2006-12-19T00:00:00+00:00 When deploying embedded systems, wouldn’t it be nice if you could easily set up remote access for troubleshooting and updates? With Embedded Linux systems, this is quite easy to do. This article will explore how to use SSH to set up remote access to an embedded system behind a firewall without using VPN or special network setup and presents a few tips for making this setup work well. The Problem</h3> With today’s complex embedded systems, testing and deployment can be a challenge. During the initial testing and roll-out, a new system may need to be carefully monitored to assure it is operating correctly. Embedded Systems programming is different than application programming in that often you are interfacing with external I/O. The behavior of things external to the embedded system are often very difficult to model during development; therefore time must be spent in the field testing the system in real-world situations. Being able to remotely access the system during testing is very convenient. As with most networked computer systems, networked embedded systems are typically deployed on a network behind a firewall. It is very easy to log in and administer any Linux system you have direct network access to – the challenge is getting to a system behind a firewall. Options for remote Access</h3> There are several general options for remote access to networks: VPN</li> Port forwarding at the firewall</li> PC remote access technologies. These are typically single session type connections that are initiated by the PC user.</li> </ul> The disadvantage of the above technologies is they require involvement from the IT department or users of the remote network. BEC is currently involved in the deployment of a system that uses embedded computers in six different buildings. Each building has its own firewalled network. Getting remote access to all of these networks is a big task and everyone is busy. An ideal solution would be a way to communicate directly with each embedded computer from a central location, which brings us to SSH … SSH</h3> SSH (Secure Shell) is a set of standards and an associated network protocol that is used to establish a secure channel between a local and remote computer. SSH encrypts all traffic (including passwords) and is generally considered to be very secure if used properly. Some uses for SSH are: remote terminal access.</li> transfer files between computers.</li> tunnelling and port forwarding between computers</li> </ul> The fact that SSH can be used to forward ports over an encrypted tunnel is what makes remote access possible using what is called a Reverse SSH tunnel. Because the embedded system is behind a firewall with no direct access from the Internet, the connection must be initiated from the embedded system. A reverse SSH tunnel is set up as follows: the embedded systems make a SSH connection to an external server that is internet accessible.</li> The SSH connection is set up so that a port on the external server (let’s pick 2900) is forwarded back over the SSH tunnel to port 22 (port used for SSH) on the embedded system.</li> The developer connects to 2900 on the external server using SSH. Because port 2900 on the external server gets forwarded over the SSH tunnel to port 22 of the embedded system, the developer is actually logging into the embedded system.</li> </ul> This connection is illustrated in the figure below: In the above figure, the external server is typically a system located in the developer’s office with only port 22 exposed to the Internet. Port 2900 is only accessible to developers on the same private network as the server. Security</h3> When talking about any type of remote access technology involving the internet, it is important to consider the security implications. The reverse SSH mechanism is fairly secure, as everything is encrypted. SSH is widely used for remote access to servers; therefore it receives a lot of testing and scrutiny. With the reverse SSH remote access setup, the most likely attack is probably to compromise the external server and then try to access the remote network the embedded system is located on through the embedded system. If the external server is properly secured, then this mechanism should be fairly safe. One of the advantages of this scheme is that the embedded system is not directly exposed to the Internet, therefore securing the embedded system itself is not all that important. It is obviously very important that the people administering the remote network understand what you are doing – that is where this article can come in handy. If there are concerns about external developers accessing a network, one mode of operation may be for the the SSH tunnel to be manually started and stopped so that administrators of the remote network know exactly when remote access is enabled. TIPS</h3> A few ideas for making this work well: It may be useful to put the reverse SSH connection in a cron job on the embedded system so that the connection is automatically set up. If you do this, I advise killing the connection and restarting every so often. Reverse SSH connections are somewhat fragile and occasionally need restarted if there is a network glitch.</li> Use the SSH TCPKeepAlive option. Some firewalls/routers will drop a network connection if there is no activity.</li> </ul> Summary</h3> Using SSH for remote access is another example of the “Big Win” with Embedded Linux – how industrial grade technologies developed for use in server and desktop systems can be readily used in embedded systems (/web/content/view/35/37/</a> ). Remote access during the deployment and testing of embedded systems can drastically reduce the amount of time and effort required to roll out an embedded system. How to build GNU/Linux for an embedded x86 computer 2006-11-29T00:00:00+00:00 Recently, I had a customer who needed an embedded Linux distribution running on an Advantech PCM-9371 single board computer (SBC). The PCM-9371 contains a low voltage Celeron or Pentium III processor. This article describes why the Openembedded build system was chosen and a few tips for running Openembedded on a x86 system. The hard way!</h2> There are many ways to put Linux on a x86 embedded system. There are hundreds of distributions out there and the temptation is always to do something “quick” and just try to trim down one of the standard distributions. Other shortcuts are often taken, such as compiling your application on a workstation that is running a different distribution and different versions of the libraries used in the embedded systems. This approach can work, but there are many possible pitfalls: Several years down the road, it will be difficult to compile applications for your target system as the libraries on the target system are very dated compared to tools and libraries used in standard desktop distributions. What this means is you will need to keep a “golden” compile machine around for the life of your project – not fun! I went through an experience like this on a previous version of this same project where it took us a week to set up a build machine to compile applications using a very old version of Debian.</li> It is difficult to gather GPL sources used to build the distribution you are using. You are required to supply GPL source code upon request to customers who purchase your product.</li> To build applications correctly, you really need a build machine that is running the same distribution as the target, unless you set up a cross-compilation or scratchbox environment.</li> Most distributions are quite large and have difficulty fitting on CF card.</li> Most distributions are difficult to customize in a clean, systematic way.</li> </ul> OpenEmbedded</h2> Considering the disadvantages of using an standard x86 distribution for this project, I chose to use OpenEmbedded (OE). The OpenEmbedded project (http://openembedded.org</a> ) is a full featured build system for building embedded Linux distributions. The development is done on a Linux workstation of your choice and the entire distribution is cross-compiled to the architecture of your choice. The entire build process is automated, including building the toolchain and an image that can be installed on the target embedded system. Some of the features of OpenEmbedded are: Over 3000 packages that can be built. About anything you would ever need for an embedded system.</li> Support for a wide variety of target architectures. ARM and x86 processors are well supported.</li> Build system is very self contained. This means you can do the builds on a variety of different host machines. Many of the tools OE needs are built by OE, so there is very little dependence on host tools besides a few basics such as Python, make, and other standard tools. This also means it is much easier to pull the build system out of the archives in several years, load it on a current machine, and do a build.</li> All source code used in the build is downloaded, so GPL compliance is easy.</li> and the list goes on …</li> </ul> Build and Installation</h2> Setting up an OE build is beyond the scope of this article, but it basically involves selecting a machine and distribution. In this case, I chose the x86 machine and the generic OE distribution. Several hours later, I had an image in the form of a tar.gz file that I decompressed to a Compact Flash (CF) card. After installing grub on the CF card, I had a booting Linux distribution that included all the basics (like ssh, ipkg package manager, shell, standard file utilities, etc) in less than 4MB of disk space. Adding GTK+ and Xwindows</h2> This system required support for GTK+ and Xwindows. To add this involved running the following commands: bitbake xserver-kdrive and bitbake gtk+. The OE build system then automatically downloaded and compiled the need packages. The image size is now around 20MB – still fits very easily on a CF card. Getting X running</h2> There are several options for running X on a x86 system. In this case, I chose the quickest way so that my customer could get started with application integration as quickly as possible. Linux and Xwindows support the standard VESA Framebuffer mode. Then can be enabled by selecting the CONFIG_FB_VESA kernel config option. The kdrive version of X can then be started by running Xvesa. In the future, we may move to an accelerated X driver if needed. Summary</h2> We now have Linux running off of a CF card with full X/GTK+ support and the customer can proceed with integration, testing and application development. Getting to this point took about 1 day of development as we did not encounter any major problems – much less time than it took to resurrect an old build environment used in previous versions of the project. Future work will involve integrating applications into the OE build system, and customizing the distribution as needed. Openembedded has proved to be a very valuable tool for getting embedded Linux systems running quickly and maintaining them in a controlled way. Adding software to a Moxa UC7408 using OpenEmbedded 2006-10-30T00:00:00+00:00 One of the big advantages of using Linux in embedded systems is the thousands of applications and drivers available for it (/web/content/view/35/37/</a> ). One of the challenges of Embedded Linux is building these applications. Building applications for an embedded system often involves cross-compiling which is frustrating on a good day. This article details how to use OpenEmbedded (OE) with the existing toolchain Moxa provides to build a number of applications for the Moxa UC7408. Web Application Support</h2> My needs for this project were to build a web application that ran on the UC7408 (See /web/content/view/55/9/</a> for a review of the Moxa UC7408). The Moxa firmware is fairly full featured and provides php for web application development. This is probably fine for many applications, but I prefer to build web applications as detailed in this article: /web/content/view/46/9/</a> . Because the rest of the application is written in Python, there are advantages to also implementing the web application in Python. I also wanted to use the sqlite database. A quick look at the php supplied with the Moxa firmware suggested it was built without sqlite support. In summary, I needed the following components: Python</li> Clearsilver</li> sqlite</li> minicom</li> vim</li> strace</li> subversion</li> wget</li> </ul> The above is a fairly small list, but what you don’t see are all the dependencies that also need to be built and installed. This ends up being many more components. Some of the items are for development only, but this is Linux – why not have nice tools running on the target system? OE to the rescue</h2> Ideally, the Moxa system would have a complete root file system built with OE running on it. But I did not have the time or budget to implement this, so I did the next best thing – use the rootfs and toolchain supplied by Moxa along with OE to compile just the applications I needed. Typically, OE builds the toolchain and Glibc, but fortunately OE provides a very slick way to use an external toolchain and glibc with OE. How to do this is detailed in the OE manual: http://www.openembedded.org/user-manual&dpage=commonuse_prebuilt_toolchain</a>. Below is the setup I used: Environment variables: TOOLCHAIN=/usr/local/mxscaleb/bin/ export PRE_BUILT=/usr/local/mxscaleb/armv5b-linux export CCACHE_DIR="${TOPDIR}/ccache" export PYTHONPATH="${TOPDIR}/bitbake/lib" export BBPATH="${TOPDIR}:${TOPDIR}/openembedded:${TOPDIR}/bitbake" export PATH="${TOPDIR}/bitbake/bin:${TOOLCHAIN}:${PATH}" export LD_LIBRARY_PATH= export LANG=C </code></pre> Local.conf file: DL_DIR = "/build/sva_oe/downloads" BBFILES := "/build/sva_oe/openembedded/packages//.bb" BBMASK = "" PREFERRED_PROVIDERS = "virtual/qte:qte virtual/libqpe:libqpe-opie" PREFERRED_PROVIDERS += " virtual/libsdl:libsdl-qpe" PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}gcc-initial:gcc-cross-initial" PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}gcc:gcc-cross" PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}g++:gcc-cross" PREFERRED_PROVIDER_virtual/libx11 = "diet-x11" ASSUME_PROVIDED += " virtual/${TARGET_PREFIX}gcc " ASSUME_PROVIDED += " virtual/libc " MACHINE = "moxa" IPKG_ARCHS = "all arm armv4 armv5te xscale ${MACHINE}" TARGET_CC_ARCH = "-mcpu=xscale" TARGET_ARCH = "arm" PACKAGE_ARCH="xscale" TARGET_OS = "linux" TARGET_FPU = "soft" DISTRO = "moxa-disro" DISTRO_NAME = "moxa-distro" DISTRO_VERSION = "0.0.1" DISTRO_TYPE = "release" INHERIT += " package_ipk package_tar debian" IMAGE_FSTYPES = "jffs2 tar" PARALLEL_MAKE = "-j 4" BBINCLUDELOGS = "yes" CVS_TARBALL_STASH = "http://www.oesources.org/source/current/" export CC = "mxscaleb-gcc-3.3.2 ${HOST_CC_ARCH}" export CXX = "mxscaleb-g++ ${HOST_CC_ARCH}" export CPP = "mxscaleb-gcc-3.3.2 -E" export LD = "mxscaleb-ld" export AR = "mxscaleb-ar" export AS = "mxscaleb-as" export RANLIB = "mxscaleb-ranlib" export STRIP = "mxscaleb-strip" TARGET_CPPFLAGS_append = " -I${PRE_BUILT}/include " TARGET_LDFLAGS_prepend = " -L${PRE_BUILT}/lib -Wl,-rpath-link, ... </code></pre> ${PRE_BUILT}/lib -Wl,-rpath-link,${PRE_BUILT}/qt2/lib " </code></pre> With the above setup, I was able to build the applications I needed with very little effort. There were a few little problems I ran into, but I was able to quickly work around them. The result is a number of packages that can be installed on the target system: bigreqsproto-dev-X11R7.0-1.0.2-r0.tar.gz python-crypt-2.4.3-ml0.tar.gz busybox-1.2.1-r1.3.tar.gz python-curses-2.4.3-ml1.tar.gz busybox-udhcpd-1.2.1-r1.3.tar.gz python-datetime-2.4.3-ml0.tar.gz clearsilver-0.10.3-r0.tar.gz python-db-2.4.3-ml0.tar.gz clearsilver-dbg-0.10.3-r0.tar.gz python-devel-2.4.3-ml0.tar.gz clearsilver-dev-0.10.3-r0.tar.gz python-distutils-2.4.3-ml0.tar.gz clearsilver-doc-0.10.3-r0.tar.gz python-email-2.4.3-ml0.tar.gz inputproto-dev-X11R7.0-1.3.2-r0.tar.gz python-fcntl-2.4.3-ml0.tar.gz ipkg-0.99.163-r1.tar.gz python-gdbm-2.4.3-ml0.tar.gz ipkg-dbg-0.99.163-r1.tar.gz python-hotshot-2.4.3-ml0.tar.gz ipkg-dev-0.99.163-r1.tar.gz python-html-2.4.3-ml0.tar.gz kbproto-dev-X11R7.1-1.0.3-r0.tar.gz python-idle-2.4.3-ml0.tar.gz libapr-0-0-0.9.12-r0.tar.gz python-image-2.4.3-ml0.tar.gz libapr-0-bin-0.9.12-r0.tar.gz python-io-2.4.3-ml0.tar.gz libapr-0-dev-0.9.12-r0.tar.gz python-lang-2.4.3-ml0.tar.gz libaprutil-0-0-0.9.12-r0.tar.gz python-lib-old-and-deprecated-2.4.3-ml0.tar.gz libaprutil-0-bin-0.9.12-r0.tar.gz python-logging-2.4.3-ml0.tar.gz libaprutil-0-dev-0.9.12-r0.tar.gz python-mailbox-2.4.3-ml0.tar.gz libcrypto0.9.7-0.9.7g-r1.tar.gz python-math-2.4.3-ml0.tar.gz libexpat-bin-2.0.0-r1.tar.gz python-mime-2.4.3-ml0.tar.gz libexpat-dev-2.0.0-r1.tar.gz python-mmap-2.4.3-ml0.tar.gz libexpat-doc-2.0.0-r1.tar.gz python-netclient-2.4.3-ml1.tar.gz libexpat1-2.0.0-r1.tar.gz python-netserver-2.4.3-ml0.tar.gz libgcrypt-dbg-1.2.2-r0.tar.gz python-pickle-2.4.3-ml0.tar.gz libgcrypt-dev-1.2.2-r0.tar.gz python-pprint-2.4.3-ml0.tar.gz libgcrypt-doc-1.2.2-r0.tar.gz python-profile-2.4.3-ml0.tar.gz libgcrypt11-1.2.2-r0.tar.gz python-pydoc-2.4.3-ml0.tar.gz libgdbm-dbg-1.8.3-r2.tar.gz python-pyserial-2.2-r1.tar.gz libgdbm-dev-1.8.3-r2.tar.gz python-pysqlite2-2.2.2-ml1.tar.gz libgdbm-doc-1.8.3-r2.tar.gz python-pyxml-0.8.4-ml0.tar.gz libgdbm3-1.8.3-r2.tar.gz python-re-2.4.3-ml0.tar.gz libgnutls-bin-1.4.0-r1.tar.gz python-readline-2.4.3-ml0.tar.gz libgnutls-dbg-1.4.0-r1.tar.gz python-resource-2.4.3-ml0.tar.gz libgnutls-dev-1.4.0-r1.tar.gz python-shell-2.4.3-ml0.tar.gz libgnutls-doc-1.4.0-r1.tar.gz python-stringold-2.4.3-ml0.tar.gz libgnutls-extra13-1.4.0-r1.tar.gz python-subprocess-2.4.3-ml0.tar.gz libgnutls-locale-en+boldquot-1.4.0-r1.tar.gz python-syslog-2.4.3-ml0.tar.gz libgnutls-locale-en+quot-1.4.0-r1.tar.gz python-terminal-2.4.3-ml0.tar.gz libgnutls-locale-pl-1.4.0-r1.tar.gz python-tests-2.4.3-ml0.tar.gz libgnutls-openssl13-1.4.0-r1.tar.gz python-textutils-2.4.3-ml0.tar.gz libgnutls13-1.4.0-r1.tar.gz python-threading-2.4.3-ml0.tar.gz libgpg-error-dbg-1.3-r1.tar.gz python-tkinter-2.4.3-ml0.tar.gz libgpg-error-dev-1.3-r1.tar.gz python-unittest-2.4.3-ml0.tar.gz libgpg-error-locale-de-1.3-r1.tar.gz python-unixadmin-2.4.3-ml0.tar.gz libgpg-error-locale-fr-1.3-r1.tar.gz python-xml-2.4.3-ml0.tar.gz libgpg-error-locale-pl-1.3-r1.tar.gz python-xmlrpc-2.4.3-ml0.tar.gz libgpg-error-locale-ro-1.3-r1.tar.gz python-zlib-2.4.3-ml1.tar.gz libgpg-error-locale-vi-1.3-r1.tar.gz sqlite3-3.3.7-r2.tar.gz libgpg-error0-1.3-r1.tar.gz sqlite3-dbg-3.3.7-r2.tar.gz libice-dbg-X11R7.1-1.0.1-r0.tar.gz strace-4.5.14-r3.tar.gz libice-dev-X11R7.1-1.0.1-r0.tar.gz strace-dbg-4.5.14-r3.tar.gz libice6-X11R7.1-1.0.1-r0.tar.gz strace-doc-4.5.14-r3.tar.gz libipkg-dev-0.99.163-r1.tar.gz subversion-1.3.1-r1.tar.gz libipkg0-0.99.163-r1.tar.gz subversion-dbg-1.3.1-r1.tar.gz liblzo-dbg-1.08-r14.tar.gz subversion-dev-1.3.1-r1.tar.gz liblzo-dev-1.08-r14.tar.gz subversion-doc-1.3.1-r1.tar.gz liblzo1-1.08-r14.tar.gz subversion-locale-de-1.3.1-r1.tar.gz libneon-bin-0.25.5-r0.tar.gz subversion-locale-es-1.3.1-r1.tar.gz libneon-dev-0.25.5-r0.tar.gz subversion-locale-fr-1.3.1-r1.tar.gz libneon-doc-0.25.5-r0.tar.gz subversion-locale-it-1.3.1-r1.tar.gz libneon25-0.25.5-r0.tar.gz subversion-locale-ja-1.3.1-r1.tar.gz libpython2.4-1.0-2.4.3-ml5.tar.gz subversion-locale-ko-1.3.1-r1.tar.gz libreadline-dbg-4.3-r3.tar.gz subversion-locale-nb-1.3.1-r1.tar.gz libreadline-dev-4.3-r3.tar.gz subversion-locale-pl-1.3.1-r1.tar.gz libreadline-doc-4.3-r3.tar.gz subversion-locale-pt-br-1.3.1-r1.tar.gz libreadline4-4.3-r3.tar.gz subversion-locale-sv-1.3.1-r1.tar.gz libsm-dbg-X11R7.1-1.0.1-r0.tar.gz subversion-locale-zh-cn-1.3.1-r1.tar.gz libsm-dev-X11R7.1-1.0.1-r0.tar.gz subversion-locale-zh-tw-1.3.1-r1.tar.gz libsm6-X11R7.1-1.0.1-r0.tar.gz tcl-8.4.11-r3.tar.gz libsqlite-bin-2.8.17-r2.tar.gz tcl-dbg-8.4.11-r3.tar.gz libsqlite-dbg-2.8.17-r2.tar.gz tcl-dev-8.4.11-r3.tar.gz libsqlite-dev-2.8.17-r2.tar.gz tcl-doc-8.4.11-r3.tar.gz libsqlite0-2.8.17-r2.tar.gz time-1.7-r0.tar.gz libsqlite3-0-3.3.7-r2.tar.gz time-dbg-1.7-r0.tar.gz libsqlite3-dev-3.3.7-r2.tar.gz time-doc-1.7-r0.tar.gz libssl0.9.7-0.9.7g-r1.tar.gz tmp libx11-6-X11R7.1-1.0.1-r1.tar.gz update-rc.d-0.7-r0.tar.gz libx11-dbg-X11R7.1-1.0.1-r1.tar.gz util-macros-dev-X11R7.1-1.0.2-r0.tar.gz libx11-dev-X11R7.1-1.0.1-r1.tar.gz vim-7.0-r0.tar.gz libx11-doc-X11R7.1-1.0.1-r1.tar.gz vim-doc-7.0-r0.tar.gz libx11-locale-X11R7.1-1.0.1-r1.tar.gz vim-help-7.0-r0.tar.gz libxau-dbg-X11R7.1-1.0.1-r0.tar.gz vim-syntax-7.0-r0.tar.gz libxau-dev-X11R7.1-1.0.1-r0.tar.gz vim-tutor-7.0-r0.tar.gz libxau-doc-X11R7.1-1.0.1-r0.tar.gz wget-1.9.1-r6.tar.gz libxau6-X11R7.1-1.0.1-r0.tar.gz wget-dbg-1.9.1-r6.tar.gz libxdmcp-dbg-X11R7.1-1.0.1-r0.tar.gz wget-doc-1.9.1-r6.tar.gz libxdmcp-dev-X11R7.1-1.0.1-r0.tar.gz wget-locale-bg-1.9.1-r6.tar.gz libxdmcp6-X11R7.1-1.0.1-r0.tar.gz wget-locale-ca-1.9.1-r6.tar.gz libxml2-2.6.22-r3.tar.gz wget-locale-cs-1.9.1-r6.tar.gz libxml2-dbg-2.6.22-r3.tar.gz wget-locale-da-1.9.1-r6.tar.gz libxml2-dev-2.6.22-r3.tar.gz wget-locale-de-1.9.1-r6.tar.gz libxml2-doc-2.6.22-r3.tar.gz wget-locale-el-1.9.1-r6.tar.gz libxml2-utils-2.6.22-r3.tar.gz wget-locale-es-1.9.1-r6.tar.gz libxt-dbg-X11R7.1-1.0.2-r0.tar.gz wget-locale-et-1.9.1-r6.tar.gz libxt-dev-X11R7.1-1.0.2-r0.tar.gz wget-locale-fr-1.9.1-r6.tar.gz libxt-doc-X11R7.1-1.0.2-r0.tar.gz wget-locale-gl-1.9.1-r6.tar.gz libxt6-X11R7.1-1.0.2-r0.tar.gz wget-locale-he-1.9.1-r6.tar.gz libz-dbg-1.2.3-r1.tar.gz wget-locale-hr-1.9.1-r6.tar.gz libz-dev-1.2.3-r1.tar.gz wget-locale-hu-1.9.1-r6.tar.gz libz1-1.2.3-r1.tar.gz wget-locale-it-1.9.1-r6.tar.gz minicom-2.1-r0.tar.gz wget-locale-ja-1.9.1-r6.tar.gz minicom-dbg-2.1-r0.tar.gz wget-locale-nl-1.9.1-r6.tar.gz minicom-doc-2.1-r0.tar.gz wget-locale-no-1.9.1-r6.tar.gz ncurses-5.4-r8.tar.gz wget-locale-pl-1.9.1-r6.tar.gz ncurses-dbg-5.4-r8.tar.gz wget-locale-pt-br-1.9.1-r6.tar.gz ncurses-dev-5.4-r8.tar.gz wget-locale-ro-1.9.1-r6.tar.gz ncurses-doc-5.4-r8.tar.gz wget-locale-ru-1.9.1-r6.tar.gz ncurses-terminfo-5.4-r8.tar.gz wget-locale-sk-1.9.1-r6.tar.gz ncurses-tools-5.4-r8.tar.gz wget-locale-sl-1.9.1-r6.tar.gz openssl-0.9.7g-r1.tar.gz wget-locale-sv-1.9.1-r6.tar.gz openssl-dbg-0.9.7g-r1.tar.gz wget-locale-tr-1.9.1-r6.tar.gz openssl-dev-0.9.7g-r1.tar.gz wget-locale-uk-1.9.1-r6.tar.gz openssl-doc-0.9.7g-r1.tar.gz wget-locale-zh-cn-1.9.1-r6.tar.gz python-audio-2.4.3-ml0.tar.gz wget-locale-zh-tw-1.9.1-r6.tar.gz python-bsddb-2.4.3-ml0.tar.gz xcmiscproto-dev-X11R7.0-1.1.2-r0.tar.gz python-codecs-2.4.3-ml0.tar.gz xextproto-dev-X11R7.0-7.0.2-r0.tar.gz python-compile-2.4.3-ml0.tar.gz xf86bigfontproto-dev-X11R7.0-1.1.2-r0.tar.gz python-compiler-2.4.3-ml0.tar.gz xproto-dev-X11R7.1-7.0.5-r0.tar.gz python-compression-2.4.3-ml0.tar.gz xtrans-dev-X11R7.0-1.0.0-r1.tar.gz python-core-2.4.3-ml1.tar.gz </code></pre> Installing the needed packages is just a matter of uncompressing a subset of the above files on the target system. Summary</h2> This exercise demonstrates how a developer can quickly (took me less than 4 hours) add a large number of applications (and dependencies) to about any Embedded Linux system. Leveraging embedded Linux is being able to use the components you need. Please contact us if you would like additional details or assistance. Moxa UC7408 Review 2006-09-29T00:00:00+00:00 I am currently evaluating a Moxa UC7408 for one of my customers. The UC7408 is a small fanless industrial computer that runs Linux or Windows CE. This article provides a basic overview of this unit and a review of some of the UC7408 features and the Linux distribution Moxa provides. The basic specifications for the UC7408 are: relatively small size and fanless</li> Intel Xscale IXP-422 266MHz Processor</li> 8 RS232/422/485 serial ports</li> 8 digital Input and 8 digital output ports</li> dual 10/100 Ethernet</li> PCMCIA, CompactFlash, Wireless LAN Expansion (supports 802.11b/802.11g)</li> Runs Linux or WinCE</li> Console serial port</li> </ul> Documentation</h3> The documentation Moxa provides is adequate and fairly detailed. The moxa documents include a quick install guide, hardware user’s manual, and a 114 page user’s manual. As good as this documentation is, a general knowledge of Linux is always helpful when working with embedded Linux systems. Basics like using SSH, FTP, Telnet, and basic Linux system administration are essential when working with embedded Linux. Linux Distribution</h3> The Linux Distribution provided by Moxa is fairly full featured. A few highlights with the v1.8 Linux firmware: Provides a 26MB user JFFS2 flash partition. This partition is read/write and is about 45% full leaving about 14.5MB for user files. It is probably also possible to delete some of the files that are not required, freeing up more space.</li> Apache & PHP support</li> SSH</li> a number of other utilities are provided such as telnet, ftp, iptables, etc.</li> VI editor</li> Many command line utilities found in most Linux systems.</li> 2.4.18 Kernel</li> </ul> The Linux distribution seems to be based on MontaVista Linux and seems fairly solid. I ran through some of the basics like mounting a CF card and everything seems to work. Firmware Updates and Recovery Mode</h3> Moxa provides a mechanism to update the firmware in the system. This update is a global flash update that programs the entire flash and will erase all user changes. The update works by enabling a RAM file system and copying the new flash image to the ram file system. Running an update command that copies the firmware update file into flash. The 7408 contains a stripped down version of Linux in a separate flash partition that can be booted in a recovery mode if the User flash partition becomes corrupted or unusable. This mode can be enabled by pressing the “Reset to Default” button on the unit and powering it on. In this mode, there is no ssh support and you must ftp an image from an ftp server to the device and then reflash it. This requires you to set up an ftp server that the Moxa system can access. While this is an entirely workable solution, it may be a challenge for inexperienced Linux users. Tools</h3> Moxa provides a Linux and Windows toolchain to use with the device. I have built a number of packages with the Linux toolchain and it seems to work well. It is based on GCC 3.3.2. Cross compiling applications for embedded Linux is always a challenge and in a future article I will present a way to use OpenEmbedded to compile a number of packages using the Moxa toolchain. Possible Improvements</h3> There would be some benefits if the system was a little more open in the following areas: access to the bootloader console and documentation. This would allow developers to more easily flash their own software.</li> readily provide source for GPL components. Moxa provides a form that you can fill out to receive GPL source code on a CE. The cost is $100.</li> </ul> One of the big advantages of using Linux in an embedded system is the ability to use the many open source components available. The easier an embedded solution provider makes this for developers, the more value their system provides. See the white paper Tips for Planning Your Embedded Linux Project</a> (/web/content/view/35/37/) for more ideas on this subject. Summary</h3> Overall, the Moxa system seems very usable and the hardware is quite nice. Stay tuned for future articles about how to get the most out of this system. How to evaluate VoIP providers with Asterisk 2006-09-20T00:00:00+00:00 There are many options when selecting a VoIP provider to terminate your Asterisk PBX. One of the most important factors in this selection is the quality of the internet connection between your PBX and the VoIP provider. The route between you and your VoIP provider is one of the most important considerations when selecting a provider. This article details a simple method for evaluating this connection using Asterisk. Asterisk includes a feature that can be used to monitor the latency between your system and a peer or friend. This feature is enabled by setting the qualify setting in the iax.conf configuration file. Valid options are yes, no, or a time in milliseconds. If qualify is enabled, NOTIFY messages are sent periodically to the peer and the latency between replies is measured. The peer is determined unreachable if the number of milliseconds is greater than the qualify value or 2,000 if qualify is set to yes. If a peer is unreachable, events are logged in /var/log/asterisk/messages as shown below: Sep 19 21:48:19 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now UNREACHABLE! Time: 49 Sep 19 21:50:09 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now REACHABLE! Time: 49 Sep 20 07:16:38 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now TOO LAGGED (769 ms)! Sep 20 07:16:48 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now REACHABLE! Time: 50 Sep 20 12:09:31 NOTICE[752] chan_iax2.c: Peer 'teliax' is now TOO LAGGED (2923 ms)! Sep 20 12:17:27 NOTICE[752] chan_iax2.c: Peer 'teliax' is now REACHABLE! Time: 83 </code></pre> I am currently using two different VoIP providers and I get qualify notice messages about once every day or two with the qualify set to 400ms. This is a very useful information as it allows me to determine which provider has the most reliable route from my office. If you get very few qualify NOTICE messages, you can probably assume your VoIP connection will be fairly reliable. Have connections with two different providers also tells me when the congestion is with my Internet connection or if it is something beyond my connection. If both providers go out at the same time, it is most likely the internet connection at my office or problems with my ISP. I rarely if ever see this scenario, so I can conclude that my internet connection is fairly reliable and the congestion problems are most likely between me and the VoIP provider or problems at the VoIP provider. Overall, things are now working fairly well. As we continue to resolve problems, VoIP is now starting to become a very usable option for us. Implenting QoS for VOIP in a Linksys router 2006-08-28T00:00:00+00:00 In our quest to implement an Asterisk PBX in our office, we are evaluating several VOIP (Voice Over IP) providers. VOIP provides several benefits over traditional phone lines including: low cost (typically around $0.02 per minute)</li> available in pre-paid plans</li> can support concurrent concurrent calls on the same number</li> </ul> However, implementing VOIP successfully is not trivial. You must have a reliable internet connection and we have found you must implement some type of Quality of Service (QoS) in the router. The first problem I ran into was calls sounded great to me, but people I called mentioned the calls sometimes sounded garbled to them. This article discusses the diagnosis of this problem and how to implement QoS in a Linksys router to fix this issue. Our connection is a standard DSL connection – 5Mb/sec down and 640Kb/sec up. There is plenty of bandwidth for incoming voice, but bandwidth for outgoing voice is much more limited. A few tests confirmed this – when a ftp upload or some other bulk transfer was occurring, the outgoing voice sounded really bad. The solution to this problem is to implement some type of Quality of Service (QoS) mechanism that prioritizes the VOIP traffic over all other types of traffic. Vonage and other “boxed” solutions no doubt do this as they recommend placing their box between the modem and the router. It turns out that QoS can be implemented in a standard Linksys router/firewall. I spent some time trying to get QoS working using a Linksys WRT54G v5 router with no success, I ended up loading the dd-wrt firmware (http://www.dd-wrt.com</a>) on an older Linksys router I had laying around. The dd-wrt firmware is quite nice and is worthy of consideration if you have a v4 or earlier Linksys router or the WRT54GL version. QoS may have worked on the WRT54G v5 with Linksys firmware with some more effort, but after trying dd-wrt, I’m not going back. Setting up QOS</h2> The QoS settings can be found under Applications & Gaming -> QoS. There are many ways to assign priority to your VOIP system. I just used the IP address as shown below. 192.168.1.10 is the LAN private IP of the Asterisk system. With the priority of the Asterisk box set to high, we should be all set. It turns out that unless you cap the bandwidth, you will still have QoS problems. Enter around 85% of your up and down bandwidth in the Uplink and Downlink boxes. Capping the bandwidth is required to keep the latency low. Otherwise, your bandwidth is filled up with large packets and the priority settings are not effective for VOIP traffic. See http://www.dd-wrt.com/wiki/index.php/Quality_of_Service</a> for more information. With the above settings, VOIP calls are now clear even with other concurrent activities. UPDATE: Several people have reported being able to install dd-wrt on v6 WRT54 routers and the QoS configuration is the same. Embedded is a Service Industry 2006-06-02T00:00:00+00:00 Linuxdevices.com has recently published an article titled: Let’s call Embedded what it is – a Service Industry(http://linuxdevices.com/articles/AT6989225710.html</a>). In this article, Curt Schacker discusses how little growth there has been in the Embedded Systems industry. He claims that most embedded companies are trying to establish themselves as product companies, when the market is really a services market. Having worked for two Embedded companies in the past that went out of business, I think he is right. I have often heard the argument that service companies are too much work, they don’t scale, etc. But, to me services has always been a more natural fit for myself and an embedded systems company – one of the reasons BEC Systems exists. TRAC and Distributed Development 2006-05-29T00:00:00+00:00 I have long been a big fan of source control tools and issue tracking systems. The “concurrent” model made popular by CVS is a tremendous benefit any time there are multiple people working on the same project. For most of the projects BEC participates in, we end up setting up a source control system (usually Subversion these days) and an issue tracking system (typically Bugzilla). On one of my recent projects, I decided to try the trac integrated SCM & Project Management tool. Trac brings together several functions into a clean, easy to use web based tool for managing projects. This article relates some of my experiences and the benefits of using trac. What is trac?</h2> Trac provides the following functions (from the trac website - http://www.edgewall.com/trac/</a> ): An integrated system for managing software projects</li> An enhanced wiki</li> A flexible web-based issue tracker</li> An interface to the Subversion revision control system</li> </ul> Why is trac so interesting?</h2> Although all of these functions are available as separate tools, having them in one tool has several benefits: The tool is more likely to be used if everything is in one place</li> The ease of cross referencing wiki pages, issues, and Subversion changesets compounds the usefulness of each tool.</li> </ul> I have been using and implementing collaboration tools for years and one thing you eventually learn is that getting people to use collaboration tools is a big challenge (see http://www.linux-watch.com/news/NS9450464356.html</a> for some more thoughts on this subject). Having mutiple tools in one place greatly increases the likelihood they will be used. A typical session might go something like this: A developer might want to see what has changed recently in the code base, so he logs into trac to view the timeline – after all, most developers are mainly concerned with code.</li> As the trac timeline lists all changes (wiki and issues as well as source code changes), he notices that a wiki page with notes about driver X has changed – that looks interesting, so he view the page. While viewing the page, the developer notes there is some missing information, so he adds the information by simply editing the page.</li> Once this habit is established, it can greatly improve the productivity of the team as the project becomes more transparent to all team members.</li> </ul> Any collaboration tool must be easy to use and provide an obvious benefit to those using the tool. The wiki markup used in trac is used for both wiki pages as well as issue/bug reports. Referencing an issue in a wiki page or another bug report is simple as typing #. It is also very easy to reference code changsets in the wiki markup by simply typing r (see image below). Clicking on a issue or changeset number will take you directly to the information. The trac interface is very clean and very easy to use. Distributed teams</h2> Tools like trac allow for very effective distributed teams. The success of Open Source software has proven that development can occur efficiently in a distributed manner. BEC has integrated this model and we regularly work with people all over the country and in Europe. When there is a process for all people involved in the project to keep up with its status and progress, distributed development becomes very possible. Often, the customer is part of the development team. Project Transparency</h2> Another benefit of trac for BEC, is that we can easily provide project transparency to our customers. It is BEC’s practice to provide customers with access to the project as it develops, so we give our customers trac logins and they can follow the progress by watching the timeline and milestone pages. This gives the customer an assurance that progress is being made on the project and some insight into some of the issues encountered. Having a comprehensive view into the project also allows customers to provide early feedback and reduces BEC’s project management and reporting overhead. Summary</h2> Tools like trac improve our customer relationships, increase efficiency and reduce our costs. If you are interested in implementing tools like trac, BEC can assist you in selecting and implementing the right tools to make your projects run smoothly. Opencores SPI Controller Review 2006-05-08T00:00:00+00:00 Recently, I had the opportunity to use a SPI controller from Opencores in a customer design based on an Intel PXA255 processor and a Xilinx CPLD. The customer did not want to change the processor or the CPLD family, but needed an extra SPI interface in the device. I had my doubts, but mentioned I would look into a SPI controller from Opencores. To my surprise, the core fit in a Xilinx CPLD and worked out very well. We have since upgraded to a Altera MAXII CPLD. This review details this effort. SPI Controller Overview</h3> The SPI controller is based on the interface for a standard Motorolla SPI controller and is available at http://www.opencores.org/projects.cgi/web/spi/overview</a> . The core includes the following features: Full duplex synchronous serial data transfer</li> Variable length of transfer word up to 128 bits</li> MSB or LSB first data transfer</li> Rx and Tx on both rising or falling edge of serial clock independently</li> 8 slave select lines</li> Fully static synchronous design with one clock domain</li> Technology independent Verilog</li> Fully synthesizable</li> </ul> A datasheet type document is provided with the core that provides the necessary information for developers writing software for this core. The Wishbone bus</h3> Most of the cores from Opencores uses the Wishbone bus standard. The Wishbone bus is a synchronous bus designed to be used inside an IC. The CPLD was connected to the PXA255 processors asynchrous bus, so most of the effort required was to create an PXA255 <-> Wishbone interface module. At some point, I hope to release this module on Opencores. Simulation</h3> The simulation was done using the Icarus Verilog simulator under Linux. Below is a screen shot of the simulation displayed in GTKWave: Conclusion</h3> The Opencores SPI core worked out very well in our design and I was very impressed with quality of this core. The documentation is more than adequate. Standards like Verilog make it very easy to move designs between different vendors devices and allows designers to use tools of their choice such as the Icarus Verilog simulator. As FPGAs and CPLDs become more larger and more standand, Opencores is valuable resource for adding standard interfaces to these devices. A Really Nice Hex Calculator 2006-05-02T00:00:00+00:00 When working with embedded software, I often have a need for a hex calculator. Over the years I’ve used a number of different tools. Through college and for several years after, I used my HP48 calculator. It is a great tool. Around 1998, I discovered the Python programming language. As Python supports an interactive session, it works well as a desktop calculator – I don’t think I have used my HP48 since I discovered Python. Recently, I’ve been using Ruby more in place of Python and am finding it is even nicer. This article presents several tips and snippets of code for using Ruby as a very effective hex calculator. For those of you not familiar with Ruby, Ruby is an interpreted scripting language for quick and easy object-oriented programming – similar in concept to Python or Perl. Ruby provides an interactive Ruby shell called “irb”. After you install Ruby, you can run irb from a command prompt. You can immediately do useful calculations like the following: irb(main):001:0> a = 10 => 10 irb(main):002:0> b = 23 => 23 irb(main):003:0> a+b => 33 </code></pre> Hexadecimal numbers are prefixed with “0x” as in many other languages: irb(main):005:0> c = 0x1000 => 4096 </code></pre> It is easy to convert from hex to decimal – just type the hex number, hit return and Ruby will give you the decimal equivalent as shown above. To convert from decimal to hex, I often just use the printf statement. The syntax is similar to C, so it is very natural for embedded programmers. irb(main):006:0> printf "0x%x", (310241024) 0x300000=> nil </code></pre> Ruby provides standard shift operators similar to those found in C. For example, if you want to know the hex value when bits 15 and 23 are set high: irb(main):019:0> printf "0x%x", (1<<15)|(1<<23) 0x808000=> nil </code></pre> It is often useful to know what bits are set in a particular number when determining if a bit in a status register is high, what address bits are used, etc. Ruby provides a bit reference method in the form. fix[n] </code></pre> n is the nth bit in a binary number fix where fix[0] is the least significant bit. Some examples: irb(main):010:0> 2[0] => 0 irb(main):011:0> 2[1] => 1 irb(main):017:0> a = 310241024 => 3145728 irb(main):018:0> 31.downto(0) {|n| print a[n]} 00000000001100000000000000000000=> 31 </code></pre> For convenience, I wrote a simple module called bits.rb that extends the basic Integer number class and provides a nice display of the bits in a number. To use this, create a file called bits.rb with the following contents and drop it in your Ruby site_ruby directory. M = 10241024 K = 1024 class Integer def bits print "3322222222221111111111n" print "10987654321098765432109876543210n" print "---|---|---|---|---|---|---|---|n" 31.downto(0) { |x| print self[x] } end end </code></pre> And to use this extension to see what address bits are used to decode 8MB of address space: irb(main):020:0> require 'bits' => true irb(main):021:0> (8M).bits 3322222222221111111111 10987654321098765432109876543210 ---|---|---|---|---|---|---|---| 00000000100000000000000000000000=> 31 </code></pre> From this example, we can see that address bits 0 through 23 are required to decode 8MB of address space. =========================================== Update 2009-07-30 The following was posted as a comment to my old web site: An even easier way of converting between bases in Ruby: irb(main):001:0> puts 300.to_s(16) 12c => nil irb(main):002:0> puts "12c".to_i(16) 300 => nil irb(main):003:0> puts 255.to_s(2) 11111111 => nil </code></pre> Very nice! Specialization or not 2005-11-15T00:00:00+00:00 I have often wondered about specialization. We can often find ourselves in ruts if we specialize too much. But on the other hand, specialization can lead efficiencies in some cases. In my work as a consultant, I often find myself in new situations where I have to learn fast. As one of my associates states – problem solving is the key skill. Kevin Barnes article Three theories on how to use developers efficiently</a> is a good read about the subject. Total Phase USB Protocol Analyzer review 2005-11-03T00:00:00+00:00 I recently purchased a USB protocol analyzer from Total Phase</a>. The device will analyze full speed USB traffic and works well with Linux. On Linux, the software makes use of libusb, so there is very little setup required to get the device working – very nice. At a price of $400, there is really no reason not to have one if you are doing any type of USB development. Economics and Finding Outsourcing Talent 2005-11-03T00:00:00+00:00 Economics In One Lesson by Henry Hazlitt presents an interesting and logical view of economics. The book presents many ideas that seem right (like saving) and provides very logical arguments to back them up. Even though the book was originally published in 1946, the topics covered seem very applicable to today (The Blessings of Destruction, Public Works Mean Taxes, Taxes Discourage Production, Credit Diverts Production, The Curse of Machinery, Spread-the-Work Schemes, Disbanding Troops and Bureaucrats, The Fetish of Full Employment, Who’s “Protected” by Tariffs?, The Drive for Exports, “Parity” Prices, Saving the X Industry, How the Price System Works, “Stabilizing” Commodities, Government Price-Fixing, What Rent Control Does, Minimum Wage Laws, Do Unions Really Raise Wages?, “Enough to Buy Back the Product”, The Function of Profits, The Mirage of Inflation, The Assault on Saving) The book can be found: online: http://jim.com/econ/contents.html</a></li> download: http://www.fee.org/pdf/books/Economics_in_one_lesson.pdf</a></li> amazon.com</li> </ul> The following article presents some of the realities of finding talent in places like India. It is encouraging that the basic economic laws of supply and demand still work. Finding coders on the subcontinent</a></h3>}

BEC Systems

Human Friendly Industrial Device IDs

What makes an ID easy for humans to process?</h2> Many MFA (multi-factor authentication), one-time passwords, and authenticator apps use codes that are composed of 6 digits split into two chunks:</p>

Survey of device ID mechanisms

How to implement Zephyr web applications

Primary concerns in an MCU system</h2>

Options</h2>

Struggling to scale your IoT system? Simplify your data.

The Ability to Improve

A few more examples</h2> Below are a few more examples of situations that are very difficult to deal with in the lifecycle of a product:</p>

The ROI Objection</h2> There are two financial mindsets:</p> View R&D as a one-time capital expense where you design something, lock it down, and then try to extract as much value as possible – the “minimizing costs” mindset.</li>

The Efficiency Objection</h2> Efficiency is always a concern, and again there are two schools of thought:</p>

Requirements for Improvement</h2> Every person and every organization is different, so there is no one-size-fits-all recipe that will always work. But there are several ingredients that are generally beneficial including:</p>

Think Horses, not Zebras (Part 2)

PCIe clock amplitude</h2> While verifying the hardware of a design, we noticed that the PCIe clock was at a lower amplitude than the specification required:</p>

Why Open Source for Product Development

Think Horses, not Zebras

The Technology Trajectory

Go for IIoT Systems

Attributes of Go</h2> Go has several attributes that make it a good fit for this problem:</p>

Layered, Event, and Data-Centric Architectures in Distributed IoT Systems

Business Contracts

Technology

Adventures with a ATSAMA5D27 board

Drone for Continuous Integration

Reflections on KiCad and EDA Tools

The Humble Programmer

Why are Go applications so reliable?

Auto-formatting/linting Go code

Microcontroller (MCU) or Microprocessor (MPU)?

Accepting Constraints in Build Systems

SD Card Write Speed Tests

Getting started with Embedded Linux

Ideas for Learning Embedded Linux</h3>

Modify and install U-boot</h4> U-boot is a common bootloader for Embedded Linux systems (but not the only one). Locate the source code for your embedded board and learn how to build, modify, and install a custom version of U-boot.</p>

Maintain Yocto or Buildroot support for an embedded board</h4> There are many embedded Linux platforms out there that are not actively maintained or tested in Yocto or Buildroot. Get involved and help keep the platform current and working in the build system.</p>

Git, Versioning, and Branching for Embedded Linux Development

Separation of Source and Build Directories

Understanding the NXP i.MX6UL Pin Mux (Part 2)

Understanding the NXP i.MX6UL Pin Mux

Why Git?

Using Go in place of a Spreadsheet

Using Docker for OE/Yocto builds

Setting up a Go development environment

Go Language for Embedded ARM Linux Systems

IOT Protocols: MQTT vs CoAP vs HTTP

OpenEmbedded / Yocto Usage Survey

Rebasing a set of changes with Git

Modifying the BusyBox config in OpenEmbedded

Best practices for using CMake in OpenEmbedded/Yocto

PCB Schematics Style Guide

Library sizes for C vs C++ in an embedded Linux system

Setting up an OpenEmbedded Package Feed Server

Why systemd in Embedded Linux Systems?

Modify and install U-boot</h4>
U-boot is a common bootloader for Embedded Linux systems (but not the only one). Locate the source code for your embedded board and learn how to build, modify, and install a custom version of U-boot.</p>

Maintain Yocto or Buildroot support for an embedded board</h4>
There are many embedded Linux platforms out there that are not actively maintained or tested in Yocto or Buildroot. Get involved and help keep the platform current and working in the build system.</p>