❓ Better questions get better answers

2026-03-25T00:00:00+00:00

A common challenge in communication is getting people to open up and share what they really think. One way to improve this is to avoid leading questions, which steer a person toward a particular answer instead of letting them respond freely. Some examples:</p>

Leading question</th>	Open-ended alternative</th></tr></thead>
“What do you think about SAP?”</td>	“Can you walk me through how you currently manage your core business processes (finance, operations, etc.)?”</td></tr>
“Have you ever thought of having a PMO?”</td>	“How do you currently organize and oversee your projects and initiatives?”</td></tr>
“Have you ever thought of a web site?”</td>	“How do customers typically find information about your company and its services today?”</td></tr>
“Why don’t you use Git?”</td>	“How do you currently track changes and collaborate on code?”</td></tr>
“Why aren’t you Release First?”</td>	“Can you describe your current process for making work available for others to use?”</td></tr> </tbody></table> The key is to start with the problem we are solving instead of our preconceived solution. So … what is your process for making work available for others to use?</p> </p> 🏞️ Communicating Status Without Demanding Attention 2026-03-24T00:00:00+00:00 One precept of “managing up”</a> is that over-communicating</a> tends to work better than under-communicating. This is where releasing “often” comes in. Releases offer a natural way to communicate status because they don’t feel like something requiring a response, the way a direct email or message does. Releases done right are like a peaceful mountain stream flowing by — something people can observe and dip their hand into when there is something they are interested in. The stream does not demand attention, but provides a comforting presence that things are getting done — continuously, reliably, and peacefully. To carry the analogy a little further, consider the stagnant pond of obscurity where nobody is quite sure what is going on. Or the torrential flood of bugs and customer problems.</p> </p> 🤝 How to manage up 2026-03-20T00:00:00+00:00 The concept of “managing up”</a> can be a valuable practice for developers and middle managers. It seems counter-intuitive that we need to manage the people over us. But, simply put, managing up is intentionally improving the relationship with your boss to make their job easier, align with their priorities, and improve the outcomes for both of you. This is not manipulation, kissing up, or trying to look good, but rather communication and understanding.</p> Release First</a> is a useful tool for managing up. It gives those above us clear visibility into what we are doing in a language they can understand</a>, and even allows them to use and try it in many cases. Release First does not replace all the interpersonal skills needed to manage up, but it does establish the groundwork so you can focus on the important parts of relationships.</p> </p> 👥 The release is a language everyone understands 2026-03-19T00:00:00+00:00 A release is a good way for developers to communicate with everyone else:</p> What got done</li> What changed</li> How to use or build the latest version</li> Where to get support or provide feedback</li> A regular cadence for communication</li> A standard format everyone can follow</li> </ul> Releases can save a lot of time normally spent in status meetings and generating status reports, freeing up developers and managers to focus on the work that needs to get done.</p> And with AI, releases are easier than ever to generate and automate.</p> </p> 📦 What is a release used for? 2026-03-18T00:00:00+00:00 A release needs to happen before a product is shipped to customers for sale. Across the spectrum of the product development lifecycle, releases are used for:</p> Generate feedback (so we know where we stand)</li> Provide something others can build on (this is how we scale)</li> Deliver value to customers (this is how revenue is generated)</li> </ul> Feedback, scaling, and profit - the engines of business.</p> </p> ⚡ The essence of shipping faster 2026-03-17T00:00:00+00:00 In recent conversations about why development schedules slip and what to do about it, several themes continue to come up:</p> Openness</strong> - This is key to surfacing issues early.</li> Safety</strong> - People must feel safe being transparent and speaking up.</li> Collaboration</strong> - People must be willing to share, help others, and receive help from others or use what others have built. Effective people don’t work in silos.</li> </ul> These all come down to culture. As one person put it:</p> Process without culture doesn’t work.</p> </blockquote> The human factors</a> are the hard part, and the most difficult for technically oriented people to figure out. But, they are the key to high-performing teams.</p> Great leadership and how to affect culture is beyond the scope of this article, but Release First is a powerful tool for assisting leaders in developing a healthy culture. Why? Because releasing early and often is the ultimate act of transparency. It provides visibility into exactly what the team accomplishes and how well they accomplish it. Issues surface early. There are no proxies</a> to hide behind.</p> Releasing also gives people confidence and satisfaction as they see things moving forward. The job of a leader is to encourage people to release their work, encourage others to use those releases, and foster a culture of speaking up. The focus must remain on the path forward to the next release, not on blaming people for what went wrong in the last release. Do this, and the rest will largely take care of itself.</p> </p> 🔀 What is the difference between MVP and Release First? 2026-03-13T00:00:00+00:00 MVP (Minimum Viable Product) is a concept from Eric Ries’ Lean Startup</a> methodology. The idea is to build the smallest possible version of a product that lets you test a hypothesis with real users. Instead of spending months building a full-featured product, you strip it down to the core value proposition, ship it, and learn from actual usage whether you’re solving a real problem. If the hypothesis is wrong, you pivot early before investing heavily. MVP is fundamentally about product/market fit — are we building the right thing?</p> MVP and Release First share DNA but differ in focus.</p> Release First asks a different question: can we ship with confidence, early and often? It focuses on building the release infrastructure (CI/CD, tests, automation, versioning, update mechanisms) before the product itself. It applies broadly — not just software, but hardware, documentation, and any engineering artifact.</p> The risks they address are different too. MVP addresses the risk of building something nobody wants. Release First addresses the risk of the last 10% taking 90% of the time — the integration nightmares, the untested release processes, the meta-data illusion where project tracking says you’re on track while reality diverges.</p> </th> MVP</th> Release First</th></tr></thead> Core question</strong></td> Are we building the right thing?</td> Can we ship with confidence, early and often?</td></tr> Focus</strong></td> Scope reduction — cut features to learn faster</td> Process — build the Big Green Button first</td></tr> Applies to</strong></td> Product/market fit validation</td> All engineering artifacts (SW, HW, docs, PCBs)</td></tr> Risk addressed</strong></td> Building something nobody wants</td> The last 10% takes 90% of the time</td></tr> When it ends</strong></td> Once the hypothesis is validated/invalidated</td> Never — it’s an ongoing discipline</td></tr> </tbody></table> They’re complementary. An MVP without release infrastructure still risks blowing up at the end. And Release First without scope discipline can still build the wrong thing. The ideal is both — a small scope (MVP thinking) delivered through a repeatable, automated release process (Release First thinking) - efficiently, quickly, and confidently - over and over …</p> </p> ✅ Each release is 100% done 2026-03-12T00:00:00+00:00 One common problem in product development is developers not finishing a feature or task. They get it to 70% done and rush off to the next thing. Bugs linger, edge cases go unhandled, testing falls short, and the release process remains incomplete. “100% done” does not mean the product is 100% done — it means each release is complete, tested, and high quality. Quality products emerge from many small high-quality steps, not from a pile of slop that somehow gets polished by QA at the end of the development cycle.</p> Release First is a simple way to ensure each step in the journey is 100% done and high quality.</p> </p> 💡 A Simple Example 2026-03-11T00:00:00+00:00 The Release First mindset starts with how you handle reasonably complex information (a procedure, technical details, etc). There are two approaches:</p> Email or message the information.</li> Put it in Git and send a link or PR.</li> </ol> The first is transient, inaccessible, and difficult to build on or improve.</p> The second is permanent, reusable, accessible, and something people can build on. It is an asset.</p> Sure, there are times when email or messaging is the appropriate forum for discussions. But the minute there is something of value, turn that into a permanent, team-accessible asset. This is how teams build value.</p> </p> 🏷️ The simple approach to versioning 2026-03-10T00:00:00+00:00 The biggest barrier to starting something is an unwillingness to start simple. Let’s use versioning as an example. The natural tendency is to think that all software components need their own version, which can be a daunting task to implement, track, and manage. However, until a team is large enough to have multiple groups dedicated to different tasks, instead put everything in a monorepo</a> and use a single version for everything. Not every component needs to be released with every version, only the ones that have changed. There are several advantages:</p> There is a single changelog to review.</li> It is easy to coordinate things - components of the same version should work well together.</li> Much easier to use AI to execute tasks that involve multiple components. If there are several repos, you are always fighting security checks when crossing repos (at least in Claude Code).</li> It is easy and simple.</li> </ul> “But!!!,” you say, “The versions won’t always be sequential, some will be skipped for various components!!!” Why does this matter? If it bothers you, release every component for every version. Automation can easily do this.</p> The resulting changelog looks something like this:</p> 2026-03-06 v3.0.18 [Server] - Fix mongo iterator stopping on decode errors instead of skipping bad documents Add bson truncate tags to all int fields to handle float64 values from JavaScript [GW] - Improve response of Modbus polling. [Portal] - Add UI for e2e test status. 2026-03-06 v3.0.17 [Server] - Log rule count each processing cycle to aid debugging 2026-03-06 v3.0.16 [E2E testing] - Improved ntfy error messages to include response body for easier debugging. - Reduced log noise: only log pump and cycle mirror state changes instead of continuous tank level. - Added logging of test status on initial check and state changes. </code></pre> Start simple, and add complexity when there is a need.</p> </p> 📦 The bare minimum to call something a release 2026-03-09T00:00:00+00:00 What is the bare minimum to call something a release? Consider the following:</p> A version</a> that increments with every release.</li> A changelog</a> that describes what has changed for every version and why it matters.</li> A clear identification of any assets (IPNs</a>).</li> Unique IDs for anything built (HFIDs</a>).</li> A place to store stuff where all this gets tracked over time (Git</a>).</li> </ol> The concrete benefits:</p> Exact knowledge of what exists.</li> Clear record of who did what when.</li> Clear communication to everyone involved.</li> Nothing gets lost.</li> </ul> Doing the above for all design assets (including documentation) from day one</strong> in the development process improves thinking, communication, and effectiveness — and reduces untold confusion and wasted time.</p> This does not need to be hard. Simple text files, spreadsheets, and CSV files work great. It is perfectly fine to start small.</p> </p> ❓ Release Questions 2026-03-06T00:00:00+00:00 Some questions to consider about a release:</p> When is it created?</li> Where is it stored?</li> What does it contain?</li> How is it created?</li> Who uses it?</li> Why is it used?</li> </ul> The “Wh- questions”</a> are a powerful technique for analyzing anything we do.</p> </p> 🤔 Two types of unknowns 2026-03-05T00:00:00+00:00 There are two types of unknowns:</p> Things we don’t know.</li> Things we think are true, but are not. 1</a></sup></li> </ol> Both can inflict havoc on our plans, but the second is the most damaging. Why? The first is more associated with doing nothing. Bugs not known to be there, etc. But the second prompts people to take action and go down a path that can be difficult to undo (promised ship dates, features, etc.). It is not too hard to say “oops, I did not know about that problem - we’ll fix it.” But it is really hard to say “sorry, I was completely wrong about that.” - this takes a great deal of humility.</p> While avoiding #1 is important, avoiding #2 is critical, because that forces people to retrace their steps and admit they were wrong. While humbling oneself is beneficial for character development, practically, it is difficult (if not impossible) for most people to do, especially those in leadership positions. So let’s Release First</a>, and learn where we are wrong in little ways, rather than have a BIG 1.0</a> humiliation.</p> ^{1</sup> “It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” - Mark Twain (maybe)</p> </div> </p>} 🧊 The hardest thing to manage 2026-03-04T00:00:00+00:00 The things we know and understand are easy to manage. We can hire managers who are good at this. Tools can effectively track this. Life is good. The difficulty arises in the unknown. The iceberg analogy illustrates this well. The things we see, understand, and touch are above water. Everything else is underwater. This is often where unexpected problems arise. As systems get more complex, the size of the unknown grows. We can’t predict what will happen in the unknown; the only thing we can do is release and integrate early.</p> </p> 🎉 The BIG 1.0 Release 2026-03-03T00:00:00+00:00 BIG releases are appealing — the magical moment when everything comes together. The months or years of hard work that finally come to fruition. The excitement and fanfare of the BIG announcement. The accolades and admiration of customers and the press.</p> … except, it rarely works this way</a> unless you are Apple or Tesla. And what we don’t see with high-end consumer products is the rigorous internal release and testing programs … something lower-volume industrial products don’t have the luxury and resources to do.</p> The BIG Release mindset assumes all the code and designs will magically ship when the BIG day comes. It never goes this smoothly. Sure, we may build prototypes and do testing, but unless we do these with release processes we can reuse, it is hard to be confident we can do it again quickly, easily, and accurately. One-off prototype processes don’t scale. Only release processes scale. And the reality is that the rigor of releases in the prototype phase works better anyway.</p> The solution is of course … Release First. The problem is that The BIG 1.0 Release fits human nature better (procrastination, excitement, and glory). Release First is boring, tedious at first, and no one cares. The effort is invisible and rarely gets rewarded. Very few developers are wired for Release First. Thus, Release First is not a technical problem, but rather a cultural one.</p> </p> ❓ What is the difference between "releasing" and "shipping"? 2026-03-02T00:00:00+00:00 Someone asked: “What is the difference between releasing</strong> and shipping</strong>?” Are they the same thing?</p> In one sense they are. However, drawing a distinction here—though somewhat arbitrary—is helpful in this context.</p> Release:</strong> Make something available.</p> Ship:</strong> Deliver to the customer for sale.</p> Releasing is part of the process to get something to the point where it provides enough value that customers will pay for it. Releasing is the way useful work gets shared with others and the delivery process gets refined. Releasing enables early feedback needed to know exactly what to build. The Release is the way different disciplines communicate around what the team is building. Releases are a concrete measure of development progress. Releases build confidence in the product.</p> The problem arises when the two are considered equivalent—when there is no release until ship. Then inevitable surprises pile up.</p> </p> 🚨 Problems with not Releasing First 2026-02-27T00:00:00+00:00 What is the downside of not Releasing First</a>?</p> Integration problems</li> Manufacturing issues</li> Last minute bugs appearing</li> Lack of product fit (does not solve customer’s problem)</li> </ul> The common theme is unforeseen problems that typically surface late in the development cycle. The point of Release First is to expose unknowns early, when there is still time to address them.</p> </p> 🌅 Why optimizing the user release experience early is so important 2026-02-25T00:00:00+00:00 With Release First, the goal is to optimize the Big Blue Button</a> (make it easy for users to use a release) early. Why is this important? There are many reasons, but the most important is so that users actually use the release and provide needed feedback early.</p> Without optimizing the release experience early (when the circle</a> is small), developers soon become inundated with support issues as the circle grows. Pretty soon the developers are doing nothing but support. Optimizing the user experience early enables scaling.</p> Fix things early, when the stakes are low, not after the BIG 1.0 Release.</p> </p> 🔵 The Big Blue Button 2026-02-24T00:00:00+00:00 In the past days, we’ve been focusing on the Big Green Button</a>, but there is another side of this - what happens after the release? Does the release get used? If not, why not? If a release does not get used, there is little point in releasing.</p> There are several aspects to consider in the usability of a release:</p> Communicating to users that a release is available and relevant to them.</li> Making it easy to consume the release.</li> An efficient way to support and get feedback from users.</li> </ol> We have a Big Green Button to create a release, so let’s continue the analogy with a Big Blue Button that makes it easy for a user to consume a release. We can’t just throw something over the wall (that may be technically very good and complete) and expect magic to happen without any consideration for the user. What does the Big Blue Button include?</p> Clear documentation, tutorials, videos.</li> Well organized assets that are consistent from release to release.</li> An easy way to obtain/install/update the release.</li> Built-in help functionality, diagnostics, automated error reporting.</li> A community where the user feels included, supported, and part of something bigger than themselves. This is important to have, even early on when the internal team is the only user base.</li> </ul> One of the key warning flags is when a bunch of hand-holding and personalized help is required for someone to use a release. Can manufacturing take what the team produced and run with it? Can someone use a software service without personalized support? Can someone set up and configure a product without a dozen support calls? The point is not to avoid talking to users, but rather to talk to them after they have used the product and can add value to the development process.</p> There needs to be a symmetry between the Big Green Button and the Big Blue Button - easy to produce and easy to consume, otherwise the flow will stall. These are the factors that will differentiate developers and products in the AI age.</p> </p> 🧠 The Human Factors 2026-02-23T00:00:00+00:00 Gerald M. Weinberg</a> once said:</p> No matter what the problem is, it’s always a people problem. — Gerald M. Weinberg</p> </blockquote> At its heart, release-first addresses the human element:</p> Our propensity to procrastinate in getting releases out.</li> Our fear that something might be broken in the release.</li> Our tendency to think short-term instead of long-term.</li> Our tendency to rush on to the next feature before finishing the one we started.</li> Our tendency to skip tests when writing code.</li> Our OCD in needing to perfect something before anyone else can see it.</li> And the list goes on ad infinitum</em> …</li> </ul> Release-first at its core is a way to address human weaknesses. It is a simple tweak in our behavior that anyone can do. It is a language that everyone can understand. It can be measured by anyone without fancy tools. It is so absurdly simple that most people will write it off as “obvious” or “too simple” and instead reach to the merchants of complexity</a>. But true effectiveness comes when we reduce things to the simplest form possible, and remove all the accidental complexity.</p> So the next time you’re struggling with a release process, a testing gap, or a feature that never quite ships — remember Weinberg. The problem isn’t the tools, the framework, or the process. It’s us. Release-first works because it doesn’t try to engineer around human nature — it simply makes the right behavior the default behavior.</p> </p> ☝️ What do we mean by "First"? 2026-02-20T00:00:00+00:00 What do we mean by Release “First”? Several things:</p> First in the timeline</strong> of development.</li> First thing we focus</strong> on during implementation.</li> First priority</strong>.</li> </ul> Simply put, implement the Big Green Button</a> before implementing the actual deliverable. This might include:</p> Tests</li> CI/CD</li> Build automation</li> Deployment scripts</li> PCB release automation</li> Clear user documentation</li> etc.</li> </ul> This becomes part of the review process, part of the culture, part of the expectation, and part of the daily habit — make it easy to release.</p> Because the Big Green Button</a> does not ship to customers, forgetting and neglecting this step comes naturally. Release First is a simple hack to ensure it gets done before the rush to do other things.</p> </p> 🟢 What is in the Big Green Button? 2026-02-18T00:00:00+00:00 As previously stated, the Big Green Button</a> enables teams to quickly and confidently release. But what is the Big Green Button? Several things come to mind:</p> Automation that translates code/design/docs/models into assets others can use.</li> Automated testing that builds confidence in the release.</li> </ul> These save developers time by avoiding repetitive, tedious tasks, avoiding mistakes, and reducing support load. These investments have a very quick payback.</p> The point with the Big Green Button analogy is that once we have a Big Green Button, we don’t think about what is inside the button every time we release, our minds are free to focus on other things.</p> </p> 🟢 The Big Green Button 2026-02-17T00:00:00+00:00 The word “release” can be a noun (what gets released) or a verb (the process of producing a release). “What” and “How”.</p> Many teams produce complete high-quality releases that allow them to produce or deploy something at some point in time. However, each new release requires significant manual effort:</p> Manually testing software to make sure it is solid.</li> Manually going through the process of generating output formats, and updating drawings for manufacturing. This might involve double-checking the output files.</li> Manually updating BOMs and top level information.</li> Manually updating instructions, manuals, and other released documentation.</li> </ul> An alternative to the above is the “Big Green Button” - a single step that can convert designs/code/models/documentation into something others (manufacturing, sales, customers, stakeholders, etc.) can use. This is the critical point - do you have a “Big Green Button”? Without one, releases naturally get delayed until the last minute — losing opportunities for critical early feedback and the ability to quickly response to new issues or needs.</p> The Big Green Button is what gets the work from developers to users - quickly and confidently. And this is something BEC</a> can help with.</p> </p> 🔄 The release circle 2026-02-16T00:00:00+00:00 As previously discussed, a release is useful and quality</a>. What counts as a release? One way to think about this is to view the release as a circle with two attributes:</p> What (it contains)</li> Who (it’s for)</li> </ol> Initially, there is not much in the circle (perhaps only a specification), and only the development team and a few others in the organization are interested. But it is something that can be used and built upon. The team will change and expand it, but it will continue to exist in some form in all future releases. This might start with a spec, then a UI prototype in the cloud, then a device prototype based on a single-board-computer, then a custom system-on-module PCB, and so on. As time goes on, the functionality expands and improves.</p> The circle of who uses a release also expands over time, to include sales and marketing, management, beta-customers, and finally a general product release. Over time, the number of customers also grows to include more varied use cases.</p> The key point is to release a small slice of permanent</em> functionality early, and then continue to grow/add slices and the number of people who use the product during the product development cycle</em>. Waiting until the BIG Release is too late. It’s OK to start small. As long as it’s a release</a>.</p> But there are things that are not a release:</p> Emails</li> Messages</li> Meetings</li> Memos</li> Experiments</li> Prototypes (that are not released)</li> </ul> Why? Because this information is not permanent. While others consume it and find it very useful, it is transitory and cannot be continually used, refined, and improved over time. It is one-off communication.</p> </p> 🧩 Why things fall apart at the end 2026-02-11T00:00:00+00:00 As discussed in the previous two posts, things often go haywire near the end of a project cycle, right before the BIG Release. Why is this? Things seem to go so well at the beginning of the project. While it is impossible to detail every reason this happens, a few are listed below.</p> Technical debt</strong> occurs when problems are half-solved. Things appear to work, but edge cases and inadequate testing lurk beneath the surface. As time progresses, the system becomes more fragile, and significant amounts of time go toward fighting bugs and unplanned problems.</p> Scope creep</strong> is a common phenomenon where features continually accumulate throughout the project. Some of this is natural if it’s based on user feedback and learning while implementing. However, there is an unnatural scope creep rooted in engineering whims, just-in-case thinking, and a developer’s natural propensity toward complexity. Just because something can be done does not mean it should be.</p> Integration issues</strong> occur when one piece of work combines with another. This is often when new edge cases surface.</p> Not correctly anticipating user needs</strong> is another problem. A team may build something thinking the requirements are understood, but it is not until people start using it that the real needs emerge.</p> Lastly, there are unknowns</strong> that simply cannot be anticipated. Complex systems are just that — impossible to completely understand. People involved at every level are fallible and unpredictable. A large amount of uncertainty simply cannot be known or predicted up front.</p> The simplest way to address all of these issues is to move the end closer to the beginning — Release First.</p> </p> 🪞 The illusion of project meta-data 2026-02-10T00:00:00+00:00 Many tools exist for managing and tracking projects: Jira, Trello, GitHub projects/issues, Notion, Airtable, Asana, MS Project, etc. These all can be very useful. Why then does effective project tracking remain so elusive? As mentioned in the previous post</a>, in a typical scenario, things appear to progress well. Issues get checked off. The burn-down chart tracks the line. But out of nowhere, monsters (bugs, unanticipated feature requests, and integration problems) appear that catch everyone off guard.</p> Several factors contribute to this problem. Foremost, the meta-data is exactly that — “meta.” It is data about something; it is not the thing. There is no guarantee it is complete and accurate. If the focus stays on the meta-data, the meta-data will (perhaps unknowingly) be gamed to look good. If developers get rewarded for checking issues off every sprint, they will eventually make sure that happens, even though quality, maintainability, and other factors suffer.</p> I love tools, and use them every chance I get. I write new tools</a> to solve problems. Tools are what amplify our efforts. But tools in themselves don’t seem to be the answer. How can we ensure our meta-data is accurate and not missing things? How can we know our tools are not lying to us?</p> </p> 🚢 What is the #1 problem in product development? 2026-02-09T00:00:00+00:00 I’ll go first … a very common challenge is shipping on time and the resulting uncertainty to the business. We often add a 2-3x factor on any estimates, but sometimes even that is not enough.</p> Shipping (delivering value) is the bottom line — it generates revenue and sustains the business. Delivering new products on time and providing subsequent updates are critical. Without shipping, there is no value delivery. Often businesses collaborate with customers throughout the development process. When things are late, customers and partners lose confidence. Development team morale tanks. It’s a downward spiral.</p> This problem has an even more painful manifestation — finding out just before the BIG Release date that things are not coming together. As the saying goes, the last 10% takes 90% of the time. Surprises like this are devastating to everyone involved. Why do things fall apart near the “end” of a project? Why don’t these issues surface sooner? These are complex and perplexing problems, but worth exploring in the coming days. Thoughts on this are always welcome.</p> </p> 📦 More definitions of a release 2026-02-04T00:00:00+00:00 A release must be: reliable, useful, repeatable, and predictable</em>. This delivers outcomes of: quality, value, timeliness, and consistency</em>.</p> The second group is the outcome we want, the first group is what we focus on to get there.</p> </p> 🛠️ Do things that don't scale 2026-02-03T00:00:00+00:00 From Paul Graham’s popular essay Do Things That Don’t Scale</a>:</p> Over-engaging with early users is not just a permissible technique for getting growth rolling. For most successful startups it’s a necessary part of the feedback loop that makes the product good. Making a better mousetrap is not an atomic operation. Even if you start the way most successful startups have, by building something you yourself need, the first thing you build is never quite right. And except in domains with big penalties for making mistakes, it’s often better not to aim for perfection initially. In software, especially, it usually works best to get something in front of users as soon as it has a quantum of utility, and then see what they do with it. Perfectionism is often an excuse for procrastination, and in any case your initial model of users is always inaccurate, even if you’re one of them. 7</a></p> The feedback you get from engaging directly with your earliest users will be the best you ever get. When you’re so big you have to resort to focus groups, you’ll wish you could go over to your users’ homes and offices and watch them use your stuff like you did when there were only a handful of them.</p> </blockquote> I should mention one sort of initial tactic that usually doesn’t work: the Big Launch</strong>. I occasionally meet founders who seem to believe startups are projectiles rather than powered aircraft, and that they’ll make it big if and only if they’re launched with sufficient initial velocity. They want to launch simultaneously in 8 different publications, with embargoes. And on a Tuesday, of course, since they read somewhere that’s the optimum day to launch something.</p> It’s easy to see how little launches matter. Think of some successful startups. How many of their launches do you remember? All you need from a launch is some initial core of users. How well you’re doing a few months later will depend more on how happy you made those users than how many there were of them. 10</a></p> So why do founders think launches matter? A combination of solipsism and laziness. They think what they’re building is so great that everyone who hears about it will immediately sign up. Plus it would be so much less work if you could get users merely by broadcasting your existence, rather than recruiting them one at a time. But even if what you’re building really is great, getting users will always be a gradual process — partly because great things are usually also novel, but mainly because users have other things to think about.</p> Partnerships too usually don’t work. They don’t work for startups in general, but they especially don’t work as a way to get growth started. It’s a common mistake among inexperienced founders to believe that a partnership with a big company will be their big break. Six months later they’re all saying the same thing: that was way more work than we expected, and we ended up getting practically nothing out of it. 11</a></p> It’s not enough just to do something extraordinary initially. You have to make an extraordinary effort</em> initially. Any strategy that omits the effort — whether it’s expecting a big launch to get you users, or a big partner — is ipso facto suspect.</p> </blockquote> What Paul is talking about here is Release First</a> — the extremely tedious, manual effort required to get usable stuff in front of real users. There is no substitute.</p> </p> 🔮 Can we predict product development schedules? 2026-02-02T00:00:00+00:00 Estimating or predicting product development is difficult. Teams often end up adding a 2-3x padding to any estimate, because bitter experience has taught them not to depend on their own estimates.</p> There will always be a high degree of variability in product development, especially software development. Product development involves doing something new. Teams discover exactly what they are building along the way. No one can completely plan it up front.</p> However, we can do better. What is most damaging is not that a product is six months late, rather it is that we did not figure this out until shortly before the original anticipated release date.</p> Release First is a simple solution. By the time the planned release date comes around, you have released dozens if not hundreds of releases. Some releases may be documents, some may be hardware, and others software. But there have been many opportunities for others to use something and provide feedback. Additionally, you have had lots of practice releasing so there are no surprises. We may have learned that it will take an extra 6 months to finish the product, but we learned that early and the business has had time to adjust. Or we may have gained so much efficiency through automation, and reduced wasted effort to a level that we come out ahead of schedule (what we really want).</p> Release First eliminates surprises, and that is what matters in business.</p> </p> 🚀 How to make releases painless 2026-01-29T00:00:00+00:00 In the olden days, releases were a lot of work. The process involved manually building/testing, uploading to a distribution location, sending out notifications to interested parties, etc. Supporting multiple platforms meant even more work.</p> Today, there are better ways. With modern CI/CD workflows, pushing a tag to a repo triggers GitHub Actions to handle the rest.</p> As an example, in the BRun</a> project, there are several key pieces:</p> Tests</a> run on every PR commit to ensure nothing breaks.</li> Go makes it super easy</a> to build for other platforms. Anyone who has cross-compiled C/C++ knows the pain.</li> The amazing GoReleaser</a> tool takes care of building the release binaries and uploading them to GitHub.</li> A script</a> automatically extracts the Changelog for the release that gets added to the Release.</li> A GitHub action runs GoReleaser whenever the repo is tagged with ‘v’.</li> GitHub sends notifications to watchers of the project. (Consider watching the BRun project!)</li> </ul> The result is beautiful releases</a> for every platform (Windows, MacOS, Linux, ARM, x86, RISC-V) created whenever anything is improved, no matter how small. There is no friction to creating a release.</p> Part of making all this happen is choosing the right technology. When it comes to release and deployment, Go makes things really easy (compared to C++, Python, Ruby, Java, etc.). There are trade-offs so make sure release is factored into technology decisions. One way to ensure this happens is do it first</a>.</p> </p> 📖 When should you write the user manual? 2026-01-27T00:00:00+00:00 Most of the time, teams write the user manual at the end of a product development cycle right before the Big 1.0 Release</a>. It is one of those things that is not much fun, and teams subconsciously put it off while fighting bigger fires. But what would happen if we write the user manual first - even before the product spec? Some things that might happen:</p> We might think a little more from the user’s perspective during the development.</li> Our upfront thinking would improve through the process of writing.</li> Internal/beta releases would have documentation, allowing others to use them without a lot of hand-holding (saving developers time).</li> We would get feedback on the user manual early in the development process so it could be improved before release.</li> </ul> Good user documentation is part of a good release - it helps make the release useful</a>. With doc-driven development</a>, I typically start at the repository README.md</a> or other documentation close to the code, but why not start with the user manual and derive more from that. Start with the experience you want the user to have. Reverse the process.</p> </p> 🎬 Is a Demo the same as a Release? 2026-01-26T00:00:00+00:00 Some development teams have a weekly demo where they might show what they are working on to others in the company or the customer, which is a great idea as it encourages hitting milestones and early feedback. However, is the “demo” just another name for an “internal release?”</p> The “release” and “demo” mindsets have some differences. For a demo, quality or automation may not be a priority. Engineers may record a video or present the demo. Again, all of these are great ideas and should be done, but they are not a release.</p> The key thing about a release is that someone else, other than the developers, uses it. And the developers are confident enough to let other people use it. This difference seems small on the surface, but is huge underneath. A release comes with an expectation that drives automation and quality. The demo can become a crutch that allows teams to avoid the hard work of delivering something others can use. The mindset matters.</p> Again, the first internal release need not be feature complete and polished, but whatever slice of functionality the team delivers should be usable by others without a lot of props and hand-holding. Beware of proxies for delivering real value: demos, closing tickets, milestones, etc. While these are all useful tools, they are not an end in themselves. Measure value, not proxies.</p> </p> ⏱️ How do you find time for higher value tasks? 2026-01-23T00:00:00+00:00 Do you (and by extension your organization) find yourself mired in minutia and struggling to find time for higher value tasks? What is the solution?</p> You can hire more people to do the lower-value stuff (a perfectly valid approach). You can also own YOUR Platform</a>.</p> One of the goals of Release First</a>/Platform Thinking is to shift time from low-value tasks (tedious repetitive work) to high-value tasks (creative, innovative work).</p> </p> ⚖️ Reliability tradeoffs 2026-01-22T00:00:00+00:00 Recently, one of our servers running Arch Linux refused to boot after updates. The fix was to comment out a line in the /etc/default/grub</code> file. It took about 15 minutes to find and everything was done in a rescue terminal — no need to boot from a rescue image. This server has been running for several years, been updated hundreds of times, and this is the only time it would not boot.</p> Many dismiss Arch as a server OS because there is supposedly a higher likelihood of things like the above happening. But in practice, they rarely happen and are fairly quick to fix. So on one side of the equation:</p> Easy to administer OS, latest software, no OS upgrades, etc. This saves a ton of time.</li> </ul> On the other side:</p> Occasional effort to fix an issue like the above.</li> Downtime while doing the above.</li> </ul> For a high-profile eCommerce site, downtime would matter, but for a small business, 30 minutes of downtime makes no practical difference. The time saved running Arch greatly swamps (by an order of magnitude or two) the time spent fixing things that break. Admin time is many times more valuable than server up-time. Even spending half a day rebuilding the site from scratch (which is mostly automated) would not matter.</p> There is no one-size-fits-all. Just because Summit Federal Bank runs Crucible Linux Enterprise edition does not mean everyone else needs to. Always consider the benefit-cost ratio (BCR)</a>.</p> </p> ⚖️ Reliability, Cost, and Value 2026-01-19T00:00:00+00:00 Everything has a cost and value to the customer. How much value does the customer put on 99.999% reliability vs the value of other features they need? If one system out of a thousand fails every 3 years (0.033% failure rate per year), what does that cost compared to a set of features that allows the customer to reduce operating costs 10%, or improve their general yearly reliability by 1%?</p> Certainly, safety-critical systems where human lives are involved are impossible to quantify with simple equations. But when considering operational equipment involving machines, understanding the real concerns of the customer and delivering the most value becomes paramount. Reliability/security/performance/whatever is just one factor in that equation, especially when it costs them other things they need.</p> </p> 🔗 What defines reliability? 2026-01-16T00:00:00+00:00 Reliability is a common concern - this thing is critical - it needs to work!</p> Many things affect the reliability of a system:</p> Bugs in software we write</li> Bugs in 3rd party components (OSS, etc.)</li> Hardware failures</li> Edge condition failures (things we have not tested)</li> Tolerance stack-ups</li> Damage in the field</li> The list goes on …</em></li> </ul> Some common assumptions:</p> Hardware is more reliable than software.</li> Safety certified operating systems are more reliable than non-certified.</li> US-made components are more reliable than those from other countries.</li> </ul> Here’s the thing … for 99% of products, most of the above does not matter. The most likely source of bugs and reliability problems is the stuff we did</a></strong>. If SafeRTOS is 99.999% verified and Zephyr is only 99.9% verified, it does not make much difference if the software we wrote is only 90% verified. Likewise, if the tricky analog hardware circuit has not been tested in every edge condition and verified at every combination of component tolerances, it may not be as reliable as a simpler RTOS application running in an MCU.</p> Practical reliability is what matters, not theoretical reliability. What defines reliability in most systems is that the design can be verified. A common MCU and RTOS (like Zephyr) has already been extensively verified and tested to be reliable in most conditions by many different people. Writing a relatively simple application on top of a complex, but verified system makes the job easier. The difficult verification work gets pushed off on someone else. This same analysis also applies to concerns like performance and security. The system is only as good as the weakest link. Reliability does not always directly correlate to system complexity, but rather what percent can be proven correct.</p> There are no easy answers here, but don’t strain at a gnat and swallow a camel - latching onto theoretical reliability/performance/security/whatever, but ignoring the practical concerns.</p> </p> 🚀 Shipping faster - what are the stakes? 2026-01-14T00:00:00+00:00 What do project delays cost? The obvious thing is the loss of revenue. Time is money.</p> However, there are other less tangible consequences:</p> Customers lose interest.</li> Team loses motivation.</li> Supporting organizations (vendors/communities) may not take the team as seriously.</li> Automating and improving the process never happens as there is more and more pressure to ship and no time to do anything else.</li> </ul> Think of this as a flywheel, either speeding up or slowing down. The forces on this flywheel are imperceptible - a small push here, some drag there.</p> The simplest way to speed up the product development flywheel is to ship faster by focusing on quality releases</a>. Start by shipping quality internal releases</a>.</p> </p> 🔍 It is often more work to review something than to do it 2026-01-13T00:00:00+00:00 As an experienced person, I am sometimes called on to review something. This can be a frustrating, seemingly hopeless task. With one developer, I commented that he could write code faster than I could review it. While there are some issues you can spot from a cursory look, most issues are subtle in nature—otherwise the original designer would have probably caught them. Therefore, you pretty much have to go through the entire design exercise again (especially with hardware) to find issues. In many cases, it would have been faster to simply do it yourself. This is the quandary many experienced developers find themselves in when working with junior people. They are responsible (because they are the reviewer), but feel helpless to ensure quality.</p> This is also the quandary we find ourselves in with AI. AI agents can write code much faster than any human can practically review. Therefore, the thought that “with AI code, as long as we review it, we’re good” is delusional. We might pretend we are doing it, but deep inside we know we are not. AI coding has brought this into stark reality.</p> Too often, reviewers receive schematics or source code with little context. What are the goals? What is the spec? What trade-offs were considered? What calculations were done? What edge conditions were considered? What is the architecture? Details without the big-picture context are very difficult to work with. While I’ve not often been party to formal code reviews in a meeting context, I suspect these could work better than offline asynchronous reviews because you can ask the designer questions and perhaps get some insight into the potentially weak areas of the system.</p> There are ways we can help reviewers:</p> Clear documentation about the requirements located in the same repository as the design/code.</li> All assumptions, decisions, calculations, etc. are documented. With schematics, as much of this as possible should be included directly in the schematic. Use more pages if needed.</li> Architecture is clearly defined and documented.</li> All testing is documented.</li> All assumptions are documented.</li> Automated tests are created where possible.</li> Performance metrics and resource usage are documented and automatically tested if possible.</li> Any concerns, potential weak spots, or things we’re not sure of are clearly documented. Often the designer is the only one who has insight into this.</li> </ul> The point many miss here is that if we do the above, we’re mostly helping ourselves 6 months from now.</p> As systems become more complex, it seems the only practical path forward is automated testing. Chip designers figured this out a long time ago as the cost of errors is high with full-custom silicon. Software developers are pretty good at this with TDD (test-driven development). But, the industry needs to go one step further and embrace end-to-end testing (which includes both hardware and software). Perhaps the most important role of a reviewer going forward is to ensure adequate documentation exists and tests are in place. Anything else is futile and we’re just pretending.</p> </p> 🚀 How far do you take releases? 2026-01-12T00:00:00+00:00 Thus far, we have defined a release as something that is useful and high-quality</a>. Typically, we think of a release as something formal - v1.3.8 that is packaged, built, and announced. But we can push this further. What about pull requests? Is every pull request that is merged at the same standard? Or do we merge them and hope that something later in the process will magically transform the work into something better? “Later never comes” - especially in complex systems.</p> </p> Open-source projects have figured this out. The standard for getting a PR merged is incredibly high in most projects. Why? Because there is no business pressure to merge it, but rather the fear of having to maintain something that is substandard. The maintainers are looking out for their own selfish interests, but these interests happen to align with the realities of this kind of work. Fixing something after the fact is so much harder than doing it right in the first place. As a result, most OSS projects maintain fairly high quality at any point in their development. This is one reason Arch Linux works so well</a>.</p> While there is a place for brainstorming, prototyping, experimenting, etc., this will not get you very far developing a real product. The earlier you push the “release mentality” into the development process, the faster you’ll move. Maybe not in the initial steps, but for sure once you are a third of the way through the development cycle. Do you want to save time on the first third, or the last two thirds of the development cycle? Hint, the Law of Compounding is at work here …</p> </p> 🎯 Standards for internal vs external releases 2026-01-09T00:00:00+00:00 It is natural to think of internal releases as “low stakes” - if it crashes and burns, no big deal. It’s just an experiment, right? Near production release, we’ll toss it over the wall to QA/Manufacturing and find all the issues.</p> This is partly true, and partly false. While it is true a product should get better over the development cycle, it is helpful to apply the same standards for internal releases as external for the following reasons:</p> It forces an emphasis on quality early.</li> It results in automated testing.</li> There will be fewer surprises late in the development cycle.</li> Integration with other groups happens earlier (Quality, Sales, Manufacturing, Customers).</li> It is much easier to build prototypes when quality is high.</li> </ul> This does not mean the entire product needs to be perfect on day one - that is impossible, as the product is not done - just the work already completed, however small that may be. Is this tiny bit of work done to high standards? Is it testable, manufacturable, etc.? Is it useful</a> to someone?</p> Here’s the thing - it is much easier to apply high standards to small bits of work as you go, than retroactively fix it late in the game. It is not a waste of time, rather it is only way that works in developing modern systems.</p> If internal releases are not at the same standard as external releases, they are not releases, they are something else.</p> </p> 📦 What do we release? 2026-01-08T00:00:00+00:00 Everything!</strong></p> Documentation</li> Hardware</li> Software</li> Prototypes</li> Products</li> Lunch</li> </ul> Engineering is fundamentally the act of creating something new. And that something new needs to be useful to someone or something. And the way we deliver “useful” is the release.</a> If we’re not releasing, we’re not delivering.</p> We might be polishing, researching, thinking, perfecting, stalling, loafing, dreaming, but we’re not delivering.</p> The “release” mindset brings the focus to delivering value.</p> </p> 🤝 A release is what unifies Engineering and Business 2026-01-07T00:00:00+00:00 Some feedback on yesterday’s post (What is a release</a>):</p> What specifically flows into the two points? Value? Effort?</p> </blockquote> Perhaps a better way to phrase this is these are the two points of focus that unify the development team with the rest of the business organization. And the release is the container.</p> Documentation, specifications, innovation, new technology, etc. flow into the useful</strong> point.</p> Testing and automation flow into the confidence</strong> point.</p> The business value that flows out of the useful</strong> point is user value, profit, increasing sales, etc.</p> From the confidence</strong> point flows quality and responsiveness.</p> But these two points give us a simple way to focus and evaluate a release - is it any good? Is everything we are doing as engineers contributing to these points?</p> </p> 📦 What is a release? 2026-01-06T00:00:00+00:00 There are two aspects of a product release:</p> Something useful</strong> that others can experience.</li> Produced with a high level of confidence</strong>.</li> </ol> Everything flows into these two points.</p> </p> 🚀 Shipping faster leads to predictability 2026-01-05T00:00:00+00:00 Shipping faster does not only mean getting to v1.0 faster (or even on time), but also many internal releases at a good rate. This is the only way</strong> to bring predictability into the development process.</p> </p> 🔗 Platform thinking does not mean we don't use other platforms 2026-01-02T00:00:00+00:00 On the contrary, platform thinking readily embraces other tools and platforms. The difference is:</p> We add our own tooling, integration, and workflows into the mix.</li> We own the integration points so we can readily swap out/in 3rd party components as needed.</li> We are not overly dependent on someone else’s platform that we don’t have control over.</li> We value openness and flexibility.</li> </ol> Too often, the tendency is to lean toward all-encompassing platforms that we think will solve all of our problems. This rarely gets us to where we need to be to provide exceptional value.</p> </p> 🔍 AI coding requires better inputs and robust verification 2025-12-31T00:00:00+00:00 Some great feedback on the previous post (Impressive is not always effective)</a>:</p> “I agree. It would be interesting, however, to compare the progression of AI code generation with that of compilers - were there star assembly-code engineers saying that compilers were just a way to brute force the generation of assembly code?”</p> “I feel like there is a fundamental difference — perhaps that AI always lacks the determinism of a good compiler. And it can completely ignore the best of documentation without throwing an error code.”</p> </blockquote> Having lived through several of these transitions (Assembly -> C -> C++ -> Python -> Go -> Rust -> Zig -> AI …) there are always those who prefer to stick with the current technology thinking the new adds no value or is even detrimental.</p> AI is more just an example in the original article as it can make an impressive mess much quicker than any human. Humans can make messes as well (made plenty myself). So the point is not so much that AI is bad (or good), but rather question: “Are values in place?” And, is there a process/system/platform (whatever you want to call it) to ensure these values happen. AI can do good work with the proper constraints and workflows, as can humans. But neither does very good work without them.</p> That all said, as the above feedback suggested, AI is much less deterministic than any compiler (two consecutive runs do not produce the same result), thus needs clear guidance and robust verification to ensure the end result is quality. All this can be accomplished, but needs to be thought through.</p> While compile warnings, linters, code review, and unit tests help ensure compiled code is quality, AI is a different beast. If humans are too slow to write the code originally, they are for sure too slow to review it. It is like reviewing machine code from compilers - at some point you need to trust the compiler. The errors from AI generated code will likely be more architectural deficiencies or logical errors at a macro level. Bad architecture can bury you in technical debt, but it takes time - initially it works fine. Thus several things are critical:</p> Getting started with a good architecture. AI can help with this, but an experienced developer needs to have input.</li> Good, concise documentation. As opposed to machine generated code, humans can review this.</li> Automated end-to-end testing is critical. This is the only way to prove and maintain correctness.</li> </ul> With AI development, better inputs and more verification are critical. Trying to work in the middle of the AI coding process is futile and exhausting. The beginning and the end of the process need shored up. But this requires better workflows</a> and more discipline. While undisciplined traditional development is a nightmare, undisciplined AI development is a train wreck.</p> </p> ✨ Impressive is not always effective 2025-12-30T00:00:00+00:00 When AI generates something, the result is often “impressive.” We might marvel at how quickly lots of code is generated that mostly works. Generated documentation is extremely detailed and well formatted. AI-created diagrams are colorful, use emojis effectively and every last detail is included. However, “impressive” is often the antithesis of “effective” for one simple reason - simplicity is missing.</p> It is difficult to understand the architecture of the code because there is so much of it.</li> “Why” is nearly impossible to draw from the expansive details in the documentation. Who has time to dig it out?</li> A quick glance at a diagram reveals nothing about the big-picture without laboriously processing many details.</li> The complexity of machine-generated designs is often a brute-force approach that lacks style, elegance, and consistency.</li> </ul> Architecture, Why, Big-picture, and Elegance - these are still the foundations of great work. Learn to look past impressive things. Is the architecture clear? Are the reasons obvious? Is it easy to understand at a high-level what is going on? Is the design and implementation clean and simple? There are two paths:</p> “Impressive” is the route to compounding technical debt.</li> “Effective” is the only way to compounding improvements.</li> </ol> </p> 🥄 The beauty of human creativity 2025-12-29T00:00:00+00:00 In this age of machine-generated perfection, something is still deeply rewarding about grabbing a stick of wood and making something by hand. This past Christmas, with the help of kids and grandkids, we made a few mixing spoons for gifts. Starting from the same pattern but with varying levels of skill and technique, each spoon turned out a bit different. However, each one is special and beautiful because it has a human touch. And sometimes with variety and imperfection, a new pattern emerges that actually works better. This is one source of innovation.</p> In an age of AI-generated code, preserving the human touch remains important. Perhaps it is the well-thought-out documentation that explains exactly how this feature will help the user. Or the hand-coded API that is a pleasure to use. Explorations in new data structures and algorithms coded by hand sometimes result in amazing discoveries because thinking accompanies the creative process. Machines (including AI) can mostly only do what has been done before. But the real value lies in the new connections, the newly discovered truths</a>, the well-understood needs of the user, and the exceptional quality. These still require human impetus.</p> </p> 📖 The definition of the word 'Platform' 2025-12-24T00:00:00+00:00 The definition of the word “platform” includes things like the following:</p> Acts as foundational for other products or services, not the final end product itself.</li> Provides a stable base.</li> Typically supports multiple, distinct applications or use cases.</li> A formal declaration of the principles of a group.</li> </ul> What are the aspects of a physical platform?</p> Provide a supporting surface: Examples include train station platforms, stages, loading docks, scaffolds, and cargo pallets, all of which physically support people or objects at a useful height or position.</li> Enable safe and efficient interaction: Platforms make boarding, loading, performing, or working safer and more efficient by aligning heights, improving visibility, and organizing space.</li> </ul> Stable, foundational, supportive, efficient, visibility, safety, organization … does this provide any inspiration for YOUR Platform?</p> </p> 🔄 Platform Thinking is a habit 2025-12-23T00:00:00+00:00 Platform Thinking in the context of product development has many parallels to effective habits in personal life. With personal exercise, healthy eating, getting enough sleep, and reading good books - these are difficult to start, but easy once started, and rewarding once finished. These things work best when they are habits - a regular behavior pattern that a person tends to do automatically, often without much thought.</p> Documentation, testing, and automation in a product development context are similar. It won’t just happen. A framework is required to nudge us into a pattern that turns into an effective habit. In many ways, Platform Thinking is just Atomic Habits</a> for product development teams.</p> </p> 🎯 Who is Platform Thinking for? 2025-12-22T00:00:00+00:00 Platform Thinking is for anyone who wants to take the friction out of creating, publishing, testing, integrating, and deploying products.</p> The result? Value. Quality. Timeliness.</p> </p> ⏳ The long-term compression timeline 2025-12-18T00:00:00+00:00 Below are several timelines illustrating long-term compression</a> comparing a product development timeline:</p> 20 years ago</li> Today</li> Today with Platform Thinking</li> </ol> The big effect of long-term compression is that negative consequences can hit much earlier, delaying initial shipment (the middle timeline).</p> While it is very difficult to compress a product development timeline to initial release, with Platform Thinking, v1.0 can still arrive in a year as in the olden days. The big difference is subsequent releases ship much quicker than in any previous scenario.</p> So while the complexity of modern technology presents new challenges and requires a different approach, leveraging technology to build a platform can produce even better results than any previous scenario.</p> Long-term compression can have both compounding negative or compounding positive effects. Both the negative costs and the positive benefits come back much quicker. Which comes back depends on YOUR Platform.</p> </p> 🗜️ Long-term compression 2025-12-17T00:00:00+00:00 </p> One reality of developing products with modern technology is the compression of the short and long-term. Today, the consequences of actions come back on the timeframe of months instead of years, as it might have been ten or twenty years ago. No longer does it work to take shortcuts with the hope of shipping v1.0, making some money, and cleaning things up later. The product may never even see v1.0.</p> This is why YOUR Platform is so important. Platform thinking enables you to reach the v1.0 release in a timely manner, and efficiently maintain and improve beyond that.</p> 🚀 Platforms are not just for developers 2025-12-16T00:00:00+00:00 </p> This is an interesting post</a> from a Marketing person creating website pages:</p> I built 19 website pages in a single day while my team was out for US thanksgiving. Not mockups. Not placeholders. Actual, shippable pages: a full partner ecosystem landing page plus individual pages for every major partner we’re building with right now. It felt surreal. Mostly because a year+ ago, there’s no universe in which I would’ve believed I could do that</strong>. And it definitely wouldn’t have happened without the engineers around me who’ve taught me how to think more like a builder—patiently walking me through GitHub (😒), tools, workflows, and a way of working that used to feel miles outside my world.</p> </blockquote> I then asked how long it took before she stopped hating Git?</p> I mean it took about 5 pull requests before I finally managed to create one without an error 🥵… it’s been two weeks and I finally feel like the lightbulbs are going off and I’m understanding the flow (pulling things from main, branches, commits, PRs, basic commands I have on a sticky note lol, etc).</p> </blockquote> This is a great example of someone who took ownership of their website creation, editing, and publishing workflow. Using local files</a> and text based formats</a> unleashed the power of AI resulting in the productivity described above. Using AI, anyone can now write scripts</a>, create web pages, and code to some level. Text based formats and command-line tools are now within reach for anyone willing to learn a bit. All it takes is a willingness to break out of the cozy cocoon of GUI based tools, learn how to use an editor and terminal, and invest a little effort in creating a workflow.</p> 🔄 Breaking the AI slop cycle 2025-12-15T00:00:00+00:00 </p> “AI Slop” is a common expression describing a familiar phenomenon. It refers to sloppy code (or other assets) generated by AI that mostly works, lacks proper review, and may be poorly architected or implemented. The primary characteristic of AI Slop is that it is hard to read, understand, and maintain.</p> Like anything else, fixing problems early in the cycle works best—not generating slop in the first place instead of trying to clean it up later. Even AI tools, as powerful as they are, have trouble cleaning up slop. Once the slop is cast in code, AI (like a human) is very reluctant to throw away all the hard work and start over.</p> So how does one break the cycle? The best method I’ve found is a lightweight formal documentation and planning workflow</a>. The documentation ensures good thinking about what the problem is, what needs to be built, and why. The planning stage helps ensure the architecture is good. It is much easier to think about and rework a plan than code.</p> The documentation/planning steps also clearly delineate the responsibility of humans and AI. While AI can help explore options, it still does not have the entire context that humans have when it comes to clearly defining the problem.</p> When slop appears, this means the up-front thinking was not very good. One of the most useful skills when working with AI is recognizing when the slop zone is approaching, backing up, and getting more formal.</p> 🤔 Isn't this just agile? 2025-12-12T00:00:00+00:00 </p> A reader responded to the last post with a comment:</p> Isn’t “agile” a part of product thinking too? That doesn’t seem linear?</p> </blockquote> This is a good question. Agile covers a lot of ground. The Agile Manifesto states:</p> We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:</p> Individuals and interactions over processes and tools</li> Working software over comprehensive documentation</li> Customer collaboration over contract negotiation</li> Responding to change over following a plan</li> </ul> That is, while there is value in the items on the right, we value the items on the left more.</p> </blockquote> Typically agile focuses on fast iterations and quick feedback, which is definitely part of platform thinking. Most software methodologies focus on creating software. Platform thinking expands to include more of product development including mechanical, hardware, and integration. Much of development these days is integration of complex systems, including vast amounts of open-source software. As system complexity grows, teams need ways to deal with this complexity. Individual interactions are great and very important, but they don’t scale with complexity. And perhaps the real goal of YOUR Platform is to free people from tedious repetitious work, so individuals will have more time to interact in creative ways.</p> 🏗️ Platform thinking vs 📦 Product thinking 2025-12-10T00:00:00+00:00 </p> Platform vs. product thinking is a difference in perspective. Platform thinking does not ignore the product, but rather approaches it from a slightly different angle.</p> 🏆 What is the ultimate platform success? 2025-12-09T00:00:00+00:00 </p> The ultimate platform success is when an internal platform turns into a sellable product/service, or an open source project.</p> Perhaps some companies succeed by starting out building a platform as a product, but the greatest platform successes</a> started as internal platforms:</p> AWS</li> Kafka</li> Kubernetes</li> Slack</li> Shopify</li> </ul> Why does this work? Perhaps because developers building internal platforms have the “platform mindset.” Starting out building a platform to sell means building with the “product” mindset instead of the “platform” mindset.</p> Compare AWS, Azure, and GCP. Amazon and Microsoft use these platforms extensively internally. Google does not</a>. Google likely has valid reasons for this as GCP is not their primary product (compared to search, YouTube, etc.). But the results speak for themselves:</p> Feature</th> AWS (Amazon)</strong></th> Microsoft Azure</strong></th> Google Cloud (GCP)</strong></th></tr></thead> Market Share</strong></td> ~29-31%</strong></td> ~20%</strong></td> ~13%</strong></td></tr> Revenue Run Rate</strong></td> ~$132 Billion</strong></td> ~$123 Billion</strong></td> ~$61 Billion</strong></td></tr> Primary Internal Use</strong></td> High.</strong> Amazon.com is heavily migrated to AWS, including retail databases and Prime Video.</td> High.</strong> LinkedIn, Office 365, and GitHub are actively migrating or fully migrated to Azure.</td> Mixed.</strong> Core search/ads/YouTube/Gmail/Docs runs on internal-only PROD infrastructure.</td></tr> </tbody></table> 🔀 The division between group and personal workflow 2025-12-05T00:00:00+00:00 </p> When thinking about developer workflows, tools, and resources, dividing them into two categories proves useful: personal</strong> and group</strong>. A group resource is something that multiple people use (or should use): SDKs, libraries, linters, formatters, AI commands/context, etc. Personal tools are things that others do not use: pencil, computer, phone, editor, notes system, etc.</p> Why is this so important? Why not just make it simple and force everyone to use the same computer, editor, notes system, chair, desk, etc.?</p> Humans are users and creators of tools—naturally inclined to make, modify, and optimize tools for specific needs. Innovation happens where there is freedom. Many improvements start from a single person scratching their own itch. Over time it becomes apparent that a tool or improvement will benefit the group, so it migrates to a container or some other shared resource and is integrated into the group workflow.</p> As previously mentioned, any shared resource should be containerized</a> when possible. Containers allow the personal computer to be the dividing line between personal and group workflows.</p> No two humans do exactly the same thing. No two humans think and work in exactly the same way. While there are many benefits to having group workflows, the personal workflows must also be nourished, as these are a primary source of innovation. The key is to have a culture where the migration from personal workflow to group workflow is encouraged, supported, and rewarded.</p> To scale, an organization must have:</p> A clear division between personal and group workflows.</li> An efficient way to reuse personal workflows as group workflows where appropriate.</li> </ul> 📦 CDPD - Container Driven Product Development 2025-12-04T00:00:00+00:00 </p> When updating the Simple IoT documentation</a>, the question arose - how to install mdbook</code> and a few plugins? Of course pacman</code> has part of what is needed. In the past cargo</code> has been used, but that is slow and not a good fit for shared tools. Then it became clear:</p> Any shared tools used by the team (build, linting, formatting, CI/CD) etc. should be put in containers up front (before development starts)</strong></p> .. and use them for development builds as well as CI/CD. (this container</a> is the result)</p> Why?</p> It is pretty easy these days with Claude Code.</li> Gitea, GitHub, and likely other Git hosting platforms now provide convenient “package” registries</a>. You no longer need to go through the hassle of using the Docker registry.</li> It is easy to onboard new devs - no need to spend time installing SDKs and a host of tools.</li> Versions are locked down, eliminating the possibility of different people using different versions.</li> They get exercised a lot, so quality is high.</li> It is much easier to move into CI/CD.</li> A Dockerfile is easier to maintain than a long document describing how to manually install all the tools needed.</li> NPM and Python packages are a pain to install locally, so why not just eliminate that pain?</li> It is easier to get everyone on the team using the entire toolset (linting, formatting, etc.).</li> No longer fighting differences between developers’ machines. If one developer wants to run Arch and another Ubuntu, no problem.</li> </ul> Developers are still free to use their favorite editor, Git client, and other local tools.</p> This is another case</a> where a little time spent up front is quickly regained, then compounded by not continually redoing stuff, and fighting problems that can be solved once.</p> 📝 Tips for working with Markdown 2025-11-26T00:00:00+00:00 </p> Markdown is now a nearly universal format for documentation in many workflows. A few tips for being effective with Markdown:</p> Use a standard editor most of the time (VS Code, Vim, Helix, etc.). This is faster than graphical “Markdown” editors for most things. Additionally, this builds familiarity with Markdown formatting codes, which aids in creating nicely formatted comments in Jira, GitHub, Trello, etc.</p> Use Typora</a> to create more complex things like tables and images. This tool is well worth the few $ it costs.</p> Put images in an /assets</code> sub-directory and name image files descriptively (Typora makes this easy).</p> Use prettier</a> to format the file and wrap long lines. Add a .prettierrc</code> to every repo with the following contents: proseWrap: always</code></p> Use Harper</a> (language server</a> in your editor is best) to check for grammar and formatting issues. One unexpected side effect of this is that it encourages marking technical words that don’t have standard spelling as code, rather than adding them all to your dictionary.</p> Add dictionary words to a .harper-dictionary.txt</code> in the repo that contains the markdown file. This way others can benefit from a common dictionary. Again, this is easy with a harper-ls</code> quick-action.</p> Add a TOC if the file is long (the harper-ls</code></a> quick-action makes this easy).</p> Use the Marksman</a> language server. This makes navigating and refactoring in your editor easy.</p> Use Markdown Here</a> or Markdown Here Revival</a> in your browser or email client to write better emails.</p> While Markdown has its formatting limitations, with a little tooling it is much easier and faster to work with than anything else.</p> 🤔 Why > How > What 2025-11-25T00:00:00+00:00 Why?</strong> Is this the right thing to build?</li> How?</strong> Can it be done efficiently, repeatedly?</li> What?</strong> The deliverable.</li> </ul> These are listed in order of importance, which also corresponds to the level of difficulty.</p> The trouble with “How” and “Why” is these are less visible, which contributes to their difficulty as it takes discipline to prioritize and do them.</p> </p> 🥉 Good, fast, cheap - choose three 2025-11-24T00:00:00+00:00 A familiar phrase states:</p> “Good, fast, cheap. Choose two.”</p> </blockquote> This is similar to the development triangle</a>.</p> Which states that everything is a tradeoff.</p> However, improving all three becomes possible when doing things differently. That is the fundamental purpose of YOUR platform - to improve “how” teams build things. To develop a synergistic relationship between value, timeliness, and quality,</a> without compromise.</p> ⏳ The "long-term" may be closer than you think... 2025-11-21T00:00:00+00:00 A common objection to “Platform Thinking” (proactively build your platform from the start</a>) is: there isn’t time right now; this can be done after the 1.0 release</a> when there is ample time and resources.</p> The “long-term” can be defined as when the consequences of actions come back - when the seeds sown are reaped.</p> When building complex systems with modern technology, the gap between the short- and long-term is months instead of years</strong>. There is a compression of the short- and long-term. Neglecting the platform may have worked a decade or two ago, but with the complexity of even basic embedded systems today (Zephyr, Embedded Linux, connectivity, data, etc.), the “long-term” is much more imminent. The pain from shortcuts emerges in months instead of years. The slope of the technical debt curve is much steeper.</p> The reality with complex systems today is there is no short- and long-term - they have morphed into one and the same. Without building for the long-term now, there will not be even a short-term, let alone a long-term.</p> 📝 DDPD - 🧪 TDPD - 🚀 RDPD 2025-11-20T00:00:00+00:00 We have been discussing document, test, and release-driven development (doing these things first before implementing features).</p> This ensures the product:</p> Meets the customer’s needs (DDPD</a>).</li> Is quality (TDPD</a>).</li> Can be modified any time and released quickly (RDPD</a>).</li> </ul> Without these, teams can quickly become buried in mundane repetitive work, bugs, and technical debt.</p> But perhaps the less obvious fact is these things often go undone if not prioritized from the start. Why? Because once a feature is done, there is pressure to move on to the next one. There is no time for “cleaning things up.” To ensure these things happen, they must be done first.</p> (2025-12-05 update: And we now also have CDPD</a>.)</p> 📝 DDPD - Document Driven Product Development 2025-11-17T00:00:00+00:00 </p> Building on TDPD</a> and RDPD</a>, Document-driven product development (DDPD) offers another perspective.</p> Ideas:</strong></p> Document the feature from the user’s perspective before implementing anything.</li> Keep the documentation close to the code (same repo, etc.).</li> Use a doc-driven</a> AI plugin when applicable.</li> </ul> Benefits:</strong></p> Better thought-through features.</li> Non-technical people can review before implementation.</li> Higher likelihood of implementing what users actually need.</li> Documentation becomes a pleasant/creative/thinking task, rather than a boring requirement after implementation.</li> </ul> 🚀 RDPD - Release Driven Product Development 2025-11-13T00:00:00+00:00 </p> Yesterday we talked about Test-Driven Product Development</a>. Release-driven product development is another perspective.</p> Build the release/deployment mechanism before implementing the design.</li> Automate as much of the release as possible.</li> This includes all assets needed to build/deploy the product.</li> </ul> While this idea is somewhat common in cloud software (continuous delivery), embedded, hardware, and mechanical release mechanisms are usually lacking.</p> The benefits:</p> More prototypes will be built (because it is easy).</li> More testing will occur early (because prototypes are available).</li> Lots will be learned (happens when people are using prototypes).</li> Releasing to manufacturing becomes easy (because it has already been done many times).</li> </ul> Implementing release mechanisms first is a strategic approach, because without doing it first, it likely won’t get done. Once something is “finished,” there is pressure (from above) to move on to the next development.</p> 🧪 TDPD - Test Driven Product Development 2025-11-12T00:00:00+00:00 </p> We’re familiar with TDD (test-driven development) in the software world. Basically the idea is to create unit tests before implementing a feature.</p> What if we extend this to systems, hardware, and even mechanical?</p> Write end-to-end system tests before developing a feature.</li> Design/build test fixtures before designing the PCB. PCBs are cheap and fast</a> to build now - so build an additional one for testing.</li> Create automated checks on 3D CAD models for fit/clearance, etc. Does the new PCB fit in the product?</li> Do you have additional ideas?</li> </ul> And then automate all these so they run continually. This takes the tedium out of testing and helps ensure every development change is a BIG Release</a>. Testing first also leads to better design.</p> 🍄 Mushroom Management 2025-11-07T00:00:00+00:00 I recall a quote from the book The Soul of a New Machine</em> that describes “Mushroom Management” and goes something like this:</p> “Keep them in the dark, feed them sh, and watch them grow.”</p> </blockquote> (This book chronicles the experiences of a computer engineering team racing to build the next-generation computer at a blistering pace under tremendous pressure at Data General in the late 1970s.)</p> Contrast this with Jensen Huang</a>:</p> “They never hear me say something to them that is only for them to know. There is not one piece of information that I secretly tell the staff that I don’t tell the rest of the company. In that way, our company was designed for agility, for information to flow as quickly as possible, for people to be empowered by what they are able to do, not what they know</strong>. That’s the architecture of our company.”</p> </blockquote> Where is Data General today? Where is Nvidia? Perhaps this is comparing apples and oranges though, because as system complexity grows, Huang’s approach is even more necessary. Systems in the 2020s are many times more complex than those of the 1970s.</p> 🚀 The BIG Release vs. 🌊 a Continuous flow of Value 2025-11-05T00:00:00+00:00 How do we look at a product release? As a big one-time event, where we spend heroic efforts to pull it over the finish line? Or as a continuous flow of high-quality value into the product (even before the customer ever sees it) that is maintained with every change?</p> This is a mindset difference that leads to two different approaches:</p> The BIG v1.0 Release:</strong></p> Quality is not a priority during development, and gets hammered in with extreme effort right before the BIG Release.</li> Tools that make releasing a product easy are not built, rather everything is done manually before the BIG Release.</li> Because there is so much work to do for the BIG Release, much of it is not anticipated so schedules slip.</li> Issues surface late in the process, which causes further delays.</li> Stakes are high, pressure mounts, and people become stressed.</li> </ul> Continuous flow of Value:</strong></p> Every pull request (PR) and internal release is treated like an important customer release.</li> Tools automatically check quality, eliminating manual verification.</li> Releases are automated to enable frequent deployments without mistakes.</li> As a result, the product is always ready to release - internally or to customers.</li> Releases happen often (full manufacturing package) because it is easy.</li> Test fixtures are already done because they are created to test each feature when the feature was developed.</li> Prototypes (which are just scaled back manufacturing) are easy to build and test because a manufacturing release process already exists.</li> More prototypes get built and more gets learned from actually using the product.</li> Issues and manufacturing problems surface early in the design cycle.</li> Beta releases can be sent to early customers for feedback without hesitation because the finished features are high quality.</li> Production release is not a big engineering event, but rather a point where people start paying for the product.</li> Schedules are predictable because there is no extra work for production release, but rather just a focus on the features needed for that incremental release.</li> Stress is low, people are happy.</li> </ul> The second scenario is desirable, but what is perhaps not so obvious is the mindset shift required to get there - a relentless focus on quality throughout the development process, automating anything done repeatedly, and thorough testing early in the process. Every internal release, every PR, every day is a BIG Release.</p> 🔀 The Power of Git workflows 2025-11-04T00:00:00+00:00 Git + Git hosting platforms</a> enable both</strong> local and cloud-based workflows.</p> Local:</strong></p> Lightning fast access to files - leverage the power of your local computer</li> Use powerful native tools (editor, e-cad, m-cad, AI agents, command-line tools, scripts, etc.)</li> Test on local hardware</li> Work offline</li> Organize/document changes</li> </ul> Cloud:</strong></p> Collaboration</li> Distributed development</li> CI/CD</li> Secure storage</li> Immutability</li> </ul> These approaches are not mutually exclusive - both can be leveraged simultaneously. Software developers take this for granted, but other disciplines can also benefit from better file management tools. It is beneficial to consider how both cloud and local domains can be utilized to their fullest potential. In the end, design assets (data) need to be created, documented, managed, verified, tested, deployed, secured, and easily synchronized</strong> between both domains.</p> ✍️ Why writing is important 2025-11-03T00:00:00+00:00 Some people are wildly successful without writing. So, if the goal is only to make a lot of money or have a lot of power, this advice may not apply.</p> However, for those who want to:</p> Think better</li> Scale ideas beyond direct personal contact</li> Capture and preserve ideas</li> Grow by building new ideas on old ones</li> Have influence beyond force and charisma</li> Effectively manage up</a></li> Not have to redo things over and over</li> </ul> Then capturing your ideas in a permanent, organized, discoverable place is essential.</p> 🔄 Three levels of workflow 2025-10-31T00:00:00+00:00 There are three levels of workflow:</p> Personal (what you do to get things done)</li> Department/Discipline (How multiple people who do the same thing get things done – Example: PCB design)</li> Cross-functional (How people who do different jobs interact - Example: Engineering, Support, Manufacturing, Sales, and Marketing)</li> </ol> Effective workflow is increasingly difficult to implement as you go from 1 to 3, as perspectives, needs, and comfort with various tools differ more. But cross-functional workflows are also where the greatest value</strong> lies.</p> 🤝 The benefits of sharing 2025-10-30T00:00:00+00:00 I wrote my first Claude plugin</a> that packages up some commands that I use in about every project I work on. This plugin allows me to have these commands globally so I don’t have to keep copying them from project to project.</p> Although the commands originally worked pretty well, the process of releasing them publicly made them a lot better. A new implement-tests</code> command was added. And the commands themselves were cleaned up to be clearer, which will likely help Claude do a better job.</p> I am continually amazed by this process. Even if no one else uses or contributes to this project, I have a better tool for myself, because I shared it.</p> 🛠️ The new Artisan Age 2025-10-29T00:00:00+00:00 We live in a world of powerful tools. Milwaukee battery-powered hand tools, high-end PCB design suites, cloud platforms, open-source tools, etc. And while these tools get us a good way toward the goal, they do not do everything. Additionally, if all you ever use is the mainstream tools, what differentiates you from anyone else? What prevents others from doing exactly what you are doing - at a lower cost?</p> In a previous era, artisan woodworkers built many of their own tools (fixtures, jigs, squares, planes, storage systems, etc.). As a result, they could highly optimize for their needs and projects. Tools were not very complex, so it was possible to make them. Today, things are much more complex. The things we can buy, or in the case of Open Source use for free, are endless. Many of them are very useful, but not always exactly what is needed. There is still friction, waste, manual steps, things that don’t work together, integration pain points, etc.</p> In the past, it was very time-consuming to solve integration and automation problems in complex systems. However, today we are entering a new era where we can again build tools for complex systems - with AI. AI is excellent at scripting, automation, testing, configuration, using APIs, generating fixtures, processing data, etc. Things that would have taken days to implement and figure out, we can now do in a few minutes. In many ways, we are entering a new artisan age where we can once again craft custom tools tailored to our specific needs and workflows. The biggest win with AI is not building your products, but rather using it to build a custom platform to build your products.</p> Human Friendly Industrial Device IDs 2025-10-28T00:00:00+00:00 Do you deploy 10’s to 1000’s of devices and need to quickly identify them in the field for configuration, maintenance, or troubleshooting? This article explores how this can be made much easier by using device IDs that are easy for humans to process.</p> </figure> In a previous article (A Survey of device ID mechanisms)</a>, we explored some of the commonly used options (MAC addresses, UUID, 2D barcodes, etc.). However, most of these are optimized for consumer devices and scenarios where machines process the IDs.</p> Why are human-friendly IDs important? Consider a few scenarios:</p> 8 identical temperature sensors are located at various places on a piece of equipment, and we need to identify the location of each one.</li> 3 IO modules are attached to a gateway, and we need to select which module a sensor is connected to when creating a virtual component in a dashboard.</li> One of the above 3 IO modules in a crowded control panel has failed, and we need to replace it. How do we identify it?</li> A user calls with a technical support problem on a piece of equipment, and we want to quickly look up the details on the hardware they have (model, version, etc).</li> We need to list and verify the device serial numbers on a packing list for a shipment to a customer.</li> Devices are provisioned and configured in the shop for several sites, and during installation, we need to make sure we install the right device in the right location.</li> A device’s mDNS name defaults to the device’s serial number. Once a device is attached to a network, the user needs to access the Web UI using the address: http://<device id>.local</code>. It is helpful if the ID is simple enough for a user to read from a label and easily type into a browser.</li> </ul> In the real world, especially the industrial world, there are many cases where humans need to interact with equipment beyond the initial provisioning.</p> </figure> The problem with (only) QR Codes</h2> QR Codes are increasingly being used to identify devices. We buy a new device, scan the QR code with an app on our phone, and the device gets added to the system – pretty slick! QR Codes are optimized for provisioning, with data flowing in one direction – from the QR code to the system. However, using the QR code to identify a device that is already deployed in the field is more challenging. We need to be able to look at multiple devices in the field and quickly identify them. Also, consider the scenario where you are on the phone supporting a customer – large QR codes are not transmitted over the phone very well. QR codes alone do not solve every problem.</p> </figure> What works well in the consumer space is not always appropriate for the industrial space. While there are many consumer technologies we can leverage, what Google, Apple, and Amazon are doing is not automatically the best solution for industrial applications. It may be possible to identify a device using a QR code using a complex standard like Matter, but do we have the development resources to make this happen? Do we need the complex security and privacy features in an industrial context? Is there a simpler/better solution?</p> What makes an ID easy for humans to process?</h2> Many MFA (multi-factor authentication), one-time passwords, and authenticator apps use codes that are composed of 6 digits split into two chunks:</p> </figure> From experience, we know these numbers are easy to remember and type into a form. Even on our phones, it is often more work to copy and paste than simply look and remember. Why is this? Human short-term memory can typically hold about 7 items at once (plus or minus 2). Chunking these numbers further reduces cognitive load for users. For example, “283-178” is easier to remember than “283178”.</p> Human-friendly IDs (HFID)</h2> If we start with the format used by authorization codes (XXX-XXX</code>), which is six digits, this gives us one million possibilities (10^6). That is a lot of devices, and may be adequate for some applications.</p> If we expand this ID to also include capital letters (example: N8H-9MJ</code>), then possible combinations are over 2 billion (36^6).</p> If we eliminate letters and numbers that are easily confused (0</code>, O</code>, 1</code>, I</code>, 2</code>, and Z</code>), then reducing the base from 36 to 30 characters (30^6) gives us ~729 million combinations. Randomizing the characters in the ID also helps ensure that the IDs are visually unique and not easily confused with IDs on neighboring devices. We might also want to require a letter and a number in each 3-character chunk to make the ID easily recognizable as an HFID, which would reduce the number of combinations to ~340 million.</p> 340 million is far more than most organizations in the industrial space will ever produce. If there is ever a need for more IDs, then the number of characters could be expanded with a minor loss in usability (example: PU89-V6M2), which gives us 656 billion combinations (30^8).</p> Managing Device IDs (hfid.org)</h2> This scheme does require a central authority to create these IDs – ideally, an API that various manufacturing processes (including subcontractors) can interface with and pull new IDs in real-time. The real value occurs when multiple companies can use the same ID registry, and thus devices from different manufacturers can interoperate. For this, it would be beneficial if there were a vendor-neutral organization that generated these IDs on demand. We created a prototype of a system to do this.</p> 🌐 https://app.hfid.org</strong></a></p> Once your account is approved, you can generate new HFIDs via the website or API</a>. Check it out …</p> Why not both?</h2> Why not both an HFID and a small QR code containing the HFID on your device? Then it is easy for both humans and machines to process the ID.</p> </figure> Summary</h2> Human-friendly device IDs are a way to uniquely identify a large number of devices, yet make it easy for humans to quickly and accurately identify devices in the field. Similar to Internal Part Numbers designed for humans</a>, systems designed to work well with humans are key to reducing mistakes and improving efficiency.</p> Even if you are using devices/sensors/equipment manufactured by others that already have an ID, adding an HFID to any device you deploy allows you to identify it with less effort.</p> The human eye and brain are remarkable instruments. Implementing IDs that are easy for humans to process has countless benefits as you manufacture, track, deploy, maintain, and service your products, devices, and equipment. Quickly identifying a device is the first step to doing anything with it.</p> Don’t just use an ID, use an HFID.</p> 🏗️ The difference between Product and Platform 2025-10-27T00:00:00+00:00 Product is what</strong> you build.</p> Platform is how</strong> you build.</p> Most people build the first and outsource the second. After all, the product has the IP—the secret sauce—the valuable part.</p> However, we observe in successful companies</a> a focus on their internal platform. They build their own tools and workflows optimized for them. How you build is where the rubber hits the road. If you don’t master the “how,” getting the “what” done is frustratingly difficult. YOUR Platform is the engine that keeps product development running smoothly and allows you to keep ahead of the game. It does not matter so much if someone steals your ideas because you are already implementing the next ones.</p> It is relatively easy to copy a physical product. It is nearly impossible to copy the internal process, culture, workflows, and tools—even if they are freely shared. Why? Because so few companies have the discipline and will to own their platform</a>.</p> 🎯 A single source of truth 2025-10-23T00:00:00+00:00 An intense discussion about part/version numbers</a> driving application logic surfaced an idea to create a simplified version of the product metadata that simplified the programming logic. However, this rarely works out long-term, because several things happen:</p> The two sets of data must be related and kept in sync.</li> This approach ends up replicating the first set of data with the second.</li> </ol> Fix the problems with the first set of data, rather than trying to band-aid it with additional data. Data can be derived by machine, but it should never be created in parallel. There needs to be a single source of truth. Otherwise mistakes will happen. More mistakes than just dealing with the first set of data, no matter how painful.</p> Manually duplicating data is symptomatic of organizational, not technical problems. Duplicated data often indicates that organizational silos exist caused by a lack of:</p> transparency</li> collaboration</li> cross functional teams</li> common standards</li> common workflows</li> common tools</li> </ul> Don’t make excuses for these, fix them.</p> 📈 Scale vs. Value 2025-10-17T00:00:00+00:00 As scale increases (# of people, # of technologies, # of products, # of teams, life cycle of product, experience, amount of knowledge, etc.), what does the value produced look like?</p> Metcalfe’s law</a> states that the financial value or influence of a telecommunications network is proportional to the square of the number of connected users of the system.</p> </blockquote> In communication and economic systems, we often find Metcalfe’s law to be true. However, personally and organizationally, the results often plateau as projects/systems/organizations grow.</p> Which of the below graphs describes you personally or organizationally? How do we get from average -> great?</p> </p> </p> </p> 💻 Local First Dev 2025-10-14T00:00:00+00:00 In the last couple decades, we’ve observed the success of SaaS web applications (Google Docs, Salesforce, Basecamp, Trello, etc.). These applications have unleashed collaboration using the web platform. Before this, users might have emailed documents around (many still do this) or shared access to a file-server or used a terminal server to access common infrastructure. However, nothing can match the web for low friction/cost access.</p> One of the downsides of SaaS is that users need to re-create the context of whatever they are doing in the web application (upload files, copy images, etc.). For many applications, the benefits to collaboration greatly outweighed the cost of re-creating context. For things like issue tracking, SaaS applications work well.</p> However, for more complex workflows (like CI/CD/PDM/PLM), the drawbacks of SaaS become more apparent because they require much more context. Some products, like Onshape</a>, solve this problem by moving everything to the cloud. This works great for pure mechanical products, but what about electrical and software? Should these all operate in their own silos?</p> A more compelling vision is a mixture of SaaS and local tooling. Examples include:</p> Claude Code</a></strong>: while using AI in web sites is very useful, running AI locally in your terminal is a huge step-level improvement for actually doing work (not just answering questions).</p> Model Context Protocol (MCP)</a></strong>: allows AI to connect to local applications and data.</p> Git hosting apps</a></strong>: Gitea/Github/Gitlab have demonstrated amazing utility for software developers by mixing local tooling (Git, Actions, etc.) with an accessible cloud web application.</p> Pushing data to the edge is how to scale, and files are the obvious way to do this. While this has always been true</a>, the rise of AI is making this fact crystal clear.</p> 💬 The importance of effective communication 2025-10-13T00:00:00+00:00 Recently</a>, effective communication has been discussed. But let’s step back – why is communication so important? As developers, isn’t coding what we do? Think for a minute – how much time gets spent coding/designing vs. everything else? For most, it is probably too little time on producing and way too much on everything else. Becoming a more effective communicator is the one way to spend less time on “everything else” leaving more time to produce.</p> 🧠 Context is everything 2025-10-10T00:00:00+00:00 In conversation, I find myself saying: “what is the context of your comment?” At times we find ourselves responding before we really understand something.</p> AI does the same thing – it just spouts off, even if it does not understand your context.</p> This is one reason local agents like Claude Code work so well, vs AI in a web browser. Local agents at least have the context of your code and local files. However, it still does not have access to your complete domain context – that is your part.</p> 💡 Reflections from AI-coding a 10,000 line app 2025-10-08T00:00:00+00:00 Lessons from AI-coding a 10,000 line app</p> BRun</a> is getting usable. Claude Code wrote almost all of it using Doc-driven development</a> (manually write/update the docs, and have Claude write the code).</p> Some thoughts:</strong></p> Claude is fast – there is no way I could have written this app manually in a day or so.</li> Large changes are easy with AI.</li> Keeping docs up to date is easy.</li> Having AI do the coding freed up more time to think about the app user interface, etc.</li> Claude is a nice partner to bounce ideas off of.</li> </ul> Some things to watch out for:</strong></p> Look out for duplicated/unnecessary code. There were multiple cases where Claude was calling the same function multiple times.</li> Refactoring is not always complete. If things get restructured, check that everything got cleaned up.</li> Check the tests – Claude really goes crazy generating tests. A lot of work remains on this repo reviewing and cleaning up tests.</li> Claude often tends toward complex approaches and a lot more code than needed – like many junior devs. Close monitoring is needed to ensure the architecture takes the simple path. Code is a liability, not an asset</a> – even with AI.</li> Docs updated by Claude tend to be overly verbose. I was continually trimming them down to remove duplicated/unnecessary content.</li> Critical things like life-cycle and state management often get overlooked.</li> Race conditions are also easy to miss in AI-generated code.</li> </ul> In the end, this project went well, but it still needed expert guidance at times. Perhaps some of the above could have been improved with more specific rules. Like many things, the greatest strength is also the greatest weakness. Claude is really good at generating code – so good that it tends to generate way too much at times. The hardest part of an app like this is figuring out what it should do, how it should be configured, what the UI looks like, and debugging problems when it does not work. This takes experience and is still a lot of work to get right.</p> 📏 Less code is not the ultimate goal 2025-10-03T00:00:00+00:00 Less code is not the ultimate goal</p> Consider the following byte to binary C program</a>:</p> int main(int b,char*i){long long n=B,a=I^n,r=(a/b&a)>>4,y=atoi(++i),_=(((a^n/b)(y>>T)\| y>>S)&r)\|(a^r);printf("%.8s\n",(char)&_);} </code></pre> Unless you are trying to win the International Obfuscated C Code Contest, this is not good code. In spite of what I’ve said before</a>, less code is not always better. The highest goal is readable maintainable code. Sometimes we have to write more code to have readable code. An example is aligning the happy path</a>.</p> Less code is more about not implementing unneeded features, cleaning up dead code, refactoring to simplify and eliminate duplicated code, etc. The goal of less code should never sacrifice readability or maintainability.</p> 😊 Other ways to align the happy path 2025-10-01T00:00:00+00:00 Yesterday</a>, we discussed aligning the happy path in code. Are there other ways to align the “happy path”? (The happy path is the desired path where things are functioning as expected.)</em></p> Some thoughts:</p> Focus on your productive people - as a manager focus your efforts on making life better for the productive 95% of your team, rather than trying to fix the unproductive 5% who probably cannot be helped anyway. At least make your efforts proportional. We should be willing to help people who sincerely want help, but trying to help those who do not want help drains our soul.</p> Processes should focus on making the primary work go faster and easier. Sure, problems must be addressed, but returns are higher on improving the work desired, vs. knocking down problems. If the former is done better, less time will be spent on the latter.</p> Proactively write tests to ensure the system is doing what you want it to do, rather than wait for bugs to appear and then write tests to cover those.</p> In summary, if energy is proactively focused on what is wanted to happen, a lot less time will be spent battling things not wanted.</p> 😊 Align the "happy path" in your code 2025-09-30T00:00:00+00:00 An idiom that I learned in the Go community is to align the happy path. An example in C below:</p> happy path not aligned</h2> int process_file(const char* filename) { FILE* f = fopen(filename, "r"); if (f != NULL) { char buffer[100]; if (fgets(buffer, sizeof(buffer), f) != NULL) { // process buffer fclose(f); return 0; // success } else { fclose(f); return -2; // error: unable to read } } else { return -1; // error: unable to open } } </code></pre> happy path aligned</h2> int process_file(const char* filename) { FILE* f = fopen(filename, "r"); if (f == NULL) return -1; // error: unable to open char buffer[100]; if (fgets(buffer, sizeof(buffer), f) == NULL) { fclose(f); return -2; // error: unable to read } // process buffer fclose(f); return 0; // success } </code></pre> In the first example, it is more difficult to follow the logic due to nested if statements. In the second example, it is easy to scan down through the code aligned to the left and sequentially see what it is supposed to do.</p> 🔓 You should not be needed 2025-09-29T00:00:00+00:00 An effective process, team, or organization should be able to continue on if any individual steps away for a bit, or forever. But, most people are insecure and like to be needed, so subconsciously structure things to require them to be present for things to proceed – the gatekeeper or authoritarian mindset.</p> A few ideas:</p> Share everything you know</a> with the widest possible audience.</li> Document why</a>.</li> Automate tedious processes so anyone can execute them. This also serves as good documentation.</li> Encourage others to do things you do.</li> Prefer transparency for accountability rather than oversight or status reports.</li> </ul> Creating a process or culture that can function just fine without a single point of dependency makes that person more valuable than ever, and frees them up to do other higher-value things.</p> 🚧 Three things that hinder product development 2025-09-24T00:00:00+00:00 Not using new tech when applicable.</li> Doing the same work over and over again instead of automating.</li> Lack of collaboration.</li> </ol> ⚙️ Three aspects of product development 2025-09-23T00:00:00+00:00 Finding and leveraging existing technology.</li> Developing some of your own technology.</li> Mixing it together.</li> </ol> All three are necessary.</p> ⚖️ Technical debt compounds with you doing nothing 2025-09-22T00:00:00+00:00 Technical debt is like any other kind of debt – it accrues when nothing is done. This is perhaps the most insidious aspect of debt.</p> How does technical debt work? Mainly that software (and technology in general) moves on (especially OSS), and we do not keep up.</p> “It works, why do we need to update? That just introduces risk.”</p> </blockquote> But the longer we put this off, it becomes exponentially harder to ever catch up – just like compound interest on financial debt never receiving payments.</p> The other aspect of doing nothing is that infrastructure rots – the tools, automation, testing, and know-how that even allow making changes and improve things. So when the gap is huge and there is no infrastructure (way to close the gap), it is nearly impossible, so we declare bankruptcy and start over.</p> There is another path: keep up-to-date. This forces the infrastructure to stay fresh. Iterations expose areas that can be automated and improved. Investing in YOUR platform compounds benefits over time.</p> 📚 Without docs 2025-09-12T00:00:00+00:00 Design files (mechanical, electrical, or software) contain the technical details that allow us to manufacture something in the moment, but they often don’t contain the “why”</a>.</p> Why is this mechanical feature present?</li> Why is this circuit necessary?</li> Why is the code structured this way?</li> </ul> Most designs exist this way – without adequate documentation. What happens two years later when something quits working? What happens when a new feature needs added? What happens when someone leaves?</p> World-class engineering requires world-class documentation. There is no other way.</p> ⚖️ Code is a liability, not an asset 2025-09-11T00:00:00+00:00 A wise developer once said</a>:</p> Code is a liability, not an asset. Aim to have as little of it as possible.</p> </blockquote> This is perhaps the #1 rule for clean code. Some examples:</p> Don’t implement “just in case” features. Remember YAGNI</a>.</li> Premature abstractions</a> often do more harm than good.</li> Premature optimization is the root of all evil</a>.</li> Simple data structures scale</a>.</li> Find patterns</a> and refactor often.</li> </ul> Just-in-case development (implementing stuff not needed right now) is a powerful force, especially with less experienced developers who have not spent significant time maintaining software.</p> Understand the problem before starting implementation.</li> If at all possible, reproduce the bug before trying to fix it.</li> </ul> ✂️ YAGNI 2025-09-10T00:00:00+00:00 YAGNI</a> - “You ain’t gonna need it!”</p> If you don’t need it now, the chances you’ll need it in the future are small, or at least not the current implementation. Do the simplest thing that works now. Get feedback. Repeat.</p> Iterations</a> are the key – get better at doing iterations faster (including testing, and feedback), not implementing more stuff now.</p> 🐛Kernighan's Law 2025-09-09T00:00:00+00:00 Kernighan’s Law:</strong> Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.</em></p> Cliff’s Law:</strong> In three months, it will be three times harder to read the code you just wrote than writing the code in the first place. With AI, add a 10x factor.</em></p> Maintaining code over a product lifecycle is many times harder than the initial implementation. With AI, doc</a> or spec</a> driven development is critical. Otherwise, you will bury yourself in technical debt. Make it easy on others and your future self.</p> 🌀 Complex systems are hard to predict 2025-09-08T00:00:00+00:00 My wife makes sourdough, which is a multi-day process and at times unpredictable. Recently, the current batch was a little sluggish and not expanding like it should, so she “fed” it and left it on the counter. It took off during the night and was all over the counter in the morning.</p> Sourdough is a complex living system. Because of this, sourdough is difficult to predict 100% of the time. As computer systems become more complex, they take on some of the same characteristics of living systems. You can’t completely predict everything. What worked yesterday may not work today, due to various external factors that cannot be controlled or fully understood. Especially with networked systems – we cannot control everything else on the network, or involved with the network. We cannot control what new feature customers are going to ask for that might require us to update dependencies. We can’t predict what new edge case will trigger a bug we have not seen before.</p> The best we can do is keep tools and processes flexible, up-to-date, and as automated as possible (YOUR Platform). This enhances the ability to adapt to the external forces that cannot be controlled.</p> 📊 Simple data formats + powerful tools scale 2025-09-04T00:00:00+00:00 There is an ongoing shift in data analysis tools. Another example is datavzrd</a>. datavzrd is a zero/low-code, interactive, visual, server-free, browser-based reporting tool for tabular datasets.</p> The paper</a> makes an interesting statement:</p> Most spreadsheet applications, including Excel, lack the capability to quickly and easily visualize different columns of a dataset. They also do not support reproducibility of results for similar datasets without repeating the work</strong>.</p> </blockquote> This last point is key. For example, if data needs monthly review, tools like this make a lot of sense. Complete automation becomes possible. Manual spreadsheet work every month becomes unnecessary, eliminating tedious and error-prone processes.</p> This is another example of the trend toward simple data formats</a> and flexible tools.</p> YAML, CSV, and Markdown are the file formats of the future.</p> Simple data formats + powerful tools is the formula that scales.</p> 📚 Documentation never has to go out of date again 2025-09-03T00:00:00+00:00 AI is really good at updating documentation after implementation. While I strongly advocate writing the documentation to drive AI</a>, inevitably, the implementation takes some unanticipated turns, so making sure everything is in sync after implementation is a good task for AI.</p> Are the docs up to date? With Claude Code (or something similar), there is really no excuse.</p> However, it helps to standardize on the format the AI knows best: Markdown</a>, and it helps a lot to keep the docs close to the design</a>.</p> 📚 Keep documentation close to the design/code 2025-09-02T00:00:00+00:00 I find it works best to keep documentation as close to the code or design files as possible. There are several benefits:</p> You can update, commit, and review documentation as part of the standard working process (pull requests, CI, etc.)</li> You can use tools you are already using (like your favorite editor)</li> It encourages the use of Markdown (the format AI works with)</li> Claude Code, which you run from your working directory, has access to it. This has always been important, but with AI tools, it is even more important—more on that in future.</li> </ul> 📝 Document "why" not "how" 2025-08-29T00:00:00+00:00 Be kind to others (and your future self) and document why you did something.</p> AI is pretty good at generating the “how” any time you need it, but only you know the “why” because you have that context.</p> 🔄 Closed-loop engineering 2025-08-28T00:00:00+00:00 What are some ways we can close the loop in engineering efforts?</p> Systematic testing</li> User feedback</li> </ul> These ensure whatever we build works</strong> and is useful</strong>.</p> 🚴 The power of movement 2025-08-27T00:00:00+00:00 Rod’s thoughts on Riding a Bike</a> are thought provoking.</p> … A decision that might be bad. Yet, with it, you can steer. Just like riding a bike. If it’s standing with no movement, you can direct the handle and still be where you are. It’s when you’re moving on the bike that you give direction.</p> </blockquote> I’ve often experienced this in software development where I don’t really know how I’m going to solve a problem until I start coding, and then things start to emerge. The same happens with writing</a> and sketching</a>.</p> There is value in up-front planning, but there is power in movement, especially when you iterate</a> and close-the-loop</a>.</p> 🔄 Closed-loop humans 2025-08-26T00:00:00+00:00 What are some ways we as humans can close the loop?</p> Listen</li> Reflect</li> Write</li> </ul> Without these, we operate in a largely open-loop mode without the feedback we need to improve or correct our course.</p> Action bias is good, but it helps to know we are doing the right things and going in the right direction.</p> 🔄 Look for closed loops 2025-08-25T00:00:00+00:00 Closed loop systems are stable. Why? Because they learn and self-correct.</p> In any system, process, person, or organization, look for closed loops. If something is not working, chances are there is an open loop.</p> 🔄 Close the loop with AI 2025-08-22T00:00:00+00:00 There is a fascinating statement in a recent MIT study</a> that claims 95% of organizations are getting zero return from GenAI investments.</p> The core barrier to scaling is not infrastructure, regulation, or talent. It is learning. Most GenAI systems do not retain feedback, adapt to context, or improve over time.</p> </blockquote> AI proves useful to software developers, perhaps because so much context is available, giving it a better starting point than in most other industries.</p> However, I think we can still do better by closing the loop. The below workflow is currently working well for me:</p> </p> Closing the loop means retaining and re-using what you have learned. Interestingly, it does not really matter if humans or AI are doing the work—the same process works well with both.</p> Notice the absence of AI prompts – while an AI tool may remember prompts for a session, they are lost on the next. They are “open-loop”.</p> Code is not the most valuable asset you own, but rather the ability to generate, maintain, and improve it—the process. It is kind of like personal wealth and skills. Even if we lose the first, we can always generate more with the second.</p> 💰 Ways to reduce your AI costs 2025-08-21T00:00:00+00:00 AI is not free, as there are significant costs associated with training and hosting modern AI models. Some ideas on how to reduce costs with tools like Claude Code:</p> If you do something that is somewhat repetitive, have AI generate scripts or programs that do the work instead of using AI directly. AI excels at creating automation tools.</li> Capture context. Instead of relying solely on prompts, update documentation and have AI implement the doc changes. This approach captures a lot of context instead of losing context in prompts.</li> Start with more complete and accurate descriptions of what needs to be done.</li> Use cheaper models. In my experience, Claude Sonnet performs just as well as Claude Opus for most tasks I need to complete. Sonnet is 1/5 the cost and is generally faster.</li> If the agent is spinning its wheels, stop it, provide more details, and resume.</li> </ul> 💰 What is the cheapest PDM tool out there? 2025-08-19T00:00:00+00:00 Product Data Management (PDM)</a> is how you manage and publish your product data. The goal is to ensure all the stakeholders share a common understanding, confusion is minimized, and quality is maintained.</p> The simplest PDM system is to put a bunch of files on a fileserver and talk about them in meetings, emails, or other messages. This is a start, but it is error-prone and does not scale.</p> There are many commercial PDM tools available. CAD systems, like Onshape</a>, provide PDM features. Software developers have been doing PDM to various degrees for decades using open-source tools. You have SCCS (1972), RCS (1982), CVS (1986), SVN (2000), and Git (2005).</p> Git and its various hosting platforms</a> support a number of the features</a> needed for PDM. The cost is minimal and you probably already have people on staff who are experts at using it. For small teams, Git may just be the PDM solution you need. However, there is a learning curve, and some additional tooling may be required to make it accessible—more on that in the future.</p> 🗂️ Git has an unfortunate name 2025-08-18T00:00:00+00:00 Git</a>, which was created 20 years, and various hosting platforms</a>, has revolutionized</a> software development workflow and collaboration. The name Git was chosen by its creator, Linus Torvalds, and has a humorous backstory rooted in both practicality and world-class trolling. In British slang, “git” is an insult meaning an unpleasant or silly person, which Torvalds acknowledged with characteristic Finnish bluntness: “I’m an egotistical ___</strong>, and I name all my projects after myself. First Linux, now Git.”</p> Most of us software developers, especially in OSS culture, appreciate a little humor and self-deprecation (it’s basically a job requirement at this point). However, those from other disciplines seem to have an inexplicable aversion to anything with the word “git” in it. Perhaps this is understandable – why would I ever use a tool that literally calls me stupid in the name? That must mean I’m stupid if I use it, right? This appears to be a subconscious psychological defense mechanism that is surprisingly hard to overcome. It unfolds something like this:</p> Developer (D)</strong>: Hey, why don’t we use Git</a> for this project? It handles all our file management problems</a> with some project management tools thrown in.</li> Project Manager (P)</strong>: Great idea! (Famous last words)</li> D</strong>: Can you comment on that “Git” PR I just posted?</li> P</strong>: Sure.</li> <deafening silence></code></li> D</strong>: Can you push the new product specification you wrote to the “Git” repo in Markdown so we can track and review changes?</li> P</strong>: Sure! (Proceeds to email a Word doc like it’s still 2003)</li> <developer dies a little inside></code></li> D</strong>: Can you test the new release in “Git”? (link to Git release).</li> P</strong>: (a little while later) Can you just email the files to Nick? (Git? What Git?)</li> D</strong>: <internal screaming intensifies></code></li> D</strong>: (at the weekly status meeting) Why am I the only one tracking my tasks in “GitHub” issues?</li> P:</strong> Good question, we should all do that. (Immediately forgets and goes back to sticky notes)</li> <the sound of one developer crying></code></li> D:</strong> Why don’t people use “Git” when it has most of the file management, collaboration, and PM tools we need?</li> P:</strong> It’s too hard. (Says the person who maintains a 47-tab Gantt chart in Excel)</li> D:</strong> It’s a lot simpler than a mega Gantt chart in Excel, or a 3D model in SolidWorks.</li> P:</strong> True. (Goes back to Excel anyway)</li> <the void stares back></code></li> </ul> The only rational conclusion I can come up with is that the word “Git” somehow triggers a Pavlovian response in most people’s brains that makes them physically unable to click on anything Git-related. It’s like some kind of linguistic kryptonite. Only a minority of battle-hardened software nerds, especially those who’ve been sufficiently traumatized by OSS projects, have managed to overcome the existential crisis of using a tool that openly mocks them.</p> 📁 Git* solves the problem of managing files 2025-08-15T00:00:00+00:00 The thing you get with Git</a> is a sane way to manage files:</p> organization</strong> (team/repo/directories)</li> documentation</strong> (excellent support for Markdown)</li> efficient navigation</strong> (links in READMEs and other markdown files)</li> history</strong> (who changed what when)</li> collaboration</strong> (issues, pull requests)</li> distributed</strong> (anyone can work anywhere)</li> automation</strong> (continuous integreation (CI), continuous deployment (CD), hooks, etc.)</li> reviews</strong> (PRs)</li> workflow</strong> (enforces workflow such as requiring PRs, approvals, etc.)</li> versions</strong> (feature and maintenance branches)</li> releases</strong> (built in way to do formal releases)</li> security</strong> (granular access controls and workflow permissions, bank level security for web access (https), and better for Git operations (ssh keys))</li> </ul> Git is not primarily a way to manage projects, but rather a way to manage and collaborate around files, with some project management thrown in. It is centered around the work, which is the main point.</p> Other tools have more advanced project management features, which are needed at times. But they are missing a key ingredient: tight integration with your files. The closer you can keep your project management and other metadata to the actual work, the better.</p> 📁 The file dilemma 2025-08-14T00:00:00+00:00 There is this movement against files in most disciplines. Files are hard to manage, track, version, and secure (unless you use Git). Thus, a lot of tools have moved to SaaS (software-as-a-service) applications in the cloud. Jon Hirschtick articulates this point of view</a> very well. Cloud and APIs are the new “file.”</p> I’d like to present a few counterpoints.</p> Integration:</strong> One of the ways we add value fast is integrating systems. This is due to the network effect</a>, where Metcalfe</a> defines the value as the square of the number of nodes (or in our case integrated systems or types of data). When we can integrate multiple types of data (say hardware, software, and mechanical), the value increase is much greater than just linear.</li> AI</strong>: AI needs context to be effective. The easiest way to provide context is to run a local agent that has access to files that are in an open format. MCP</a> (Model Context Protocol) may be another way.</li> Accessibility:</strong> One of the problems with SaaS and APIs is you don’t really own the data. Ten years down the road, are you still going to have access to it? Are you still going to have the tools needed to use the data?</li> </ul> Files are flexible and scale. The success of Linux evidences this (which adopts the Unix philosophy that everything is a file), the web (it still uses a bunch of plain-text files for every site), and software development (anyone ever heard of source code?). Software is by far the largest and most complex systems humans have created, and it is still 99% based on the lowly text file. Since its inception in 1991, the Linux kernel has had more than 20,000 contributors. Onshape</a> is impressive, but it is a long way from scaling to projects of this size and longevity.</p> The reality is that a mix of files and APIs is no doubt required. However, if you want scalability and flexibility, I’m not sure you want to throw out files. GUI technologies have been trying to do this for decades, but HTML and friends</a> are still king (anyone remember Flash?).</p> 🔄 AI and documentation - close the loop 2025-08-13T00:00:00+00:00 Starting with good documentation is often a better way to use AI</a>. However, you can also close the loop and have AI update the documentation when it is finished. It can fill in details, clean things up, etc. This closes the loop. Then repeat.</p> </p> 🤖 AI is the new compiler 2025-08-12T00:00:00+00:00 There are two ways to use AI:</p> Start with a vague prompt and keep prompting until we get the result we want.</li> Point the AI at documentation, and keep updating the documentation until it is clear enough that AI can do the right thing.</li> </ul> Even though it takes more time up front, in the end, the second approach will be faster and more efficient because building a cohesive picture from a long string of sloppy prompts is difficult, even with something as capable as AI. AI coding engines are not free, so efficiency matters.</p> AI is the new compiler – IF we choose to use it that way.</p> Writing clean code is a skill. So is writing clean documentation.</p> ✨ Anyone can write scripts now 2025-08-11T00:00:00+00:00 With Claude Code (and likely many other AI technologies), anyone can write scripts now. Instead of writing in a technical language like Python or Bash, you can now write a sequence of commands with some rules in plain English, and Claude transforms these into the technical bits to make it happen. Low-code</a> is finally a reality. You can keep these as Claude commands, or have Claude write permanent scripts, programs, tools, etc.</p> It is now easier than ever to build YOUR Platform.</p> See this summary</a> or this example</a> for more information.</p> ✏️ Effective Communication (Sketching) 2025-08-08T00:00:00+00:00 Sketching is kind of like writing cursive—it engages a dynamic in our brains that inspires creativity. As we mostly use computers, it takes extra effort to pick up pen or pencil, but surprising things happen when we do. It engages broader areas of our brain, and there is a flow that comes from the lack of restraints our digital tools have. So perhaps sketching is mostly about communicating with ourselves, but we can still easily snap a photo with a phone and share it. This is perhaps the lowest friction form of drawing.</p> Here is an example</a> in thinking through a hard problem where sketching really helped.</p> One thing that helps me cultivate a sketching habit is to always have paper and a pencil or pen out on the desk when working. The daily worksheet</a> is always on my desk—the back is a good place to sketch and write notes. I also need to get into the habit of using a journal</a> more.</p> 🎨 Effective Communication (Drawing) 2025-08-07T00:00:00+00:00 We’ve already explored 4 modes of communication</a> (audio, written, images, video). Drawings are another, and today it is even easier to do with tools like Excalidraw</a> and Draw.io</a>. It is easy to copy drawings from both into other applications.</p> A clean drawing/diagram brings another level of professionalism to your work.</p> Drawing with AI is another option, but I have not figured out how to use that yet.</p> One thing that has worked well for me is to use a single Excalidraw drawing as an infinite canvas. This way I can easily see content I’ve created before, reuse it, or use it for inspiration.</p> </p> 📹 Effective Communication (Video) 2025-08-06T00:00:00+00:00 Video is now an important way to communicate. The most obvious type is Zoom calls. But there are other ways we can use video such as screencasting, recording demos, etc.</p> Position a decent camera at a good angle. Shooting up your nose probably isn’t a good camera position.</p> Screencasts are a scaling superpower. Record how you do something and share it with people learning. Have them write the documentation or checklist.</p> Screencasts may be a better way to demo. Record and share the demo ahead of time, then discuss it in a meeting.</p> With a screencast, it’s even more important to have good audio. In a meeting, people will overlook bad audio because you have a captive audience. But in a recorded video, people will likely quit watching if the audio is bad. So get a decent podcasting mic</a> or headset</a>.</p> Learn OBS Studio - this is a professional way to record screencasts that is free. I’ve found that setting up a 1080p (1920x1080) section</a> of my screen to record works well. Drag, resize, and stack the applications you want to display in this section of the screen, then just record that single scene. This is much simpler than setting up a separate scene for every application.</p> If you are working with hardware, get a document camera to video stuff on the lab bench.</p> And don’t forget your phone takes excellent video - we all should probably make better use of this capability.</p> When possible, record in MP4 format. This format plays easily in most browsers, so you can simply upload a file to a file sharing site, and people can automatically play it in their browser.</p> I’ve yet to figure out lighting, green screens, etc. - lots to learn yet. Systems like this can always be improved, so keep improving.</p> 📸 Effective Communication (Images) 2025-08-05T00:00:00+00:00 “A picture is worth a thousand words.”</p> </blockquote> There are many times when a screenshot or a photograph can drastically improve the rate of communication. Therefore, figure out ways to do this efficiently.</p> Make sure your computer system has an efficient way to capture and mark up screenshots – preferably in the same application. Spectacle in KDE is excellent. I’m sure there are good options for other environments.</p> Also have a quick way to get photographs off your phone into a message. I use the Google Magnifier app on Android, sharing to a “Note to self” in Signal, then copying the image in Signal desktop on my PC. This is much faster than waiting for the image to upload to Google Photos.</p> What works for you?</p> ✍️ Effective communication (Written) 2025-08-04T00:00:00+00:00 For starters, consider Michael Lynch’s “Underused Techniques for Effective Emails”</a>.</p> A few points in summary:</p> Put the most important information first.</li> Use headings, code blocks, and other formatting.</li> Be succinct.</li> Provide context by quoting and replying (bottom posting</a> in emails and quoting in PR/issue threads). Don’t make your readers read the entire discussion and try to extrapolate the context of what you are trying to say.</li> </ul> 🤖 How I use Claude Code to help publish this blog 2025-08-01T00:00:00+00:00 I have been using Claude Code to help with the following tasks on this blog:</p> proofread (including detecting passive voice</a>)</li> add tags</li> add an emoji to the title</li> </ul> These are not rocket science, but they also take time and are something AI does pretty well.</p> This video</a> shows how the process works.</p> 🎧 Effective communication (Audio) 2025-07-31T00:00:00+00:00 Effective communication requires that those receiving information from us can easily process it. I’ve been on way too many calls where you can barely hear the speaker due to echo, background noise, or other audio problems. Some things I’ve found helpful:</p> Use a headset. This eliminates any possibility of feedback from speakers into the mic.</li> Use a good noise-canceling/rejecting mic, especially in noisy environments. Some headsets (like the Sennheiser SC series</a>) block background noise amazingly well. Good podcast mics (like the AT2020</a>) are usually directional and block noise pretty well, especially when positioned close to your mouth</a>.</li> Use a Bluetooth headset</a> with your phone that has noise blocking in the microphone. Most noise-canceling headsets focus on blocking background noise for your benefit. On calls, optimize for your listener’s benefit.</li> Pro tip: never use the speakers and mic in your laptop, even if it is a Mac.</li> Pro tip 2: don’t use speakerphone mode on your phone – especially for group calls.</li> </ul> Don’t be the one who disrupts a group call with crummy audio. You are a professional – use professional gear.</p> 📧 Top vs bottom posting in email 2025-07-29T00:00:00+00:00 I was thinking about computing history this morning, and I wonder how many young people even know that bottom posting in email</a> is even a thing? Some of the older OSS projects still have this culture</a>, but I suspect that most emails sent today use top posting.</p> I find myself getting lazy and top posting, but it is so much more considerate of the reader (especially with modern email readers that thread messages) to bottom post and only keep the text needed to understand the context or your reply. The old emails are still in the thread and easily viewable – there is no reason to repeat the entire discussion chain in every email.</p> Smart clients like Gmail somewhat alleviate this concern by hiding previous message details, but it often hides too much, and you have to expand parts of the old message to see what a response is referencing. And with top posting, you often have to repeat the context of your reply vs. just including your reply directly below the relevant context.</p> This is one reason discussions in Git* issues and other forum type threads can be so much more effective – you can more easily follow the discussion over time. But you can get near the same efficiency with email if you take a little care.</p> 🎉 200 posts 2025-07-28T00:00:00+00:00 This marks the 201st post I’ve sent – 2/3’s of the way through the 300 bad posts</a>.</p> How is it going? I still enjoy the process and the thinking it brings. However, it is getting a little bit more difficult recently to keep the discipline, especially with busy project and summer schedules.</p> I continue to have many more ideas than time to write, so coming up with ideas is not a problem.</p> The publishing process has been further streamlined using Claude Code – will share more details on that in the near future.</p> If you have any feedback, please reply – always welcome!</p> 🔗 Integration, not isolation (part 2) 2025-07-24T00:00:00+00:00 We recently discussed integration</a>, not isolation from a technical perspective.</p> In an organizational context, we can have integrated cross-functional teams, or isolated silos.</p> And the easiest way to avoid silos</a> is transparency and common systems. These things are hard – they don’t just happen automatically.</p> 🔗 Integration, not isolation 2025-07-23T00:00:00+00:00 In a recent discussion</a>, the comment was made that containers are not always a good fit for edge/embedded devices, as they are designed for isolation (which is useful at times for cloud workloads), but with embedded, we want tight integration. Justin Schneck advocates file-system layers as an alternative. Isolation from the kernel is a feature in the cloud, but in embedded systems, we often need direct access to the kernel (integration).</p> We, as an industry, perhaps focus too much on isolation (security, protection, every app bundles its own stack of dependencies, etc.). Short term, this is convenient. Long term, this can lead to bloat, inefficiencies, and brittle systems.</p> Can we have integration AND decoupling? That is a topic for future thinking…</p> 📈 What do we mean by scaling? 2025-07-22T00:00:00+00:00 This can mean many things:</p> More units sold</li> More customers served</li> More people doing the work</li> More features</li> More innovation</li> More models produced</li> More model options</li> More quality</li> </ul> We could go on, but basically we mean more of something over time. And the “over time” part means we need to be thinking long-term. You cannot scale with short-term thinking—this is a logical fallacy. This is why YOUR Platform is so important.</p> 🔄 Open loops -> technical debt 2025-07-21T00:00:00+00:00 If something is corrected, it needs to be fed back and incorporated, so you don’t keep redoing the same stuff over and over and eventually bury yourself in technical debt.</p> Effective organizations close the loop.</p> 🧩 Composing is hard, complexity is easy 2025-07-18T00:00:00+00:00 We’ve recently explored the idea that composing simple things is the way to scale</a>. But this is harder than just cranking something out. Some examples:</p> It is much easier to create an Excel spreadsheet than write a Python script to process a CSV file and create a chart which can be later automated in a CI process.</li> It is much easier to create lots of custom structs with lots of fields than to create simple structs that can be used for multiple purposes.</li> It is much easier to add additional functions to an interface than to create additional well-thought-out interfaces.</li> It is quicker to hack on a Raspbian image than to set up a proper Yocto build for a Raspberry Pi single board computer that can be more easily maintained long-term.</li> It is quicker to draw a schematic with generic parts and manually assign manufacturer part numbers/footprints, than set up a proper parts library that will make subsequent designs and changes easier.</li> </ul> If your task is a one-off project, or something only you will ever interact with, then the easy path may be fine. However, if you want to scale, the hard path is the way forward.</p> 🎼 Compose software like music 2025-07-17T00:00:00+00:00 Music is an interesting thing – a beautiful arrangement is composed of very simple constructs – a handful of simple notes on a relatively simple scale. The brilliance is in how these simple constructs are arranged.</p> Likewise, brilliant software is composed of simple data structures and interfaces. Just like music, there is no limit to the ways these simple constructs can be arranged.</p> Conversely, complex constructs are very limited in how they can be used.</p> Put the complexity in the arrangement, not the constructs.</p> 🌱📈 Simplicity scales 2025-07-16T00:00:00+00:00 When programming, we learn that simple interfaces</a> and simple data structures</a> are good, because they are flexible.</p> Simple file formats</a> are the same. We can do way more with a CSV or Markdown file in terms of presentation and automation than we can with Excel or Word documents.</p> This seems counterintuitive – Excel is a very powerful tool, and the right tool for many uses. However, an Excel file is not as composable as simple formats. As an example, consider the assets</a> of a Perplexity labs project. They are Python scripts, CSV, Markdown, etc. Why? Because a powerful tool, like AI, can do more with simple formats.</p> Complex formats (Ex: Excel) allow a single person to do a lot in the short term, but simple formats allow a team and automation to do way more over time.</p> The way to scale YOUR Platform is to compose simple interfaces, data structures, and file formats. A little more work short term, but way more powerful long-term.</p> 📏 Goodhart's law 2025-07-15T00:00:00+00:00 Goodhart’s law</a> states: “When a measure becomes a target, it ceases to be a good measure”.</p> Metrics are useful, but they are just a tool, not the goal. If Metrics become the goal, then they undermine collaboration, trust, and transparency.</p> 📈 Why GUI development tools don't scale 2025-07-10T00:00:00+00:00 This is an interesting read: GUIs are Antisocial</a></p> This hits on an important point… GUI tools don’t scale, because they are not social. Two of the most complex efforts humans have accomplished are chip design and software development – both are firmly entrenched in text based formats and command line tools</a>.</p> 🔭 Text based formats and command-line tools are the future 2025-07-08T00:00:00+00:00 For the past three decades, we have been told that graphical user interfaces (GUIs) will revolutionize computer use and even programming. In many ways, they have made computers very accessible for the average person – especially phone UIs.</p> However, the command-line tools and text files are still the standard for building stuff. Most useful programming is done in text based programming languages. Even most UIs are still implemented in text based languages such as HTML and CSS. In spite of the many graphical options that have come and gone – HTML remains the king.</p> Why do text based formats and command line tools work so well?</p> Text-based formats are easy to work with (collaborate, edit, tooling).</li> Command-line based tools are easy to automate (build systems, CI, CD).</li> </ul> The coming AI storm makes these factors even more important. Because AI coding is all about tooling, collaboration, and automation.</p> If we dig a little deeper, text-based formats and command-line tools are “open” – you can do things that were not originally planned. Open formats and tools do not lock you into a single paradigm, but rather allow you to build YOUR Platform to meet your needs.</p> 🔄 Flipping the playbook for AI coding 2025-07-02T00:00:00+00:00 A lot of focus these days is on “good AI prompts.”</p> Some of the best prompts may be documentation and tests. Write your CHANGELOG.md</code> entries first, then update the documentation, then write the tests, then tell the AI to update the code to match.</p> We tend to get it backwards—asking AI to generate tests and documentation after the code. Flip this and do the docs and tests first.</p> Your thinking will be better, and you’ll move faster.</p> 🧑‍💻 The first step in AI coding 2025-07-01T00:00:00+00:00 ✔️ Should be “How does this work?”</p> ❌ NOT “Implement this feature.”</p> Understanding should precede any work, especially in complex systems.</p> 👥 Alternatives to meetings 2025-06-26T00:00:00+00:00 ✔️ Instead of presenting in a meeting, record a screencast and share it.</p> ✔️ Publish your work often.</a></p> ✔️ Utilize notifications in workflow tools.</p> ✔️ Utilize a team chat tool such as Signal, Slack, Discourse chat, Discord, etc.. This does wonders for bonding teams (especially distributed teams). You can always mute it when doing deep work.</p> 🥣 The two ingredients for agility 2025-06-24T00:00:00+00:00 ✔️ Low friction</strong>: tooling and workflows that allow anyone to make changes easily. (good build systems, clean libraries, release automation, etc.)</p> ✔️ Confidence</strong>: what we are doing is not going to break things. (CI, end-to-end testing, review, etc.)</p> #1 is obvious. #2 is subtle – if we don’t have confidence in the work, we will avoid it.</p> ♻️ How to iterate quickly on hardware 2025-06-23T00:00:00+00:00 Today, we mostly think of iterating quickly on software, but it is becoming increasingly possible to also do this with hardware. A few ideas:</p> Printed circuit boards (PCBs) are now relatively cheap and quick to manufacture in low quantities from companies like JLCPCB (you can have a new PCB in less than a week at a very reasonable cost).</li> Companies like JLCPCB and Macrofab offer prototype assembly services.</li> It is practical to assemble your own</a> prototype PCBs quickly.</li> You can now do more CI/CD type things with PCB design tools. KiCad has a powerful DRC rules engine and the ability to script a lot of operations (including generating outputs) from the command line. Run ERC/DRC in CI and generate outputs in CD. This allows you to make and deploy changes with a high level of confidence.</li> Verified parts libraries</a> allow you to easily reuse electrical components in new designs.</li> Spice simulation tools allow you to verify designs before building a PCB – again, you can have a higher confidence that things will work.</li> 3D printing in production</a> allows you to make mechanical improvements without expensive tooling changes.</li> Intelligent part numbering schemes</a> and PLM tools</a> allow you to efficiently and reliably manage changes.</li> Use OSS tools like KiCad</a> where possible so anyone can make changes.</li> Use Git workflows</a>.</li> </ul> The above is YOUR Platform. Here’s the thing – the more complex and the higher value your product is, the more important it is to be able to iterate quickly and confidently on both hardware and software. As an example, Tesla updates the software in their cars frequently and continuously makes mechanical improvements (sometimes as many as 120-160 changes per week). If Tesla can do it with cars, you can probably do it with your low-volume industrial product.</p> 🤝 Modern product development is a collaborative endeavor 2025-06-20T00:00:00+00:00 There is no way around this – even if you are a Company of One</a>. You must collaborate with:</p> ▶ OSS communities who provide the tools and software you use,</p> ▶ Your customers – if you are not learning from them, you probably won’t understand and fulfill their needs,</p> ▶ and Your team.</p> While the last item on the list is obvious, the first two are less obvious, but perhaps just as important.</p> ✨ Use the most advanced technology that is comfortable in your product design 2025-06-19T00:00:00+00:00 Some examples:</p> ▶ When designing a PCB, put down the smallest packages you are comfortable working with. Example: put down 0603’s instead of 0805’s, even if you don’t need the space right now.</p> ▶ Use a MCU with the most performance and RAM/Flash that is reasonable. Use a STM32 instead of a PIC16.</p> ▶ Everything else being pretty much equal, use Go instead of Python.</p> Why?</p> ✅ You may not need the space now, but the next iteration or variant of the product will likely be smaller, so why not plan for that now if it does not cost you anything?</p> ✅ Modern embedded systems that support field updates are flexible and can do a lot. New features will be requested. Plan for that now.</p> ✅ Python may be good enough now, but what happens when you need to add a handful more threads, more features, deploy to other architectures, and need higher performance?</p> Here’s the thing – it is impossible to predict all the directions a product will take. Set yourself up now to be able to reuse what you’re doing.</p> 🔭 Product VISION == Smart technology choices</p> 🔄 When to update application dependencies? 2025-06-18T00:00:00+00:00 Most modern languages (Node.js, Python, Rust, Go, Zig, Elm, etc.) have their own package manager. So when should we update package dependencies? How about at the beginning of every PR cycle? I’m working on making this my standard practice when starting a new feature in an application – first, update all dependencies. There are several advantages to this flow:</p> Updating frequently means you are dealing with small changes.</li> Updating at the start of a dev cycle also gives you a standard time to do this. Otherwise, things like updating dependencies tend to get neglected.</li> Updating at the start of a dev cycle (vs end) gives you some time to exercise new modules during development before merging.</li> You automatically get security updates without having to track these too closely.</li> It feels right.</li> </ul> This also requires good tests, so that you can update with confidence. There might also be an issue with a team if multiple developers are updating dependencies at the same time. So in larger teams, perhaps one person or an automated process should do this. The key thing is that it does get done, and implement the needed automation to make this easy.</p> ➡️ Frontload the boring stuff 2025-06-16T00:00:00+00:00 Some examples:</p> Think about and write tests before</em> implementing a new software feature.</li> Automate deployment of a new service before</em> developing it.</li> Create eCAD libraries before</em> doing the schematic/layout work.</li> Document the architecture</a> before</em> implementing it.</li> Open a PR before</em> doing the work, and explain what is going to be done.</li> Create a Changelog</a> entry describing the benefit to the user</a> of what you are doing, before</em> writing the code.</li> </ul> In the end, you will go faster and do better work. Why? Because you are frontloading the tedious details that are difficult to find time for after</em> the work is done.</p> This is the difference between the prototyping and product development mindset.</p> 🌿 A natural way to get more testing/automation 2025-06-06T00:00:00+00:00 One thing I’ve noticed is when I release more frequently, I tend to automate. If releases only happen once every year, it is easy to neglect automation, CI/CD, etc. Automated tests are not needed as it is just quicker to manually test everything. But, with frequent releases, there is a subtle push toward scripting stuff, getting CI/CD running, etc. Instead of demanding more tests/automation, perhaps decide to release more often, and the automation will naturally follow.</p> 🌒🌓🌔The four phases of product development 2025-06-05T00:00:00+00:00 </p> Each phase is produced by someone or something. And the problems arise when we mix up who does what.</p> Great/innovative ideas usually don’t come out of a committee. Truth is a solo pursuit. Evaluate ideas in meetings, but don’t try to come up with them there.</li> When a team is involved, collaboration is required. The better your collaboration culture/workflow, the more efficient your team will be.</li> Manually building, testing, and deploying software is a waste of human time – leave that to machines when possible.</li> Developers are horribly inept at testing/evaluating their own work. Any outside viewpoint is useful – especially with user interfaces.</li> </ul> The faster you can do this loop, the faster you’ll learn, and the quicker you can build something useful.</p> ⏩ Get on with it ... 2025-06-04T00:00:00+00:00 While many great ideas and algorithms germinate in deep work and solo thinking, bringing a modern product (especially one containing software) to life require many iterations with feedback and learning. So lets propose a new release every week</a>. But, the common objection is: we can’t do a quality release every week? A monthly release seems more reasonable, then we have plenty of time to get it right.</p> You can’t build a great product until you’ve built 52 bad ones.* If you only do an iteration every month, then you are looking at 4.3 years to get to good. If you do an iteration every week, then it only takes 1 year.</p> So instead of pining away about how can I possibly build a good iteration every week, quit worrying about building a perfect release, and instead focus on getting through the bad ones as quickly as possible.</p> This is a parody on Jonathan Stark’s “Weekly vs. Daily”</a> post.</p> * This may be an exaggeration, but if we are honest with ourselves, it is often astounding how many revisions it takes to get something working, let alone great.</p> 👥 Who are users? 2025-06-03T00:00:00+00:00 To get really valuable feedback, we need users. If the product has not been released yet, find someone – external CI system, CEO, beta users, manufacturing people, accountants, your mom …</p> Often, when it leaves the developer’s computer, we find it does not even build, let alone run and do something useful!</p> A user is anyone</em></strong> other than the developer.</p> ♻️ How to define an iteration? 2025-06-02T00:00:00+00:00 I’ve found it useful to define an iteration as a formal release (software that is packaged and users use, a PCB that you build, a prototype, a web app that you deploy, etc.). You don’t learn many things until you build and, more importantly, others use - even if your users are only internal. Practice formal releases during development, and then the process will be polished when you go into production.</p> 🧑‍💻 How can we increase developer productivity? 2025-05-29T00:00:00+00:00 “The way you get programmer productivity is not by increasing the lines of code per programmer per day. That doesn’t work. The way you get programmer productivity is by eliminating lines of code you have to write.The line of code that’s the fastest to write, that never breaks, that doesn’t need maintenance, is the line you never had to write.” – Apple WWDC ’97 Steve Jobs Closing Keynote</a></p> </blockquote> How can we reduce the lines of code we need to write? Several ways come to mind:</p> Use OSS components</li> Build on someone else’s platform</li> Don’t implement “just-in-case” features</li> Removed unused code (linting tools can find this for you)</li> Don’t implement unnecessary abstractions</li> Simplify</li> Reuse what you have already done (YOUR Platform)</li> </ul> 📈 What is the quickest way to store and track data? 2025-05-28T00:00:00+00:00 Do you need a quick way to track some data? Serial numbers, part numbers, test results, defect metrics, etc.?</p> There are many ways, but one of the quickest and most flexible is CSV</a> files in Gitea. You are already using Git</a>, right?</p> With CSV files in Gitea, you can:</p> View the data as a table</li> Easily edit the data using a variety of tools (Libreoffice, Gitea web UI, etc.)</li> Track and review changes</li> Manage access/permissions</li> Work in a distributed fashion</li> </ul> Additionally, it is easy to add automation to CSV files in Go or Python. While databases can be very useful for larger datasets, they are often overkill for a lot of things today, as most datasets (especially human-generated ones) now easily fit in memory. And when it comes time to use a database, it is easy to import your existing CSV data into a database.</p> 🌱 OSS can decrease or increase your code liability 2025-05-27T00:00:00+00:00 A wise person once said</a>:</p> “Code is a liability, not an asset. Aim to have as little of it as possible.”</p> </blockquote> Whether the open-source code in your projects is an asset or a liability depends on how you handle it. Almost all code in complex systems needs to be maintained</a>.</p> If you do not update the OSS code in your projects, you are essentially forking it</a>, and it is now “your code” and you own it. It is a static entity that is not gaining value unless you put direct effort into it. Additionally, there will likely be issues that you now need to solve. Very few OSS project maintainers are going to be interested in helping you fix a problem in a 5-year-old snapshot.</p> Alternatively, if you keep the OSS code in your projects up-to-date, then many issues automatically get fixed (security, compatibility, performance, bugs, new features, etc.). Additionally, if there is a problem, then maintainers will likely work with you to fix it. In this scenario, OSS code is an asset – it is gaining much more value than your direct effort. However, you must commit to:</p> updating OSS components regularly</p> </li> having an automated way to comprehensively test the system</p> </li> engage with the OSS projects you use, build relationships, and contribute where you can (documentation and bug reports are contributions)</p> </li> </ul> Some effort is required on your part – it’s not a free lunch. However, if you do this right, the benefit-cost ratio (BCR) is high – well worth it in the long term</a>.</p> 🤝🔍 Trust, but verify 2025-05-23T00:00:00+00:00 Apparently, this is an old Russian proverb that Ronald Reagan popularized</a>.</p> This is good general advice in life – don’t automatically distrust people and go looking for problems, but when something does not look or feel right, verify.</p> As engineers, when we are building something new, this is important.</p> ✅ Are our assumptions correct?</p> ✅ Have we checked the edge conditions?</p> ✅ Did this change break anything else?</p> ✅ What are we missing?</p> As systems become more complex, two tools come to mind:</p> ▶ Simulation (a faster/cheaper way to verify)</p> ▶ Automation (testing, deployment, etc. – do things humans don’t have time for)</p> In complex systems, blind trust does not get us very far, especially in ourselves. As engineers, the work we should verify the most is our own.</p> 🌽 What is the easiest way to avoid silos? 2025-05-21T00:00:00+00:00 Silos occur when a person or group of people operate in isolation from the rest of the organization. One of the reasons you have failure in organizations is one silo does not know that the other silo is doing.</p> Silos are deadly in our organizations. How can we avoid them? The efficient way is to use open, transparent, and common systems where possible for:</p> Issue/project tracking</li> Part number management</a></li> Information storage (design files should be versioned in Git)</li> Releases</li> Change control and review (Pull requests)</li> Documentation</a> (Markdown, CSV, etc)</li> </ul> Software developers build really good workflow tools. Why, because they can build their OWN tools. For this reason, no matter what your discipline, I recommend starting with Git</a>. For small teams, this can get you most of the way there. For larger teams, you may need something more comprehensive like odoo</a>. But, start small with Git, cover the basics, and add more complex systems when needed.</p> 🤖 The biggest problem with AI 2025-05-20T00:00:00+00:00 “It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” - Mark Twain (maybe)</p> </blockquote> AI is really good at summarizing existing information. It is less good at thinking, although it can do some amazing things. The biggest deficiency I see right now is AI does not know how to ask questions and doesn’t know what it doesn’t know. It just makes assumptions and pretends like it knows what it is doing. People like this are some of the hardest ones to work with. They act like they know everything and never think they are wrong. Smart people, blind to their own weaknesses, make the subtlest, hardest problems to find.</p> Until AI is self-aware of its own weaknesses and knows how to ask questions, we cannot fully trust it to operate like a professional human. It needs watched very carefully. AI can still be very useful, and it will likely get better, but for now, I think this is the current state.</p> 🤖 What is AI good for? 2025-05-19T00:00:00+00:00 AI is really good at summarizing existing information – Perplexity Pro mostly has replaced Google/Duckduckgo, etc., and is worth every penny. AI is really good at finding API documentation – much quicker than searching through docs, and often provides a useful snippet of how to use an API. Occasionally, I’ll use AI to write a quick Python script where I don’t care how it works.</p> The most useful scenario is when you are having an intelligent back-and-forth conversation with AI, and you understand what is going on. What does not work is just lazily expecting it to do all your work so you don’t have to understand it – especially the hard parts like architecture. If AI is doing things that you don’t understand, ask it to teach you. Write tests so you can be sure it is right. Trust, but verify.</p> I’m currently working on a project with a smart intern working on the frontend code. He is working fast and appears to be vibe coding some of it. Some of it is a bit of a mess, but it mostly works. Some parts have obviously never even been tested. Is vibe coding in this case a net gain? Probably, but I’m not sure yet – will know more once we try to maintain this code.</p> On a project that is complex enough to require some architecture, coding is not the hard part</a>. Architecture, requirements, integration, testing, debugging, support … all this stuff that is required to get it right swamps any coding effort. As one person said</a>, “Code is a liability, not an asset. Aim to have as little of it as possible.”</p> At this point, don’t aim to have AI write all your code for you. Use AI to help you write better code faster that YOU understand.</p> 📝 What is my process for capturing experience? (part 3) 2025-05-16T00:00:00+00:00 Reflection is another important aspect of the experience capture process. Do we carefully evaluate and critically think about what happened? We don’t do this to blame ourselves or others, but rather to recognize areas where we might improve. Reflection can also include recognizing things that went well and what we need to keep doing.</p> Some examples:</p> Personally spend 5m journalling at the end of each day. What went well? What did not? What are we thankful for?</li> Blameless postmortems</a></li> Periodic project review – what have we learned? Is what we learned captured?</li> Pull requests – have we captured everything we have learned when we did this work? Are there documentation and tests so the work can be extended and reused?</li> Look for opportunities to automate. Automation is one of the best ways to capture experience.</li> </ul> If we do not reflect, we are destined to keep repeating the same mistakes and needlessly doing the same work over and over.</p> One of the most tragic and common forms of human nature is to just keep doing what we are doing, even if it is not working.</p> 📝 What is my process for capturing experience? (part 2) 2025-05-15T00:00:00+00:00 In part 1</a>, we discussed the importance of getting information into permanent mediums. Below are a some that I use:</p> Workflowy. This is my 2nd brain with 1000’s of notes, links, howtos, etc. It is easy to share Workflowy notes with others as needed and multiple people can collaborate on a note with just a link – no account needed. I can’t recommend Workflowy enough – it is so flexible and powerful.</li> Checklists</li> Templates (typically a Google Doc)</li> OSS projects Github</a></li> Simple IoT</a></li> Yoe Distro</a></li> GitPLM</a></li> </ul> </li> My daily newsletter</a></li> TMPDIR Community Forum</a></li> TMPDIR Podcast</a></li> Sketches</a></li> Youtube General</a></li> SimpleIoT</a></li> Yoe Distro</a></li> </ul> </li> Automation (Ansible, scripts, CI, CD, etc.)</li> </ul> I try to not directly email useful information if possible, but rather write it down somewhere permanent and then share the link.</p> 📝 What is my process for capturing experience? 2025-05-14T00:00:00+00:00 After yesterday’s post</a>, I was asked: “What’s your process to capture experience?”</p> The most important thing is to get information out of transient mediums into permanent mediums.</p> What are the attributes of a permanent medium?</p> Searchable</li> Persistent (exists over time, backed up)</li> Easily accessible to all relevant parties over time, even those who join late</li> Well organized</li> Has context</li> Improved over time</li> Turned into automation where possible</li> </ol> The first two attributes are the easiest. While email, chat, and meetings are useful in teams for communication and notifications, they are NOT permanent mediums for information. Even though they may be persistent and searchable, they fail on the other attributes.</p> 🧠 How to leverage experience? 2025-05-13T00:00:00+00:00 To leverage experience, we need to capture it. And to capture it, we need process. Without this, leveraging the experience of our best people is at best random. If we can capture experience, then we have it forever, even when our best people inevitably move on.</p> 🔭 How to encourage vision? 2025-05-09T00:00:00+00:00 Innovation is fueled by vision. Without vision, things can be directionless, drifting toward obscurity. So, how to encourage vision? Perhaps the first thing is not to discourage it. Ironically, the more successful an organization is, the more likely this is to happen. This happens when people mistakenly confuse experience with vision. In the realm of technology, how things were done yesterday, while a useful guide and foundation, will not get us to where we need to go tomorrow.</p> 🧠🔭 The tension of experience and vision 2025-05-08T00:00:00+00:00 🧠 Experience</strong>: Guardrails for our journey, temperament for our actions.</p> 🔭 Vision</strong>: The view of where we are going, the source of energy and enthusiasm for our enterprise.</p> Experience is negative, a restraint. Vision is positive, a motivator. Both are necessary in the right balance.</p> Again, dysfunction ensues when we mix these up.</p> Guardrails based purely on what we think will happen (vision) are not very effective.</p> Building a product only on what has worked in the past (experience) is not very compelling.</p> 🛣️ Two approaches to life 2025-05-07T00:00:00+00:00 There are two ways we can approach life:</p> Take responsibility and try to solve some of our own problems.</li> Expect others to solve all of our problems.</li> </ol> Some examples:</p> Owning the integration points</a> in our product vs. outsourcing everything.</li> Critically evaluate what we hear vs. blind trust.</li> Growing/raising some of our food vs. buying all of it.</li> Learning to find fulfillment in doing vs. allowing others to fulfill us by entertaining.</li> Making/building some of what we need vs. buying everything.</li> Running Arch Linux on our computers, where we control the system vs. an OS vendor.</li> Proactively learning about nutrition and health vs. total dependence on mainstream medicine.</li> Owning YOUR Platform vs. expecting someone else’s platform to solve all our needs.</li> </ul> In the short term, the first approach rarely makes business sense. However, long-term, we learn that the second approach is an illusion.</p> 🏆 The entire story 2025-05-06T00:00:00+00:00 In debates about technology, people bring up points</a>, such as an Arch Linux update occasionally breaking and not booting; thus, Arch Linux is bad.</p> While the above point is true, it is also irrelevant because it is not the entire story. What about:</p> The hours upon hours I’ve saved updating and installing stuff because pacman is so fast.</p> </li> The 100’s of packaged tools and applications that save me from installing them manually.</p> </li> The AUR, which has everything that is not already packaged, making it easy to install even the most obscure piece of software.</p> </li> The benefits of using the latest and best versions of everything. After all, if something is getting better, may as well benefit from it now vs later.</p> </li> The simplicity of PKGBUILD, which allows me to create and modify my own packages easily.</p> </li> </ul> I could go on, but you get the point. After running Arch for over 10 years, let’s conservatively say I’ve saved an hour a week by using Arch – that is 520 hours. I’ve spent maybe 5 hours working through broken stuff in Arch in 10 years. That is a benefit-cost ratio (BCR) of 104. So yes, the 5 hours are real, but they are irrelevant.</p> 👤 It's all about the people 2025-05-05T00:00:00+00:00 It is very true, we need great people in our organizations. And I have the utmost respect for those who can motivate and lead people. I also have the utmost respect for the capabilities of humans – that divine spark of creativity and the ability of the human brain to solve problems and make new connections.</p> However, to do much in this age requires people to work together and build on the work of the past. What one person does, another person needs to continue. The lone accomplishments of any one individual may be impressive, but without continuity and scale, there is likely to be very little lasting impact. Without systems in place (YOUR Platform), the brilliant work of your people will not be utilized to their full potential.</p> 🧠🔭 Experience and Vision 2025-04-29T00:00:00+00:00 In the realm of product development, there are two powerful forces – experience and vision.</p> Experience (the past) is a guide – a help to avoid serious blunders and mistakes. Experience can help us avoid dead-end tangents. Experience gives us a gut feel that is critical at times. Experience helps us be more efficient if we can make fewer mistakes.</p> Vision (the future) is where we are going. It is seeing things that have not been done before. It is exploring new combinations of technology to build compelling products. It is adopting new technology, including tools, software, components, and workflows. It is about understanding the needs of your customers.</p> Without experience, we are on the path to collapse.</p> Without vision, we are on the path to obscurity.</p> 📊 Past performance ... 2025-04-28T00:00:00+00:00 “past performance is not indicative of future results”</p> </blockquote> The above phrase is a disclaimer we often see associated with investments.</p> However, this applies to more than investments …</p> A technology that was cutting edge in the past may not be the best option for the future (example building web applications in PHP vs Go).</li> People who did good work in the past usually continue to do good work, but not always. (people can lose interest, suffer burnout, have personal problems, etc.)</li> Workflows used in the past are not always the best option for the future (example, requiring patches on mail-lists will probably alienate a significant portion of the developer population who were raised on pull-requests).</li> As the level of integration in integrated circuits continues, the IC we used in the past may not be the best solution compared to more integrated solutions available in the future.</li> </ul> How we proceed in the future needs to be a combination of what we learned in the past (experience) and the technology available in the future (vision). Again, dysfunction arises when we mix these two modes up.</p> ✔️ Legacy code 2025-04-25T00:00:00+00:00 Michael Feathers defines legacy code as code without tests. He further elaborates in his book</a>:</p> “Code without tests is bad code. It doesn’t matter how well written it is; it doesn’t matter how pretty or object-oriented or how well-encapsulated it is. With tests, we can change the behavior of our code quickly and verifiably. Without them, we really don’t know if our code is getting better or worse.”</p> </blockquote> Without automated tests, any modification to the codebase is risky, as there is no reliable way to know if a change introduces defects. This lack of feedback forces us to rely on manual testing or institutional knowledge – often lost over time – which makes maintenance difficult and error-prone. It is not about the age of the code or who wrote it, but about whether the code can be changed safely. If you cannot verify behavior through tests, the code is “legacy” regardless of its age or origin.</p> 🚀 Git driven CD -- you know what got deployed 2025-04-24T00:00:00+00:00 Git driven CD (Continuous Deployment) typically means whenever you merge to main</code> , or tag a repo, something automatically gets deployed. It can be something as simple as a static website being updated.</p> One of the biggest advantages to this is that you know for sure the code was checked in and tagged. And you know exactly what code was deployed. There is no manual process involved to mess it up.</p> This may not seem like a big deal, but it is. As humans, we get lazy and forget to do the simple things like manually tagging a repo – especially on small teams where process is not emphasized. However, CD is a way to have a process without having a process. You do it in the name of saving time, but it also gives you enough process that things are now also consistent.</p> 💸 Technical debt 2025-04-23T00:00:00+00:00 I like the definition of technical debt from this article:</a></p> Technical debt is anything that slows down your ability to update and upgrade your work</p> </blockquote> 🛠️ Another reason to self-host Gitea 2025-04-22T00:00:00+00:00 You own your data – forever. You don’t need to keep paying monthly fees if you decide to switch to a different system just to access your historical data and conversations.</p> I’ve been advocating that development discussions should happen in Git</a> issues/PRs/projects whenever possible, so the conversations have context. One requirement for this to work long term is that you need access to these conversations for the duration of a product, and perhaps beyond.</p> There is a hosting cost, but it is minimal ($5/mo on a Linode server). There is also some ops cost to keep things running and up to date, but in my experience, Gitea is cake to self-host compared to things like WordPress or Email. The benefit-to-effort ratio is huge.</p> 👷 The tension between solo and collaborative work 2025-04-21T00:00:00+00:00 As systems become more complex, we emphasize collaboration more and more. But we must also remember that truth is still largely a solo pursuit. Truth by committee is almost an oxymoron. The new algorithms, the keen insights, and the disruptive inventions are usually not the work of a committee, but rather a lone person exploring the boundaries of knowledge, careful observation, and making new connections from what already exists. And really, invention is not anything new, but rather a discovery of the universal truths that exist in our universe – the laws of nature.</p> Increasingly, we need to be good at both – the focus and solitude required for deep work, and the collaboration required to build complex systems. The great ideas will generally come from the former, but the ability to do anything useful with them requires the latter.</p> Dysfunction arises when we mix these two modes up.</p> 🕹️ What can we control? 2025-04-18T00:00:00+00:00 There are some things we can control:</p> Products have a robust field update mechanism.</li> Automated testing is in place.</li> We have automated deployments.</li> We have processes to quickly and confidently make and release hardware changes.</li> </ul> And some things we can’t control:</p> A customer asking for a new feature we did not anticipate.</li> Bugs showing up in the field.</li> Security problems in a dependency.</li> Cloud vendor infrastructure problems.</li> Supply chain issues (have we forgotten COVID already)?</li> </ul> But if we do the first list, then we are prepared to handle the second. However, we tend to focus most of our energy on the 2nd list (onerous contracts with vendors, extensive manual testing of the “final” release, etc.) and think we are controlling the situation, when in reality we are not – it’s just an illusion. Really, it is change that we fear, as change is associated with a lack of control. But it is change that gives us control.</p> 😩 Boring, error-prone, repetitive, tedious work 2025-04-17T00:00:00+00:00 Some examples:</p> manually testing software</li> manually constructing a bill of materials</li> searching your email for some bit of information</li> scrolling through logs looking for potential problems</li> manually configuring a new server</li> manually deploying software updates</li> manually checking for spacing violations in a PCB layout</li> building physical prototype circuits and trying a range of different components</li> organizing parts or physical materials without an easy way to identify or sort them</li> </ul> If this describes what we are doing, there may be an opportunity for workflow or automation improvements. This is the kind of work computers do best – to free us up for what humans do best.</p> 🧑‍💻 Will AI cause coding to disappear? 2025-04-16T00:00:00+00:00 AI is not a binary off/on thing, like many of the industry commentators like to think. Many of us are already using AI to improve our productivity. For me, Perplexity has largely replaced Google search.</p> Technology has always brought productivity gains: ASM -> C -> Code generators -> Libraries -> OSS -> Google search -> Zephyr -> etc. AI is just another step.</p> Coding has always been the easy part. Requirements, innovation, collaboration, architecture, integration, debugging, etc. – these are the hard parts. There will always be a place for skilled humans … people with intuition, creativity, experience, taste, vision, and style. People who see and can do what does not already exist.</p> If all you do is translate well-defined requirements into code, then yeah, I’d be worried. However, there is a lot more that needs to be done that is of much higher value, and not enough people to do it. Step up!</p> Survey of device ID mechanisms 2025-04-15T00:00:00+00:00 Are you implementing an IoT system that requires you to identify many connected devices? This post reviews different types of IDs and provides some technical details to help you decide if any of these are appropriate for your application.</p> </figure> Note: some of the below information was generated by Perplexity and may not be 100% accurate. The goal of this article is to get a sense of all the different types of IDs in use. To implement anything, use the official documentation.</p> UUIDs</h2> UUIDs (Universally Unique Identifiers) are often used in distributed software systems where multiple distributed systems create objects that must be uniquely identified. UUIDs are typically 128-bit numbers represented in a 36-character format that might look like: d50ae174-b65f-4ab9-b938-5f99b67ac50b. Base64 encoded UUIDs can be as short as 22 characters and look like: 1QrhdLZfSrm5OF-ZtnrFCw. UUIDs are typically constructed using the time and the MAC address of the device generating the UUID.</p> These IDs are attractive because no central organization or database is required to prevent conflicts - they can be created anywhere at will with a statistically insignificant possibility that there will ever be a collision. However, these IDs are large and best processed by machines, not humans.</p> MAC Address/EUI-48</h2> Network devices use a MAC (Media Access Control) address, also called (EUI-48), to uniquely identify a device on a network. These are 48-bit numbers typically displayed as 12 hexadecimal digits and grouped into 6 pairs (00:1A:2B:3C:4D:5E</code>). The first half of the MAC address is called the OUI (Organizationally Unique Identifier). IEEE is the registration authority</a> for MAC address blocks. You can purchase 4K, 1M, and 16M size blocks; the cost ranges from $1,080 to $3,680 at the time of writing.</p> You can also purchase EEPROM integrated circuits, like the 24AA02E48</a> from Microchip, that have a EUI-48 number already programmed into it. This is a nice option for lower volume products where you don’t want to mess around with managing the EUI-48 numbers.</p> One advantage of using a MAC address is that it may already be programmed into the device you are using, so it does not require any extra effort to allocate and program the ID.</p> Bluetooth Low Energy (BLE)</h2> BLE uses “MAC” (EUI-48) hardware addresses from the same pool as Ethernet. Additionally, the following IDs are used:</p> Identifier Type</th> Size</th> Purpose/Usage</th> Example</th></tr></thead> Bluetooth Address (EUI-48)</td> 48 bits</td> Device-level unique ID (public or random)</td> 12:34:56:78:9A:BC</td></tr> UUID (16/32/128-bit)</td> 16/32/128b</td> Service/characteristic identification</td> 0x180D, 12345678-...</td></tr> Company Identifier</td> 16 bits</td> Manufacturer/vendor identification</td> 0x004C (Apple)</td></tr> System ID (Device Info)</td> 64 bits</td> Encodes device's MAC address (optional)</td> (varies)</td></tr> Local Name</td> Variable</td> Human-readable device name</td> "Charge HR"</td></tr> Appearance/Class Codes</td> 16 bits</td> Device type/classification</td> 0x0340 (Heart Rate Sensor)</td></tr></tbody></table></figure> Bluetooth Mesh</h2> Bluetooth Mesh networking introduces several unique identifiers (IDs) at various layers of the protocol to manage devices, messages, and network security. These IDs serve different purposes than traditional Bluetooth or BLE device addresses.</p> Unicast Address (Node Address/Element Address)</strong></p> Each node (device) in a Bluetooth Mesh network is assigned a unique unicast address</strong> during the provisioning process. This is a 16-bit value that uniquely identifies the node or its elements within the mesh network.</li> A node can have multiple elements, each with its own unicast address in a contiguous range.</li> </ul> </li> Group Address</strong></p> 16-bit values used to address multiple nodes simultaneously (e.g., all lights in a room). Nodes can subscribe to one or more group addresses, enabling efficient publish/subscribe communication.</li> </ul> </li> Virtual Address</strong></p> 128-bit values derived from a UUID. They allow addressing of a logical group of nodes, regardless of their physical location or provisioning sequence. Virtual addresses are mapped to a 16-bit address space for use in messages.</li> </ul> </li> Device Key (DevKey)</strong></p> A unique 128-bit cryptographic key assigned to each node during provisioning. It is used for secure communication between the provisioner and the node, especially for configuration and management messages.</li> </ul> </li> Network Key (NetKey) and Application Key (AppKey)</strong></p> Network Keys</strong> are shared among all nodes in a mesh network and are used to secure network layer messages.</li> Application Keys</strong> are used to secure application-specific messages, allowing for separation of different functionalities (e.g., lighting vs. sensor data).</li> </ul> </li> Model Identifier</strong></p> Each mesh model (defining device behavior, such as a light switch or sensor) is assigned a unique identifier:</li> SIG-adopted models</strong>: Identified by a 16-bit model identifier.</li> Vendor-specific models</strong>: Identified by a 16-bit vendor ID and a 16-bit model ID.</li> </ul> </li> Message Sequence Number</strong></p> Every message sent by a node includes a sequence number</strong> to prevent replay attacks and ensure message uniqueness.</li> </ul> </li> Device Property ID</strong></p> For sensor and property models, each property (e.g., temperature, humidity) is assigned a unique Device Property ID</strong> as per the Bluetooth Mesh device properties specification.</li> </ul> </li> Node ID</strong></p> Some documentation refers to a Node ID</strong> (or nodeId), typically a 2-byte value, as a unique identifier for a mesh device within the network, used for addressing and control.</li> </ul> </li> </ol> IPv6/EUI-64</h2> IPv6/EUI-64 addresses are a 64-bit number commonly used in IPv6 and typically displayed as 8 groups of four hexadecimal digits: 2001:0db8:85a3:0000:0000:8a2e:0370:7334</code>. Consecutive groups of zeros can be replaced with a double colon. For example, 2001:0db8:0000:0000:0000:0000:0000:0001</code> can be shorted to 2001:0db8::1</code>. Note, leading zeros in the group are also omitted. IPv6 address blocks are assigned by the Internet Assigned Numbers Authority (IANA), which allocates large blocks of addresses to Regional Internet Registries (RIRs). These RIRs then distribute smaller blocks to Internet Service Providers (ISPs) and other organizations. The ARIN (American Registry for Internet Numbers) provides blocks of IPv6 addresses. The annual fee for a 3X-Small block (288 addresses) is $262.50 at the time of writing.</p> There are also EEPROM and other ICs that have EUI-64 numbers preprogrammed, such as the Microchip 24AA02E64</a>.</p> QR Codes</h2> QR codes provide the opportunity to easily read a UUIDs or some other long ID on every device using a phone camera. This relieves some of the complexity of humans having to deal with long IDs.</p> Entering a known device in the field into an application is easy with QR codes. However, finding a device (one among many) in the field can be difficult unless the application has a feature designed just for this.</p> </figure> There is no central registry for QR codes. Instead, the process of QR code generation and use is decentralized and relies on the vast number of possible QR code combinations to avoid conflicts.</p> SNMP/Private Enterprise Numbers (PEN)</h2> A SNMP Private Enterprise Number (PEN)</strong> is a unique identifier assigned by the Internet Assigned Numbers Authority (IANA) to organizations, individuals, or entities. It is primarily used in network management protocols like Simple Network Management Protocol (SNMP) to identify objects defined by an enterprise within the Management Information Base (MIB) structure.</p> PENs are part of the Object Identifier (OID) hierarchy under the prefix 1.3.6.1.4.1</code>, which corresponds to .iso.org.dod.internet.private.enterprise</code>. This prefix ensures that each PEN is globally unique.</p> Any entity can request a PEN assignment from IANA at no cost</a>.</p> Automobile VIN</h2> The modern 17-character Vehicle Identification Number (VIN) format was first conceptualized and standardized by the United States National Highway Traffic Safety Administration (NHTSA) in 1981. This standardization aimed to unify the previously inconsistent vehicle identification practices across manufacturers and facilitate more effective tracking of recalls, registrations, and thefts. The VIN system introduced by NHTSA was later adopted globally and is now governed by the International Organization for Standardization (ISO), specifically under ISO 3779 (content and structure) and ISO 4030 (location and attachment)</p> An example of an automobile VIN (Vehicle Identification Number) is:</p> 1P8ZA1279SZ215470</strong></p> VINs are always 17 characters long and use a combination of capital letters and digits. Each section of the VIN provides specific information about the vehicle:</p> The first three characters (World Manufacturer Identifier) indicate the manufacturer and country of origin.</li> Characters 4–8 describe features such as model, body type, and engine.</li> The 9th character is a check digit for security.</li> The 10th character indicates the model year (e.g., “S” for 1995, “R” for 1994 or 2024).</li> The 11th character shows the assembly plant.</li> The last six characters are a unique serial number for that vehicle.</li> </ul> KNX Serial Number</h2> KNX is a popular building automation standard. The KNX serial numbers</a> follow the following format:</p> Length:</strong> The KNX serial number is always 6 bytes (octets) long.</p> Manufacturer Code:</strong></p> Octets 6 and 5</strong> (the first two bytes, when read from left to right) contain the official KNX Manufacturer Code assigned to each manufacturer by the KNX Association.</li> </ul> </li> Device-Specific Portion:</strong></p> Octets 4 to 1</strong> (the remaining four bytes) are managed by the manufacturer.</li> This part may encode information such as production site, date, series, batch, or simply a sequential unique number.</li> </ul> </li> </ul> BACnet</h2> BACnet Device IDs, also known as Device Object Identifiers or Device Instance Numbers, are unique numerical values assigned to each BACnet device on a network to identify them distinctly.</p> The Device ID is a 22-bit unsigned integer ranging from 0 to 4,194,302 (decimal). The value 4,194,303 (0x3FFFFF) is reserved and cannot be used as a valid Device ID.</p> This Device ID corresponds to the object instance number portion of the BACnet Device Object’s Object_Identifier property.</p> The full Object_Identifier is a 32-bit value composed of:</p> Object Type (10 bits): For a Device object, this is fixed as 8.</li> Object Instance (22 bits): The Device ID itself.</li> </ul> BACnet IDs are not assigned during manufacturing, but rather during setup/installation.</p> Matter</h2> Matter devices include a Device Attestation Certificate (DACs) generated through a hierarchical Public Key Infrastructure (PKI) process, ensuring each device is uniquely authenticated. Each device is cryptographically verified, making spoofing difficult.</p> The DAC is part of a chain of trust:</p> Product Attestation Authority (PAA):</strong></p> Root certificate issued by the Connectivity Standards Alliance (CSA) or an approved third party.</li> Stored in the Distributed Compliance Ledger (DCL), a global registry of trusted certificates.</li> </ul> </li> Product Attestation Intermediate (PAI):</strong></p> Intermediate certificate signed by the PAA.</li> Typically includes the manufacturer’s Vendor ID (VID).</li> </ul> </li> Device Attestation Certificate (DAC):</strong></p> Unique per-device certificate signed by the PAI.</li> Contains the device’s VID, Product ID (PID), public key, and serial number.</li> </ul> </li> </ol> Additionally, devices use rotating keys, which makes tracking difficult and preserves privacy. End users scan a QR code to onboard devices.</p> Matter attempts to be:</p> Authentic (not counterfeit)</li> Secure (resistant to hacking)</li> Private (resistant to tracking)</li> Interoperable (works with any ecosystem)</li> </ul> Serial Numbers</h2> Other custom serial numbers take many different forms. Some examples are shown below:</p> Format Type</th> Example</th> Description</th> Typical Use Cases</th></tr></thead> Sequential</td> D000001</td> Incremented by one, often padded with zeros</td> Manufacturing, inventory</td></tr> Alphanumeric</td> TBL-101A234</td> Letters and numbers, often with category or model encoded</td> Electronics, equipment rental</td></tr> Random</td> X9F4-7T32-ZY6</td> Unpredictable, generated by algorithm</td> Security-sensitive items</td></tr> Date/Batch Embedded</td> 20230715-001-123</td> Encodes date and batch with sequence number</td> Quality control, traceability</td></tr></tbody></table></figure> Systemd network interfaces</h2> Systemd’s predictable network interface name encodes the physical path of the device. Example of how enp0s13f0u2c2</strong> is constructed:</p> en</strong>: Ethernet device (prefix for Ethernet interfaces).</li> p0</strong>: PCI bus 0</li> s13</strong>: PCI slot 13 (device 13 on the bus)</li> f0</strong>: PCI function 0</li> u2</strong>: USB port 2 (the device is connected via USB, possibly a USB Ethernet adapter)</li> c2</strong>: USB configuration or child port 2 (sometimes used to indicate a specific port or configuration on a USB hub)</li> </ul> Summary</h2> I’m sure there are many other examples – please leave a comment if you’d like to share other interesting formats. In a future post, we’ll explore a new standard for device IDs that is optimized for human interaction in the field. Subscribe below to stay informed of the latest developments.</p> ✅ Guarantees 2025-04-15T00:00:00+00:00 Sometimes we want guarantees:</p> Are you sure this will work?</li> Can we lock this down to a rock-solid stable release?</li> Is it fully tested?</li> Will this vendor be able to provide this part forever?</li> Do we have all the features the product will ever need?</li> </ul> The reality is that in complex systems, nothing is guaranteed. There will be problems, especially during integration and new use scenarios. No supplier is guaranteed. We can’t anticipate all the use cases of a complex system.</p> So the more important questions are:</p> Can we start integration and testing earlier?</li> Do we have automated testing?</li> Can we easily update deployed systems?</li> Can we pivot to new components as required?</li> Do we have processes and workflows in place to quickly and confidently make any needed changes (hardware or software)?</li> </ul> 🔄 The 2nd key to effective process 2025-04-14T00:00:00+00:00 If the people at the top don’t use the process, hardly anyone else will either. What we do often speaks way louder than what we say.</p> (1st key …</a>)</p> 🛠️ Git* -- the new standard for workflow 2025-04-11T00:00:00+00:00 It started with Github, a place to host your OSS projects. Over time, several other options emerged, and each has its place:</p> Github</a>: excellent for OSS projects where you want to attract the most developers.</li> Gitlab</a>: great for a large hosted enterprise solution.</li> Gitea</a>: a great solution for small/midsize teams building products. Can be run on a $5/mo Linode or Digital Ocean cloud server.</li> Forgejo</a>: is a fork of Gitea focused more on community and security.</li> </ul> I’ll call the above Git</strong>. These tools set a new standard for workflow integration</a>.</p> 🛠️ The defining characteristic of a good workflow tool 2025-04-10T00:00:00+00:00 As mentioned in the previous post</a>, modern workflow tools provide efficient access to information, collaboration, and automation.</p> The real power is when these three things come together</strong> in one environment. The defining characteristic of a good workflow tool is … integration</strong>.</p> There are powerful project board tools like Jira or Trello. There are great communication tools like Discourse, Discord, Slack, and Email. There are popular code review tools like Gerrit. But good things happen when you bring all these things together into one workflow tool, which I will call Git: Github, Gitlab, Gitea, or Forgejo. Then everything has context, and relationships between different types of information are easy to create. Integration in a tool is a powerful thing where the overall value is much greater than the sum of the individual features.</p> 🛠️ The critical features of workflow tools 2025-04-09T00:00:00+00:00 Tools are what enable knowledge work in the information age. Several critical features:</p> Efficient access to information</li> Collaboration</li> Automation</li> </ol> We’ll discuss more in the future posts …</p> How to implement Zephyr web applications 2025-04-08T00:00:00+00:00 Are you considering implementing a web UI in your Zephyr RTOS</a> application and perplexed by the endless number of web technologies available today? This article traces my journey through trying several options and presents some of the tradeoffs.</p> </figure> Modern microcontrollers (MCUs) have many connectivity options (Ethernet, WiFi, Bluetooth, Cellular, etc.), and when coupled with the Zephyr OS, implementing a web application on these devices often makes sense. Once you start doing this, you soon realize that this is not Ruby-on-Rails. Rather, Zephyr is still a very constrained environment (see my article on the differences between MPUs and MCUs</a>). While we can still leverage modern web technologies, we need to be selective because of the constraints of MCU systems. An example of a web UI in the Zephyr SimpleIoT project</a> is shown below:</p> </figure> Primary concerns in an MCU system</h2> Size</h2> The memory resources on MCU and MPU systems are vastly different</a> – a difference of over 1,000x.</p> </td> MCU (Internal)</strong></td> MCU (External)</strong></td> MPU (Server)</strong></td></tr> Flash</strong></td> 0.5-2MB</td> 16MB</td> 500GB</td></tr> RAM</strong></td> 0.25 - 1MB</td> 16MB</td> 16GB</td></tr></tbody></table></figure> While it is possible to use larger memory devices with MCUs, like an SD card or SDRAM, the above table is for the more common scenarios.</p> If we want to serve the web application entirely from a Zephyr device, then the web assets must fit in flash. The RAM size also limits the number of connections we can handle and the size of the data payloads we can transmit and receive.</p> Stability (includes maintainability and security)</strong></h2> The constraints of an MCU drive us toward doing more in the browser (frontend) because resources there are free and plentiful. This typically ends up being a Single-Page Application (SPA) architecture, where the web application is a free-running application that fetches and sends data to the MCU (backend). All rendering happens in the browser instead of the MCU. Because of this, the web application needs to be stable. Extra care must be taken so that the application does not crash and freeze or pop up annoying messages telling the user to reload the page, which can easily happen with JavaScript.</p> We also need to consider how the application is hosted. A typical web application is hosted in the cloud in one place. If there is a problem, we have complete control over updating and fixing it as needed. Embedded applications are a completely different matter – there may be thousands of devices. If there are problems, it is not as simple as updating a single server. We may have a firmware update mechanism in the device, but it is often up to the user to decide when to update. Users expect embedded devices to be stable. Many of them are used in critical applications. An instability in the web UI of the device does not give a positive impression, even though it may only be a web frontend crash, and the device is still functioning fine.</p> </figure> It seems simple to write an initial prototype, but maintaining a web application over time is a different matter. Security can also be a concern as many web frameworks pull in mountains of JavaScript dependencies via NPM, which are difficult to audit for security problems.</p> State Management</strong></h2> One of the hardest parts of a web application is maintaining state. The reason for this is that user interfaces can be deeply nested and complex. The naive approach is to distribute the state into each UI component. The difficulty with this is keeping everything in sync. There has been a continual stream of new frontend state management libraries (Redux, MobX, VueX, React Context, XState, react-easy-state, etc.). This is not an easy problem.</p> Options</h2> Serve site/assets from external server</h2> One way to work around the asset size constraint in MCUs is to serve large assets (like a Bootstrap CSS file) from an external server.</p> </figure> There are several potential issues with this:</p> ❌ The external asset must always be available, and if it goes away or changes, then the web application on the device must be updated to point to the new location.</p> ❌ If the entire site is served from an external server, then you must host all versions of the web application or be very careful to maintain backward compatibility to all previous versions of firmware running on the Zephyr device.</p> ❌ You may need the Zephyr UI to function if the Internet is down or unavailable (example: air-gapped secure setups or during device setup). In many systems, device state is mirrored to an upstream server, so the only time you would use the local UI is during setup or when the Internet is down.</p> ✅ The most reliable way to host a Zephyr web application is on the device itself.</strong> You certainly have many more options if you host it externally, but there are tradeoffs that must be considered.</p> No Build, just HTML, JS, CSS</h2> DHH and others are advocating No-Build</a> web solutions. However, they have Rails for a backend, and we have Zephyr. I investigated this approach, but manually mutating HTML elements in JavaScript in a complex app is messy and scattered – you’ll eventually create a framework anyway, so you may as well start with something close to meeting your needs.</p> </figure> HTML/JS/CSS development is powerful and flexible, but is tailored to design, not programming. In its raw state, it is like developing a program where everything is a global variable. If you have a dynamic web application responding to real-world events, this quickly becomes a programming problem that is difficult to manage with 200 global variables. You need an abstraction on top of this. Again, a powerful templating engine on the backend helps a lot, which we do not have in Zephyr applications.</p> React, Vue, Angular (non-compiled frameworks)</h2> Solutions like React are too large for many MCU applications as they typically require a megabyte or more of web assets to be transferred from the host. Additionally, you have the issue of JavaScript crashes and the continual bloat and churn of the NPM package ecosystem. Updating to new versions of packages can be a lot of work as APIs sometimes change. While this solution can work for large-scale cloud-hosted apps, it is not ideal for embedded web UIs where you often don’t have control over the device and can’t easily patch the system if there is a problem.</p> </figure> I’ve been writing/maintaining a 50K SLOC Meteor/React app for the past 10 years – it works and has been very successful. However, we’ve had a handful of frontend crashes. Being a cloud app, we were able to deploy fixes for this quickly, but it is still an inconvenience for users. In another project developed by a very experienced JavaScript developer, we also had a number of JavaScript crashes in the frontend. There are likely ways to reduce or avoid these stability problems, but it is not inherent in the system/language. It requires a high degree of carefulness and discipline. It is not easy to refactor JavaScript and know that you have fixed all the issues caused by your code change.</p> </figure> There are slimmer solutions like Preact which may solve the size issue, but you likely lose some of the integration with all the libraries written for React.</p> htmx</h2> htmx is the cool kid on the block right now – supposedly simple and easy to use. I tried this for the first iteration of a Zephyr web application and got a prototype running, but concluded it works best if you have a powerful backend that can do most of your rendering, and the frontend just handles a little bit of reactivity. Additionally, with htmx, you typically open up a network connection for every widget to get updates, and this requires a lot of connection resources on the backend, which are expensive in a Zephyr system.</p> </figure> Svelte</h2> Svelte is a compiled language that generates small assets, so it works well from a size perspective. It also appears to be a well-maintained project with good tooling. However, there are two downsides: 1) it is still JavaScript, so you have potential stability issues when maintaining applications long-term. 2) It puts the “JS” in the “HTML” instead of the “HTML” in the “JS”</a> (like React or Elm). This is largely a matter of preference, but as a programmer, I prefer the latter. The success of React indicates that many others do, too.</p> Functional Languages</h2> Functional languages are becoming more common in frontend development because they solve many reliability and refactoring challenges. Some options include ClojureScript, Elm (discussed below), PureScript, ReasonML, F#, Scala.js. Some of these options likely have smaller asset sizes than non-compiled options.</p> Elm</h2> Elm</a> is a compiled language that produces small assets. It is like React JSX in that it puts the “HTML” in the “Elm</a>“. Elm is a functional language that solves many of the challenges with web programming, including stability and managing state. Runtime errors in an Elm application are very rare. I wrote the Simple IoT UI in Elm, which is now 18,610 lines of code, and have maintained it since 2018. It has never crashed, and it is easy to add new features. Code density/reuse in Elm is very high. If it compiles, it generally works, so very little time is spent in the browser console debugging stuff. The elm-ui</a> package offers a powerful style and layout system that eliminates most of the pain with web programming and allows someone (like me) who is not a dedicated web developer to maintain a pretty nice web application without learning the intricacies of CSS.</p> </figure> As you maintain an Elm project, you’ll find that the language pushes you to a cleaner design. It nudges you in the right direction. The opposite generally happens in JavaScript projects, where a lot more discipline is required to keep things from becoming a mess.</p> Elm allows for creating powerful functions like the following for a text input form:</p> textInput Point.typeBitRate "Bit rate" "250000" </code></pre> This line of code renders a form element and automatically posts a new point to the backend of type BitRate</code> when the content is changed, nothing else in the system needs to change for this to happen. There are no “handlers” that need to be attached to this element. Elm also has some of the best tooling in this industry – elm-review, elm-land, etc. This tooling is largely made possible by the language.</p> Elm is a small project and community, but it is a stable/friendly community that has existed for many years, and many companies are successfully using Elm</a>. The Elm package system is fairly comprehensive and has most of what you might need to implement web apps. Since Elm packages are pure Elm, the entire ecosystem is stable and relatively secure</a>. There is some concern about the future of Elm as the compiler does not get frequent updates. Some have produced new versions of the compiler to meet special needs. The packages and tooling around Elm are where most innovation is happening right now, and it is phenomenal. Elm got it mostly right, so there is not much that needs immediate improvement in the core.</p> Summary</h2> There is no silver bullet in technology. With any choice, there are tradeoffs. Most people stick with mainstream JavaScript for frontend programming as the “safe” approach. However, in Zephyr/MCU-based systems, there are constraints that we don’t face in server environments.</p> There are significant challenges in the HTML/JS/CSS world. While it is nice to have options, the continual churn of new frameworks (jQuery -> Backbone -> Ember -> Angular -> Meteor -> React -> Angular2 -> Vue -> Svelte -> Next.js) (and many others) indicates there is some fundamental problem with the Javacript approach that has not been solved in a general way. The number of NPM package dependencies pulled into a typical project is staggering and nearly impossible for small teams to audit for security and longevity. NPM installs routinely break for various reasons. NPM suffers from some of the same problems as Python, as many packages compile C code in non-standard ways during the package install process, which can be fragile.</p> There is also the issue of what might be viewed as an impedance mismatch – what the technology mega-companies produce and use is not always the best fit for small companies/teams with limited resources. Facebook faces different problems on a different scale than a small embedded team. While React is right for them, we can’t assume it is right for everyone. Most of us are not facing Google-scale problems</a> that require a system as complex as Kubernetes. Ansible and a few servers work just fine. Facebook has near-infinite resources. They have ways to deal with the inherent stability problems of JavaScript. A small company/team does not have these resources, so different technologies like Elm may be a better fit.</p> When faced with the need for a Zephyr web application, I tried htmx first, then looked at “no build”. Neither of these worked very well for me. Having worked on decent-sized React and Elm projects, Elm looked like a better fit for the next try due to asset size, reliability, and maintainability. Additionally, I had already solved the problem of how to effectively deal with data in a distributed IoT system</a>, and had frontend code already written</a> to interface with this dataflow in Simple IoT. It was easy to port the Simple IoT frontend to a smaller version tailored for MCUs, and thus far it seems to be working well. Check out the Simple IoT Zephyr Networking Example</a> to learn more.</p> ✉️ Email filters 2025-04-08T00:00:00+00:00 Email is an amazing technology – the only widespread decentralized communication tool we have. But most of us are flooded with email. The fundamental problem is when we get emails from so many different sources, projects, and contexts flowing into one inbox. The context switching as we go from one email to the next is very expensive.</p> Email filters which automatically put emails into folders can help tame this problem. Gmail already tries to do this automatically with their Primary, Promotions, Social, and Updates. However, we can also do things like the following:</p> filter emails for every project/context into a separate folder. You can trigger of senders, subject, etc.</li> filter notifications from workflow tools (Github, Trello, etc.) into folders. If there are separate projects, then create separate folders for each project.</li> filter email newsletters into two folders: news-high-priority, and news-low-priority – the former is the ones you want to read, and the latter you just browse occasionally.</li> create a “personal” folder that emails outside of work get filtered into.</li> </ul> This is automation that makes information easier to process. Of course some emails will still slip into your inbox, but for most of our email, everything is grouped into a single context. This greatly improves our efficiency in processing them.</p> The critical thing is to use an email client that makes it easy to create filters. Thunderbird works pretty well</a>.</p> 👥 Everybody Works and the impact on meetings 2025-04-07T00:00:00+00:00 In a recent podcast titled Everybody Works</a>, the owners of a 60-person company discuss the impact of full-time managers and one of them is more meetings:</p> And the primary problem with management is when it’s full time and it’s only that it can’t do anything by itself. So the way 40 hours are manifested is through other people. It’s, hey, can you join this meeting? Hey, can we do a one-on-one? Hey, can we do some planning? Hey, can we do? Every bit of it is something that pulls someone who’s actually working on things away from that work and into the managerial orb. And I think we just finally came to the realization we did not need that. We do not need that at our scale, that we can supplement the work that we’re doing with a handful of managerial practices or protocols, things that don’t even require human to run.</p> </blockquote> The following article also discusses the subject:</p> We once more have no full-time managers at 37signals</a></p> 👥 Efficient meetings 2025-04-04T00:00:00+00:00 There are a lot of good ideas for effective meetings:</p> Whose meeting is this? A simple checklist</a></li> Who has time for meetings?</a></li> </ul> There are good reasons to have a meeting (hanging out, brainstorming, setting priorities, making decisions), but in this age sharing information is probably not one of them. People should have access to information before the meeting. The goal of the meeting should also be known ahead of time. It is like planning your day the night before — your subconscious has time to work on it and you are prepared. Prepared people are efficient.</p> 👥 Meetings and human relationships 2025-04-03T00:00:00+00:00 Improving human relationships is sometimes used as a justification for meetings, and face-to-face meetings can certainly improve human relationships.</p> But more generally, we can say that positive interactions improve human relationships. And the more positive interactions we have, the stronger our relationships will be.</p> Modern workflow tools provide a tremendous opportunity for human interactions at a scale that is impossible to accomplish with meetings. They provide transparency. The stream of activity is visible to all, and anyone can participate (or ignore) at any time. There are so many more opportunities for positive interaction than a weekly, or even daily status meeting.</p> However, it is necessary to have a culture that encourages and promotes transparency and frequent, positive interactions. It starts with the example of those at the top. Do you provide positive feedback, give helpful suggestions, and ask thoughtful questions in the course of work? Do you follow the development activity and use the communication features of the workflow tools such as Issue/PRs/project boards comments?</p> 👥 Why meetings? 2025-04-02T00:00:00+00:00 There are many reasons to call meetings:</p> Accountability</li> Knowledge transfer</li> Transparency</li> Status</li> Review</li> Oversight</li> Improve/maintain relationships</li> Brainstorming</li> </ol> However, in this age of excellent collaboration tools (Git workflows, project boards, etc.), meetings are an inefficient way to handle the first 6 items on the above list. If our workflow does not provide complete transparency into what everyone is doing, then there may be opportunities to improve our workflow.</p> ⚖️ Horizontal vs Vertical Scaling 2025-04-01T00:00:00+00:00 In the context of product development:</p> Horizontal Scaling</strong>: Selling more units, getting more customers.</li> Vertical Scaling</strong>: Add value to already sold/deployed units.</li> </ul> Bigger vs. Better – both are valid ways to scale and produce value. In this age of powerful connected devices, vertical scaling is becoming increasingly important. And that is where YOUR Platform matters.</p> 👌 Struggling to scale your IoT system? Simplify your data. 2025-03-31T00:00:00+00:00 I recently published a new blog article about IoT data. Have you struggled with adapting your IoT system to new applications and requirements? Is handling configuration updates at both the cloud and edge a challenge? Is synchronizing required data between cloud and edge instances a challenge? This article describes a simple way to model data that eliminates many of these difficulties.</p> Read here …</a></p> Struggling to scale your IoT system? Simplify your data. 2025-03-29T00:00:00+00:00 Have you struggled with adapting your IoT system to new applications and requirements? Is handling configuration updates at both the cloud and edge a challenge? Is synchronizing required data between cloud and edge instances a challenge? This article describes a simple way to model data that eliminates many of these difficulties.</p> We can scale an IoT system both horizontally</strong> (deploy more units) and vertically</strong> (add features and address new applications). While the ideas presented in this essay are focused on vertical scale, any simplification will likely help with horizontal scale as well.</p> Previously, we explored data-centric architectures in IoT systems</a>. This post expands on this by describing how to represent data using Nodes and Points in a way that drastically simplifies IoT data storage, exchange, and synchronization. This is in contrast to encodings and mechanisms used in traditional web and cloud systems.</p> How are IoT systems different?</h2> IoT systems 🌐 have the following properties, which differentiate them from traditional computer/web/cloud applications:</p> 🌐 IoT systems are inherently distributed (equipment is located in many physically different locations).</p> 🌐 Data must exist concurrently in multiple places (cloud and many edge instances).</p> 🌐 Data is not fully replicated on every instance (a cloud server instance might contain the entire dataset, whereas an edge instance only has a tiny fraction of the dataset that is relevant at that site).</p> 🌐 Instances are not symmetrical (edge instances are very different than cloud instances. It is not a cluster).</p> 🌐 There can be many instances (thousands of edge devices).</p> 🌐 The network is even less reliable (edge systems are often in uncontrolled environments).</p> 🌐 The system must continue to operate during network interruptions (it is not a cluster).</p> We can attempt to address these challenges by adding constraints such as: 1) data only flows in one direction, 2) device configuration is only changed in the cloud, and 3) the data payload format will not change over time. However, if we think realistically about these constraints, they are impractical for many systems. In distributed computer systems, change is the constant. Because a system can be networked, its capabilities are endless. The ability to connect to new and different systems means new systems and new value can always be created, and things will change</strong></em>. If you don’t, somebody else will.</p> The “struct”</em></h2> Cloud/Web APIs have traditionally used various encoding schemes and patterns such as Protobuf, JSON, REST, gRPC, etc. Tools exist to manage schemas, such as GraphQL and JSON Schema. Synchronization mechanisms can be used to manage conflicts (Raft, CRDT, etc.). Most of these schemes assume that the complex struct</em> with multiple and sometimes nested fields is the fundamental element of data representation, storage, and transmission. An example is shown below:</p> type EmployeeData struct { ID int Name string Age int Email string Phone string Street string City string Country string PostalCode string EmploymentHistory []struct { Company string Position string Years int } Skills map[string]struct { Proficiency string YearsExperience int } LastUpdated time.Time } </code></pre> Structs are great – they are a powerful tool for organizing data in a computer program. They are so useful that we have also assumed structs are the best way to store and transfer data outside a program. Thus, we have tables in SQL databases, REST APIs, Protobuf/JSON encoding, etc. This works fairly well and is likely the right solution in many domains. However, we have the following problems with using the struct</code> outside a computer program:</p> ❌ Updating the struct</code> tends to be an all-or-nothing proposition. Fetching the data from a REST API gives you the entire struct</code>. Likewise, when updating, you send the entire struct</code>. There are tools, such as GraphQL, that attempt to solve this problem.</p> ❌ Schema changes are painful. Many very elaborate tools are dedicated to solving the problem of database migrations.</p> ❌ API schema changes are even more painful.</p> ❌ Distributed systems are hard, especially if the struct</code> may be updated by multiple systems concurrently and then requires merging. CRDTs and various other distributed system algorithms attempt to solve this problem, but they are complex.</p> ❌ Nested data is problematic and compounds issues with schema changes and synchronization.</p> Data in IoT Systems</h2> What are desirable features for handling data in a distributed IoT system?</p> ✔️ We can generate/change data anywhere (in the cloud, at the edge, etc.).</p> ✔️ Synchronization between instances is granular, simple, and reliable.</p> ✔️ Data can flow in any direction.</p> ✔️ One mechanism/transport can be used for all types of data (config, sensor samples, etc.).</p> ✔️ All data changes in the system are recorded in history.</p> On the surface, this seems like a hard problem. If you have a struct</em> that exists in multiple places, and one field gets changed in one location, and another field gets changed in a second location, how do you reconcile the differences and record what has changed?</p> As mentioned before, there are solutions such as CRDTs or sending data changes over the wire. Some systems only store changes (mutations) and then compute the current state from all the changes. However, all these schemes are mostly just adding layers of complexity on top of an existing paradigm and are rarely a good fit for IoT systems where you accumulate many thousands of data samples (changes) over time. Maybe it’s time to change the paradigm …</p> Points and Nodes</h2> What if we rethink our concept of data representation? Instead of trying to make a generic struct</code> the fundamental unit of data, we take a more granular approach and focus on each piece of data in a struct. Can we create a simple unit of data that is easy to store and synchronize, and then build on top of this? This same unit of data is used for everything, including configuration, sample data, notifications, etc. The data unit we have arrived at in the Simple IoT project</a> (after many iterations) is the “Point.”</p> </figure> The data type is typically a string</code>, int</code>, or float</code>, but it can be anything you want.</p> Points are organized in a hierarchical tree of nodes, where each node contains an array of points.</p> </figure> Because a point</code> is defined by both a type</code> and a key</code>, it is possible to construct arrays and maps by populating the key</code> field with either an array index or a map key. This allows us to easily represent one level of structure (arrays and maps) in a node as shown in the following example:</p> </figure> If a deeper structure is required, then nest additional nodes.</p> Benefits</h2> There are multiple benefits to this data architecture.</p> ✔️</strong> Data is granular.</strong> An edge instance can update a node’s sensor reading, and the user can update a node’s config setting in the cloud simultaneously. The points are easily merged as they are received into the node at both locations, without conflicts. This works very well because different users/instances tend to update different points. The exception to this may occur when a user at the edge and a cloud user update a config item at the same time. In this case, we simply use the point with the latest timestamp.</p> ✔️</strong> The storage and communication infrastructure does not change.</strong> One of the biggest problems in traditional storage mechanisms and network APIs is that schema and encoding changes can be very painful. If you add a field to a data structure, every piece of code that touches the data as it travels throughout the system needs to be updated. With nodes and points, only the ends of your system need to be updated. Everything in between (transmission, storage, sync, etc) can stay the same.</p> </figure> ✔️</strong> Introspection and monitoring are easy.</strong> Since our data format is simple and fixed, it is easy to watch and decode traffic anywhere to see what is going on.</p> ✔️</strong> Integration is easy.</strong> Because we have standardized the data to be very simple, it is easy for one service to provide data that another can use.</p> ✔️</strong> Everything is data that can be acted upon.</strong> In the past, we might have had a rules engine that could respond to sensor data changes. Since config changes are now the same kind of data, the rules engine can now respond to and make config changes as well.</p> ✔️</strong> History is easily logged for all changes.</strong> Likewise, since config and sensor sample data are now the same type of data, everything that happens in the system can be easily recorded in history. The system is automatically auditable.</p> ✔️</strong> It is efficient.</strong> Because the data is granular, only what changes is transmitted and processed.</p> ✔️</strong> It works both internally and externally.</strong> While this data scheme was primarily developed to efficiently communicate between systems connected by networks, it also works very well for communicating between threads in an application. When an application is architected this way, it is very easy to extend, it as the communication scheme internally and externally is the same. This architecture scales.</p> Objections</h2> 🚩Objection #1: But isn’t this a lot of overhead to store a type/key/timestamp/etc with every bit of data?</strong></p> We need to focus on where the bottlenecks/hard/expensive parts of the system are. Synchronizing data from many different instances over networks is the hard part of IoT systems. Local storage and CPU processing are relatively cheap in comparison. Networks are also relatively slow compared to the additional processing time required to process this data. CPU and storage are now abundant, even on microcontrollers. Development, operations, and deployment are not cheap.</p> 🚩 Objection #2: Nodes and Points are hard to deal with.</strong></p> True, dealing with raw nodes</code> and points</code> is not as convenient as a struct</code>, but we can create tools</a> that easily convert structs</code> to and from nodes/points</code>. We do it all the time – JSON, Protobuf, etc. Computers are very good at encoding/decoding data – let them do it.</p> 🚩 Objection #3: How do you protect your data if anyone can update anything?</strong></p> AuthZ/AuthN is a valid concern and has not been fully solved yet. One idea is to authorize access to data at the node level – each user or instance would have access to a set of nodes. However, this says that an edge device could change its own config parameters. Perhaps this is OK because, without physical security, you don’t have true security. It may also be useful to differentiate betweenread and write access</a> to node data. Ideas are welcome!</p> 🚩 Objection #4: If points for a single struct are flowing in at different times, how do I know when a change is complete and when to process it?</strong></p> This is also a valid concern. One option is to set a timer that gets reset any time the config changes, and the changes are only processed once the timer expires. With a little creativity, we can probably come up with other schemes to signal the change is complete – perhaps a special point that is sent at the end of a sequence of points.</p> 🚩 Objection #5: What if the data is more complex than what can be represented with a simple data type like a number or string?</strong></p> You can store anything you want in the point data field (JSON, Protobuf, etc.) This should provide enough flexibility for things like messages, where it probably does not make sense to separate the data into separate points.</p> 🚩 Objection #6: How do you deal with schema changes?</strong></p> There may be instances where there are point type name changes, node topography changes, etc. In this case, each client, which provides the backend code for a node type, should be responsible for handling any schema changes. This localizes the migration logic to each client (including external clients that plug into the system at runtime) rather than requiring a central system to understand all migrations. A node may contain a version point if necessary.</p> 🚩 Objection #7: Do I have to represent all my data as points?</strong></p> If data needs to be distributed between instances on a network, then it is very beneficial to use points. However, for local processing of data, there is nothing preventing you from using more conventional techniques, storage, etc.</p> 🚩 Objection #8: How do you store data as nodes and points? SQL databases are not designed for this.</strong></p> That is an excellent question and part of our research in the Simple IoT project. We are currently using SQLite, but we are moving to NATS Jetstream</a>. Stay tuned!</p> 🚩 Objection #9: Will this scale to a bazillion devices?</strong></p> Probably not. If you need to connect a bazillion devices, then perhaps AWS IoT or something else is a better fit. However, what we are presenting here is a flexible architecture that will scale over time to many different applications and configurations as your needs evolve and change. How do you want to scale? Think carefully about this, as scaling in one dimension sometimes limits how far you can scale in another.</p> 🚩 Objection #10: This will not work for data sampled at 100 bazillion samples per second.</strong></p> For moderate data rates, transmitting blocks of data with every point by putting an array of data in a point data field may work. For high-rate data that flows, a custom data encoding and transport will likely be required.</p> 🚩 Objection #11: You can’t depend on time in a computer system, so last-write-wins (LWW) is not reliable.</strong></p> True, time, like networks, is not always reliable. However, in IoT systems, most of the time, you do not have the same data being edited in multiple locations, so LWW is only used in the rare case where two human users make changes at different locations. A little clock drift is not a huge deal. You can also do things like check if two systems have similar times before you synchronize data between them, and verify a system has a valid time before writing any data. You can also require that every instance in the system has its own RTC clock and that some engineering effort is made to ensure it is reliable and reasonably accurate.</p> 🚩 Objection #12: Why not send commands, events, changes, etc.?</strong></p> Sending “commands” is a common paradigm in IoT systems, but it is the siren song into complexity. How do you deal with cases where the command does not make it due to a network problem? There are valid cases for events such as orders or notifications, but in most cases, commands/events are an unnecessary abstraction and only complicate things. A sensor is at value X. The config setting is Y. Keep things declarative. Just send the data itself, keep some amount of history, and if you want to know when something changed, look at history. Utilizing new technologies like NATs Jetstream</a> is a critical piece of this puzzle.</p> 🚩 Objection #13: But this is different !?!?</strong></p> Yup, I’ve never seen anything like it before either. But it works.</p> Summary</h2> When we want to advance something, we often need to simplify it first.</strong> Piling on layers of complexity will only get us so far – at times, we need to go back to ground zero and re-think things.</p> There are many ways to specify contracts between systems, such as APIs and schemas. A more fundamental contract focuses on a more granular unit of data. Points in an IoT system are like atoms in matter. A Node tree in an IoT system is like the DNA in a living organism. The composition of points and structure of nodes (like atoms and DNA) is infinitely flexible. It is this simplicity and flexibility that allows atoms and DNA to define so many different types of matter and species of organisms. It is time to throw out the custom struct as the defined unit of data between systems and focus on something more granular, flexible, and scalable.</p> Want to learn more? Check out and follow the Simple IoT project</a>.</p> 🔄 The secret to process 2025-03-28T00:00:00+00:00 Process/workflows/tools should be good enough that they help the people doing the work. They are not just an administrative tool. Some examples:</p> Git helps me manage my own work, so I use it even on personal projects where collaboration is not needed.</li> Gitea issues are so fast and easy to use that when working on hard problems, I automatically create an issue to log progress. It is easier and faster than keeping notes anywhere else.</li> Gitea releases are so quick and easy to use that I use them even for internal releases there rather than composing an email.</li> Jitsi Meet is easier than a phone call, so we use it for most calls and have the benefit of screen sharing, video, multiple participants, etc.</li> Code formatting tools help me as a developer, so I gladly integrate them into my editor.</li> Part numbers are so easy to pull and use that they help me manage parts and assemblies for my prototype builds, so I pull them early in the design process and integrate them in CAD libraries.</li> Tests are easy to create and run locally, so I create them as part of my development process.</li> Documentation is integrated with the design files and easy to create, edit, and track. Thus, documentation is an integral part of the development process vs. an afterthought.</li> End to end tests are easy to run by anyone, so when there are issues to debug, they are tool that is regularly used by developers.</li> It is easy to purchase things needed to do my job and improve the company as everyone is issued a credit card, and expense reports are generated automatically. There is no motivation to bypass the process.</li> </ul> Tools/workflows/process should help the people doing the work go faster, not slow them down. If this is the case, you won’t need to force them on people. If this is not the case, the process needs work.</p> ⚙️ Fix systems, not people 2025-03-27T00:00:00+00:00 We can rarely “fix” other people. Short of a divine conversion experience or a life-changing event, people don’t fundamentally change, no matter how much we plead, beg, threaten, cajole, encourage, etc. Certainly, some are more gifted with the ability to motivate, nurture, and mentor others. But, even this has limits. So we are left with two options:</p> Get the right people on the bus.</a></li> Work on the systems and processes that help all of us work and communicate more effectively.</li> </ol> 📈 Scaling 2025-03-26T00:00:00+00:00 We often think of scaling as growing a company’s size or the number of units sold/deployed/etc. That is one way to provide value to the world.</p> However, there are other ways to scale:</p> improving the experience</li> improving reliability/security</li> new markets</li> deploying updates to existing systems</li> better customer support</li> giving users more control/flexibility/customization</li> higher profit margins</li> </ul> Think carefully about this as scaling in one dimension often limits how far you can scale in another.</p> 🌅 Simple stuff is boring 2025-03-25T00:00:00+00:00 That may be. But doing the simple stuff well is also rewarding. And which is better: exciting or rewarding?</p> To do the simple stuff well feels good. It feels right. That is because it is right. And what gets the simple stuff done? YOUR Platform.</p> 👌 The simple/basic things are the most important to get right 2025-03-24T00:00:00+00:00 It pretty much any endeavor, consistently doing the simple things are what make the difference.</p> Doing simple exercises consistently.</li> Eating whole foods.</li> Not overeating.</li> Taking time to share and communicate.</li> Showing up.</li> Being on time.</li> Meeting commitments.</li> Not overcomitting.</li> Using a headset when communicating.</li> Fixing bugs.</li> Keeping code clean.</li> Following the process.</li> Having a process.</li> Working transparently.</li> Making things easier for others.</li> </ul> Simple things are foundational – if you don’t get these right, the rest crumbles.</p> ✅ The problem with software 2025-03-21T00:00:00+00:00 I’m currently reading The Checklist Manifesto </a> by Atul Gawande. Gawande is a surgeon who wants to find a way to reduce the mistakes and their catastrophic consequences during surgery. So he investigates other industries such as air travel and the construction of large buildings. What he finds is the checklist is at the center of their processes. (more on this later …)</p> With software, our failures (with some exceptions) are usually not as catastrophic as people dying in surgery, airplanes falling out of the sky, and large buildings collapsing. Another difference with software is that is easy and cheap to fix by deploying an update (hence the prefix “soft”). People, airplanes, and buildings are not this way.</p> Thus we can get sloppy. Motivation to “get it right” can wane as pressure to ship increases and we can “always fix it later.” The problem is that this approach encourages accumulating technical debt. Technical debt will just as surely crush the soul of a programmer and eventually the company, just not in dramatic news-catching ways.</p> Building a sustainable technology company requires an extreme level of discipline because there is no external force requiring you to do it right. For many technology industries, there is not any regulatory agency watching over them – it is entirely up to YOU, and that is where YOUR Platform comes in.</p> 🌻 Downstream, things should be better 2025-03-20T00:00:00+00:00 For everyone downstream of us who uses or is influenced by what we create or do, things should be simpler/better/easier – because we did something. This is the formula for creating wealth.</p> The alternative is that we produce no value for others. It would be easier for someone to do whatever we’re doing themselves, or in extreme cases what we are doing should not be done at all. This is the formula for debt.</p> Wealth can include more than money – good will, positive energy, intellectual property, spiritual well-being, relationships, community, health, etc.</p> Whenever debt creation is greater than wealth creation for too long, the family/organization/company/country we’re a part of collapses.</p> In whatever sphere we’re a part of, do we create wealth or debt.</p> 🪙 Complexity and Simplicity -- two sides of the same coin 2025-03-19T00:00:00+00:00 To get to simple for someone, it often has to get more complex for someone else.</p> An elegant, easy to use UI was a lot of work for someone to design.</p> A powerful, modern programming language that makes it easy for software developers was no doubt very complex to implement.</p> Modern CPUs that allow us to do so much are very complex to design and manufacture.</p> In technology, anything that appears simple/nice/easy likely has something very complex behind the scenes.</p> Again, this makes business sense if many people can benefit from the simplicity for every person that deals with the complexity.</p> What does not work is if we push complexity onto our users/customers in an effort to keep things simple for ourselves.</p> ⬆️ The business case for improvement 2025-03-18T00:00:00+00:00 As discussed previously, improvement is sometimes hard – really hard. So how do we justify it?</p> The payback comes when we improve something once, and then something else gets lots easier many times.</p> An example might be automating a release process. If it takes 1 hour to manually do a release, and it takes 6 hours to automate it, then in 6 releases we break even and releases after that is gain.</p> We want the balance to tip heavily toward the gains from repeatedly using the improvement, not the effort spent implementing improvements.</p> </p> For this to work out, we need a way to leverage improvements in the future. We need tools, workflows, automation, and processes to capture improvements – YOUR Platform.</p> ⬆️ The path of improvement 2025-03-17T00:00:00+00:00 Improvement is not always easy, actually it is usually hard – really hard, and sometimes a complex path. But in the end, something likely got simpler for someone</a>.</p> Getting to complex is easy and the natural state of things (chaos).</p> The path and the resulting state are two different things.</p> We can do the hard/complex things and get to a simple/nice/sustainable state.</p> Or we can only do the the easy things which results in a complex/messy/unsustainable state.</p> Take your pick.</p> </p> ⬆️ What is improvement? 2025-03-14T00:00:00+00:00 Improvement often means something got simpler for someone:</p> We refactored the code and it is now easier to read.</li> The build system is more reliable.</li> The automated tests are better, so less manual testing is needed.</li> The documentation is better, so it is easier to understand something.</li> The circuit is more reliable, so support has less field problems to deal with.</li> A product has more needed features, so it is easier to sell.</li> </ul> If things are not getting simpler for someone, we need to question – are they improving?</p> 👷 Make less work for others, part 2 2025-03-13T00:00:00+00:00 A few examples of how to make “less work” for others:</p> try not to introduce bugs or break stuff</li> communicate in workflows like Tickets/PRs where there is context and messages can be easily processed</li> document your work – including “why”</li> focus on the business priorities</li> be self-directed – see what needs done and do it</li> work in public (commit early and often</a> and release your work often</a>).</li> document how work was tested</li> better yet, provide examples/scripts/tests of how to use/test your work</li> </ul> It should be obvious:</p> what we are doing</li> why we are doing it</li> how we did it</li> how we tested/verified it</li> and how to use it</li> </ol> 👷 Do we make more work for others or less? 2025-03-12T00:00:00+00:00 This question really gets to the heart of our value to others. In everything we do, we need to think – am I delivering this work or information in a way that is easy for others to understand and use? Or do they need to do a bunch of unneeded/extra work?</p> Organizations composed of people who make less work for others scale.</p> Organizations composed of people who make more work for others collapse.</p> 🔄 The business model for updates 2025-03-11T00:00:00+00:00 In the old’n days, a project was shipped and done, and we moved on to the next one.</p> However, with modern connected systems, we can deliver a lot of value</a> after the initial release. Most of our engineering effort may be spent after the initial release maintaining and improving a product. How do we fund this? Some ideas:</p> Build the cost into the product up front (many consumer products use this model)</li> Pay for maintenance with new sales (be careful this is not a pyramid scheme)</li> Monthly/yearly subscription fees (might include a cloud service)</li> Fees for add-on features (high-end test equipment like oscilloscopes often use this model)</li> Pay for every update (hard to get customer buy-in)</li> </ul> The biggest mistake is not to plan for maintenance costs, or to surprise your customers with them. Be up front with the costs, but be more up front by communicating the value that you are continuously delivering.</p> 🔄 What is the best software release frequency? 2025-03-10T00:00:00+00:00 Different projects/products have different needs, but generally the trend in our industry is to release new software more often. There are several reasons for this:</p> Modern processors (even embedded microcontrollers) are very powerful and expandable, so it is possible to add additional features after the initial release.</li> Many devices are now connected, so updates can be deployed “over the air” automatically, rather than executing some manual operation.</li> Continuous Integration (CI) and other workflows are improving the quality of software such that we are more confident the software is always tested and working. Long manual testing/QA periods are no longer the best practice.</li> People are used to software updating on their phones/computers, so it is generally more accepted.</li> Connected systems need more updates</a>.</li> Connected systems bring security concerns which require updates.</li> </ul> Perhaps the main reason the release frequency is increasing is more cultural than anything. In previous times, software updates were perceived as “breaking things.” Especially in embedded systems, the mantra was: test it extensively, get it right, and lock things down. Updates are risky. However, the world has changed.</p> In the end, a software release means more value is being delivered to your users. And if our goal is to deliver value, then that includes releasing new software more often.</p> 🚀 How can Arch Linux get by without releases? 2025-03-07T00:00:00+00:00 Arch Linux does not have discrete releases – it is a rolling distribution where individual packages are continuously updated. Most people still raise an eyebrow at the mention of Arch Linux – how can that possibly work and be stable?</p> Having run Arch for many years, I can affirm it works very well</a>. It is fast, efficient, and lets me run the latest software of everything with minimal fuss.</p> Today, a Linux distribution release is an artificial notion. A distribution’s main job is to deliver a collection of software to a computer. Each application has its own release cadence. Imposing a distribution release cadence on top of the package release cycle does not make sense.</p> But the distribution’s job is ensuring everything is stable and works together? Really? Does a distribution really test every one of the 1000’s of packages before every release?</p> If there is a serious problem in an application, it will be fixed soon in the main development branch. Requiring these fixes to be back-ported to an old release in a distribution is a lot of work. In some cases, it is nearly impossible to backport a change if things have changed significantly in the codebase.</p> A Linux distribution delivers the most value when each software package can be deployed to its users as soon as it is ready. Arch does this and then gets out of the way.</p> ⚖️ Truth is discovered, not invented 2025-03-06T00:00:00+00:00 I once advised a young man – If you have some down time on a plane, in a hotel, etc., you might consider writing an essay or two about some of the issues facing you personally, or your organization. This is one of the best ways to think and discover truths. I’m often surprised and amazed what comes out.</p> Meetings would be much more effective if writing were part of the preparation (Amazon does this</a>). But, alas, most think we can make progress by more discussion, meetings, emails, etc.</p> Truth is discovered, not invented. Truth cannot be forced. It is only when we approach the creative process with this humility that we can make progress. Designs devoid of truth are unwieldy, unreliable, hard to maintain, etc.</p> Modern, powerful tools like AI, CAD, computer languages, etc, may give us the illusion that great inventions start with us, but truly great designs/algorithms/processes are discovered, a simple reflection of truths that have existed for eternity. In the process of simplifying our designs, we often come closer to truth.</p> I wish I had understood the power of writing and the essence of truth earlier in my career.</p> 🔗 Connected systems are different 2025-03-05T00:00:00+00:00 In distributed computer systems, change is the constant.</p> Because a system can be networked, its capabilities are endless. The ability to connect to new and different systems means new systems (and value) can always be created, and things will change</em></strong>.</p> If you don’t, somebody else will.</p> 🛡️ How to get to secure and reliable 2025-03-04T00:00:00+00:00 The best way to get to secure and reliable software is to fix the core issues.</p> If your programming language is not reasonably secure and reliable, adding more external tools to check and track down problems is helpful, but is likely not the most direct approach.</p> Likewise with architecture – getting the core architecture right is important. Adding external “watchers” and “fixers” to compensate for an architectural problem only complicates things, and complexity is the #1 enemy to both security and reliability.</p> 🏭 Semi-structured part numbers 2025-02-28T00:00:00+00:00 Some time back I recorded a video on Semi-structured part numbers.</p> https://youtu.be/tlsl01uxUfE</p> Be more efficient.</p> Make less mistakes.</p> We tend to ignore simple things like part numbering systems.</p> But doing the simple things right is what frees us up to work on the hard things.</p> 👤 Who is our "customer"? 2025-02-27T00:00:00+00:00 Your “customer” is someone who derives value and uses what you produce. Customers are both internal and external to your organization.</p> While it is good to focus on the flow of primary business value to the end-customer (that is what pays the bills), we can also think of everyone in the organization as being our customer. How can we provide value to everyone?</p> ⚠️ Toward a Theory of Medical Fallibility 2025-02-26T00:00:00+00:00 I recently skimmed an essay from 1975 titled Toward a Theory of Medical Fallibility</a>. A few quotes from the article:</p> Therefore, where there is scientific activity, there is a partial ignorance – the ignorance that exists as a precondition for scientific progress. And since ignorance is a precondition of progress, where there is the possibility of progress there is the possibility of error. This ignorance of what is not yet known is the permanent state of all science and a source of error even when all the internal norms of science have been fully respected.</li> Error may indeed arise from the present state of scientific ignorance or from willfulness or negligence. But it may also arise precisely from this third factor which we have called necessary fallibility in respect to particulars.</li> The encouragement of this inflated belief in the competence of the physician is of course reinforced by the practice of not keeping systematic and accessible records of medical error. Yet everyone knows that this is a false confidence.</li> Indeed, the only profession of which we know which fully and publicly documents predictive successes and failures is that of horse-racing correspondents in Great Britain.</li> The first reaction of physicians to the invitation to dispense with the masks of infallibility is likely to be a humane alarm at the insecurity which a frank acceptance of medical fallibility might engender in the patient. But we wonder whether the present situation, in which the expectations of patients are so very often disappointed during medical treatment, is not a worse source of insecurity.</li> </ul> </blockquote> As computer/software systems get more complex, are there parallels?</p> 🛤️ Two paths with experience 2025-02-24T00:00:00+00:00 As we gain experience, we can take two paths:</p> learn from mistakes and all the things that went wrong, and apply this knowledge to do better in the future.</li> assume because we’ve done something before the next time we do something similar it will “just work.”</li> </ol> If I’ve learned anything working in technology, things rarely “just work.” Nothing is easy.</p> Some can accomplish #1 because they have a great memory. For the rest of us, we need some process to capture what we have learned and a mechanism for applying this in the future.</p> A checklist is a great place to start.</p> 🛤️ Two paths 2025-02-21T00:00:00+00:00 There are two paths we can follow as a team or an organization:</p> what we do makes it easier</strong> for everyone else to do their job.</li> what we do makes it harder</strong> for everyone else to do their job.</li> </ol> #1 is the formula for scaling</strong>. #2 is the formula for stagnation and technical debt</strong>.</p> And which path a team or organization takes depends on culture</strong>. What are the expectations</strong>? Are they documented</strong>? And most importantly, what example</strong> do we set?</p> 👥 Your team as the customer 2025-02-20T00:00:00+00:00 Previously we defined our “customer”</a> as someone who derives value from what we do. While it is useful to think of how the primary business value is flowing, hopefully we are providing value to our team as well as people outside our team. Some ideas:</p> Are Git commit messages helpful and easy to understand?</li> Are commits and PRs small enough to comprehend?</li> Is our work transparent?</li> Do we deliver what we said we would?</li> Is the code we write easy to follow and maintain?</li> Do we document what we are doing and why?</li> Do we share tests scripts and other automation that helps people use what we created?</li> </ul> 🚀 Why are releases important? 2025-02-19T00:00:00+00:00 Since many products/systems run software these days, they can be improved after the initial release. Most software projects have “releases” when a new version of the software is deployed or made available.</p> Why are releases so important? Some reasons:</p> Communicates to customers an update is available and what changed.</p> </li> Provides a cadence for doing work.</p> </li> Encourages development teams to “ship”. Shipping is motivating.</p> </li> Delivers more value to the customer.</p> </li> Helps us isolate problems if we know what is running and what changed.</p> </li> </ul> 📄 Who is a changelog for? 2025-02-18T00:00:00+00:00 Most projects have releases, and with each release, we describe what is new/changed/fixed in a changelog file. keep a changelog</a> has a lot of good ideas.</p> Who is a changelog for? It is for our customers – those who use whatever we produce. The changelog communicates to our customers what is new and valuable for them</em></strong> in each release.</p> Once we realize who the changelog is for, then this can influence how we write changelog entries. A few ideas:</p> keep it is as succinct as possible (don’t waste people’s time)</p> </li> don’t use technical jargon that our customers don’t understand</p> </li> be humble – if there are bugs that got fixed, acknowledge they were bugs. Don’t try to embellish or gloss over bug fixes by calling them features, etc. People can see right through this.</p> </li> explain changes from the lens of benefits to the customer/user, not from the developer’s perspective</p> </li> </ul> 👥 Who is the customer in organizational hierarchy? 2025-02-17T00:00:00+00:00 In a business organization hierarchy, where E (engineer) reports to M (manager), who is the customer in the E - M relationship?</p> Our first inclination may be to think M is the customer – M is the boss, E should make M happy. But, lets think about this a bit more. Who does E create business value for? Is E writing code that benefits M, or producing a printed circuit board that M will directly use?</p> Hmm, does not seem so. E’s customer is probably purchasing and manufacturing (in the case of a PCB). M does not directly produce tangible outputs. M’s job is to help E get the job done. Therefore, it would follow that E is M’s customer. Certainly, M has authority and needs to make decisions at times. But, in the end, the job must get done. Business value must be produced. In product development, authority or decision-making in themselves do not produce business value. The more people we have under us, the more people we serve.</p> Do we exist to serve, or be served? The higher up we go, the more important it is to understand servant-leadership</a>.</p> 👤 Confusion about when we are the customer 2025-02-06T00:00:00+00:00 In our modern, entitled, self-serving consumer culture, we tend to be confused about when we are the customer or not. If we work for pay, our job is to do something to benefit someone else – our customer. We are not the customer in this scenario, someone else is. We are supposed to make someone else’s life easier or better. In our job role, are we serving our customer, or do we expect to be served? This mindset probably does more than anything to determine if we are useful.</p> 🕊️ Simple -- from whose perspective? 2025-02-05T00:00:00+00:00 “Just keep it simple” – we sometimes hear this. But from whose perspective?</p> Legal/IP</li> Management</li> IT</li> Financial</li> R&D</li> Security</li> Sustaining engineering</li> Purchasing</li> Manufacturing</li> Quality</li> Sales/Marketing</li> Customer support</li> End User/Customer</li> </ul> How do we prioritize simplicity for the above list?</p> Hint, the one at the bottom is probably the most important. Do you constantly keep your customers in mind as you make decisions? “Customer” may include multiple from the above list.</p> 🕊️ Simple to maintain vs simple to implement 2025-02-04T00:00:00+00:00 Recently</a> we talked about the tension between simple to use vs. simple to implement.</p> Maintenance is similar. If you want something that is easy to maintain, you need to put in the work upfront.</p> The Elm programming is a classic example of this. The initial coding takes longer than generating some Javascript from AI prompt. You need to figure out how to best represent the problem with Types. But after you get it right, it is so easy to maintain</a>.</p> 🤝 Integrate early 2025-02-03T00:00:00+00:00 Integration is where we bring together separate parts of a system.</p> When things come together, problems appear: performance problems, stability problems, electromagnetic noise issues, mechanical interference, incompatibilities, manufacturing issues, usability issues, aesthetic issues, etc.</p> It is much better to discover these problems early, and the only way to do this is to integrate everything early in the project. And the only way this happens is if it is prioritized by management and supported by process.</p> Iterate Hardware Like Software</a></p> 🕊️ Simple to use vs simple to implement 2025-01-30T00:00:00+00:00 To make something simple to use, often it is not simple to implement. For example, in the SimpleIoT UI, the following line of code is all that is required to add a text input field to a node. No handlers need to be attached, etc.</p> textInput Point.typeDescription "Description" ""</code></p> However, the code that implements the textInput</a> function is a little more involved.</p> Generally, the better you want the user experience to be (even if your users are developers using your API or library), the more work you’ll need to put into things behind the scenes. There is a fundamental tension between “Simple to use” vs. “Simple to implement” – you can have one or the other – usually not both. We all use many “components” in product development, but an excessive off-the-shelf mental, especially at the top hardware and software levels of your product, won’t get you very far.</p> Simple != Easy</p> 🤏 Handling the small details 2025-01-29T00:00:00+00:00 We’ve discussed</a> the necessity to fulfill your small commitments as a stepping stone for fulfilling larger commitments. We can expand this thinking – handling the small details well enables you to focus on bigger things.</p> In product development, we have many details to be concerned about:</p> consistent code formatting</li> testing new code</li> testing the system</li> ensuring there are no errors or DRC violations on a PCB layout</li> generating a documentation package (including BOM) for each PCB release</li> generating documentation for each project release</li> notifying relevant people when things change</li> reviewing code changes</li> organizing parts in the lab and in production</li> deploying software updates</li> publishing various communications</li> ensuring everyone on the team has access to needed information</li> </ul> We can do all of the above manually, and it is not a big deal to do once. However, these details soon start to pile up until pretty soon that can be all a person does. Fortunately, there is a better way – all of the above can be handled to some degree with intelligent workflow, automation, and tooling.</p> We have so many good options for automation available today – scripting languages, APIs, CI/CD systems, flexible DRC engines in CAD tools, Git based workflows, commercial tools and services, etc. Implementing and using these tools is not always easy and requires an investment in time and learning. However, this is much easier than manually managing tons of details forever and frees you to focus on more important things.</p> 🛠️ Selecting tools 2025-01-28T00:00:00+00:00 There are many considerations when selecting a tool:</p> will it do the job?</li> what does it cost?</li> can I get support?</li> is there a community around it?</li> will it be available and maintained long-term?</li> what does the future look like for new features?</li> is it accessible/available to others?</strong></li> </ul> The last point is often overlooked – “That is not my problem.” However, if you are building a platform, this should be a primary consideration as tool accessibility is important for collaboration and maintenance. This is the difference between a sustainable business and a few talented individuals.</p> 🛠️ Two classes of tools 2025-01-27T00:00:00+00:00 There are two classes of tools: Personal</strong> tools and Common</strong> tools.</p> A personal tool might be your editor – each developer has his own preferences and what editor one person uses does not affect another.</p> However, it is important to have a common tool for formatting code. So you decide on a formatting tool and everyone configures their editor to use that.</p> Some more examples:</p> Personal</th> Common</th></tr></thead> Computer OS</td> File/doc formats</td></tr> Phone</td> Git PR/Issue system</td></tr> Notes system</td> Build systems</td></tr> Email client</td> CI/CD</td></tr> Browser</td> CAD Tools</td></tr> Git client/setup</td> Compilers</td></tr> </tbody></table> Sometimes sticking with the common stuff is slightly less convenient than doing it “my way.” However, you need to decide if you want to be part of a sustainable business or just a few talented individuals.</p> 📝 Documentation is part of the process, not an afterthought 2025-01-24T00:00:00+00:00 One of the best productivity hacks ever is to change your thinking about documentation. Most documentation is done after the fact, which is a rather distasteful task at this phase.</p> If we learn to use documentation as part of our thinking/design/implementation process, it can be enjoyable, and help us do a better job in the first place. A few ideas:</p> Write code comments before we write the code, describe what we are going to do, etc.</li> In a schematic, document constraints and design criteria as a note in the schematic before drawing a new section of the schematic. As you go, think in your notes.</li> Write the user documentation before implementing a feature – this may help us think more from the user perspective upfront.</li> If we are stuck on a problem, write documentation about it and check it into Git. This allows us to still ship something</a>.</li> As we are doing something, create careful notes about all the resources we need to do the job, summarize important parts with screenshots, etc. It will be much easier next time.</li> Create a checklist before you do something.</li> When someone asks you about something, instead of writing an answer in an email, create a document and point them do it</a>.</li> </ul> A small shift in our habits can have a big impact!</p> ✨ Demos vs. Products 2025-01-23T00:00:00+00:00 I’m a huge fan</a> of demos – so is Khem</a>. They are a very important part of product development. They are a way to ship early and get critical feedback. Demos during the course of real product development are essential.</p> There are some challenges or risks with demos. One is that when managers see a demo, they sometimes assume the product is nearly done when this is rarely the case.</p> Other demos are for the purpose of evaluating some technology with little concern as to how it is implemented. The world has exploded with “maker” stuff (Arduinos, Raspberry PIs, etc). These are very useful for one-off lab experiments, but experience has taught me that it is generally a bad idea to ship products (at least industrial products) based on “maker” technology.</p> Products have considerations like cost, manufacturability, maintainability, reliability, and availability over a lifecycle that can be 5, 10, or more years for industrial products. We can ignore these concerns for the demo, but they can’t be ignored for the product.</p> 🧩 The three legs of YOUR Platform 2025-01-22T00:00:00+00:00 Automation</strong> – start with a checklist.</p> Collaboration</strong> – #1 rule is don’t view email as a permanent place to store information.</p> Process</strong> – the habits of your organization.</p> 🔍 The importance of small Git commits/pull-requests 2025-01-21T00:00:00+00:00 Why are small Git commits/pull-requests important?</p> small changes are easier to review. Large PRs are painful to review, thus they tend to be rubber-stamped and not reviewed, or rejected in OSS projects.</li> you get to ship more often</a>.</li> other people are motivated by seeing work get done (positive peer pressure).</li> if we are heading down the wrong path, this can be corrected earlier before wasting a lot of time.</li> our work is more transparent, and transparency is a powerful tool.</li> </ol> Even if we are working on a personal project that no one else sees, small commits helps us organize our thoughts, reflect on what we’ve done, and receive the motivation boost that comes from incrementally shipping, even if only to ourself.</p> 🎆 100 daily emails 2025-01-20T00:00:00+00:00 Today marks the 100th “daily” email I’ve sent out. (I don’t actually send on weekends, or days when I’m out of the office). Some observations:</p> Daily is easier than weekly – I was skeptical when others said this, but I think it is true.</li> If I skip a few days, it can be hard to get going again. There is something about the process of doing something daily that energizes the creative process and keeps it going.</li> Ideas are everywhere – when I write consistently, the mind starts to see ideas everywhere. There are 32 article ideas in my backlog – Ideas come in much faster than they go out.</li> I spend 10-20 minutes a day writing these posts. Often writing one will spur the idea for the next, and I’ll write that one too. I think it could be even faster.</li> Tooling is important. I have scripts that automate creating a post and publishing it. Kit is configured to pick up the post from my site’s RSS feed and then mail it out. There is nothing slow or painful about the process that pushes me away from doing it.</li> </ol> If you have feedback, please let me know! I’d love to know what topics are interesting to you or not.</p> 🔁 The importance of Git Commit/Push often 2025-01-15T00:00:00+00:00 There are many forms of “shipping” your work as we recently discussed</a></p> Shipping does not necessarily mean a product release, but rather releasing your work external to you. It can be a Git push, running tests in CI, a prototype, a document that others can read – basically something of lasting value</em></strong> that others can interact with. (Email does not count).</p> As a developer, the simplest way to ship your work is a Git push. It does not have to be perfect. In most cases, it does not even have to be done. That is the beauty of software, it is soft – it can be improved/finished later. A Git commit does not even have to be code – it can be documentation, specs, etc. A Git commit/push is the simplest and most effective way to demonstrate you are providing value to the project or organization. Yet how many of us are consumed with ephemeral activities that in the end provide no lasting value?</p> 💭 Thinking, Designing, Developing, Engineering ... 2025-01-13T00:00:00+00:00 If we are thinking without writing, are we really thinking?</p> If we are designing without sketching/drawing, are we really designing?</p> If we are developing software without iterations/testing, are we really developing?</p> If we are engineering without building prototypes, are we really engineering?</p> Something happens when our work is implemented in an external form that is staring back at us. It speaks to us, exposes the flaws, and shows us where improvements are possible. When our work is good, we can recognize it and find motivation for the next phase.</p> Yes, it is possible to do some things without shipping, but when we ship our work in the form of writing, drawing, prototyping, and iterating, a world opens up that we could never see before. New connections are made.</p> Optimize your habits/processes for shipping something – your work will get better.</p> 🔧 envsetup.sh -- a very useful automation technique 2025-01-10T00:00:00+00:00 During a Android development stint, I learned about the envsetup.sh</code></a> concept. This is a script that contains numerous functions get populated in a shell envrionment when the script is sourced. There two ways to source the file – you can type:</p> source envsetup.sh</code></p> Or use a ‘.’ for a shortcut for source and type:</p> . envsetup.sh</code></p> Most of my projects and Git repos now contain a envsetup.sh</code> file (examples: Simple IoT</a>, Yoe</a>, TMPDIR Podcast</a>). Typically all of the functions in the file have a common prefix. This gives them a namespace and allows you to see all the functions quickly in your terminal by typing <prefix><tab><tab></code>.</p> Why shell functions instead of something like Make, Earthly, Nix, Task, Mage, Pants, Bazel, Python …? Shell script is about the simplest abstraction on top of typing commands. Thus, if mostly what you need to do is execute a sequence of terminal commands, then shell functions are very efficient at doing this. A shell function is often the best documentation for how to do something – it is almost like a powerful checklist. There are no dependencies – every Linux and MacOS system comes with a shell built-in. envsetup.sh</code> functions can often be used directly in CI/CD and other build systems (like Yocto/Bitbake tasks).</p> There are some common objections …</p> Shell is not cross-platform to Windows. That is unfortunate, but I rarely use Windows, and most developers I interact with use either Linux or MacOS systems. However, this may change with Zephyr as MCU developers traditionally have used Windows.</p> Why not use something more powerful like Make? With modern programming languages and build systems like Go, Rust, and Yocto, the is no need for yet another build system. The complex issues like dependencies are already handled. Thus, all you need to do is remember how to kick it off.</p> Why not use Python – everyone uses Python for everything – it is the most popular programming language ever! Python is great if you need to do something more complex, but it is a more complex programming abstraction. And mostly what you need to do is record a sequence of commands, any abstraction costs you more than benefits.</p> Why not a shell script that you execute instead of sourcing a script that populates functions? I’m not sure – the sourcing just seems more ergonomic to me and simpler. If you need to run a script to execute a command, then you need to process command line arguments, etc. You are now programming, and programming is what we are trying to avoid in envsetup.sh.</code> Also, the functions are not as discoverable as when they are populated in your environment.</p> Shell scripting is not without its tradeoffs – when you do need to program something in shell, it can be very painful and errors can be subtle and hard to catch. However, AI tools are very effective at writing shell script, so this can be a big help. Linters and formatters are also very good at catching problems early and making your scripts more readable (more on this later).</p> In summary, think of your envsetup.sh</code> file primarily as documentation. Create one in every project and instead of documentation how to do something, create a shell function. Your future self will thank you.</p> 📧 Effective email is either a pointer or notification 2025-01-08T00:00:00+00:00 Email is the only decentralized communication platform that has widespread usage. And this is more amazing considering it was invented in 1971. Email is an amazing technology and has stood the test of time.</p> However, there are challenges with email:</p> It can be difficult to find information in emails at a later date.</li> Visibility is limited to the recipes of the email.</li> Discussions often don’t happen in the context of the work being done.</li> Information is not organized very well.</li> </ol> I follow a few rules regarding email:</p> Whenever possible, don’t send information in email, but rather point people to information in a more permanent location (Git repo, Wiki, Discourse forum, etc.). If the information does not exist, create it, and then send the email.</li> Keep project-related discussions in Git pull requests, issues, forums, etc.</li> For transient discussions, use chat tools like Discourse chat, Signal, Discord, Slack, etc. This is more fun, and keeps email from clogging up with chatty stuff. This keeps the information accessible to everyone involved. It is also way more efficient as discussions and information are stored in the context of the work.</li> </ul> It is still very useful to receive email notifications when a piece of information or a discussion is updated. This is quick to process and gives everyone interested the opportunity to watch the stream of activity as it flows by. Most systems give you options for tuning how many email notifications you receive based on your time/interest.</p> Accessibility and context – the keys to effective communication and information.</p> 🎓 The problem with experts 2025-01-07T00:00:00+00:00 At a recent family holiday gathering, we happened to be discussing how crazy family life gets at times – you are halfway through cooking something and realize you don’t have the ingredients on hand, so you need to improvise. Some of the men in the group described how they always get everything out first before starting, which ironically I also do</a>. Some of the females commented this was unnecessary as they had done it thousands of times, so it was easier to get stuff as needed. They have a pretty good sense of where stuff is and what they have on hand – 99.9% of the time it works pretty well.</p> Perhaps this is a general difference between men and women – women are generally better at multitasking. But I think there is also another factor at play – with food preparation, the women in our families are experts, and the men are amateurs. Experts are very good at what they do. They don’t need a process to prop them up. However, in the cooking example it is more difficult to scale these methods to their husbands or children.</p> The real value in expertise is in scaling it. Do we document what we do? Can we make it easy for others to follow? Can we make it easy to find the things we need? Can we create processes that reduce errors and blockers? In the kitchen, these differences between the amateurs and experts are mostly amusing and not all that critical, but in business and product development it is. Experts need to be creating and contributing to platforms. This is the difference between a sustainable business and a few talented individuals. Unfortunately, many experts don’t personally feel they need a platform, so have little motivation to contribute to one.</p> When you hire, expertise is important, but so is Platform Thinking.</p> 🍳 Preparation - lessons from the kitchen 2025-01-06T00:00:00+00:00 Most Sunday mornings, I make gluten-free waffles for breakfast (see recipe at the end of this post). I like to get everything out (ingredients and utensils) before I start.</p> </p> This seems to make the process go faster and smoother. There is an efficiency in focusing on getting everything first, and then cooking.</p> So I was wondering if there is an analogy to product development. What are the ingredients for product development?</p> clear definition of the goal and next steps</li> any relevant documentation</li> tools (editor, CAD tool, issue tracking systems, build systems, tests, release processes, etc.)</li> collaboration with others on the team (issues, PRs, discussions, etc.)</li> </ol> Is this scattered all over the place, or can you quickly assemble all of this and hit the ground running?</p> It helps to organize your work in a notes system</a>. This has links to relevant information and notes so you immediately have all the relevant context.</p> For some tasks like code review, tools can prepare the task for us. Github/Gitea/Gitlab pull request (PR) workflows have the code, discussion, and proposed changes all in one place.</p> With any task, having all the ingredients at hand makes for a nice result! Preparation leads to focus.</p> </p> Gluten-free waffle recipe:</p> </p> 😊 Joy 2025-01-03T00:00:00+00:00 Platforms help bring the joy back into engineering.</p> How? – by assisting you with the tedious error-prone tasks, and freeing you to focus on the valuable creative work.</p> Satisfaction comes from being useful and creating value. We need to get to the place where we can do that work.</p> 🪞 Platform reflection 2025-01-02T00:00:00+00:00 How has YOUR Platform done this past year?</p> Have you burned down technical debt?</p> Have you automated repetitious or tedious tasks?</p> Is your test coverage increasing or decreasing?</p> Are you releasing more often?</p> Are deployments getting easier?</p> Are you deploying with more confidence?</p> Is your documentation getting better?</p> Is it easier for new people to get up to speed?</p> Do you have any new checklists?</p> Have any existing checklists been automated?</p> 🙏 Humility 2024-12-27T00:00:00+00:00 An essential mindset for platform thinking is humility – the realization that we as humans are finite, we make mistakes, we don’t always perform at the same level, we get lazy and tend to take shortcuts.</p> YOUR Platform can compensate for much of this as it does the boring tedious work, checks for mistakes, deploys reliably, provides structure, etc.</p> But we need to acknowledge this first, and that is the first step.</p> 🔄 The biggest benefit to frequent updates 2024-12-23T00:00:00+00:00 Perhaps the biggest benefit of frequent updates is that the changes are small, making them easier for humans to absorb and adapt to.</p> The conventional wisdom in our industry is that frequent updates mean things are always breaking. This has not been my experience; rather, things are continuously improving in small ways, and the changes are much easier to adapt to.</p> Arch Linux is a classic example – always improving, rarely breaks, and when it does, you can usually find a fix in a few minutes. Zephyr is another example – always improving, and rarely problems merging in the new changes.</p> When things break or change unexpectedly, you are only dealing with one issue at a time, not dozens or even hundreds.</p> This is more organic and enjoyable – the way natural systems work. Small continuous changes (either good or bad) are better than large step changes and much easier on the system.</p> 📅 Dates in filenames 2024-12-20T00:00:00+00:00 One of the most common labels we use today is dates. We use them in file names, reports, database fields, etc. There are many ways to enter a date:</p> 2024-12-20</code></li> 20 Dec 24</code></li> 12/20/2024</code></li> 20/12/2024</code></li> Dec 20, 2024</code></li> etc.</li> </ul> RFC 3339</a> (YYYY-MM-DDTHH:MM:SS+HH:MM</code>) is increasingly becoming an international standard. Many programming languages (Go, Rust, Javascript, Elm) support RFC 3339.</p> What about when naming files? It is often helpful to put a date in directory or file names so you can easily sort to find the latest or construct a timeline of events. In this case it makes sense to format the date general to specific</a> YYYY-MM-DD and put the date at the beginning of the filename:</p> 2024-10-20_project-requirements.docx</code></li> 2023-11-01_architecture-rev1.pdf</code></li> 2024-11-11_architecture-rev2.pdf</code></li> 2024-12-15_release-v1/</code></li> </ul> With a little thought, even simple things like file and directory names can become useful metadata that is easy to sort and query. Consistency and standards – simple things you can do to make things better.</p> 🔄 Lifecycle management in dynamic systems 2024-12-18T00:00:00+00:00 One of the hard parts of dynamic systems is the lifecycle management of various bits of running code.</p> It is easy to create stuff, but are you cleaning up properly when things disappear or change?</p> We can order these in difficulty:</p> Create something: (easy)</li> Delete something: (hard)</li> Change something already running: (very hard)</li> </ul> On the surface, this looks easy – what is the big deal, right?</p> But we soon realize differently. Have you ever heard of memory/resource leaks and race conditions? Even worse, code that we don’t know is still running.</p> Implementations usually start simple – no problem.</p> But the scope grows, the system becomes more dynamic, and we start to scale. Scaling requires concurrency. Concurrency is hard, but when the system changes while running, concurrency gets really hard.</p> This is the nature of IoT systems – because they reflect the real world, and the real world is always changing.</p> 🧪 Abstractions and Platforms 2024-12-17T00:00:00+00:00 I’ve come to appreciate the Go communities advice on abstractions.</p> Avoid creating abstractions prematurely. Every abstraction has a cost, and often duplicated code is simpler and easier to maintain than a premature abstraction.</p> </blockquote> Don’t take DRY (Don’t Repeat Yourself) to an extreme. It is perfectly fine to duplicate code several times and only create an abstraction once you see a clear pattern and understand better the problem the abstraction is solving.</p> For instance, in Simple IoT, we implemented a number of subsystems (rules, modbus, 1-wire, etc.) before we noticed a pattern – all of these systems needed lifecycle management, which is very tricky to get right. So it made sense to create a client abstraction</a> that handled lifecycle in one place. But it was not clear what we really needed until we had implemented several subsystems.</p> Follow the YAGNI (You Ain’t Gonna Need It) principle – only create abstractions when you need them, not for hypothetical future use cases.</p> The same applies to platforms.</p> Other than avoiding architectural pitfalls and selecting the wrong technology, focus on building out YOUR Platform to solve your current needs, not your future needs. This is one of the best ways to know if your architecture and technology are right.</p> Platforms for the sake of platforms are detrimental and counter-productive – kind of like the IT department that loses its focus that they are there to serve the business needs and becomes a barrier to getting things done. They must serve the business goals, not themselves.</p> A platform in itself provides no value if you are not shipping something of value.</p> The same can be said of any supporting role or metric – finance, performance, etc.</p> If you are not shipping (internal prototypes and test versions count as shipping), nothing else matters.</p> 🤝 Keeping commitments 2024-12-16T00:00:00+00:00 When we commit to something, do we follow through? How do we get better at this when we always have more to do than time to do it?</p> The thought came recently that it probably starts with keeping commitments to ourselves, which is the essence of self-discipline. If we can’t keep promises to ourselves, likely we won’t keep them to others either.</p> If we plan to get up at 6 AM, do we?</p> If we plan to walk every day, do we?</p> If we plan to work on something today, do we block out the time and make it happen?</p> Seemingly simple things, but important and sometimes hard in this busy life.</p> If we can’t do the small things, we’ll likely not succeed in the bigger things.</p> This is foundational in Platform Thinking – at the personal, group, or organization level. If you do the small things right, this will enable you to do the big things.</p> 📝 Personal vs. collaborative documentation 2024-12-13T00:00:00+00:00 Yesterday</a>, we discussed the importance of a personal note-taking tool. We can also extend this concept to a team or company using Wikis, Markdown in Git, shared Google Docs, etc.</p> Collaborative documentation is very valuable – IF you can get group participation. Over the years at various companies, I’ve set up Wikis as collaborative documentation efforts. Most of them resulted in only me contributing. Group initiatives like this need to come from the top, otherwise there is very little incentive for people to contribute. As a result, I’ve pretty much given up on Wikis for small teams/companies – it is nearly impossible to get momentum and maintain the information. This is sad because so much knowledge is lost and effort wasted as the same things are rehashed, relearned, and re-communicated.</p> As a result, I now focus primarily on:</p> My own personal notes system</li> Markdown in project Git repos</li> </ul> If I’m going to be the only one creating and reading notes, it may as well be in my own personal notes system which is most convenient for me.</p> For project documentation, this is best maintained in the Git repo as close as possible to the source code or design files. There it has a better chance of documentation being maintained if it is worked into the PR workflow and review process.</p> In the course of product development, documentation is the hardest deliverable to produce because its effects are hard to measure. Initially, the product still functions without documentation. Documentation costs short-term but produces compounding gains long-term. It is an investment in the future. If your team or organization is not interested in making this investment, you can always make it personally.</p> 🔧 What is your most important tool? 2024-12-12T00:00:00+00:00 Your editor? Your CAD tool? Your 2GHz oscilloscope? Your phone? Your computer?</p> In the information age, we are flooded with information and opportunities.</p> Many of the problems we solve are ones we’ve never seen before, or at least not very often.</p> Most of us have more to do than we can ever hope to get done in multiple lifetimes.</p> We are exposed to way more knowledge than we can ever fit in our finite memories. Therefore, a tool to efficiently capture, organize, and find your personal knowledge may be your most important tool.</p> If we figure out something, we can make a note so we don’t have to figure it out again the next time.</p> If we repeatedly do something that requires multiple steps, a checklist can make this easier.</p> If we learn something valuable, a note makes this useful in the future.</p> Creating notes encourages reflection.</p> With a good notes system, we can organize and prioritize what we do.</p> Over time this helps build a picture or model of what we know and have experienced. This model helps us leverage what we have done in the past in the future.</p> There are many good notetaking tools and methods (Notion, Workflowy, Logseq, Bullet Journal, etc.) so there is very likely one that will fit you. Some things I look for:</p> low friction to use</li> stores text and images</li> can be used on multiple mobile and desktop platforms</li> can export the information so that it can be moved to another system if necessary</li> has a stable history and likely future – something you can use for 5-10+ years</li> easy to share information with others</li> and most importantly – you own the information and can take it with you. IE, don’t store your personal knowledge base in your employer’s MS Teams OneNote app.</li> </ul> Workflowy has been perfect for me, but there are lots of other good options.</p> Over time your personal notetaking tool becomes a valuable knowledge base that you can take with you anywhere you go – part of YOUR Personal</a> Platform.</p> 📧 Inbox zero 2024-12-11T00:00:00+00:00 Thanks to a nudge from Khem</a>, I’m back to Inbox-zero.</p> The big question is what to do with emails that need action or have valuable information in them? Some put them in a special folder. Others tag them. Others forward them to lots of people.</p> The best solution is to move the action or information to another system such as a project ticketing system, or Workflowy (my personal notes system). I then archive the email and dig it out of the archive if I need to reply later.</p> Email is a great notification system, but it does not work very well at the following:</p> collaboration</li> knowledge capture</li> workflow</li> </ul> Find other solutions for the above such as:</p> Github/Gitea/Gitlab issues, projects, wikis, etc.</li> Markdown in Git is a great way to capture knowledge from emails</li> Trello</li> Personal notes system</li> </ul> Collaboration, knowledge capture, open access to knowledge, and workflow are key components of YOUR Platform. Getting information out of email is critical, and following an Inbox-zero policy is a great way to help make this happen.</p> 🏆 Goals vs. Systems 2024-12-10T00:00:00+00:00 “You do not rise to the level of your goals. You fall to the level of your systems.” – James Clear, Atomic Habits</p> </blockquote> Goals and vision are important, but we also need a way to get there.</p> 🏁 Easy to start vs. easy to finish 2024-12-09T00:00:00+00:00 There are many ways to classify how we do things. One way is “easy to start” or “easy to finish.”</p> We may try to use off-the-shelf “maker” style embedded boards (Arduino or Raspberry PI) instead of a SOM (System-on-module) designed for industrial use.</p> An interpreted language like Python is easy to use and definitely has its place on your workstation, tooling, etc. But deploying, cross-compiling, upgrading, and maintaining a Python stack on thousands of embedded Linux edge nodes is not easy – Go is much better over the long haul.</p> Skipping documentation, CI/CD, and other basic best practices is certainly quicker to start, but long-term will slow you down.</p> We can skip tests when writing code (often faster up front), or we can take the time to write tests as we implement features. Likewise, when we debug a problem, we can just fix the problem, or we can reproduce the problem first with a test, and then fix it to ensure it does not happen again.</p> Ignoring architecture up front is easier initially, but taking the time to think through architectural issues like decoupling, life-cycle management, and high-level performance, pays huge dividends long-term.</p> Outsourcing PCB design is easy to start, but owning the custom PCB designs in your product (especially the integration points) gives you the flexibility to easily fix, improve, and adapt.</p> When designing a PCB with constraints (such as high-speed or high-voltage), we can manually route the board or invest time in creating DRC rules and documentation that clearly details the board’s constraints. The first approach might be quicker to get rev 1 out, but the latter makes it much easier to maintain the design over time, onboard new people, etc.</p> When designing hardware, we can skip creating CAD libraries, assigning house part numbers and instead hand-create BOMs. But long term it becomes very difficult to manage even a small inventory parts in the lab, let alone production. We have duplicates, things are lost, and mistakes are made.</p> We can do things like we’ve always done them (certainly easier to start), but there are new technologies, tools, and workflows that can greatly improve our productivity.</p> We certainly need to push hard to get to an MVP (minimal viable product), but this should be done by cutting features, not the approach. However, most people do exactly the opposite – they want all the features up front and rather cut out the foundational investments in technology and process.</p> Here’s the thing – “easy to start” puts you on the path that accumulates technical debt. As time goes on, things become more complex, fragile, and difficult to manage. This approach does not scale.</p> “Easy to finish” puts you on the path to burn down technical debt. Over time you look for patterns and ways to simplify things. You continuously improve all aspects of the product. You select the best technologies at the time and adopt new technologies when it makes sense. This approach scales.</p> Starting is easy, and finishing is hard. What are you optimizing for?</p> 🎧 Use a headset optimized for calls 2024-12-06T00:00:00+00:00 One thing most of us do routinely is voice calls or video conferencing. For this to work, each party needs to have a microphone and a speaker.</p> The simplest setup is to use the microphone and speaker built into your phone, laptop, or car. Sometimes this works pretty well.</p> But, often we are environments where this is background noise. Simple microphones, especially those some distance from our mouths, cannot discern background noise from our voice. AGC (automatic gain control) algorithms tend to amplify any sound to capture your voice. The most baffling scenario is those walking through a busy airport holding a phone in front of them in speakerphone mode. Another disruptive problem is echo which sometimes occurs when using a laptop mic/speaker for video conferencing. Trying to talk when your voice is being echoed back is nearly impossible.</p> Conversely, our ears are very good at differentiating what we want to hear from local background noise. Our brain and ears are very sophisticated instruments</a> and include features such as using the time delay and intensity difference between each ear to pinpoint a source, selective focus, frequency tuning, adaptive processing, etc.</p> Some headsets attempt to block out background noise to the microphone. The most effective ones have a direction microphone located very close to your mouth (on a boom), or a microphone array and DSP processing. Two examples I’ve used are the Plantronics - Voyager 5200</a> (Bluetooth) and the Sennheiser SC 160</a> (USB).</p> </p> </p> There are likely many other good options. I use the Plantronics with my phone and the Sennheiser with my computer. Both are very effective at blocking out background noise and I’m sure there are many other effective models.</p> The result is whomever I am communicating with can clearly hear what I am saying without being distracted by background noise in my environment. This makes me more effective.</p> Some headsets have noise noise cancellation for the speakers. This is very nice for deep work when we are trying to focus and need to block out all sounds other than perhaps some music. But they are of marginal use during a call because our ears are already pretty good at focusing on one sound and during a call we have the discussion to focus on.</p> Consider having separate headsets for deep work (noise-cancelling speakers) and calls (noise-reducing microphones) as part of YOUR Personal Platform</a>. This may seem like a minor thing, but the small things add up. Using a good microphone is a courtesy to those listening to us. It is a courtesy to the entire group on the call that our background noise is not interrupting the call. It is professional. It shows we care about others.</p> For more discussion and review, see this thread</a>.</p> ⚡ Are your processes proactive or reactive? 2024-12-05T00:00:00+00:00 There are two ways to implement a process.</p> The traditional way is as a check to make sure people did everything right. This could be required documentation that is written after a project is finished, a separate QA department that runs tests after each release, or various certifications like UL, FCC, CE, etc. There are certainly times when things like this are necessary; however, writing documentation after the fact is never much fun, and dealing with failure reports from the QA department is not all that pleasant, and the FCC testing facility has been called “the house of pain.” We’ll call this the “reactive” process approach.</p> An alternative approach is the “proactive” process. Structure your documentation efforts as part of the development process – as a way to help you think and as a vehicle for collaboration. QA can also be implemented proactively as automated tests that are run during development.</p> Proactive processes focus on well-designed tools that help the people doing the work get it right vs. tools that help managers who oversee the work. Proactive process tools are well-designed, fast, and a pleasure to use.</p> Think what could happen if UL focused on providing KiCad templates and DRC rule sets for various high-voltage design scenarios that would help engineers do their work instead of a bureaucratic morass that is very painful and expensive to work through. The first approach encourages innovation, whereas the current approach greatly hinders it.</p> Some reactive processes are necessary, but where is your focus? The more you can shift your processes from reactive to proactive, the better off you’ll be. This is what Platform Thinking is all about – building YOUR Proactive Platform.</p> ⚖️ Architecture vs. Optimization 2024-11-27T00:00:00+00:00 In 1968, “The Art of Computer Programming” by Donald Knuth was published with the following quote:</p> “We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.”</p> </blockquote> This was in the age when computer resources were scarce.</p> We now live in the age of abundant computing resources, so it seems this statement would be many times truer now.</p> Performance is important when we select technologies. For instance, Go or Rust is significantly faster than Python and it is important to think about issues like this up front. This is an architectural issue.</p> Deciding if we will use a message bus or not is an architectural issue.</p> However, if we receive data over a network, especially in distributed IoT systems, the network is relatively slow. So it rarely pays to optimize how we process data from remote systems until we reach a pretty large scale. This is an optimization issue.</p> Creating an abstraction to reduce some duplicated code is an optimization.</p> With architecture, we need to think about it up front.</p> It rarely pays to optimize up front – that is best done later after some learning.</p> And if we spend too much time optimizing, we never get to the learning. And if we don’t learn from testing a working system, we’re likely optimizing the wrong things.</p> Architecture vs Optimization – there is a difference.</p> Creativity 2024-11-25T00:00:00+00:00 “Creativity is just connecting things. When you ask creative people how they did something, they feel a little guilty because they didn’t really do it, they just saw something. It seemed obvious to them after a while. That’s because they were able to connect experiences they’ve had and synthesize new things“ – Steve Jobs</p> </blockquote> (Thanks Rod</a> for this quote!)</p> As technology advances, being good at making connections (integration) is the critical skill, as we can’t create it all ourselves.</p> Part of YOUR Platform is building a high-quality toolkit of things to connect.</p> And knowing how, when, and where to connect.</p> 🔗 Coupling 2024-11-22T00:00:00+00:00 Good platforms emphasize loose coupling where one part of the system can change without affecting another.</p> Not abstractions for the sake of abstraction.</p> But rather well-thought-out data structures and patterns.</p> A message bus like NATS is an excellent way to decouple a program. One part of the system publishes information, and any other interested part can subscribe.</p> But for this to work, you must have simple and flexible data structures that don’t change often. Otherwise, your system is now tightly coupled again.</p> Simple data structures and messaging</a> – a sane way to scale YOUR Platform.</p> ⏰ A platform stuck in time 2024-11-21T00:00:00+00:00 Vintage is great in some contexts—I like old woodworking hand tools. Even in our modern jobs, some vintage tools, like a pencil and paper, are still very useful.</p> But information technology moves forward, and we rarely benefit from using decade-old phones, computers, programs, etc.</p> New information technology and tools are often simpler to use and more powerful. Some examples:</p> C -> Go/Rust/Zig</p> </li> Apache -> Caddy</p> </li> Vim -> Helix</p> </li> x86 -> ARM -> RISCv (MPU)</p> </li> 8051 -> PIC -> ARM -> RISCv (MCU)</p> </li> FreeRTOS -> Zephyr</p> </li> Kafka -> NATS</p> </li> Rotating storage -> Solid state</p> </li> </ul> Information technology, and perhaps computer technology in general, is different. The physical constraints are largely removed as we scale into GHz and Terabytes, which opens the doors to human innovation and collaboration.</p> Innovation is largely a process of discovery, so you can’t predict where it will go and what will happen. It is a result of development iterations that are now happening at a massive scale and distributed across the globe.</p> Therefore, when you lock yourself into some version of technology in the name of stability, cost savings, or whatever, you are also limiting the potential for your product – for interesting things to happen that no one could have predicted.</p> It takes humility to acknowledge this – that you can’t see the future, the best you can do is prepare for it.</p> YOUR Platform is the vehicle that keeps your product development moving forward with technology so that you are not stuck in time as you watch it pass you by.</p> 🧪 What is a good abstraction? 2024-11-20T00:00:00+00:00 When writing a program, abstraction is a powerful tool.</p> Every abstraction has a benefit and a cost. Hopefully, the value > cost.</p> There are good abstractions and bad abstractions.</p> Good abstractions break things down into simpler steps that others can understand.</p> Bad abstractions may be brilliant but are nearly impossible for anyone but the author to understand.</p> Who are you optimizing for? Yourself, or everyone else?</p> This often tells the difference between a good and bad abstraction.</p> ⚡ Improve development efficiency, reduce technical debt 2024-11-18T00:00:00+00:00 The term “platform” can mean a lot of things, but this is primarily what I’m talking about in the context of product development:</p> Improve development efficiency, reduce technical debt</em></strong></p> Some examples:</p> do more with what you have</p> </li> iterate faster</p> </li> make less mistakes</p> </li> regularly update to new software versions</p> </li> collaborate better</p> </li> use better tools</p> </li> automate where you can</p> </li> reduce legacy (untested) code</p> </li> deploy with confidence</p> </li> react quickly to new customer needs or problems</p> </li> </ul> Of course, everyone wants these things and claims to be doing them. But in your gut, you know if these things are getting better or worse.</p> And the only way I know to get better is to work on YOUR Platform.</p> 🎯 Rituals and Culture 2024-11-14T00:00:00+00:00 Recently, we discussed</a> personal habits.</p> Organizations have habits too – we call them rituals and culture.</p> How does the organization react when things don’t go as planned?</p> How does the organization motivate and demotivate people?</p> How does the organization encourage openness and transparency?</p> A lot of this probably comes down to those at the top – the art of leadership.</p> Organizations are made of people, so the struggles we face personally we also face organizationally.</p> We need some structure and tooling to help us do our job.</p> To ensure the boring parts are taken care of.</p> To catch errors we overlook.</p> Platforms.</p> Does your organization celebrate platforms? If so, how?</p> 📈 Two ways to scale 2024-11-13T00:00:00+00:00 Scaling is the goal of many businesses – grow revenue, serve more people, sell more devices, grow the product line, etc.</p> There are two ways to scale:</p> Increase volume with a small number of direct customers. You might be making an integrated circuit that is sold to a company making consumer devices, or you might be making a consumer product that is sold through a few outlets like Amazon and Walmart. Although you might have thousands of end customers, you only have a few direct customers.</p> </li> Increase the number of direct customers. Many of these customers may be low volume, but the aggregate can be large, sometimes described as the long tail</a>.</p> </li> </ol> (For this discussion, we’ll assume you already have a product or technology that meets a real need).</p> With approach #1, you can brute force about anything. As long as the product is something people want, you can dedicate application engineers to help your customers make it happen. Sometimes the design/development team interacts directly with customers. This is expensive, and the reason many industrial products are costly is that they rely on direct support.</p> However, this does not scale past a handful of customers. Direct application engineering support is expensive. If you want to scale the number of customers, you need to do something different.</p> Open-source projects are an interesting study in this regard. They address well the needs of many users. How do they do this?</p> There are two primary aspects: A) Quality, and B) Community.</p> For any complex technology to scale without a lot of hand-holding, the quality needs to be high. There has to be good documentation. The source code or design needs to be readable. The architecture needs to be simple and solid so it can be understood. You need consistency (data structures, code organization, etc.). Testing needs to be automated and robust so there are not a lot of unexpected problems.</p> The second thing that is needed is community. A community is the only way to support a large number of users or customers of a complex technology. With complex systems, the problems you run into are often things you have never seen before. If all of these problems need to be solved from scratch with direct one-on-one support, you will quickly become buried. Additionally, you’ll find that experienced users are often better at solving user problems than technology creators. If this support is done in a public place, you soon build up a knowledge base that can be reused.</p> Quality and Community are the ways to scale the number of direct customers or users, and OSS-style workflows and communities are the best way to accomplish this.</p> If you want to address the long tail, the days of NDAs with every customer are past. The future is open. Not necessarily 100% open source, but certainly open documentation and support. And “open” provides feedback that increases quality – it all works together.</p> 🔍 How to get the little things done? 2024-11-11T00:00:00+00:00 Previously, we discussed habits</a>. Habits are the way we get the small, but important, things done.</p> Some people are naturally neat, organized, and focused – I’m not. Stuff tends to pile up in my workspace and I don’t even notice it.</p> One thing I’ve been using some is the Pomodoro technique</a>. This is a time management method where you use a timer to break down work into intervals – typically 25 minutes in length separated by 5m breaks.</p> The primary goal of the Pomodoro technique is to increase focus, and that is a worthy goal.</p> However, another benefit of this technique is to allocate time for the little things, like organizing your workspace which is a good use of your 5m breaks.</p> These are the little things that clear space to do the big things – part of YOUR Personal Platform.</p> 😊 How can you make your commercial source code customers happy? 2024-11-08T00:00:00+00:00 When engaging with a community around an open-source project, Github is likely the lowest friction option. It makes sense.</p> But there are other scenarios – one is where you are delivering closed-source commercial source code to customers.</p> Git makes sense for delivering any source code, no matter what the context – closed, open, etc.</p> Using zip files to deliver source is antiquated. Since Subversion (2000) and Git (2005), there are now other options.</p> With Git, is easy to deliver updates, manage changes, tag releases, and track what has happened.</p> Tools like Gitea/Forgejo, that wrap Git, provide nice ways to interact around the source code – issues, pull-requests, projects, etc. Issues and pull-requests are much better ways to communicate about code than email or other messaging methods. The communication is close to the code and preserved instead of being disconnected, and eventually lost.</p> Gitea/Forgejo also give us unlimited organizations and users that we can create and manage. What this means is every customer can have their own organization. This provides issolation between customers. Each now has their own space to raise issues, discussion PRs, etc. You can mirror the source code they have purchased to their own organizational space, and continue updating it as long as they are a customer. If they decide to quit purchasing updates, you can leave their Gitea organization and source code intact, but simply quit updating it. This is ensures you do break build processes they already have in place and is a measure of good will.</p> Why not deliver the source-code to the customer’s Github organization? You have less control in Github. Simple things like users not being able to create an account are now under your control. With Gitea, you have more control of the repo visibility. It is less likely someone might accidentally make a repo public in the future.</p> Would you like the velocity, stability, and collaboration that OSS projects have? With tools like Gitea/Forgejo, you can mirror these workflows into any software development effort.</p> 🚀 Platforms vs Architecture Astronautics 2024-11-07T00:00:00+00:00 The excellent Superstruct Manifesto</a> addresses the tendency for developers to want to “rewrite” things, wholesale switch to new frameworks, jump on new tech, etc. vs. delivering real business value.</p> Another description for these people is Architecture Astronauts</a>.</p> Platforms, at least the way I envision them, are the opposite of this.</p> They are incremental, not wholesale.</p> They are a part of our daily thinking, not some separate grandiose effort that will solve all of our problems.</p> They are largely invisible.</p> They leverage what we already have and make it better.</p> They are the responsibility of everyone, from the bottom to the top, not some elite team.</p> However, like personal habits, company platforms take discipline to implement and are sometimes difficult to get established.</p> There are exceptions, but if platform work is exciting and interesting, this is a warning flag – you may be venturing into architecture astronautics. Generally, we need to block out time in our schedule to work on platforms as it does not just happen because it is fun – kind of like brushing your teeth and exercising.</p> But in the end, platforms make the difference whether you have a sustainable business or just some talented individuals. And just like exercising, or cleaning up our workspace, it is very satisfying afterward.</p> 💪 Habits -- a critical ingredient in YOUR Personal Platform 2024-11-06T00:00:00+00:00 We defined</a> your platform as the collection of technology, tools, and processes that enable you to differentiate and consistently deliver your products and services.</p> At a personal level</a>, habits are critical – they enable us to do the boring/hard stuff that makes us effective.</p> Getting up early gives us undistracted time to think.</p> Regularly exercising helps us in so many ways.</p> Patterns and tools</a> that help us plan and focus are sometimes necessary.</p> Bad habits (sleeping in, overeating, reading instead of working, etc.) are easy to fall into.</p> Good habits are amazingly hard to establish, but worth it, and a big part of YOUR Personal Platform.</p> 🚀 Advanced technology and small players 2024-11-05T00:00:00+00:00 Integrated circuits (IC) vary widely in complexity from the small devices that are easily documented and understood to complex beasts that would require a stack of books and years to fully document.</p> If you are a high-volume manufacturer of consumer goods, you can easily get all the attention and support you need from IC vendors. They will help you design your product.</p> However, if you are a low volume manufacturer, it is nearly impossible to get direct support for complex ICs. Support resources are limited and IC vendors can’t help everyone in the world design their products.</p> So they allocate support engineers to a small number of higher-volume customers.</p> It does not matter how important you think you are, or how much money you have, if you don’t have volume, it will be hard to get direct support from complex IC vendors. Their margins are small and they make it up in volume.</p> However, there is a path forward for low-volume manufacturers.</p> You can buy system-on-modules (SOM) from vendors who do the hard parts of integrating a processor, RAM, power supplies, and other complex ICs on a module.</p> Additionally, they provide open-source software for these modules. Open-source has changed the game for software by providing everyone access to advanced technology.</p> The third aspect of this is community. If a technology vendor has a vibrant community around their products, then the community can support the low-volume customers. This is the best and most sustainable model.</p> The difficult part in all this is discerning where the line is. What technology is accessible to low-volume manufacturers and what is not?</p> A few red flags:</p> SOM or other module technologies that don’t have a clear support model for industrial customers.</li> The technology is not open and requires NDAs to access documentation.</li> The documentation is inadequate and difficult to understand. The only way you’ll figure it out is by the vendor helping you.</li> There is not a vibrant community around the products.</li> Supporting software is not open-source.</li> </ul> The accessibility of technology today has spurred tremendous innovation. Open-source and community have changed the game enabling even small players to use very advanced technology in their products. But you must understand your limits. There is technology that is accessible to small players and technology that is not.</p> IC vendors who put the pieces in place (openness, open-source, and community) can benefit from the long tail</a> of distribution, which increasingly is where innovation is happening.</p> What can kids teach us about platforms? 2024-11-01T00:00:00+00:00 When helping my children with math, one of the biggest struggles is to teach them to break math problems into small steps. They are smart and think they can do it in their head, and inevitably make mistakes because everything gets muddled.</p> When you break a math problem into steps, it is easy – because each step is easy.</p> And writing out the steps is easy. It’s not rocket science.</p> But what is not easy is the discipline to write out the problem step-by-step instead of rushing to the answer.</p> YOUR Platform is the same way – it is breaking down problems into simple steps.</p> It’s not rocket science.</p> But it does take discipline to create a checklist, document a process, break things down into smaller steps, automate some things, look for patterns, etc.</p> Getting to simplicity is hard, but once you get there, everything is easier.</p> 🏛️ What are the pillars of a good platform? 2024-10-31T00:00:00+00:00 Simplicity</strong> - you can make changes when needed, and understand the system enough to do so.</p> Consistency</strong> - useful patterns are reused throughout the system where appropriate.</p> Reliability</strong> - the design/implementation is sound and there is enough testing that you don’t worry too much when deploying updates.</p> Velocity</strong> - iterations are fast and easy to deploy.</p> 🛣️ Kayaks and Platforms 2024-10-30T00:00:00+00:00 Recently we purchased a few used kayaks that had been built 20 years ago from Pygmy Boats kits</a>. (Unfortunately, Pygmy is not longer supplying kits, but similar kits can be purchased from Chesapeake Light Craft</a>)</p> </p> We had the pleasure of visiting with the former owner (Mike) for an hour or so and he generously shared his knowledge about the boats, and kayaking in general.</p> The more I listened to Mike, the more I realized these boats were his platform for getting on the water.</p> Building a boat like this from a kit is a lot of work, but in the end you have:</p> a nice boat – performs as good as boats 4x the cost</li> one that is customized to exactly what you need (features, rigging, seat, bracing, etc).</li> can be easily modified in the future</li> </ul> Additionally, you have the know-how and tools to repair and maintain it.</p> If the hull is damaged, you can get your extra plywood, fiberglass cloth, and epoxy, and fix it.</p> If you don’t like the rigging, you can fasten new points and fill the old holes.</p> If you don’t like the seat, and can hand-carve a new one out of a block of foam.</p> If the boat is scratched up, a little sandpaper and varnish will have it looking like new.</p> It’s not for everyone, but for those who want to own their kayak platform, a stitch and glue</a> boat is nice option.</p> 💻 What is the purpose of technology? 2024-10-29T00:00:00+00:00 To do more with less effort.</p> To simplify our tasks.</p> Sometimes this backfires and technology ends up giving us more to do and complicates our lives.</p> It becomes a monster and we are consumed feeding it.</p> Thus, the selection and implementation of technology is critical.</p> How do you objectively measure this?</p> Perhaps we need a term “technical efficiency”</p> technical efficiency = time saved/time feeding</code></p> Hard to measure, but important to reflect on.</p> A number greater than 1 is good, less than 1 is bad. Ideally, the number should be much greater than one and also increasing.</p> Like with so many things, the key metric is not where we are now, but where we are heading.</p> ✨ The easiest path to reliability ... 2024-10-25T00:00:00+00:00 … is simplicity.</p> Not that we dumb things down and avoid new technology.</p> That is false simplicity.</p> As we discussed previously</a>, the problem is most often not with the underlying technology, but rather the stuff we do.</p> Therefore, if we can simplify what we do, it will get better.</p> This is a fundamental purpose of YOUR Platform – to simplify what you do. To make it more repeatable, deployable, reliable, and stable.</p> Technology and tools can help, but in the end what you do and how you do it is your responsibility. You can’t buy a solution for this, nor can AI do it all for you.</p> 🔍 Where do reliability/stability problems typically come from? 2024-10-24T00:00:00+00:00 99% of the time, system bugs/problems come from the stuff you do.</p> The code you wrote.</p> The hardware you designed.</p> The system you configured.</p> The custom PRC system you created.</p> Rarely are we affected by stability problems in the Linux kernel, the Zephyr RTOS, an OSS library, or a bug in a chip.</p> Why? Because these components have many more users and developers than the in-house pieces we develop.</p> “Given enough eyeballs, all bugs are shallow.”</a></p> Why are we so hesitant to update the open source system software in our product, but deploy new application updates that one developer created without a thought?</p> And ironically, the best way to avoid the rare problem in OSS components is to keep up-to-date.</p> We should be most suspect of the parts of the systems with few developers and users.</p> This is why YOUR Platform is so important – because you reuse it, more developers work on it, and more users use it.</p> And by extension why it is so beneficial to open source parts of what you do. Even if you get only a few more users, it will get a lot better, making your products better.</p> 🔝 Support from the top 2024-10-23T00:00:00+00:00 In a recent discussion with Matthew Rassi</a> about Lean, I asked Matthew if he has ever witnessed Lean being successful in organizations where there is not support at the top. He answered with a resounding no.</p> Likewise, I’ve never observed “Platform Thinking” being successful company-wide without support from management.</p> If Platform thinking does not start at the top, it is a hard go.</p> These are cultural issues, and perhaps the most important influence those in leadership positions have is culture.</p> However, you can still apply platform thinking at the personal or sometimes team level</a>.</p> 🎯 Doing more with less, rather than less with more 2024-10-22T00:00:00+00:00 Building your platform does not mean you need to invent a bunch of stuff from scratch, or purchase expensive tools.</p> Rather, it is making better use of what you already have.</p> Automating your current workflows and deployment.</p> Reusing the designs and IP you already have.</p> Increasing the rate of iterations.</p> Automated testing so that you can deploy with confidence.</p> Platforms are not rocket science, and you don’t need fancy tools to get started.</p> A checklist can do wonders.</p> A few shell scripts and a cron job can get you a long ways.</p> Automation can be written in the languages you are already using on the project.</p> Building testing and diagnostics into your products rather than needing expensive, external fixtures.</p> Leverage the CI features built into tools like Github and Gitea.</p> Platforms are the boring work that clear the path and you give you time and space to do the interesting work that adds value to your products.</p> 🆘 How to ask for help 2024-10-21T00:00:00+00:00 Modern systems are too complex for any of us to know everything about them.</p> Many of the problems we encounter we have never seen before.</p> Communities built around various technologies are what fill this gap.</p> So the superpower in this age is first having the humility to ask for help, and then knowing who and how to ask for help.</p> Before asking anything, do a reasonable amount of work.</p> Someone once told me he puts thirty minutes into a problem before asking a co-worker for help, which seems like a good baseline. Any less than that you are likely needlessly interrupting and distracting someone from their work. Any more than that you may be spinning your wheels needlessly. The nature of the problem may justify more at times.</p> Sometimes our questions are met with silence.</p> Don’t just remind people that you need help.</p> Do more work, and bring more information, and then ask again.</p> If you ask more questions, always bring more information.</p> Help people help you.</p> 🔧 How to keep a product maintainable? 2024-10-18T00:00:00+00:00 How do you know if you have the ability to maintain your product?</p> With complex systems, having all the design files and source code does not always mean a lot.</p> Do you know how to build the source code?</p> Do you know why the hardware is designed the way it is?</p> Do you know how to test and verify changes?</p> Do you regularly use these tools?</p> The only way to know for sure if you can efficiently make a change to your product is to practice doing so.</p> If you don’t, things soon ossify to the point where the cost and risk of any change are prohibitive.</p> 🤝 What happens when someone else tries to use your work? 2024-10-17T00:00:00+00:00 It does not build on machine X …</p> It does not work in scenario Y …</p> It is hard to understand by developer Z …</p> As developers, we naturally avoid the rough edges of whatever we are working on – it hurts.</p> When other developers try to build/run our stuff, it is amazing what they find.</p> The more people you have building and running your stuff, the better it will be.</p> Even two people vs. one makes a huge difference.</p> The platform approach says anyone should be able to easily check out, build, and run any code on any machine.</p> We’ve run this experiment with OSS projects for a long time – it works.</p> Encourage others to build and run your code – it will make YOUR Platform better.</p> 📌 Where do you put stuff? 2024-10-16T00:00:00+00:00 There are a lot of ways to organize product development information.</p> You could do something like:</p> productA</strong> firmware</li> hardware</li> yocto-build</li> docs</li> manufacturing</li> </ul> </li> productB</strong> firmware</li> hardware</li> yocto-build</li> docs</li> manufacturing</li> </ul> </li> </ul> The above is nice in that it keeps all the stuff for each product together, but discourages re-use between projects.</p> If you want to build a platform, then it may make more sense to do something like this:</p> Firmware</strong> Boards productA</li> productB</li> </ul> </li> Apps</li> Common</li> </ul> </li> Hardware</strong> Part Libs</li> productA</li> productB</li> </ul> </li> Yocto</strong> …</li> </ul> </li> Manufacturing</strong> Common testing framework</li> </ul> </li> Product</strong> productA</li> productB</li> </ul> </li> Doc</strong> (general technology information)</li> (general processes, etc)</li> </ul> </li> </ul> Each type of code/design files live in a common location if possible. The Product</code> location is only for information that is truly unique to a specific product (documentation, top-level BOMs, etc) that does not fit anywhere else. In many cases, each of the top level categories are a separate repo. The Product</code> repo may pull the other repos in as submodules, so you can version everything together.</p> If you want to build a platform, it helps to be intentional about this. Reuse and cross-product learning won’t just happen.</p> 🌱 When edge nodes grow up 2024-10-15T00:00:00+00:00 Yesterday</a> we discussed how to partition functionality in hierarchical distributed systems.</p> In the past, it was difficult to do much at the edge nodes because they could not do much.</p> Resources were limited.</p> It was difficult to update software in them without being on-site and connecting a PC-based programmer, or before that replacing an OTP (one-time-programmable) chip.</p> But today things are different.</p> A powerful 32-bit MCU is as now as cheap as your father’s 8-bit PIC and your grandfather’s 8051.</p> It has plenty of memory for network/communication stacks, and in-system update routines.</p> We now have powerful operating systems like Zephyr that we can run on these edge nodes.</p> If we continue our analogy to human systems, when a person is young or inexperienced, we don’t push a lot of responsibility on them.</p> However, when they grow up, we want them to do more, make decisions, and look around and do what needs done.</p> Modern MCU’s are capable of doing a lot including running AI algorithms</a> – maybe it is time our edge nodes grow up.</p> 🔠 How do you partition systems? 2024-10-14T00:00:00+00:00 In modern networked systems, there is often the debate how to partition the system.</p> Below is one example where you have three levels: cloud, gateway, and IO (input/output) nodes.</p> </p> All of these devices have a processor and can run software and do stuff.</p> So where do you do stuff? There are two basic philosophies:</p> Push as much of the processing upstream to the cloud (where you have “infinite” resources) and never update downstream devices.</p> </li> Do as much at the Gateway and IO nodes as possible, make decisions when possible with local data, and only push processed data and events upstream. This requires routinely updating the downstream devices as needs change and better algorithms are developed.</p> </li> </ol> Initially, approach #1 seems simpler and may be appropriate in some cases.</p> You can centralize logic and decisions as much as possible.</p> It is easy to update the system – you only have to update one central piece.</p> Until you need to scale, or there is a network disruption …</p> And have 50 IO nodes per site that are sending frequent unprocessed data all the way to the cloud, and decisions are coming back down, even when nothing is changing.</p> And then you eventually have 1000 sites.</p> And then there is a cloud outage and the system is in limbo until the cloud issue is fixed. All data during the outage is lost.</p> And someone damages the CAN bus cable and an actuator is left in the ON state and a tank overflows because no one is making decisions.</p> And you are storing Gigabytes of unprocessed historical data that you never use.</p> Distributed systems are hard, but in the end they scale and are more resilient, especially when you are forced to be distributed by device locations. Within reason, the more processing you push to the edge, the better. “Distributed” in these systems is not an architectural choice, but is forced on us by topology. Why not leverage this since we need to deal with it anyway.</p> We can compare this to human systems. How well do autocratic, centralized organizations work? Where someone at the top makes all the decisions and the rest are minions?</p> They may work OK at a small scale, but even that is questionable.</p> A student of history soon observes that successful organizations push responsibility down. People at the top provide oversight but are only involved in local decision-making when coordination is required. Trusted people feed necessary information up as required.</p> Is it any different with hierarchical, distributed digital systems?</p> 🏷️ How do you consistently name things? 2024-10-10T00:00:00+00:00 There are only two hard things in Computer Science: cache invalidation and naming things. – Phil Karlton</p> </blockquote> Plenty of opinions abound on style, naming conventions, formatting, etc.</p> But there is one principle I try to follow when naming things – go from general to specific. Example:</p> UserEmailAddress</code></li> </ul> For functions, you may have a module that has multiple operations. In this case, the operation is a specific operation of a more general module.</p> atmtcp_v_dev_open()</code></li> atmtcp_v_dev_init()</code></li> atmtcp_v_dev_close()</code></li> </ul> Even though open_atmtcp_v_dev()</code> reads better as an English sentence. Programs can be thought as both prose and data, and sometimes there are advantages to considering function and variable names as data.</p> General->specific naturally organizes things. It is easy to browse variables in an editor that may list all the variables in a module. Searching/refactoring is also easier.</p> Sometimes when dealing with two unrelated types of information, it is not obvious which is more specific. In the below example, VINST/VPEAK</code> are measurements (type), and AX/AY/...</code> are channels (instance). Which should go first?</p> VINST_AX</code></li> VINST_AY</code></li> VINST_AZ</code></li> VPEAK_AX</code></li> …</li> </ul> Generally I prefer to put the “type” first, and then the “instance” – especially if there are multiple instances. But this could probably go either way.</p> Another way to think about this is VINST</code> may be a map/dictionary/hash-table property of an object, and AX/AY/AZ</code> may be keys into this map.</p> Use nouns for variables and verbs for functions.</p> Does this make sense? How do you name things?</p> 🔄 How can we be more consistent? 2024-10-09T00:00:00+00:00 What does not work very well is long standards and endless code reviews where we shame people into compliance. There are better ways.</p> We now have tools that can proactively format our code. We have linting tools that check things. Use them. Even if no one else reads our code, they help us.</p> Write tests. Perhaps their greatest value is that they give us a new perspective on our work, which leads to consistency.</p> We can have CI hooks that check for various things – check out the Zephyr project if you want a good example of this.</p> Being nudged by a CI tool to be consistent is a much better experience than being pulled over by the consistency police.</p> Does YOUR Platform have tooling that encourages consistency?</p> 🔄 Does consistency matter? 2024-10-08T00:00:00+00:00 If you have a single developer on a single project, then perhaps consistency does not matter too much.</p> However, if you want to scale, either products or developers, then consistency matters.</p> Why?</p> So that code does not get drastically reformatted every time someone makes a change, making Git diffs impossible to review.</p> So that any developer can easily understand and make changes in any part of the codebase.</p> So that new products can leverage previous efforts.</p> So that new developers can be more easily on-boarded.</p> So that we can see patterns and simplify systems.</p> So that our systems are tested.</p> So that documentation can be easily found.</p> Linux Torvalds is being lambasted</a> for encouraging some consistency in Git commits.</p> But if you read his actual email</a>, the request seems quite reasonable.</p> The Linux kernel has a well-defined coding style</a> that all contributors are expected to follow.</p> Have we considered the impact this emphasis on consistency has had on the Linux Kernel’s success?</p> The irony of all this is that consistency is usually done in the name of the “team” or “reuse”. But if we reflect a bit, we are mostly just helping ourselves.</p> We can read and understand our own code in 6 months.</p> We can find stuff.</p> We can more easily make changes and improvements.</p> We have tools helping us.</p> A little bit of consistency goes a long way in building YOUR Platform.</p> 🎲 Isn't it risky to update your dependencies? 2024-10-07T00:00:00+00:00 This is a common objection I hear when building industrial systems: “We want to lock things down to a super stable/tested LTS (Long Term Support) release and then stay on that release for a long time – it’s risky to update dependencies.”</p> Is it?</p> How often do you update your browser?</p> Your phone?</p> How often does Windows or MacOS force you to update your computer OS?</p> Do you worry every time it updates?</p> I’ve run Arch Linux for years and update routinely without worrying.</p> I update to new versions of Gitea every time they come without a concern.</p> I routinely update to the latest HEAD of Zephyr on projects during development and have rarely had a problem.</p> The same with about every software component I use.</p> Yes, there are safety-critical control systems that have stringent testing requirements, but we’re talking about complex connected systems that are mainly concerned with moving data around.</p> Where security is a concern.</p> With rare exceptions, modern OSS projects get more stable with each release, and to a lesser extent with each Git commit.</p> They have defied the laws of entropy.</p> How? With OSS workflows, testing, continuous integration (CI), more real-world usage, more user feedback and contributions, etc.</p> With good CI, changes don’t get merged to main until they are tested pretty well.</p> Transparency, community, and OSS workflows are powerful – really the only practical way to build complex technology.</p> The next time you seek the cozy cocoon of an LTS release for a dependency in YOUR Platform, think about what you might be giving up … features, improvements, community connection, and likely also stability.</p> 🏗️ Platforms are for building systems 2024-10-04T00:00:00+00:00 If you are building a one-off, non-connected device, you can get by without a platform.</p> This is why so many design-shops don’t get platforms – they are designing something then moving on to the next project.</p> But if you are building a connected system, you have a much bigger problem to solve.</p> You now have a distributed system, and distributed systems are hard.</p> You are now living on the Internet with all its associated security concerns.</p> You have a system that has the potential to do so much more than it is doing now.</p> You have a system that has almost unlimited potential to be expanded.</p> You (potentially) have a platform.</p> 💸 The cost of updating dependencies, or not 2024-10-02T00:00:00+00:00 As developers, we are often lazy when it comes to updating dependencies.</p> A short-term productivity hack is to not update them.</p> Leave our Yocto build at an old version.</p> Never touch go.mod</code> or package.json</code> – everything is working and I can keep focusing on coding features.</p> Don’t update our tools – we don’t have time.</p> … until things break, there is a security problem in a dependency, or we need a feature in a new version of something, etc.</p> And then things grind to a halt.</p> As Khem recently shared, Maintenance is costlier than development</a>, so even though development is important today, maintenance is more important – for tomorrow.</p> Part of YOUR Platform should be selecting technologies that can be updated regularly with little pain, and a process to do this.</p> It is the question of paying a little bit continuously, or a lot all at once later, and the latter is often so painful that in many cases it is impractical.</p> Investing in YOUR Platforming compounds positive gains – neglecting technical debt compounds negative gains.</p> 🆚 What is the difference between YOUR Platform and other platforms? 2024-10-01T00:00:00+00:00 We all use other platforms – operating systems, cloud services, middle-ware, hardware modules, etc.</p> It is tempting when building a product to piggyback entirely on someone else’s platform (AWS, .NET, one of the 100’s of IoT Platforms, etc.)</p> Society tells us – you can’t host your own service, deploy your own updates, design your own hardware, implement reliable systems, etc.</p> But at a small/medium scale, none of these things are very hard.</p> If you do them, you can simplify and optimize for your needs.</p> YOUR Platform is partly the ability to leverage other platforms, but also to build your own – where you are in control of the critical integration points.</p> 💪 Good enough is not good enough 2024-09-27T00:00:00+00:00 </p> My grandfather had this sign on his desk for years, and it embodied his philosophy for work.</p> He did things right.</p> He read manuals for his equipment cover to cover.</p> He understood how things worked.</p> And because of this, he could fix and maintain machines. Lots of people brought their machines to him.</p> Is your product development process trapped in “good enough?”</p> Do you understand the technology used to build your products?</p> Have you read the manuals?</p> Do you know how to fix things when they break?</p> Is your process maintained? Can you update the pieces?</p> Platforms are long-term prospects.</p> And because YOUR Platform is never good enough, over time it gets really good.</p> ⚠️ When things go wrong ... 2024-09-26T00:00:00+00:00 What do we do?</p> Do we focus on who/what to blame?</p> Or do we figure out a path forward.</p> How we are going to prevent this problem in the future?</p> Not by shaming someone into paralysis, but by fixing the process.</p> By improving YOUR Platform.</p> The opportunity for the individual/organization who made the mistake to help fix the process is a graceful way out and preserves their dignity.</p> 🚀 Do you own your deployment? 2024-09-23T00:00:00+00:00 Saturday morning, I got a call from a customer – something was not working due to a bug we had deployed Friday (no we don’t have very good tests) 🤫.</p> The fix was easy, I tested it locally, and then tried to push it to a Git hosting service we are using, but the Git service was down.</p> Now what? Our Ansible deployment script pulled directly from git, built the program, and then deployed it.</p> While I could reverse engineer the build from the Ansible scripts and do it manually, that would have taken time and introduced the possibility of another error.</p> So I pushed the repo to my Gitea server, tweaked the repo line in the Ansible script, and deployed the update – not a big deal.</p> This brings up a question though – we don’t usually think of deployment as critical infrastructure – not a big deal if it is not working – until you need to fix something quickly in production.</p> What if the deployment was wrapped up in some CI/CD workflow that only worked in vendor X’s cloud service?</p> Maybe simple deployments are actually a better – a shell script that lives in the project repo that you can run anywhere. This could still be called by a CI process for the normal workflow.</p> All computing systems have the potential to fail – it does not matter how big vendor X is – their stuff can still fail.</p> Networks occasionally have problems.</p> DNS can have issues.</p> Systems get hacked.</p> No matter how many layers of complexity we pile on top of this.</p> In networked computer systems, the simplest path to resiliency is the ability to QUICKLY rebuild systems, whether that is your workstation, laptop, server, or deployment system.</p> 🔧 What will you improve today? 2024-09-20T00:00:00+00:00 Platforms are all about improvement – at the personal, team, and company levels.</p> One approach is to each day write down something you are going to improve and set aside a small block of time daily to work on it. Make this part of your personal process.</p> This does not have to be something big – clean your workspace, create a checklist for something you don’t enjoy doing, automate something, write some documentation to help others on the team, write a test for some troublesome code, refactor something, improve CI/CD, …</p> The internal improvements you do today, while not directly seen by your customers, will help you deliver something better tomorrow.</p> What will you improve in YOUR Platform today? Reply and I’ll compile a list and share it in a future post.</p> What is the role of vision? 2024-09-19T00:00:00+00:00 Yesterday</a>, we discussed the difficulty in predicting the future and the folly of “just-in-case” development.</p> However, we should not discount the need for vision – the sense for where we need to go in the future.</p> Not necessarily specifics, but a gut-feel for where things are heading.</p> Clearly seeing problems that could be solved.</p> To understand the changes that are happening in our respective industries.</p> To realize the potential of new technologies that most don’t understand.</p> This vision does not translate to “just-in-case” feature development, but rather building a platform that prepares you to meet the future.</p> Where you can quickly build the features your customers need as you learn the specifics.</p> Where iterations cost very little.</p> Where new features and technologies can be integrated, validated, and deployed – confidently.</p> 🔮 Preparing for the future 2024-09-18T00:00:00+00:00 Yesterday</a>, we discussed a fundamental characteristic of a platform: it will go places you never imagined.</p> How do we prepare for this?</p> There are two ways:</p> Try to predict where things will go in the future and build specific features into your platform now – just in case …</li> Build the tooling, processes, and workflows so that you can easily and confidently add functionality when it is needed.</li> </ol> It is obvious which approach works better – if we try to predict the future (at least the specifics), we are more often wrong.</p> We can either try to build the future now, or prepare to meet it as it comes.</p> The line between these two is often difficult to discern.</p> ✅ The Platform Test 2024-09-17T00:00:00+00:00 How do you know if you own YOUR Platform?</p> If a customer needs a new hardware interface or connector on a product, can you easily add that?</p> If a security vulnerability is found in a piece of software in your stack, can you fix it?</p> If a customer wants to use a new USB peripheral in a product, can you release a new SW version with that driver included?</p> Can you do all the above quickly, easily, and with confidence?</p> If so, you likely own YOUR Platform.</p> Here’s the thing – embedded systems today are general purpose, updatable, and expandable. That means people will do things with them that you never imagined.</p> A fundamental characteristic of a platform is that it will go places you never imagined.</p> Are you prepared for that?</p> 🤖 What is the role of AI in your platform? 2024-09-16T00:00:00+00:00 We hear much today about how AI is going to do our jobs better than us.</p> AI is powerful and has access to a lot of information and does a lot of things humans can’t do.</p> It is really great at writing shell scripts, boilerplate in programs, figuring out how to use an API, summarizing public information on the Internet, etc.</p> But these tasks are not the differentiator.</p> The real value is going deep.</p> Information found only in books, which AI does not have access to.</p> Experience found only in people and encoded in platforms.</p> AI in a product can be useful for extracting information from YOUR data.</p> General AI tools are platforms in themselves, but probably not YOUR Platform, and probably not the creator of the primary value of YOUR Platform.</p> YOUR Platform is for extracting and leveraging the deep – the value people need.</p> 🔄 What is the first thing you should implement? 2024-09-06T00:00:00+00:00 When building a new system, what is the first thing you implement?</p> There are a lot of approaches – one is to implement a minimal proof of concept.</p> However, I think an easy way to update software in any part of the system (Linux edge devices, microcontrollers, cloud applications) is something that should be done as early as practical.</p> Requiring a technician to go on-site with a computer and specialized software does not count.</p> Neither does a multi-step manual process of manually building something, scp’ing files to a server, manually restarting stuff, etc.</p> Deploying new software to any system should be easy and quick.</p> A single click, or putting a single file on USB disk and rebooting a device, or merging to main and continuous delivery (CD) delivers our new software to the cloud.</p> We may think nice update mechanisms are for our customers in production, and they certainly are.</p> But update tooling also helps us during development.</p> An easy way to update systems encourages us to do it more often, because it is easy.</p> We iterate faster.</p> Fixes and improvements are deployed instantly.</p> We test more systems during development.</p> We can easily adapt as we learn more about the problem we are trying to solve.</p> We can involve more non-technical people in development and testing.</p> What helps us scale in production also helps us scale during development.</p> Instead of doing update last, do it first – as an integral part of YOUR Platform.</p> 🚫 No Golden Machines 2024-09-05T00:00:00+00:00 As humans, we can become attached to “golden” machines.</p> A very expensive bicycle set up just right, a well-tuned tool, a nicely configured workstation, a server that we have set up just right …</p> We like buying expensive things or the iterative process of tweaking things just right.</p> However, this generally does not move YOUR Platform forward.</p> Everything on this earth is ephemeral.</p> Laptops are damaged. Bicycles are stolen. Tools break. Severs crash.</p> So then we become obsessed with protecting our golden machines – locks, excessive security, redundancy, monitoring, etc.</p> The problem with being over-protective is that it is time-consuming and hinders our using something in the first place.</p> We’re afraid to use it in case we might mess it up.</p> We’re afraid to change it because we don’t understand the history of tweaks.</p> What if instead we said: “NO GOLDEN MACHINES”</p> If our laptop gets run over, we can quickly set up a new one.</p> If the server crashes, we can quickly deploy a new one.</p> If we need to use a favorite editor on a different computer, it only takes 2 minutes to set it up.</p> If our bicycle is stolen, we buy another reasonably priced one and have the skills to set it up.</p> YOUR Platform is best built from machines that we can easily use, replicate, and scale, not golden machines that we have to protect.</p> 🙋 How to help yourself? 2024-09-04T00:00:00+00:00 Did you ever have the experience going back into a project you have not touched for 6 months – why did I make this change?</p> Or did you ever struggle to get all the dependencies installed to build a project you wrote?</p> Or did you ever wonder – will this change break something I’m not thinking of?</p> Or did you ever question – how do we actually deploy this now that it is updated?</p> Ironically, when you try really hard to help others use your work – documentation, tests, CI/CD – you are mostly helping your future self.</p> 🔎 Seeing patterns 2024-09-03T00:00:00+00:00 Patterns are important.</p> Seeing patterns allows us to simplify things by using common data structures, re-using code/design, finding solutions to common problems, etc.</p> Being able to see patterns also allows us to troubleshoot effectively.</p> But, to see patterns, we need to have history and data in a form that is easy to see and process.</p> For troubleshooting, this may be a common place to collect notes – say every time a system fails, we record the incident in a single shared Google Doc.</p> Tagging is also effective – we might tag a dataset for different types of events, and then be able to quickly filter on different combinations of tags.</p> To see patterns in source code, we need to have source code organized such that it is easy to see and navigate through all of it – for most of this, this means a monorepo</a>.</p> Transparency is critical – can we easily see things other people are working on, or is each person or team working in walled-off silos?</p> Patterns are not invented, but rather discovered.</p> Platforms are all about identifying and leveraging patterns.</p> And seeing patterns requires transparency, thoughtful organization of our assets, and rigorous logging of what happens over time.</p> Do these things, and patterns will naturally emerge, for YOUR Platform.</p> 🗝️ What is the best tool for private Git repos? 2024-09-02T00:00:00+00:00 Several days ago, I made a case</a> for using Github to engage users around your OSS project.</p> But is this the hammer you should use to drive every nail?</p> For private projects, I think Gitea is actually better in most respects.</p> It is very fast, clean, and most of the basic functions work as good or better than Github.</p> And you can host all but the largest repos on a $5/mo Linode with unlimited users.</p> Groups and permissions are very flexible.</p> Branch protection works.</p> In all, it is very good!</p> I wrote an Ansible role</a> that works very well for updating your Gitea instance. I’ve updated through ~70 Gitea versions with almost no problems.</p> Gitea may be a better option for YOUR internal Platform – more details to follow …</p> 🐶 Lessons from a 1-year old dog 2024-08-30T00:00:00+00:00 As I was going through my morning routine with our 1.3 year old dog, Reese, who is very energetic, it occurred to me how well short training sessions every</em> morning are working.</p> A 12hr session with her once a month will do virtually nothing.</p> Improvement is not really what my dog wants to be doing – she would rather chase squirrels, dig holes, jump on people – do something heroic.</p> Likewise, most of us don’t really want to clean our office, write a CI script, refactor some messy code – we want to design things, code a new feature, make a sale, build 1000 widgets, etc.</p> Improvement is hard and sometimes painful.</p> And the pain is proportional to the size of the improvement dose we are facing.</p> If we can break it down into small enough chunks, it is manageable, and actually enjoyable.</p> Platform improvements are rarely moments of deep inspiration, but rather just buckling down and doing what you know should be done.</p> Discipline.</p> And this sometimes works best if you schedule a time block every day – 25 minutes, start the timer, go …</p> You only have 25m, get going, now!</p> And after our morning walk, Reese now walks toward where her leash is hanging – she is actually looking forward to these short, regular sessions.</p> Improvement is best continuous.</p> </p> 🔍 Solving problems or symptoms? 2024-08-29T00:00:00+00:00 The following quotes from the book “The One-Straw Revolution” caught my eye:</p> The more elaborate the countermeasures, the more complicated the problems become. … When a decision is made to cope with the symptoms of a problem, it is generally assumed that the corrective measures will solve the problem itself. They seldom do. Engineers cannot seem to get this through their heads. These countermeasures are all based on too narrow a definition of what is wrong.</em></p> </blockquote> Although Masanobu Fukuoka’s book is about agricultural systems, the concepts apply to systems in general.</p> With security, do we pile on additional layers of checking, detection, etc., or do we use a simpler and more secure technology to start with?</p> For reliability, do we implement elaborate distributed/redundant systems that check each other, or improve our testing of a single system so that it rarely fails?</p> When deployment mistakes are made, do we add more bureaucracy and red tape, or do we improve the process/tooling such that it is difficult to make the mistake in the first place?</p> When something is not working, do we add on layers of complexity to fix it, or try to simplify and identify the root cause?</p> One way to improve this is ask “why?” five times.</p> 🐙 Github and why does it matter? 2024-08-28T00:00:00+00:00 For the last</a> two days</a>, we have been discussing aspects of several high-quality Yocto BSPs.</p> Another characteristic of these three BSPs is that they are all hosted on Github.</p> Yes, there are many other platforms you can host Git repositories on.</p> You can easily do your own Git hosting (I personally use Gitea for private Git repos).</p> But, for better or worse, if you want to engage with a world-wide community around a software project, Github is the easiest, lowest friction place to do that.</p> Most developers have a Github account, and this means they can easily create issue tickets, submit pull requests etc.</p> Some projects, like the Linux kernel, are so popular and well-established it does not really matter how they interact with the community. They can be rude, use older methods of handling changes like patches on a mail-list, etc. and it does not matter. It might even be argued that these “barriers to entry” are helpful in weeding out the noise in a very large project. Perhaps this is true.</p> But for most of us, we are not at that scale. We don’t have that luxury. Even most downstream Linux trees are hosted on Github these days.</p> If users find it difficult to engage and interact with us, they won’t bother, especially the younger generation of developers.</p> Most developers are (rightly) focused on their projects – their work, not yours. Thus, it is an act of generosity if they take the time to interact with or contribute to a supplier’s project. They are doing it because they want to – not because they have to.</p> And if users can’t easily interact with a supplier, does that supplier really understand their concerns?</p> Github has set a new standard for transparency, tooling, and interaction. It is the lowest friction platform for social coding. And social matters today.</p> This all has implications for how users/customers interact with YOUR Platform (if public) and how you select technology for YOUR Platform.</p> 🔁 Tracking upstream and why does it matter? 2024-08-27T00:00:00+00:00 Yesterday, we discussed three Yocto BSP layers that are exceptional:</p> meta-tegra</a> (69 contributors)</li> meta-freescale</a> (174 contributors)</li> meta-raspberrypi</a> (157 contributors)</li> </ul> The above three layers make an effort to keep up with upstream developments. This may mean regularly merging upstream, doing a build, fixing issues – keeping up.</p> As a result, they are always ready for the next release of the Linux kernel, Yocto, whatever.</p> A little bit of continuous effort is much easier than a monumental effort every four years.</p> Why does this matter for small teams/companies?</p> There are many reasons, but when working with complex open-source software, we’ll eventually need support/help.</p> And this help typically comes from the community around open-source projects.</p> And the community is focused on the current development, not a 4-year LTS release.</p> Additionally, the latest releases are where security problems get fixed, features implemented, and value added.</p> YOUR Platform benefits most from being where the value is being created.</p> 🔧 What makes a good Yocto BSP? 2024-08-26T00:00:00+00:00 As we evaluate technology to use in our platforms, Yocto Embedded Linux BSPs often come into the mix.</p> Recently we talked</a> with Matt Madison, who maintains the meta-tegra</a> BSP layer, which provides Yocto support for Nvidia’s embedded processors, which are increasingly becoming popular in edge AI applications.</p> We are using meta-tegra in several projects, and it has been a good experience.</p> What came out of our discussion is that user/community involvement</strong> is what makes a good BSP.</p> meta-freescale</a> and meta-raspberrypi</a> are other examples of community-oriented BSP layers that are very high quality.</p> Chip makers have different priorities and concerns than users. And the only way to understand your users is to get them involved. And the best way to get them involved is to work on an OSS project together.</p> We’ve run the experiment in the Yocto BSP space for 14 years, and the results are in.</p> When evaluating a complex technology like processors that rely on open-source, if a supplier does include their users in the development process, there is a good chance they don’t really understand your concerns.</p> ⏰ The present or the future? 2024-08-23T00:00:00+00:00 Planning has its place – we need a vision for where we are going.</p> But YOUR Platform is best focused on the present, not the future.</p> Because the future of a complex system is pretty hard to predict.</p> When we focus on improving our current efforts with refactoring, testing, automation, documentation, CI/CD, etc. in the simplest way possible …</p> We pave the way to the future, by reducing friction in the present.</p> 👥 Platform != People 2024-08-22T00:00:00+00:00 The right people in your organization is absolutely essential – no question about that.</p> And dealing with “people issues” is a critical and essential skill.</p> But, if your organization is dependent on a star individual and you are in big trouble when he/she is gone, then you don’t have a platform.</p> We need great people, but if they can’t work in the context of YOUR Platform, then you will have trouble scaling and hit a major speed bump when they leave.</p> People come and go.</p> YOUR Platform is what gives your business the consistency and resiliency to keep going – smoothly.</p> ✅ What is the simplest form of automation? 2024-08-21T00:00:00+00:00 The simplest form of automation is a checklist or playbook.</p> If there is something you need to do more than a couple times that involves multiple steps, a checklist is a great place to start.</p> Then we no longer have to dread doing a multi-step task because it is easy – all we need to do is go through the documented steps.</p> Checklists are surprisingly effective at reducing friction and mistakes. Many of us do this personally for things like travelling as it is easy to forget something if we don’t.</p> And the beautiful thing about checklists is they can easily be turned into more advanced forms of automation (scripts, continuous delivery, etc) because you have already thought through and debugged the process, which is often the hard part and where we get stuck.</p> Checklists can turn into scripts, which can turn into continuous delivery.</p> Part of YOUR Platform.</p> 📈 How does your product value increase after the initial sale? 2024-08-20T00:00:00+00:00 How does your product increase in value after the sale?</p> Software/Firmware/Cloud updates.</p> Software is soft for a reason – it is meant to be changed, improved.</p> And sometimes we install “apps” to add new functionality.</p> And with this improvement comes increased value.</p> It is kind of like your phone – with each new OS update, things generally get better.</p> Each app you install (potentially) provides some utility.</p> Why, because modern phones are platforms.</p> Is your product a platform, or is it at least built and deployed using a platform?</p> Does its value increase over time?</p> At industrial product scale, you don’t necessarily need to support user installed apps, but you need to at least be able to add functionality, fix things, and deploy these updates – efficiently.</p> How? With YOUR Platform.</p> ⚠️ Hack-it, ship-it, forget-it ... 2024-08-19T00:00:00+00:00 Hack-it, ship-it, forget-it … is a race to the bottom where cost is the only thing that matters.</p> The initial delivery is what your customer gets – forever.</p> These types of products only depreciate.</p> And cannot be maintained.</p> Alternatively, the value of your product can increase</em> over time.</p> Where you fix problems quickly.</p> And add features in a timely fashion as needs/markets change.</p> Where you overcome the laws of chaos and improve instead of decay.</p> Because you own YOUR Platform.</p> Which business do you want to be in?</p> 🏆 What is the #1 purpose of YOUR Platform? 2024-08-16T00:00:00+00:00 The purpose of YOUR Platform is to reduce friction in delivering value to your customers.</p> We are in business to deliver value to our customers.</p> If we can do this better, we can deliver more value.</p> There are lots of ways we can do this.</p> Automation.</p> Better tools.</p> Additional features.</p> Quality/stability improvements.</p> Release more often.</p> Respond more quickly.</p> 🌱 Culture 2024-08-15T00:00:00+00:00 What type of culture do platforms thrive in?</p> Where ideas are valued and evaluated on merit and truth.</p> Where initiative is rewarded.</p> Where mistakes result in process improvements, not scoldings.</p> Where openness and transparency are the standard.</p> Where anyone can improve anything in the system – there is no protecting turf.</p> Where OSS tools are used when possible.</p> Where things are continuously improved.</p> Platform cultures are organic.</p> We can learn a lot from Open Source projects here …</p> 🏗️ Three levels of YOUR Platform 2024-08-14T00:00:00+00:00 We can look at platforms at three levels.</p> Your personal platform is the know-how that you can personally reuse from task-to-task, project-to-project, and job-to-job. But, it is not real useful outside yourself.</p> A product platform may be what a team uses to efficiently build, maintain, and produce variants of a product.</p> A company platform is when you are able to reuse know-how, technology, tools, and processes on one product and leverage that on the next product, and when an improvement on one product improves another.</p> All three are valuable, but the higher up the stack you can go, the higher the value.</p> 👣 What are the first steps? 2024-08-13T00:00:00+00:00 What are the first steps to building YOUR Platform?</p> Ironically, the first step is not some grandiose new Platform vision, but rather to set your house in order.</p> Clean up what you already have.</p> Fix legacy code (code without tests).</p> Set up CI/CD.</p> Implement end-to-end testing so that you can deploy with confidence.</p> Write documentation.</p> This paves the highway for efficiently moving forward, instead of slogging through the mud.</p> 🔄 What is YOUR Platform? (v2) 2024-08-12T00:00:00+00:00 Had a nice visit with Rod</a> and he asked me to describe what a Platform is in 20 words or less, so here we go:</p> YOUR Platform is the collection of technology, tools, and processes that enable</em></strong> you to differentiate and consistently deliver your products and services.</em></strong></p> sorry Rod, that is 22 words :-)</p> We have discussed</a> this before, but I want to revisit this question occasionally and see if the definition improves over time.</p> What got added this time is “differentiate” and “consistently”.</p> Thoughts?</p> ❓ But what if ... 2024-08-08T00:00:00+00:00 In the last post</a>, we discussed the fallacy of using complex technology when the scale does not justify it.</p> But, what if we get a lot more customers, or sell a lot more systems? Should we not plan for the future now? Use technologies now that will work at large scale? We don’t want to paint ourselves into a corner such that we end up with a system that won’t meet our needs in the future.</p> Certainly, we should keep the future in mind, but the most important way to plan for the future is to streamline your processes and development workflow now so that you can continually move toward the future as it comes.</p> NOT try to implement the future now. That is costly, and you’ll never get it right that way.</p> Some decisions are important now and some are not.</p> You don’t need to use Kubernetes now – you can add that in later. But it is important to use some type of automated configuration and deployment – perhaps a simpler solution like a simple Github Action, or Ansible.</p> The same with micro-services. It is easy to split things apart later as it makes sense. But it is difficult to deal with a mess that is prematurely split up.</p> You can plan by using a decoupled architecture, data structures that are flexible, and modern programming languages that are efficient.</p> If you have an efficient workflow where it is easy to make and deploy changes, the correct architecture will naturally emerge. You will continually simplify things as you see common patterns.</p> And simple patterns are what scale to any problem.</p> 🔍 You are not Google 2024-08-07T00:00:00+00:00 A blog article published 2017 titled You Are Not Google</a> is a good read. This article makes the case that most companies are not operating at the size of Google, Amazon, and LinkedIn. Thus, most of us are solving different problems than the big 5 tech companies.</p> YOUR Platform needs to fit your problems at your scale.</p> If one server will work, then you are saving yourself untold pain and suffering by not pre-maturely building a distributed system.</p> If one monolothic application can run your product, then you are saving a lot of hassle not implementing microservices up front.</p> Keep it simple – you can move to distributed micro-services later – AFTER you succeed at a smaller scale.</p> And if you succeed at small scale, then there will be plenty of resources to adjust your architecture later.</p> The quicker you can get to a MVP, the better, and simple gets you there quicker.</p> But what if ??? … stay tuned :-)</p> 💰 Owning YOUR platform is not more expensive 2024-08-06T00:00:00+00:00 Owning your platform sounds expensive.</p> “We can’t afford ops people to manage servers.”</p> “Security is scary.”</p> “We don’t want to manage a database.”</p> However, at the scale of most industrial products, none of these things are very hard.</p> A couple servers can easily handle all your traffic.</p> A backup and restore plan can handle failures in most cases. Redundancy and high-availability are overkill.</p> Cloud services require management too – often you end of managing the cost instead of the fundamental issues listed above. Six of one, half-dozen of the other – take your pick. And then you still have to deal with problems – often due to the distributed nature of cloud services where the network is the weak link.</p> See DHH’s posts on “leaving the cloud”</a> for an example. They are saving millions owning and managing their own servers and it is not any more work.</p> You can hitch up to one of the major cloud vendors and go all in on their services – that is what everyone else is doing.</p> Or you can do something different. Take the technology that is being developed for the cloud, use the cloud where it make sense, and then optimize for YOUR Platform.</p> ⚖️ False vs Real Platform Development 2024-08-05T00:00:00+00:00 A common objection to investing in a platform is that it is “too expensive.”</p> “We want to get it right, lock it down, and then crank out our product and make some $$$.”</p> Many managers and owners are wary of platform development because they have experienced false</em> platform development – endless tinkering, the costly pursuit of perfection, etc.</p> However, real platform development is focused on shipping as soon as possible and then improving.</p> Real platform development is realizing that complex, connected systems are dynamic, living things that need maintained. They do not operate in isolation.</p> Real platform developers have the humility to realize they can’t see and understand the big picture up front until they build something and get feedback.</p> The key to controlling costs is not locking things down such they cannot change, but rather the opposite – improving your process so that improvements are efficient to develop and and painless to test and deploy.</p> 🎆 Do something different 2024-08-02T00:00:00+00:00 If you only build on other people’s platforms, how are you different?</p> And if you are not different, what value are you providing?</p> We are not talking different for the sake of being different – outlandish, rebel, etc.</p> We are talking being different in the sense of offering something better – something that no one else is doing.</p> Something you can do pretty well.</p> Something that helps others.</p> And to do this, you need to own YOUR Platform, otherwise, why would anyone come to you?</p> 🐜 The smaller you are, the more import YOUR Platform is 2024-08-01T00:00:00+00:00 This may seem like a contadiction – how can small companies afford to own and control their Platform? Should not this all be outsourced? “Platforms is what large companies do.”</p> Actually, the reverse is true. Netflix can afford to build on AWS and be locked into the AWS platform because they have enough volume that they have clout and the ability to negotiate. Amazon will make sure Netflix gets what they need.</p> Small companies cannot get this type of support.</p> Does this mean “small company” should never use AWS? No, certainly not – AWS is a good option for many things.</p> But it does mean that we only use stuff in AWS that we can migrate elsewhere if needed.</p> Use managed Postgres instead of some proprietary AWS.db.</p> Use NATS running on EC2 or managed at Synadia instead of a proprietary event system.</p> Owning YOUR Platform does not mean you build all of it, but rather you have some level of control over it. You have options.</p> And the smaller you are, the more important this is.</p> 😨 Control is a scary thing 2024-07-31T00:00:00+00:00 Control is a scary thing – because then we are responsible.</p> And if we are responsible, then we can’t blame someone else.</p> But if we are not responsible for anything, then what are we doing?</p> How are we adding value?</p> Why would anyone want what we are doing?</p> 🎛️ What is control? 2024-07-30T00:00:00+00:00 What is control in the context of Your Platform?</p> Control is simply the ability to respond to opportunities and problems.</p> In modern systems we are very dependent on others as there is no way we can do everything ourselves.</p> But we can also strategically retain enough control so that we can adjust and adapt as conditions require.</p> We have more options today than ever before.</p> We may think putting all of our eggs in the “big company” basket is the path to safety.</p> But if we open our eyes, we soon realize this does not always work.</p> “big company” may discontinue a service we depend on.</p> “big company” may have an outage.</p> “big company” may have security problems.</p> There is nothing wrong with using “big company” as long as we have the option to move when problems or opportunities require it.</p> Lock-in is a good short-term strategy, but is rarely beneficial long-term.</p> 🎯 Why do we need some level of control? 2024-07-29T00:00:00+00:00 In today’s connected systems, some level of control of your Platform is needed.</p> Security problems needed fixed.</p> Problems needed debugged.</p> Problematic subsystems need swapped out.</p> Suppliers go out of business.</p> Parts go obsolete.</p> New customers require new features.</p> We may want to leverage new technologies.</p> New Open Source projects become available.</p> We obviously can’t control every detail, but we at least need enough control to adapt and respond to change or the unexpected.</p> Control is required because we can’t predict everything up front.</p> With complex systems, this is more true than ever.</p> 🏁 The long game 2024-07-26T00:00:00+00:00 Platforms are long games.</p> If you are building a one-off consumer product that will not be maintained, improved, or ever change, then you probably don’t need a platform.</p> However, if you are building an long-lived industrial product that will be maintained, or a series of related products, then it gets very expensive to start over every time.</p> To effectively maintain multiple unrelated products, or a system that that is difficult to update, improve, test, and deploy is a recipe for cripling technical debt.</p> The expectation today is things will improve over time, because software can be updated. It is not really optional any more.</p> And with the complexity of systems today, the odds you’ll get perfect on the first release are near zero.</p> Build for the long game.</p> ♾️ It's about not starting over every time 2024-07-25T00:00:00+00:00 Platforms are about reuse.</p> Not starting over on every new product.</p> Leveraging what you have done before is one way to scale your efforts.</p> Reuse takes practice and experience to get it right.</p> Effective platforms are an art.</p> 🌐 Complex Systems, Simple Platforms 2024-07-24T00:00:00+00:00 The word “simple” in the context of modern systems may seem like an oxymoron.</p> We can’t deny that modern systems are complex – they need to be to do the things they do.</p> The role of a Platform is to reign in this complexity and create a way to effectively manage and interact with the system.</p> Platforms narrow our focus to the problem we are solving.</p> Platforms decouple us from being too entangled or dependent on any one technology or system.</p> Platforms are the abstraction on top of complex systems that make them workable.</p> 🔹 Simplicity 2024-07-22T00:00:00+00:00 What is the role of simplicity in your platform?</p> Simplicity makes it easier for people to use your platform.</p> Simplicity helps ensure maintainability.</p> Simplicity moves us toward reliable and secure platforms.</p> Simple is not easy, actually it’s much</strong> harder than complex. But platforms are a long game.</p> 🏢 What is your Platform? 2024-07-18T00:00:00+00:00 What is a Platform?</p> There are many ways to look at this. Some might think of AWS, GCP, iOS, Android, etc.</p> These are certainly platforms, but they are not your</em> platform.</p> Your platform is the collection of technology, tools, and processes you use to do what you do.</p> This may be building products, delivering services, developing people, etc.</p> ✨ The Beauty of Constraints 2024-07-17T00:00:00+00:00 Constraints are what guide us to good solutions.</p> Software that is simple enough that average developers can actually understand it.</p> A user interface that is intuitive.</p> A system we can deploy without a dedicated SRE team.</p> Hardware created with open tools that anyone in our organization can maintain.</p> An architecture that can scale in unforeseen directions.</p> Flexible core data structures that rarely change.</p> Platforms are all about constraints.</strong></p> 😱 The tragedy of no constraints 2024-07-16T00:00:00+00:00 We never have enough time, resources, funding, CPU, storage, etc.</p> However, think what would happen if we had infinite resources.</p> We would never get started because we have forever to do it.</p> With unlimitted resources, we tend to make a mess and get sloppy.</p> Getting everything we want leads to bloat and lethargy.</p> Platforms are a long game, and but no contraints encourages short-term thinking.</p> 👋 Introduction 2024-07-15T00:00:00+00:00 Product development is hard.</p> Technical debt tends to accrue at a frightening pace.</p> A “Platform” approach helps you reign in the chaos.</p> And provides a framework to reuse, scale, and leverage your efforts.</p> This newsletter provides tips to help you in the journey to “Own your Platform.”</a></p> The Ability to Improve 2024-03-05T00:00:00+00:00 </figure> (while this article contains many general ideas, it is written from the perspective of product development.)</em></p> If you don’t continually improve, you soon lose the ability to do so …</strong></p> This thought came after an associate described a company that is struggling to manufacture one of their products:</p> They have been making a product since way back in the last century. EOL (end-of-life) is catching up with them. The circuit is really hairy analog stuff with ridiculously high precision. Pots everywhere to get it dialed in. But their voltage regulator is drifting in value significantly, whereas the old parts are rock steady.</p> </blockquote> How do we get into situations like this, and how can they be avoided?</p> A few more examples</h2> Below are a few more examples of situations that are very difficult to deal with in the lifecycle of a product:</p> A part becomes obsolete in a printed circuit design, and no one knows how to use the CAD system it was designed in. Rather than updating to an open/modern CAD system</a> years ago at a relaxed pace, exorbitant efforts are now spent trying to track down a few more parts or find a replacement. When this fails, it is now a frantic crash-and-burn project to revise the design with no time for testing and verification.</li> A serious security problem is found in a product, but the software build system has fallen into disrepair and no one knows how to set it up, and it will not run on current operating systems.</li> A new Linux kernel driver is needed to support a new USB peripheral, but the kernel we are shipping is 7 years old, and the SOC/SOM vendor has not provided any updates. Updating to the latest kernel would take months. Backporting the new driver is nearly impossible due to the many changes in the kernel and its subsystems.</li> Manufacturing was outsourced to another country years ago. Then COVID happened and the manufacturer shut down for an extended time. No one knows where the design files are or how to build the product to move it to another manufacturer.</li> The manufacturing process for a complex product has been the same for years. No one knows why things are done the way they are, but are afraid to change anything for fear that problems may crop up.</li> A code base has not been refactored as it grew and no tests were written. Adding new functionality is very difficult and usually causes bugs in unrelated areas, requiring an exorbitant amount of manual testing with each release.</li> An application does not have a cohesive architecture. Global variables and unrestrained function calls between various subsystems are common. As time goes on, the application becomes less and less stable. Fixing one bug often causes another. All developer resources are consumed in a circle of support, continuous bug fixing, and damage control.</li> A new customer is interested in purchasing a product, but it is missing a critical feature. The original development was contracted to an outside development firm, and there is no knowledge in-house on how to modify the software. The managers at the company are scrambling to find someone who can figure out how to make the changes needed.</li> </ol> Technical debt</em> is another term to describe systems that are not adequately maintained – eventually, it takes so much effort just to keep things going that no time is left for improvements.</p> Maintaining Systems</h2> How do you maintain the ability to react to changes, security problems, part shortages, and feature requests? How can you add value to your products over time? Maintaining systems (both hardware and software) is what keeps them malleable and flexible. If you don’t maintain stuff, then it soon becomes brittle and harder and harder to change. Modern systems are not ship-it-and-you-’re-done, but rather more similar to a living organism. If it is not maintained, it is dying.</p> The only way to make sure you can</em> maintain something is to maintain it.</strong> Make it better. Replace a part with a cheaper/better one. Improve something. Add a feature. Make it faster. Streamline and simplify manufacturing. If we don’t continually improve, then it becomes difficult to do anything without starting over. It is much easier to absorb a series of small costs over time than a large expense all at once. Small changes over time allow us to improve our processes. If we are not under pressure and something goes wrong, then we simply revert to the previous version or write tests to keep it from happening again.</p> This does not have to be hard, but it does require an investment in methodology and workflow. It requires a commitment to open tools, automation, and testing. It requires process – not the type that results in stifling bureaucracy but rather improves efficiency through automation and transparency.</p> There are several concerns related to change including return on investment (ROI), quality, and efficiency.</p> The ROI Objection</h2> There are two financial mindsets:</p> View R&D as a one-time capital expense where you design something, lock it down, and then try to extract as much value as possible – the “minimizing costs” mindset.</li> View R&D as an ongoing operating expense, improve things over time, and continually increase value – the “maximizing value” mindset.</li> </ol> The first is easier to measure and forecast. Financial people like and understand this model. The second requires intuition and vision. It involves some risk. As your processes become more efficient, the value you get from an improvement begins to outweigh the costs of making the change. Over time, these improvements have a compounding effect.</p> </figure> It depends if you are playing the short or long-term game …</p> The Quality Objection</h2> Stability and Quality is always a concern with any change in a system. With changes, mistakes will be made – especially early on. When this happens, there is an opportunity to learn and proactively improve the testing and verification process for next time, or reactively place blame, lock things down even harder, and remove responsibility. When someone introduces a bug, the question should be: “Did you create a test so it does not happen again?” If no mistakes are being made, not much is likely being done. The upside of growth and improvement should far exceed the costs associated with the occasional mistakes made in the course of changes. Done right, quality should follow the same value curve as the value shown in the graph above. The key is to not get discouraged by an initial drop in quality and focus on improving the processes, not minimizing or restricting change and responsibility.</p> The Efficiency Objection</h2> Efficiency is always a concern, and again there are two schools of thought:</p> It is more efficient if we just do things the way we’ve always done them, not risk change, and get what we can out of our investment. Change brings risk.</li> Efficiency can always improve, bringing lower costs and more value.</li> </ol> Again, the key is to be patient with the initial declines and persevere until you get to the point where efficiency is increasing and the cost and value lines cross.</p> Case Studies</h2> Tesla is an interesting case study on continuous improvement</a>. From a design and manufacturing perspective, their accomplishments are notable. There is always a better way to design or manufacture something.</p> Google’s blameless postmortem</a> practice enables proactive improvements for the future instead of reactively placing blame for the past.</p> Amazon’s “You build it, you run it”</a></em> philosophy allows people who know the most about the system to improve how it is run instead of going through endless rounds with gatekeepers who don’t understand the system.</p> Toyota’s implementation of “kaizen</a>” has produced excellent results</a>. Kaizen is a concept referring to business activities that continuously improve all functions and involve all employees from the CEO to the assembly line workers.1</a></sup></p> These practices are used at large successful companies, but in concept, they are simple and can be applied at any scale.</p> The Irony of Kaizen</h2> We can develop products with the thought – we are going to get this perfect and then go into production. Logically it would seem that if we expect to make improvements after it is in production, then we must be starting with an inferior product with flaws. However, Toyota has proved the opposite. Even when a model is first introduced, their quality is likely better than the average in the automobile industry. Kaizen does not mean we slop something together of inferior quality, but rather we do the best we can but have the humility to realize it can always be better. Kaizen is not limited to the assembly line, but all aspects including design, prototyping, quality control, testing, etc. If all of these processes are continually getting better, then the first item produced will also be better.</p> Requirements for Improvement</h2> Every person and every organization is different, so there is no one-size-fits-all recipe that will always work. But there are several ingredients that are generally beneficial including:</p> Metrics</li> Testing</li> Openness</li> Simplicity</li> Leveraging outside improvements</li> Iterate faster</li> Owning your platform</li> The right people</li> </ol> Metrics</h2> Not all change is an improvement – change for the sake of change is just churn and generally negative. You have to be able to measure something to know if things are improving or not. Improvement is very difficult without some metrics, because we are then just guessing and going off our intuition, and human intuition is often distorted by emotion, social pressures, and a host of other distractions.</p> Testing</h2> Testing is trendy today and many talk about it, but fewer individuals and teams have the discipline to implement testing consistently. If you are not confident enough to deploy new software to production after your tests pass</a> without doing some manual checking, then you possibly need better tests. Automated end-to-end and performance testing is critical. Consider the following statement about SQLite:</p> The reliability and robustness of SQLite is achieved in part by thorough and careful testing.</p> As of version 3.42.0 (2023-05-16), the SQLite library consists of approximately 155.8 KSLOC of C code. (KSLOC means thousands of “Source Lines Of Code” or, in other words, lines of code excluding blank lines and comments.) By comparison, the project has 590 times as much test code and test scripts - 92053.1 KSLOC.</p> https://www.sqlite.org/testing.html</a></p> </blockquote> Most of SQLite’s tests are not open-source. This is the real IP of SQLite.</p> Openness</h2> Openness is another necessary ingredient for improvement. This takes many forms:</p> Open access to tools, design files, and source code. Open-source software and tools are preferred when available.</li> Good documentation is available to all involved.</li> Work being done is in the open and can be reviewed by anyone at any time. Pull requests are created when work starts to provide visibility into work being done rather than when the work is finished. Transparency is a powerful ingredient for quality.</li> Metrics, including financial information</a> and customer feedback, are accessible to the team.</li> Anyone on the team to allowed and encouraged to fix or improve anything. There are no “turf battles” where only certain people are allowed to do certain things. Within practical limits, anyone can do any work, but the process (transparency and testing) ensures that substandard work will not be automatically used.</li> </ul> The above does not mean everything is open-source to the world, but rather those involved with the product – a team, organization, company, customers, partners, etc. We can learn a lot from the success of open-source projects and their associated culture and methodologies.</p> Simplicity</h2> As the complexity of systems continues to increase, we need to also be aware that humans interacting with these systems are finite – we can only handle so much complexity. Thus, efforts need to be made to simplify where possible. We often hear the phrase: “keep it simple!” But, from whose perspective? Design, Manufacturing, Users, Service, Ops, Support, Sales, Financial, CEO, etc. If we look at how much time these various groups interact with the product, we can arrange them in an inverted pyramid:</p> </figure> The CEO</em> is only one person and hopefully, there are many Users</em>, so if we can simplify things for the Users</em>, that will probably provide more benefits to the product than simplifying things for the CEO</em>. Likewise, if Design</em> can spend a little more time making Manufacturing</em> and Support</em> simpler, that is probably a good investment.</p> Simplifying a system’s architecture often provides the biggest wins. This can take the form of common data structures, how data is structured, how data is communicated and synchronized, lifecycle management, etc. A simplification at this level has the potential to improve the entire system.</p> Faster Iterations</h2> The most efficient way for humans to design complex systems is through iteration</a>. Some call this failing fast</a>. The faster you can design/develop, test, get feedback, and repeat, the more likely you will be successful.</p> </figure> There is certainly a place for some up-front planning, but you don’t want to get stuck there</a>. Build a minimal slice of your product</a> as soon as possible, even if it only has a few features. This can even be done with hardware – put down only the bare minimum parts on the PCB and build something. The rest can be added later. If it takes too long to build a PCB prototype, find ways to speed it up, or switch to another vendor. Automation and process is very important for fast iterations. CI (continuous integration) tells you quickly if your code fails tests. CD (continuous delivery) continually builds and deploys your work so that it can be tested in the complete system and evaluated by others. CAD and PLM automation</a> allows you to quickly package, track, and build new versions of PCBs.</p> A focus on iteration compensates for our many human weaknesses. It reveals our blind spots. It improves teamwork and project transparency. It helps avoid the scenario where the lone genius works for three months behind closed doors while the rest of the team is largely stalled. It ensures we are building what the customer actually needs. It encourages automation to avoid the tedium of manual testing and releases. It accelerates learning. Above all, it improves the efficiency of the entire process. It ensures we can quickly make changes and release them when needed, with confidence.</p> Owning Your Platform</h2> Owning your platform means you “own” and take responsibility for key parts of whatever it takes to build and maintain your products. It might be a printed circuit design, a build system, an application architecture, automation, infrastructure, process, etc. Owning your platform likewise is an investment for the future. It gives you more control and the ability to improve things. You are not locked into what another company provides at one snapshot in time. Certainly, there are limits to this and we cannot create an entire system from scratch so we all must use software and components that other companies create. However, we can build a system that has flexibility so that we can easily swap out one component for another. We can understand it well enough to test it. We can manage the build systems so that if something needs to be fixed or updated, we can do that. We build our device software when possible instead of relying on binary images that we can’t control. We are not afraid of developing our own automation and process software instead of relying entirely on outside vendors. Perhaps the most important thing is to own the integration points. This might include:</p> Top-level PCB (printed circuit board) designs (we can change electronic components and submodules as needed)</li> Software build systems (we can add, remove, and update software components)</li> PLM (project lifecycle management) (we write our own or use a system that can be easily extended through programming to meet our needs)</li> Core application architecture (can be adapted to specifically meet our needs)</li> Hosting (we can run our own servers or move applications between hosting providers as needs change)</li> CI/CD (can be adapted to any task/technology)</li> </ul> Integration is the most difficult (but also the most valuable) part of product development, so it is tempting to outsource integration. But this often locks you into one vendor and does not give you the flexibility to scale to higher volumes or additional products, as this running commentary</a> illustrates.</p> Ideally, you can reuse parts of your platform across many products, thus compounding your efforts over time. Even if your design files and source code are open-source, owning your platform (which includes so much more than design files) is what gives you the ability to efficiently design, test, maintain, and produce something. This is the hard part. If you are moving fast, a static snapshot of your product design at any point in time is of marginal value to competitors.</p> Leveraging Outside Improvements</h2> Leveraging outside improvements is another important activity. This can include upgrading to a new open-source software component, designing in a new integrated circuit, etc. It is easy to get stuck in the mindset that the “old” was “better.” In some aspects of life, this may be true, but with technology, things are generally getting better. Open-source software is generally more stable as time goes on due to better testing. Newer integrated circuits are more capable and lower cost. Many products today are largely an integration of many outside technologies, many of which are now open-source</a>. To improve, we must continually track and integrate these improvements. Developing using the mainline</a> (upstream) branch of open-source projects is a good strategy that companies are increasingly adopting</a>. Knowing when to adopt</a> new technology is a key skill.</p> If you use an outside development firm to develop parts of your product, it is critical that you receive a deliverable that is maintainable. Several suggestions for ensuring this include:</p> run the project like an open-source project. All activity is transparent and is committed to Git often.</li> own the integration points and testing.</li> use small iterations when possible. This ensures you are engaged and what is being built is what you need.</li> use open tools</a>/technology that everyone has easy access to where possible. Ensure that you can build or generate usable outputs from the design/source files. Even if you don’t have the expertise to use the tools, there is a better chance you can find someone else who can.</li> </ol> The Right People</h2> There is a saying often used by hikers: “Leave it better than you found it.” If we are working on a section of code that does not have tests, do we write them? If a manufacturing problem is found, do we improve the process to try to prevent the problem in the future, or just write it off? Do we give more responsibility and opportunity to everyone involved to improve things, or do we let a few mistakes restrict responsibility to only a few people?</p> For this to all work, you have to have people with integrity – who are genuinely interested in improvement and not just gaming the metrics. You have to have people with the discipline to write tests, which goes above and beyond just getting the job done and the ticket closed.</p> This all applies personally too … do you want to improve how you use your time? Then measure how you spend your time and share these measurements with someone. Use one of the many tools available to help track your time. Create your own tooling and process, and then improve it. Look for better ways to do all this.</p> Fundamentally, people don’t usually change much, so if someone is not interested in improving themselves, they likely will not be interested in improving the organizations they are a part of and be a drag on the process. So perhaps the most fundamental thing to learn in an interview is not only past accomplishments, but also if the candidate still has that burning desire to learn and improve themselves. It is a frustrating situation when a person who is wired for improvement is part of an organization that is not, or vice-versa. Both types of people and organizations exist.</p> Small Changes vs Large Changes</h2> We can make small or large changes. This might be the difference between an improvement vs. a pivot, slow and steady vs drama. The drama captures attention in the news – the new charismatic CEO brought in from the outside to shake things up. You typically don’t see great companies in the news, but you do see them in investment portfolios. In his book “Good to Great,”</a> Jim Collins studied several companies that produced exceptional results and concluded the following:</p> We keep looking for change in the wrong places, asking the wrong questions, and making the wrong assumptions. There’s even a tendency to blame Wall Street for the “instant results” approach to change. But the companies that made the jump from good to great did so using Wall Street’s own tough metric of success: a sustained leap in their stock-market performance. Wall Street turns out to be just another myth—an excuse for not doing what really works. The data doesn’t lie.</p> Now picture a huge, heavy flywheel. It’s a massive, metal disk mounted horizontally on an axle. It’s about 100 feet in diameter, 10 feet thick, and it weighs about 25 tons. That flywheel is your company. Your job is to get that flywheel to move as fast as possible, because momentum—mass times velocity—is what will generate superior economic results over time.</p> Right now, the flywheel is at a standstill. To get it moving, you make a tremendous effort. You push with all your might, and finally you get the flywheel to inch forward. After two or three days of sustained effort, you get the flywheel to complete one entire turn. You keep pushing, and the flywheel begins to move a bit faster. It takes a lot of work, but at last the flywheel makes a second rotation. You keep pushing steadily. It makes three turns, four turns, five, six. With each turn, it moves faster, and then—at some point, you can’’t say exactly when—you break through. The momentum of the heavy wheel kicks in your favor. It spins faster and faster, with its own weight propelling it. You aren’t pushing any harder, but the flywheel is accelerating, its momentum building, its speed increasing.</p> This is the Flywheel Effect</a>. It’s what it feels like when you’re inside a company that makes the transition from good to great. Take Kroger, for example. How do you get a company with more than 50,000 people to embrace a new strategy that will eventually change every aspect of every grocery store? You don’t. At least not with one big change program.</p> Instead, you put your shoulder to the flywheel. That’s what Jim Herring, the leader who initiated the transformation of Kroger, told us. He stayed away from change programs and motivational stunts. He and his team began turning the flywheel gradually, consistently—building tangible evidence that their plans made sense and would deliver results.</p> “We presented what we were doing in such a way that people saw our accomplishments,”Herring says. “We tried to bring our plans to successful conclusions step by step, so that the mass of people would gain confidence from the successes, not just the words.”</p> Think about it for one minute. Why do most overhyped change programs ultimately fail? Because they lack accountability, they fail to achieve credibility, and they have no authenticity. It’s the opposite of the Flywheel Effect</a>; it’s the Doom Loop.</p> Companies that fall into the Doom Loop genuinely want to effect change—but they lack the quiet discipline that produces the Flywheel Effect</a>. Instead, they launch change programs with huge fanfare, hoping to “enlist the troops.” They start down one path, only to change direction. After years of lurching back and forth, these companies discover that they’ve failed to build any sustained momentum. Instead of turning the flywheel, they’ve fallen into a Doom Loop: Disappointing results lead to reaction without understanding, which leads to a new direction—a new leader, a new program—which leads to no momentum, which leads to disappointing results. It’s a steady, downward spiral. Those who have experienced a Doom Loop know how it drains the spirit right out of a company.</p> https://www.jimcollins.com/article_topics/articles/good-to-great.html</a></p> </blockquote> Change is easiest to implement and absorb if it is incremental and not a drastic overhaul. Incremental change allows you to measure, adjust, and backtrack if needed. Incremental improvements can be implemented by anyone without permission and can then be evaluated and improved by others.</p> It is also important to understand the difference between foundational truths/values (things you should not quickly change) and inefficiencies that and ripe for improvement. The foundational principles of your business provide stability and need to be understood and valued.</p> Summary</h2> Do you want to be able to respond quickly to unexpected opportunities or issues that come up? The solution is simple – practice doing so now. If you regularly practice on the little things that by themselves may not really be worth the time to do, several things will happen:</p> you will be ready and practiced to make critical changes that will make a big difference such as fixing a critical security problem, responding to supply chain shortages, or making a change necessary to land a big sale.</li> the compounding effect of small improvements will eventually provide a sizable return in the future.</li> each small improvement is a small push on the flywheel of momentum. This adds positive energy to your organization and this momentum helps carry you through difficult times.</li> </ol> A culture of improvement is a long-term game. There will be some up-front pain as you develop your processes, but this is much easier than the long-term pain of accumulated technical debt which cripples your ability to do anything. You can compound the value from improvements or the cost of technical debt – take your choice.</p> ^1</sup>https://en.wikipedia.org/wiki/Kaizen</a></p> </div> Think Horses, not Zebras (Part 2) 2024-03-01T00:00:00+00:00 There is a popular quote</a> in medical circles:</p> When you hear hoofbeats, think of horses, not zebras. — Dr. Theodore Woodward</p> </blockquote> I recently posted several debugging experiences</a> where it was beneficial to examine simple scenarios before complex ones. Below are a few more …</p> NVMe slot not working</h2> An NVMe slot is fairly simple – it has a clock and one to four PCIe lanes. However, in a recent new hardware design, the slot was not working. We reviewed the kernel boot logs, scoped the clock signals, etc. We did notice the clock amplitude was lower than other designs (see below). The initial suspect was signal integrity as the PCIe data lines run at very high speeds, so we reviewed the routing, board stackup, impedance calculations, etc. Eventually, I compared the PCIe connector pinout to another design and it appeared that the RX and TX data signals were swapped. Sometimes RX vs TX (receive vs transmit) can be a little confusing based on which perspective the signals are referenced to (the host system or the target device). In the case of PCIe, it appears the signals are all named from the perspective of the host. After modifying to swap the RX and TX signals, the NVMe slot worked fine, even though the modification wires drastically violated signal integrity and impedance requirements. PCIe must be a fairly robust transport!</p> </figure> </figure> PCIe clock amplitude</h2> While verifying the hardware of a design, we noticed that the PCIe clock was at a lower amplitude than the specification required:</p> </figure> What we were measuring:</p> </figure> The Vcross was about 170mV, and the spec required it to be between 250 and 500mV. To verify our measurement technique, we measured the PCIe clock on another system and it it looked correct. The default output of the clock chip should have given us the correct amplitude. The clock chip was connected via an I2C bus, so we read back the registers to make sure they were set to the defaults. We then set the amplitude to the maximum the chip would output, and it still was not within range. Finally, we noticed the following in the datasheet:</p> </figure> Sure enough, we had terminations resistors for these signals on the PCB. After the resistors were removed, the clock amplitude was correct.</p> Cellular connection problems</h2> Cellular IoT systems can be tricky to troubleshoot. They are often installed in harsh environments (rain, cold, hot, damp, caustic sewer gasses, etc.). Occasionally a modem goes bad and connections fail. In one recent case, we replaced a modem and it still would not connect. The antenna, cable, and connections were the next suspect. After spending considerable time swapping components and not making any progress, I reviewed the configuration and noticed the APN was not set. After setting the cellular APN, it worked.</p> Zephyr Nanopb build error</h2> We recently updated the version of Zephyr we were using in a project. In the process, a protobuf generation process broke and another developer found a solution – nanopb was now an optional dependency that needed to be activated. However, the build still failed in my workspace. We discussed diffing our build workspace, but in the back of my mind, I was thinking – there has to be a simple reason. While reading the documentation, it occurred to me to check the west</code> version – sure enough, I was running 1.0 and the latest was 1.2. After updating, it worked.</p> What can we learn?</h2> Again, in the above cases, the initial assumptions were more complex than the actual problem. Suggestions from the previous article</a> are relevant here:</p> Double-check connector and component pinouts.</li> Compare to working designs.</li> Re-read the entire datasheet carefully.</li> </ul> Additional suggestions learned from these examples include:</p> Verify the configuration of the system is correct, never assume it is.</li> Make sure you have the correct version of the tools installed (ideally your build system would check this for you).</li> Re-read documentation for tools and software components. This will often spark ideas on where to look for problems.</li> </ul> Why Open Source for Product Development 2023-12-01T00:00:00+00:00 (this topic is also discussed in a podcast episode</a>)</p> One of the paradoxes in product development is Open Source. How can you personally or as a company benefit from participating in Open Source projects? Why should you share your great ideas and code? How can you build a business or a career by giving things away? How can an open-source project be a reliable supplier without contractual guarantees? Most successful technology companies today participate in open-source projects. Why do they do this?</p> Open source is difficult to fully understand and much has been said about it. There are many approaches to licensing and funding. Many question the equity and sustainability. Many get bogged down in ethical, moral, and political arguments. There are messy problems that don’t seem to have good answers. But we need to go deeper and examine the core issues of why companies are turning to open-source.</p> We will consider several product development perspectives:</p> Our motivation is to produce something of value</strong> that society can use and benefit from.</li> Software technology is advancing at a rapid rate of change</strong>.</li> Technology in modern systems is increasingly complex</strong>.</li> How can we maintain some level of control</strong> over the resources we use to ensure they will meet our needs?</li> And how does organizational culture</strong> impact our ability to deal with these issues and build modern systems?</li> </ul> </figure> Value</h2> The value of software is hard to quantify, and breakthroughs are difficult to predict. New software algorithms, methods, languages, and structures are sometimes much more effective than previous ones. It is these non-linear breakthroughs that we must leverage. Software is an unbounded sphere of opportunity, free from the constraints of physical locality, physical devices, duplication costs, and, in many cases, computing resources. Modern computers, even those we hold in our hands, are capable of running more software than we can write in multiple lifetimes and storing a lot of information. Systems that improve human efficiency and capability potentially add value and wealth to society. The best software can do more with simpler expressions and less effort. The best software can scale to larger problems. The best software is easier to maintain and thus the rate at which value is added over the long term increases. The best software is more reliable and can be easily deployed without worrying about it crashing or malfunctioning.</p> Complexity</h2> We should clarify our use of the term complexity – we are talking about inherent and necessary complexity, not artificial, human-created complexity (some call this accidental complexity</a>). The human body is incredibly complex, but this is necessary for it to do all the things it does. Modern microprocessors are complex, but they need to be in order to do the amount of processing they do. The Linux kernel is complex, but only because this is required. In modern systems, software is potentially orders of magnitude more complex than hardware. We might spend X hours designing the hardware and XY hours writing and maintaining the software over the lifecycle of the product. This is hard to comprehend because the hardware can be seen, but the software is largely invisible. Software is increasingly the driver of value in modern systems, whether the systems are products or processes.</p> Change</h2> The development of hardware versus software requires a different mindset and process. With hardware, the effort is mostly in the initial development as we design, specify, and integrate the components needed. Once designed or qualified, a hardware component can be used for 5 or 10 years with very few changes. Its value is driven by minimizing variance (change), which is how we define hardware quality. Software systems are completely different – the inherent value in software is that it can be changed</strong> over a product lifecycle and its value subsequently increases, so the value in software comes from maximizing (instead of minimizing) changes. Software in a product may be updated hundreds or even thousands of times over its life. This is why it is so difficult for a company that is really good at hardware to also be good at software, or vice versa – a different culture and mindset are required.</p> Control</h2> How can we secure and have some level of control over the critical software components we need to build products? First, we should clarify the scope of the problem. Modern systems are increasingly assembled from hardware and software components we did not create ourselves. In a Linux-based system, there are many millions of lines of source code. The few thousands of lines of custom application code we might write for a product, which may represent many man-years of effort, are only a drop in the bucket compared to the rest of the software running in the system. However, it is easy to ignore the “rest” of the software, as it is largely invisible. How you manage the “rest” is critically important and perhaps the hardest part. And this “rest” is increasingly being supplied by open-source projects. Many engineers are good at creating things from scratch, but fewer are good at understanding, modifying, debugging, and managing software technology that others create. This skill is increasingly valuable.</p> Ideally, we would create everything we need and not be dependent on anyone else. However, we won’t get very far on our own if we need to create our own microprocessors, operating systems, programming languages, databases, and the myriad of other components needed to build systems. Likewise, these components by themselves are of little value unless they are combined into a larger system to do something useful. No one person, company, or even nation can build and maintain all the technology building blocks required to build modern systems. Human success is largely dependent on the degree to which we are able to work with others. The scope of the Linux kernel is beyond even the largest companies or governments. Even if an organization had the manpower, it would likely not have the culture required to pull off a project like the Linux kernel, the PostgreSQL database, or many other useful OSS projects. For smaller organizations, the capability to build even the most basic building blocks is often beyond our reach.</p> Culture</h2> Thus far we have established that to create value</strong> with modern systems, we must be able to build complex</strong> systems using software technology, much of which we did not create, and be able to change</strong> this software over time to deliver more value. In all this, we must have some level of control</strong> over this software. If there are problems, can we fix them? Can improvements be implemented and new features added? Secondly, can we influence development over time if we are not doing it ourselves? When we source software technology, there are many options, but let’s reduce it to two for the purpose of discussion:</p> Find a vendor with a proprietary software product. Initially, it fits our requirements – at least during the sales cycle. As changes/fixes are needed in the future, this vendor is hired to provide these updates or we work around them.</li> Find an open-source software project. We may purchase support from a company supporting this project or we may make needed changes ourselves.</li> </ol> There are no doubt times when #1 is the best solution, but increasingly #2 is being chosen. Why is this?</p> Let’s again review the challenges with software development:</p> Obtaining the software we need (value</strong>), as we only have the resources to write a tiny fraction of it.</li> Making changes</strong> over the product lifecycle, as this is where value is added.</li> Having some level of control</strong> and influence over the software we use.</li> Managing the extreme complexity</strong> of modern systems.</li> Providing an environment and finding the people where software innovation can happen (culture</strong>).</li> </ol> As we review these challenges, we start to realize that the primary benefits of open-source software are cultural. Software innovation is not created with a manufacturing-type mentality but is more art than science – a process of discovery, experimentation, collaboration, inspiration, and simplification. Developing great algorithms and code is a creative process that cannot be easily planned or scheduled, other than allocating the resources and time to work on it. The development of software innovations is not always predictable and cannot be forced. It requires the rare combination of the right people, the right environment, the right level of collaboration, and the freedom to experiment and innovate. All these conditions rarely exist inside corporations. Companies are realizing they cannot always hire the people needed or provide the environment for the level of innovation required. So the next best thing is to support open source projects where this innovation is happening. There are two ways to do this: time and funding.</p> Many authors of innovative software projects are driven by the craft itself, not wealth. Funding is necessary, but they are not working on a project to become rich – there are much easier ways to make money. They are driven by the intrinsic challenge, the joy of discovery, and the reward of building something useful. The first step in making sure an open-source project will be maintained and available in the future is to ensure that the authors have the funding they need. There will never be perfect equity in that authors will receive exact compensation proportional to the value they create, but this matters little to most OSS authors</a>, and in reality, this is also the case with most developers employed by companies. Many developers are more than satisfied with earning a good living and having the freedom to work on whatever they choose. Most are generous and glad to see their work being used. However, there are cases where OSS authors do not receive funding comparable to what they could earn working elsewhere, so it would bewise for companies to audit all the OSS components they are dependent on and ensure that these projects are adequately funded</strong>. Contributing financially also makes it more likely the authors of the project will respond to the company’s needs, but we need to be careful that this process does not become restrictive to the point where the environment for innovation is destroyed.</p> Contributing time (code, documentation, support, feedback, etc.) to open-source projects is another way to help ensure a project will be useful and available to companies needing it. Contributing to the community around an open-source project is critical for the project’s health and is part of the flywheel of innovation. If someone is a valuable contributor, then they will also have some influence on the direction and priorities of the project. They will become skilled at understanding and using the project. They will also become better programmers, as they work with some of the best in the world, and are exposed to well-written software. Open source projects provide an environment for innovation that is often not possible inside a company, so having your engineers do some work in OSS projects provides them with an alternative environment to do software development.</p> Why Open-Source Works</h2> It is somewhat of a puzzle why the open-source environment works so well for software innovation. Transparency is a powerful tool for great work. When you do things in public, you do them better. In OSS projects, there are no barriers to working with the best people in the world. Freedom is likely another ingredient. When there is the latitude to experiment, it is more likely that new and better methods will be discovered. Formal software development methodologies are good at managing more predictable efforts to solve business problems, but they seem to be of less value in OSS projects at driving innovation. Each successful OSS project is different than the previous one and is a unique combination of various factors that cannot be easily replicated.</p> Many people still stumble at the fact that if my company contributes to an open-source project, we are helping our competitors. This is true. However, if we don’t participate in open-source projects, we likely won’t get to where we need to go. It may no longer be optional. Open source is not like walking into a discount store and picking something off the shelf at a tremendously good deal. It is not a one-time decision where we select something and are done. It is an ongoing process of continual change and improvement. And the better we are integrated into this process, the more we can leverage and use open source. Economic opportunity is not a fixed-sized pie, where we are all competing for the largest slice. With software, the duplication costs are near zero, so this is a perfect opportunity for economic growth or an overall increase in society’s wealth. We can work to grow the size of the pie, so everyone can have a larger slice.</p> Often in writing, the author learns and understands things he did not know before, and this essay is no exception. The benefits of open source are not primarily in getting stuff for free or pooling resources in a socialist sense for the greater common good, but rather in providing an environment and connecting the right people for innovation to happen.This dynamic has happened occasionally in the past inside companies such as Bell Labs</a> and Lockheed Martin’s Skunk Works</a>. But this environment is obviously difficult to replicate or more companies would do so. Even the largest and most successful technology companies, such as Google and Facebook, aggressively support and promote open source. At the time of this writing, Google employs over 27,000 engineers – why would they need open-source projects? With that number of developers, why can’t they create anything they would ever need? Google’s support for OSS projects is not an altruistic gesture, but rather a calculated business decision – they are supporting the processes and people that will produce the software technology they need. This is a long-term game that requires experience, skill, understanding, and patience. It is an example of humility, which is the source of all true greatness – the realization that there are some things I cannot do, even if I have huge financial resources. It is the realization that there are people out there who are much better at certain tasks than anyone in my company or anyone I can hire. It is the realization that my company organization is not the perfect environment for the development of needed software breakthroughs and highly refined building blocks, so we need to support projects that are. Because it is counterintuitive, many companies will not make this leap, especially those who have been successful in the past without it, or those building products that in the past did not require advanced software systems. Ironically, it is often the smaller companies that need OSS components the most that are resistant to adopting and contributing to OSS projects. However, the opportunity is available for anyone to participate, support, influence, and use the most advanced software innovations available today. And these innovations are largely happening in OSS projects. Your engineers can work with some of the most gifted developers in the world and learn skills that will greatly enhance their careers and benefit your organization. You will be plugged into an engine of innovation that produces value. You will have more control over the needed building blocks. You can be part of the process that produces the valuable software you need, rather than just a consumer.</p> Open-source projects are far from perfect. They are messy at times, as human efforts tend to be. Their future is not guaranteed, but nothing in this world is guaranteed. The lack of formal business contracts/methods is perplexing to most business people. However, open source appears to be the best model we’ve discovered thus far for software innovation, and the results are amazing. Open source is the only supply chain that can deliver the needed value</strong>, innovate and change</strong>, manage the increasing complexity</strong>, give you some amount of control</strong>, and provide the culture</strong> necessary for innovation. Will you be a part of it?</p> Think Horses, not Zebras 2023-09-22T00:00:00+00:00 There is a popular quote</a> in medical circles:</p> When you hear hoofbeats, think of horses, not zebras. – Dr. Theodore Woodward</p> </blockquote> “Zebra” is the American medical slang for arriving at a surprising, often exotic, medical diagnosis when a more commonplace explanation is more likely.1</a></sup> What does this have to do with product development? Like the medical profession, we often diagnose problems – we call it debugging</a>. Below are three recent cases where I would have been helped by applying this approach a little more rigorously.</p> Simple IoT Startup</h2> I recently encountered a bug</a> where the compiled version of Simple IoT would not work on one of my computers. Interestingly, go run</code> worked fine. What was the difference?</p> At first, it seemed that a race condition was at play – the compiled version was faster, did not have the race detector enabled, and would likely have different characteristics. SIOT is a highly concurrent application with many clients running in parallel. However, after spending some time debugging, I could not find any issues with startup, rather the NATS client in SIOT could not get data from the embedded NATS server. Again, race conditions were assumed, so I tried an external vs. embedded NATS server – no difference.</p> I finally observed the network traffic with Wireshark and could not observe any NATS traffic in the failing version. The NATS client was not even sending any requests. I then looked at /etc/hosts</code> and there was no localhost entry. After adding this, everything worked properly. See this discussion</a> for more analysis of this issue.</p> In this case, a race condition was the zebra and the missing entry in /etc/hosts</code> was the horse.</p> i.MX8 Stability</h2> We recently ran into a problem with a new product that uses a Variscite VAR-SOM-MX8</a> in a custom baseboard. The software was stable on the Variscite development baseboard, but the same SOM (System on Module) would crash in the custom hardware. The system would never crash until after the Yoe updater initramfs ran and the main rootfs was loaded. Hardware was the immediate suspect – the power supply is probably not able to handle the dynamic load when multiple cores spin up, etc. However, the power looked stable and we could not find any hardware issues. In the end, we found a missing device tree line ("fsl,imx8qm")</code> in the Linux kernel DTS file for the new design. This missing construct caused some critical kernel initialization code not to run.</p> In this instance, a subtle power problem was the zebra and a missing line in the DTS file was the horse.</p> STM32H7 USB Instability</h2> In a recent STM32H7 design, we wanted to use a High-speed USB Phy with the STM32H7 and Zephyr. This is not supported by Zephyr yet, but with a few hacks</a>, we got it working. We are trying to push a fair amount of data over this link (10’s Mb/sec) so some optimization and performance work was required on both the sending and receiving end. In the mix of all this, we got a new revision of the target hardware, and at some point started seeing an increase in error rates and the target not enumerating and connecting properly to the host. With a lot of moving parts and a lot of data flowing through the system, it is sometimes hard to pinpoint the problem. In this situation, we tried to narrow the scope by testing older/newer versions of Zephyr and comparing STM32H7 reference designs to the customer hardware. We found the reference designs performed much better. We also learned that a previous version of the custom hardware also performed like the reference designs, so it appeared there was a hardware problem with the latest custom hardware.</p> When something worked before, and now is experiencing high levels of communication errors, the first thought was signal integrity. On the PCB design, there is a fairly long trace from the USB Phy to the connector, so this was an obvious suspect. Although we were careful to design the PCB such that the impedance of these traces was controlled, there is always the possibility that the PCB vendor did not implement the board stack-up correctly or some other strange issue. One of the engineers analyzed the trace impedance on the old and new boards with a network analyzer. This is a fairly involved test procedure that requires cutting traces, soldering coax cable to the PCB, and a fair amount of expertise. The USB traces between the two revisions looked equivalent and the impedance measured was correct. The next suspect was the high-speed signals between the MCU and Phy – we had changed some of the length matching between the two versions, so perhaps we messed something up. In the process of looking at these signals, the following was observed when scoping the data line:</p> </figure> The data line was running at two different voltage levels. The problem ended up being the power to the VDDIO pin on the phy was floating, so the signals from the phy to the MCU were not being driven at the right voltage levels. Simply scoping all the power pins on the Phy would have found this problem in a few minutes.</p> In this instance, signal integrity was the zebra, and a floating power pin was the horse.</p> What can we learn?</h2> In all of the above cases, the initial assumptions about the cause of the problem were considerably more complex than the actual problem turned out to be.</p> Problem</strong></td> Zebra (complex)</strong></td> Horse (simple)</strong></td></tr> SIOT Startup</td> Race conditions</td> Host networking config</td></tr> i.MX8 Stability</td> Dynamic power supply problem</td> Missing line in device tree file</td></tr> SMT32H7 USB errors</td> Signal integrity</td> Floating Phy IO power</td></tr></tbody></table></figure> Hindsight is always 20/20, so it’s easy to say we should have solved all these problems more quickly than we did. But there is always something to be learned, and I think as humans, our minds generally go to the more exotic scenario. Part of this may be overconfidence in our ability to get the simple things right, like copying a DTS file from a reference design, or supplying power to a chip. How can we possibly mess up something so simple? So we tend to overlook the simple and obvious and jump to the obscure and exotic, which exercises our most advanced abilities and knowledge. The more experienced we are, the more likely we may be to think this way. To overcome these human tendencies, it is helpful to acknowledge first that they exist (humility), and then put in place a process for future endeavors to help guide us in our debugging efforts. Consider the following maxim:</p> When debugging a problem, always verify and eliminate the simple/obvious possibilities first. Work from simple to complex.</strong></p> This is a pragmatic approach because verifying the simple possibilities usually does not take much time, so little is lost in starting there. It may be boring or tedious, which is why we subconsciously tend to avoid these, but we can save a lot of time if we just follow this simple rule. A general process might be:</p> isolate the problem as much as possible (remove unused functionality, etc.)</li> write down everything you know, possibilities (all of them), and ideas of what you might try.</li> sort your ideas from simple to complex and work on the simple ones first.</li> document all tests and results.</li> meet as a team regularly to discuss results and brainstorm the next steps.</li> </ol> Some specific ideas:</p> verify basic network connectivity any time a network is involved, even if on the same machine.</li> don’t ignore any problem in the system. Often problems are related, and solving one problem sometimes solves another.</li> check that all components involved have good power.</li> compare to a known working reference design (both operation and design).</li> ask for help in relevant communities.</li> review and verify component pin-outs.</li> read the datasheets of the components involved.</li> read the errata for components.</li> work backward from a known working setup.</li> with web apps, open the console, and look for errors.</li> carefully review boot and build logs and diff with logs from working systems.</li> narrow the scope of the problem wherever possible.</li> try multiple systems.</li> carefully review any changes between working and non-working versions. Verify the operation of any new circuits or code.</li> don’t keep doing the same thing and expect different results.</li> </ul> Debugging is an art and a part of the development of any complex system. We must realize that our intuition often fails us in these situations and, with any human weakness, we must have a process to compensate. This debugging process can turn what is often a frustrating and scattered experience into something much more enjoyable and productive. Think horses first, and then zebras!</p> ^{1</sup> https://en.wikipedia.org/wiki/Zebra_(medicine)</a></p> </div> The Technology Trajectory 2023-01-03T00:00:00+00:00 See also a podcast episode</a> where we discuss this topic.</em></p> Modern systems require the integration of technology, often from many sources. This can come in the form of processors, integrated circuits, operating systems, open-source software components, databases, messaging systems, cloud infrastructure, programming languages, build systems, frameworks, etc. All technology follows a trajectory – an example is shown below. The timeline may be short or long. Your usability threshold may be low or high, sloped right or left, etc. But the fact remains that most technology will eventually be replaced by something newer. We can debate whether or not newer is better, but the fact remains that time marches on.</p> </figure> Why can’t we use a particular technology forever? With hardware, components may not be available. With software, we potentially have a little more control, but if we want to use a software component beyond the above cycle, then we have to commit to taking over maintenance. This means:</p> if there are bugs, we will fix them</li> if there are security problems, we will fix them</li> if there are build problems with new versions of compilers and other libs, we will fix them</li> </ul> For complex, networked projects, this can be a significant undertaking.</p> But a far bigger issue is that of value. Open source software and corresponding ecosystems are continually improved over their life and value is added. If you use projects in your development that are being improved, then value is being added to your product. The rate at which value is being added correlates to the slope of the curve above.</p> To extract the most value from technology, it is generally best to adopt technology when the trajectory is rising and there is project momentum, not when everyone else has moved on. There are several reasons for this:</p> the rate at which value is being added is greatest during this period. This makes what we are building better.</li> you have an opportunity to participate in and influence the development</li> you have more time to benefit from the technology</li> more support is available as there are more people participating in the project</li> in programming languages, there are more maintained 3rd party packages available</li> </ul> We also need to select technologies that have a reasonably long life cycle as switching out a complex component can be a lot of work.</p> There are many other perspectives by which to evaluate technology – this is just one of them. Technology follows a trajectory from inception, growth, maturity, decline, and finally obsolescence. How do you know where a particular technology is on the curve? It is probably partly gut feeling, part experience, part asking others, and perhaps there are analytical ways to approach this as well (search engine ranking, etc). But the most important thing is to use this concept as a framework for objective thinking and evaluation. Effective technology selection is a long-term effort. We need to have a process. We may have to watch a project for a year or more to get a good feel for it. Most of us can’t significantly change the trajectory of most technologies, but we can choose where on the curve we will participate. Inefficiencies occur early and late on the curve. If we are always too early, we likely won’t get much done. If we are always too late, then our products will be a few steps behind everyone else’s.</p> Go for IIoT Systems 2022-12-09T00:00:00+00:00 As developers, we have many options for programming languages. On one hand, it is great to have choices. On the other hand, it can be a little overwhelming. Many times in life, we need to make decisions before we have the experience to know the best answer. There are many opinions as to what is best, and this article is just another one. There is no one-size-fits-all – a lot depends on your team and the task. In this article, we will examine the challenges we face in developing industrial IoT (IIoT) systems, review desirable attributes in a programming language, and discuss how Go meets these needs.</p> Peter Bourgon defines the industrial programming context</a> as:</p> in a startup or corporate environment;</li> within a team where engineers come and go;</li> on code that outlives any single engineer; and</li> serving highly mutable business requirements.</li> </ul> Additionally, an industrial IoT system can be thought of as a system that gives users access and control of remote systems. There are many variations of this, but let’s focus on a simple case that is typically composed of the following pieces: Edge device (remote equipment, sensors, etc), Cloud, and Browser.</p> IoT Components</figcaption></figure> All of these pieces must be dealt with at some level and will require some programming. The fundamental requirement of this system is networking – data must be moved between the above three instances. IIoT systems are inherently distributed systems. Distributed systems are hard</a>, thus the distributed problem needs to be solved well if you want to have a scalable, reliable IIoT system. It is natural to focus on the end application at the edge and add IIoT as an afterthought. However, this rarely works very well.</p> The context of this article is that of an average programmer working in small teams where engineers do a lot of different things. Many of us are not full-time programmers doing deep theoretical work, but find ourselves building these complex systems. Some of the challenges/requirements we face include:</p> resources are constrained at the edge (CPU, Disk space, etc)</li> networks can be unreliable</li> config/state needs to exist in multiple places (edge & cloud)</li> software updates need to be deployed to remote places (edge & cloud)</li> end-to-end (edge <-> browser) real-time response is often desired</li> many parallel operations are happening (collecting data, running rules, managing the system, etc)</li> physical access to edge systems so debugging problems can be difficult</li> edge systems typically use Embedded Linux, which can be a challenge to develop and maintain</li> IoT systems are inherently concurrent and distributed</li> many of us are doing this part-time, are not computer science gurus, and need pragmatic solutions.</li> </ul> Attributes of Go</h2> Go has several attributes that make it a good fit for this problem:</p> excellent reliability. I’ve written about this before</a>. (our systems are not crashing in the field)</em></li> it is a compiled language (reasonably fast and small)</em></li> runtime is small, embedded, and efficient (this means there is very little overhead to distributing the entire runtime with each binary. This makes deployment easy.)</em></li> statically linked (this makes it much easier to deploy updates as everything you need is embedded in one binary file)</em></li> easy to embed assets (again, this makes it easy to deploy in one file without containers)</em></li> is garbage collected, so we don’t have to worry about managing memory (we don’t have memory leaks, invalid pointers, and other problems that can cause field failures)</em></li> typed language (compiler catches more problems early instead of having crashes in the field)</em></li> popular and well-supported (will continue to improve and be around for a long time)</em></li> excellent concurrency support (a lot is going on in these systems, so this helps)</em></li> popular in cloud and distributed systems (there are plenty of stdlib features and 3rd party packages – metrics, monitoring, logging, communication, etc to help us build an IoT system)</em></li> language is simple (easy to learn, read, and maintain)</em></li> tooling is excellent, compile is blazing fast, no Makefiles, etc, built-in support for profiling, and easy to cross-compile for other systems (we spend less time messing around and more time deploying value to our customers)</em></li> </ul> Go is not the best solution in every case, but if your situation matches what is described above, it might be a good fit.</p> Go removes a lot of complexity</h2> Cloud and edge systems typically run Linux. You may outsource parts of the cloud portion and perhaps even parts of the edge device, but you are generally responsible for the edge device. It must run 24/7. You likely need to deploy updates, maintain security, and add functionality over the product lifecycle to meet customer needs. You will run into issues and need to be able to debug them. You can’t walk over to an edge or cloud system and press the reset button or hook up your JTAG debugger. Most embedded software in the past has been developed for MCUs (microcontrollers) and written in C/C++. This is a fine solution for disconnected control-focused products, but a Linux MPU (microprocessor) solution provides you with more flexibility and functionality for data-focused connected devices that process/store a lot of data and have complex user interfaces. Linux provides unparalleled support for a large number of devices, USB and networking functionality is reliable and mature, and it scales to processing and storing large amounts of data. In an MCU platform, you may write most of the software in the system. In some cases, you may use an RTOS (Real-time operating system), but even these are relatively small as resources are limited. In a Linux system, you are running millions of lines of code you did not write. We can use the iceberg analogy – the part underwater includes Linux and system libraries which you don’t see most of the time. The part you see above water is your application, which is a tiny portion of the total software in the system.</p> </figure> Part of the value of modern systems is the huge number of software components you can reuse in your system as there are software libraries/packages for many tasks. Building modern systems is partly the creative combination of existing/reusable technology and the creation of new technology (the code you write).</p> </figure> So the question arises: How am I going to get up to speed on all this?</strong> One solution is to minimize the amount you need to interact with the stuff underwater. If you build applications with C++, the compiler does not give you a lot of help in linking to libraries and cross-compiling to edge system architectures. (Cross compiling is compiling native code on one architecture for another – typically developing on an x86 workstation and then running on an ARM Linux target system). Thus, you need to rely on complex build systems (CMake, Yocto, Buildroot, etc). Every time you add/update a library/package, you are faced with the following questions:</p> How do I integrate this library into my app build?</li> How do I cross-compile to my edge system?</li> </ul> It’s not easy, thus projects such as Yocto and Buildroot exist. All developers need to have a cross tool-chain (compiler and libraries) installed on their development computer that matches the libraries in the edge system. Deploying an application update often means deploying additional libraries, or library updates. So we add container technology to try to manage all this, which is another layer of complexity.</p> Some solutions (like Raspbian for the Raspberry PI) advocate developing directly on the target device to avoid the complexity of cross-compiling. This works fine for small one-off maker projects, but for the development of large industrial programs, it is a poor solution as it is like developing on a PC from the 90s. Sure, cross-compiling is easier, but everything else is much harder.</p> Python or Node.js may look attractive in that they are easier to program than C++ and the program is interpreted on the target device, but many of the underlying packages for these languages depend on C++ code, as interpreted languages are slow, as a result the build problem is worse because it is even more difficult to cross-compile this code in the Python and Node.js build systems. Additionally, a separate run-time is required on the edge system which is relatively large and must be kept in sync with the application.</p> Go sidesteps most of this because it is a mostly pure ecosystem. There are no host library dependencies in a pure Go application – everything it needs is embedded in a single binary file (typically 2-10MB compressed). The Go standard library is rich and has many needed functions. Additionally, there is a rich selection of pure Go 3rd party packages</a> that cover most other needs. Adding packages to a Go application is as simple as go get some/package/name</code>. There are no development tool requirements other than a standard Go installation and an editor. No complex build tools are required to add 3rd party packages or to cross-compile to different target systems (macOS, Windows, Linux). It has built-in support for profiling. If it crashes, you always get a stack trace. It’s so easy, and the learning curve is small.</p> Many software tools today add layer upon layer to solve the problems with the underlying technology. While this helps in some cases, it often adds additional complexity. You can’t gloss over complexity – when things go wrong, you still need to understand the entire stack anyway. Abstraction has its place but can only take you so far. If there are problems in one technology layer, adding additional layers sometimes just makes it worse. Go threw everything out and started over</a>. Simplicity is the only sustainable approach to technology.</p> Go is a relatively new language (released in 2009), so many people have not heard of it. Learning a new language does take some time. Go has a small learning curve, but that is nothing compared to the learning curve of building, deploying, and maintaining C++ code in edge systems. Go allows you to quickly become productive and spend very little time fighting tools over the product lifecycle. Moving your Go application forward over time often does not require any changes to the base Linux system because they are decoupled.</p> Go is familiar and simple</h2> Google originally developed Go</a> as a solution to the pain they were experiencing with C++, and they are still investing in it</a>. Go syntax is similar to C and was developed by some of the same people</a> who were involved in the development of the C language and Unix back in the 1970s. Thus, it feels very familiar to those of us who have spent time developing in C, Java, C#, etc. Go eschews the complexity that has overtaken C++. Language features are added slowly and carefully. Compile errors are generally easy to understand and debug. Backward compatibility has been excellent. Since 2014, I’ve implemented several systems in Go and have not had any significant issues after updating the Go or 3rd party package versions. Go developers take API compatibility seriously, so this helps.</p> Go is a simple language but does not force you to do things correctly (as a language like Rust does). You can still make a concurrency mess with channels. Go has nil</code> values, so you can still crash your program if you try to reference a nil</code>. If you don’t check returned error values, you’ll probably run into problems. But, if you follow a few simple best practices and idioms, it works well and is a nice experience. Simple programming languages are much easier to read and maintain.</p> Go Concurrency Support</h2> IIoT systems have a lot going on, so you need good support for concurrency. Interpreted languages like Python and Node.js are weak in these areas. Go has excellent support for concurrency by providing channels and the go</code> keyword as language features. This allows for running concurrent tasks and communication between them with very little overhead. Go channels are easy to misuse, so it requires a little effort and practice to learn how to use them correctly, but once understood, it is a very simple and reliable way to manage concurrency in a system.</p> Go is an Internet language</h2> While older languages like C/C++ and Python have been excellent tools for solving programming problems in the past (and are still the best in some domains like MCU development, machine learning, etc), the problems we are solving in IIoT systems are different today in several ways:</p> we are dependent on large ecosystems of 3rd party packages to implement all the functionality we need</li> we need better tools for concurrency and distributed systems</li> we need simpler models for network programming</li> we need software that is easier to build and deploy</li> </ul> Communication between systems is the fundamental problem IoT systems solve and Go is good at this. Go is the best-in-class for a new breed of software: an Internet programming language</a>. As evidence of this, most cloud and distributed software infrastructure is written in Go these days (Kubernetes, Docker, NATS, etc). Cloud companies that scale to thousands of systems are very sensitive to cost, thus they need efficient platforms. This efficiency in Go also benefits us at the edge where resources are constrained. An added benefit is that you can write your edge and cloud applications in the same programming language.</p> But Go has no GUI support …</h2> Although there have been many excellent efforts to develop a native GUI library for Go, none of them have emerged as the standard. There is a good reason for this … the browser is the standard UI platform these days</strong>. Nothing else can come close to touching its capabilities. It is cross-platform, reasonably performant, and very well-supported. The browser has so many options for programming, packages, etc. The browser has built-in support for video and other forms of media and is highly optimized. Developing applications in the browser is more difficult to get started than in a C++ GUI toolkit. But there is a much richer ecosystem of packages and functionality available in the browser platform. The browser platform is stable and you are ensured of backward compatibility – it will be around and well-supported for a long time. The web platform is built on solid engineering principles that have proven to scale. Many UI technologies designed to replace the browser programming model have come and gone: Java, Macromedia Flash, and Microsoft .net – yet the standard browser programming model of HTML5 + Javascript has displaced all of these and is growing at an astounding rate. Google Flutter</a> is the latest attempt to displace the web platform, but it remains to be seen yet if it will be successful. Most C++ GUI toolkits are highly dependent on a single company that may be here today and gone tomorrow (look at the history of Qt</a> for an example), or licensing requirements may change</a> at any time – there are a lot of unknowns. Plus, with the web platform, you can view the UI remotely just as well as locally. You can also use the same UI building blocks in the cloud as in the edge device. If designed correctly, web applications also function very well on mobile devices, eliminating the need to develop separate mobile apps. With the web platform you can address all of the following:</p> on-device LCD UI (runs a browser full screen)</li> remote access to the device using a browser</li> cloud application UI</li> mobile device access (browser, PWA, Cordova, etc)</li> </ul> The experience of a native phone app might be slightly better than a web application on mobile devices, but how many of us can afford to develop 4 different UI applications for a product (native, web, Android, iOS) when one will do in most cases? The question is not what is possible or technically best, but what is practical with the limited resources we have.</p> Languages like Elm</a> simplify front-end (web) development in many of the same ways that Go has simplified back-end development.</p> But Go is not the best at …</h2> Very true – Go is not as easy as Python, as widespread as C++, as popular as Javascript, as safe as Rust, or as good at concurrency as Elixir. But, Go is almost as good as the best at everything and its tooling is much better than any other language ecosystem – especially for distributed applications that need to be deployed to remote and different architectures. It does not have any of the significant drawbacks other languages have. And especially for networked IoT systems, it is an excellent choice. The below decision matrix likely reflects my biases, but illustrates this point (green is best, red is worst):</p> </figure> But what if I need to use C++ …</h2> There are certainly cases where you may need to use C++ or some other language. You may choose to implement a native UI using Qt or LVGL. Machine learning code is often written in C++. There is no reason your system needs to be written in one language – Go can be used in combination with other languages. There are several ways this can be implemented:</p> call the C++ code directly from Go (CGO)</li> run the C++ and Go code as separate applications and communicate between them using a message bus</li> </ol> Method #2 is preferred in many cases as it keeps your Go code clean, and the C++ code can be built using standard C++ build systems instead of trying to force the Go build system to build C++. Simple IoT</a> is one framework that is designed for exactly this scenario:</p> </figure> This architecture keeps your Go code easy to build/develop/deploy. Your core logic, which manages config/state and communicates with other systems, can be written in Go. This core logic must be reliable and easy to maintain. Specialty code like machine learning models can be dedicated C++ applications. NATS is a modern message bus written in Go that is easy to embed in any Go program and is a great way to connect different applications.</p> </figure> Optimize for where you will spend most of your time</h2> Linux-based IoT edge systems have plenty of memory, storage, and processing power. Thus, features are continually added over the product lifecycle, which can be 5-10 years in an industrial system. The time spent maintaining and improving a product often swamps the initial development. Therefore, it makes sense to optimize for the long term – adding features and deploying updates over the next ten years. Go makes it easy to build and deploy updates with the latest Go and package versions.</p> Friction</h2> Developer productivity is difficult to fully understand. We often think of the features we need in our languages/tools. Commercial software vendors focus on features. Development methodologies are designed to increase productivity by better organizing what needs to be done. But another aspect that is easy to overlook is the amount of time we spend messing around working on stuff that does not directly add value to our product. I call this “friction.” This friction can manifest itself in unforeseen problems that are more common in complex systems: learning curves, build issues, issues with libraries/packages, lack of flexibility, security problems, deployment problems, bugs, race conditions, stability problems, technical debt, lack of control over parts of the system, etc. Go does an excellent job of reducing friction in the development process. You focus on what needs to be done, and the rest is mostly invisible.</p> Summary</h2> In this article, we explored many of the attributes of Go that make it ideal for implementing IoT systems. Go provides a modern programming ecosystem where the build tools are first class, the language is simple, and there is little friction to get things done. Go also provides features such as garbage collection and concurrency support in the language which are very helpful in IoT systems where data is distributed and a lot is going on. Go is an Internet language that is increasingly used to develop distributed systems, which includes IoT systems. It is a stable (almost boring in many respects) technology and is a great option for developers who are familiar with C/C++ and want to expand into Embedded Linux/IoT – the skills you have are very applicable in Go.</p> What is the main constraint in your projects?</p> Application performance?</li> Getting things done and shipped?</li> </ol> If #1, then perhaps C++ or Rust is a good choice. If #2, then take a look at Go.</p> Go is not the best solution for every team/project, but for the scenario described in this article, it has worked incredibly well. With Go, we can develop and deploy remote systems with confidence.</p> Layered, Event, and Data-Centric Architectures in Distributed IoT Systems 2022-06-14T00:00:00+00:00 With the advent of the browser, cloud, embedded Linux systems, and networked microcontrollers, distributed systems are everywhere. There are many models for communication in distributed systems – we will look at the tradeoffs between three of them. There are many perspectives to consider – initial implementation, client libraries, maintenance, adding features now and in the future, client compute and storage requirements, network bandwidth, data structure, etc. This article will discuss a number of these concerns and present several options.</p> The motivation for this discussion is synchronizing data in IoT systems where we have the following requirements:</p> Data (state or configuration) can be changed anywhere, at edge devices or in the cloud, and this data needs to be synchronized seamlessly between instances. Sensors, users, rules, etc. can all change data. Some edge systems have a local display where users can modify the configuration locally as well as in the cloud. Rules can also run in the cloud or on edge devices and modify state.</li> Data bandwidth and allowance is limited in some IoT systems (example Cat-M modems (< 100kb/s) and cellular IoT plans (< 50MB/month)). Additionally, connectivity is not always reliable, and systems need to continue operating if not connected.</li> </ol> The first is the Layered</strong></em> model. In this model, various instances talk directly to other instances. If any information needs to be shared, it is sent directly through an API (application programming interface) call using REST, CoAP, and other protocols – sometimes traveling through multiple layers.</p> </p> The layered, or API model’s advantage is simplicity – almost anything can connect to a REST endpoint. Most networked devices have built-in support for HTTP. Many companies provide services through REST APIs. Polling is typically used in the layered model to determine when things change, which limits the response time to the polling frequency. There are mechanisms that can be used in a layered architecture to obtain real-time response such as HTTP long polling, WebSockets, Server SIde Events (SSE), callbacks/webhooks, CoAP subscribe/notify, etc. However, an event bus is generally a much cleaner implementation if real-time response is needed. Firewall and security concerns can also make callbacks impractical on edge devices. If a device at the end of a layered architecture needs new information, often multiple layers need to be modified to handle this new information, which can make changes expensive.</p> A second model is the Event</em> or Message Bus</em></strong> model. Implementations include NATS, MQTT, Kafka, and many others. In this model, every instance in the system is connected to a message bus. All data communication is routed through the bus. Clients can post or subscribe to topics of interest so that updates happen in real-time.</p> </p> Some advantages of the event bus include real-time response and streaming workflows. All communication and information is on the bus, so if an instance needs to send or receive some information, it simply subscribes or publishes it. Event buses often leverage one connection to the message server so that each message does not need to establish a new connection, which can bring some efficiencies and reduce needed resources</a> (such as number of TCP connections). However, the event bus protocol is considerably more complicated than REST, so a message bus library is typically required to implement the client application.</p> The third model is the Data-Centric</strong></em> model. In this model, each client instance contains a set of data it is interested in. In the below example, the cloud system has an aggregate of all the data in a system, and the browser and edge instances are subscribed to a subset of the data they are interested in or have access to, represented by the different colored blocks.</p> </p> The data models the entire system, so if an action (setting an output, sending a notification, etc) is required, a bit of data is changed, and then the system takes action based on this data. The needed data is replicated to each instance in the system and is automatically synchronized in any direction when it changes.</p> To all you Go programmers out there, this may immediately raise a big red flag. What? Communicate by sharing common data?</p> Don’t communicate by sharing memory; share memory by communicating.</em></p> R. Pike</em></p> </blockquote> Pike’s comment is typically used in the context of an application where multiple threads (goroutines) share data in memory, and locks are used to keep multiple threads from writing to it at the same time. This model can be hard to reason about due to race conditions. An analogous model in distributed systems may be a common database that all instances use and all config/state is stored in this remote database. Locks may be analogous to database transactions. A common database is a good solution for many cloud based distributed systems where everything is always connected and, and this model allows data to be modified from multiple locations. However, with IoT systems, we have the following constraints that make it impractical to use a single remote database as our primary configuration/state data store for everything in the system:</p> We often don’t have the network bandwidth or data allowance (ex: LoRaWAN or Cat-M modems and low cost cellular IoT plans)</li> Network connections are unreliable, and edge instances need to continue working (collect data, run rules, etc.), even if the network is down.</li> Network latency can be long – especially with low bandwidth connections.</li> </ol> This leads us to a “local first”</strong> data model, where all configuration and state needs to be stored local to an edge device. However, this data is often viewed or edited in the cloud, so it also needs to be present there. We could craft new data structures and messages for every new bit of data and send custom messages over a REST API or event bus every time anything on either end changes. However, a simpler approach is to design a generic data model that can hold any kind of data, and then a mechanism to synchronize this generic data model. Then, adding functionality is much simpler as we can write/read data wherever needed, and synchronization between instances happens automatically with no extra effort. However, we have to learn to work within the constraints of the data model. A new feature might involve tweaking the UI and writing some code on the edge instance. Often, nothing in the middle needs to change:</p> </figure> Each instance has its own local copy of whatever data it needs. But instead of the developer needing to manually manage the synchronization of data between systems using custom messages, changes to the data are automatically synchronized in any direction.</p> The data-centric architecture requires more computing resources and storage to run it – typically an embedded Linux system. The below chart illustrates the trade-offs between client complexity and programming simplicity.</p> </p> For less capable networked devices such as microcontrollers (ex: Arduino), you may be limited to a layered approach, as HTTP may be the only supported protocol. However, as microcontrollers advance, many of them are now supporting event buses such as MQTT. The cost of a data-centric architecture is an up-front cost when building the base system. As time goes on, the incremental costs of adding features is much lower. Since adding features is typically a continuous, long term exercise, the initial cost is paid back many times over. A data-centric architecture also introduces some discipline in how data is structured, stored, and synchronized. This helps avoid the technical debt that rapidly accrues in less structured approaches.</p> All architectures have their place. If the application is fixed in functionality, and you only need to send a temperature reading to the cloud once per hour and this is never going to change, then a microcontroller with a layered architecture is fine. If you need to send data in both directions and real-time response is needed, then moving up to an event bus makes sense. If you have devices that have a decent amount of configuration that can be modified at the device and in the cloud, runs rules locally, and features will continually be added to the system, then a data-centric architecture is beneficial. The data-centric architecture requires a more complex system to start with, but the incremental effort to add features down the road is drastically reduced because the developer does not need to worry about the data storage/synchronization problem. Additionally, the data model is available at each instance, so programming is simpler.</p> It is possible to support multiple architectures in one system. For example, in Simple IoT</a> all data modifications are sent over an event bus which is the primary programming API</a>. A client can connect to the event bus of a local or remote Simple IoT instance. If the local and remote instances are connected using an upstream sync connection</a>, then writes to the common data-set of either instance are automatically synchronized to the other. If a client does not want to implement a local data store, they can connect to a remote instance’s event bus. NATS JetStream</a> is another example of a store built on top of an event bus. Some event bus implementations like NATS allow for easy request/response transactions</a>, which is similar to a layered REST/RPC type interface. Thus clients in a system like Simple IoT can range from simple HTTP clients, to mid-range event-bus clients, to advanced Linux based, data-centric clients with a local data store.</p> In computer science, focusing on data is typically the right approach.</p> I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships.</em></p> Linus Torvalds</em></p> </blockquote> Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.</em></p> Fred Brooks</em></p> </blockquote> Distributed systems are no different – the focus needs to be on the data. Synchronizing data efficiently and reliably between distributed instances is a hard problem to solve, and the focus of much research and development effort today (example: the Riffle project</a>). This requires some up-front thought, planning, and constraints, but the payoff is simpler data models that result in systems that are more reliable and cost effective over the product life-cycle. Data-centric architectures free developers from the tedium of synchronizing data and allows focus on the application problem at hand. Data-centric architectures also provide discipline and structure around the data synchronization problem that helps avoid technical dept build-up which is a great impediment in the flow of value to end users.</p> Each distributed system architecture has its place, and with a correct implementation, we can support multiple architectures in one system. This gives us the option to support simple as well as advanced clients, depending on the need. However, as the cost of advanced MPUs systems running Linux such as the Raspberry PI continue to fall, deploying advanced IoT clients to the edge makes a lot of sense.</p> In future articles, we’ll examine in more depth the techniques used to store and synchronize data in data-centric architectures.</p> Thanks to Khem Raj</em></em>, Bruce Stansell, Gina Brake, and Collin Brake for reading drafts of this.</em></p> Business Contracts 2022-03-21T00:00:00+00:00 Business contracts are often executed when two companies work together. At the most basic level, a contract should describe a common understanding between the two parties. The contract should assist and aid the relationship. However, many contracts are written by lawyers with the primary goal of covering every legal loophole and taking every legal advantage possible for their clients. This often results in contracts that are very long, filled with legal jargon, benefiting only one party, and in general, a burden to process.</p> For a business relationship to be successful and sustainable, there has to be value</strong> for both parties and there has to be trust</strong>. If these two key ingredients are lacking, there is little motivation to continue the relationship.</p> We can look at business relationships as a positive-sum game, or a zero-sum game. A positive-sum game is where the relationship is structured as a win-win – both parties find good value in the relationship and are motivated to continue it. There is a belief that the “pie” of opportunity will grow through their combined efforts. The engagement is voluntary and not constrained by force or compulsion. Zero-sum thinking is where one party is always trying to gain the maximum advantage over the other party. This is the “fixed pie” mentality where we think the resources in business are fixed, and if anyone else gains something, it is my loss. This attitude is often found in purchasing departments where they are trying to obtain the lowest possible price for a good or service, net 60 payment terms, etc. I recall in one case a manager of a company exclaiming, as we went into a meeting with a vendor, “Let’s go beat ’em up!”</p> Business relationships are more complex than getting the lowest price or the greatest legal advantage. The people/companies you want to do business with are likely already busy. They likely have plenty of opportunities. They will seek engagements with those who play the positive-sum game. If a company plays the zero-sum game, they will end up doing business with second rate companies who don’t have any other choice. They won’t get good service, and trust in the relationship will be lacking. They will be forced to hop from one vendor or customer to other. There will be few long term relationships. As Bob Metcalfe once wrote</a>, “First, don’t “hire” anyone. B people hire C people – they collect resumes and choose the person they want to honor with a job. A people recruit A people.“</em> The same applies to business relationships.</p> With this background, let’s examine the topic of contracts a company may present to an independent contractor or consultant providing services to the company.</p> Again, the best developers (the ones you want) are successful and busy. They will not mess around in business relationships where people are playing the zero-sum game. They are successful because they are trustworthy and expect the same in return. They are generous with their time and knowledge because they understand the universal law that the more you give, the more comes back to you.</strong> But they also expect the same in return. They are confident in their abilities and do not take measures that lock their customers into their services, but on the contrary, make themselves “easy to fire</a>.”</p> A fundamental problem with trust is that people generally don’t tell you when they don’t trust you. Actions that damage trust are likely unintentional (otherwise you would not do them), but you never get feedback on this. So, I’m giving you a hint now – an onerous contract, with one-sided indemnification clauses, damages trust. It tells the other party that “I might take advantage of you.” It implies that if bad things happen due to a product flaw, “you could be on the hook.” It sows seeds of doubt. Most often, this is likely not the thought of the company presenting the contract, and the contract language is just the result of lawyers trying to do an exceptional job of protecting their client from any possible liability. However, a contract is still a contract. If disaster strikes, the contract will be used in court by various parties trying to recoup damages. Products are often sold over a long time (5-10 years or more). Companies often change owners. Therefore, it is very important that a contract be fair and just. When a developer is presented with an onerous contract, immediately the question arises – can I trust this company? Is it worth it? Even if the project goes forward, it will be with some reserve and guard. It will not be with the wholehearted attitude that is an ingredient for success.</p> As we consider indemnification phrases in a contract, we need to establish who should assume liability for a product: 1) a consultant/contractor who is billing by the hour, or 2) the company selling the product over a span of many years? Risk should be proportional to reward. A hourly contract developer has no reward for a product’s success, other than the baseline costs of doing the development. The company selling the product is the one who will potentially reap the multiplying rewards of a successful product. Therefore, they should be the ones who bear the risk as well. It is not just to expect otherwise. In the situation where a developer is providing consulting or contracting services to a larger company developing a product, it is important that the contract clearly state that:</p> The Company assumes sole responsibility for all testing of the Work, including but not limited to software and hardware, and also assumes sole liability for any use or misuse of the Work developed by Contractor for the Company.</strong></p> An argument may be made to the developer – you are not a lawsuit target because you are not a large company with deep pockets – so just sign the contract. This may be true, but what if the consultant does grow the business to the point where there are more assets? Products live a long time – a lot can change. As mentioned before, a company can change hands. If there is a lawsuit, then anything is possible.</p> An individual developer has limited control of the development of a complex project and the resulting product. (We are not talking about fixed-bid turnkey product design where a larger company designs the entire product, but rather hourly development services). Typically, there is extensive involvement from other developers. The contract developer does not control how the product is manufactured, marketed, sold, supported, or even how other parts of the system are developed. In complex systems there will be bugs, and often there are time to market pressures that prevent extensive testing and verification. These are all factors that are out of the developer’s control and are choices made by the product owner. Again, this further illustrates that those who are ultimately in control of a product (the owner) should be responsible for any liability the product might bring.</p> This brings us to a common concern of the company – they want to be sure the product is developed to professional standards. A contract phrase might be: “Contractor hereby warrants that its work shall be of professional quality and performed consistent with generally accepted industry standards”</em> Who defines these standards? How will they be defined in court? The reality is that few developers have the time and resources to really do the job they want to do. Schedules are always pressing, forcing compromises. Ultimately the product owner makes the decisions concerning these trade-offs – not the developer. Time to market is critical. Therefore it is difficult for a developer to warrant any work will be at any particular industry standard – as the product owner is ultimately in control. The universal laws of reality apply: “Good-Fast-Cheap” – pick two – all three do not exist. In any product development effort, there are compromises.</p> There is a very practical way to handle the concern that a company’s product is being developed to their standards, and that is to encourage collaboration and openness between all developers on the project. Make sure all source code and design files are shared in Git repositories that are easily accessible to all developers – both employees and contractors. Encourage all developers to work on any part of the system needed to get their job done. Google does</a> it</a> – maybe you should, too. Discourage engineering “turf” where developers feel they “own” something in the sense that no-one else should mess with it. This is essentially the process used in Open Source Software, and successful companies have adopted these practices – the results speak for themselves. If this works in a large company, is there any excuse for not doing this in a small company? Open source projects don’t have contracts that specify all contributors have to do a good job, yet open source projects contain some of the highest quality code available. Peer review and doing your work in public (even if only within a company) is a powerful motivator. Because the development is in an open environment, it is quickly evident who is going a good job and who is not. The same dynamic can happen in a company. If there is open collaboration, frequent integration, and early testing, you will soon get a good feel if developers are efficient and are generating work that is of “professional quality.” Share weekly reports with the entire team of what each developer did and how much time they spent on the project. This will produce a level of accountability among peers that will far exceed what any manager can force. If someone is spending excessive amounts of time and not getting anything done, other developers will start to question it. And you will know of problems early in the project at a time when you can do something about it, rather than after a product has been sold in quantity and some horrible defect arises that results in lawsuits. Which is the better option? To summarize:</p> If a company wants to ensure quality work is being done by their developers, then they must foster a collaborative, transparent development environment where:</strong></p> All development is done in the “open” in Git repositories that the entire team has access to.</strong></li> Any developer can contribute to any part of the system required to do their job.</strong></li> Integration is done often.</strong></li> Product testing, by the owner, is started early in the development cycle.</strong></li> </ol> No amount of contract legalese can come close to being as effective as the above.</p> Whether you intend it or not, your legal contracts are both a reflection and driver of your company culture, and its resulting success. They impact the people you will attract – both employees and contractors. Do your contracts reflect a fair and generous mindset where you are willing to take responsibility for the things you should take responsibility for? Do they turn away or create reserve in people you need the most? This is a serious issue because the best people often come through referrals. Will a developer recommend their talented and busy friends if they are required to sign uncomfortable contracts that must be negotiated to even approach an acceptable level? Any friction in this process is slowing you down. Yes, a one-sided legal contract that takes everything you can get may save you some liability in the unlikely event of a product lawsuit. But it also has the effect of introducing friction in relationships that are key to your product being completed in the first place. Which is more important? Which should have priority? In summary, a contract should:</p> clearly represent the concerns of both parties</li> document what is expected</li> not be one-sided</li> be fair</li> avoid complex legal jargon as much as possible</li> aid, not hinder the business relationship</li> include provisions for how the relationship will be terminated, if needed</li> </ul> I’ve been working as a consultant for 20 years. My most successful and longest relationships have been with companies where I have never executed a single contract. Is this a coincidence? Is it cause or effect? I don’t claim to know, but my experiences resonate with the words of Seth Godin:</p> A handshake deal, on the other hand, is about the future. Either side can claim loopholes or wriggle out of a commitment, but the consequence is clear—if you disappoint us, we won’t be back for more. The participant in a handshake deal is investing in the future, doing more now in exchange for the benefits that trust and delight and consistency bring going forward.</em></p> from: https://seths.blog/2014/10/handshakes-and-contracts-the-future-and-the-past/</a></p> </blockquote> Somehow an essay about contracts has morphed into a discourse on company culture, development practices, and ultimately success. This is the power of writing – it helps you connect concepts that you did not previously realize were connected. In the end, everything is connected. Your contracts, your company culture, the type of people you attract, etc. It all matters.</p> References:</p> https://www.technologyreview.com/s/400489/invention-is-a-flower-innovation-is-a-weed/</a></li> http://www.unixwiz.net/techtips/be-consultant.html</a></li> http://sethgodin.typepad.com/seths_blog/2014/10/handshakes-and-contracts-the-future-and-the-past.html</a></li> http://badsoftware.com/contract1.htm</a></li> https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext</a></li> https://medium.com/@Jakeherringbone/you-too-can-love-the-monorepo-d95d1d6fcebe</a></li> https://www.clearviewlegal.co.uk/ohlawdy/contracts-are-communication</a></li> </ul> Technology 2020-11-27T00:00:00+00:00 The fire of information technology – will you use it:</p> to create</strong> or consume?</li> to comfort</strong> or torment?</li> for education</strong> or entertainment?</li> as a tool</strong> or weapon?</li> to temper</strong> or incite?</li> for community</strong> or faction?</li> for harmony</strong> or rebellion?</li> to encourage</strong> or discourage?</li> to spread light</strong> or darkness?</li> to bless</strong> or curse?</li> to compliment</strong> or ridicule?</li> to save time</strong> or waste time?</li> to be generous</strong> or selfish?</li> to heal</strong> or hurt?</li> to share ideas</strong> or focus on self?</li> to be open and direct</strong> or talk behind backs?</li> to focus on things we can do</strong> or complain about things we have no control over?</li> to spread truth</strong> or falsehoods?</li> to express thankfulness</strong> or discontent?</li> to preserve our autonomy</strong> or be manipulated and exploited by others?</li> for productive</strong> pursuits or indulgence in vanity?</li> to engage in real life</strong> or escape into some virtual universe?</li> </ul> Do we have integrity such that our digital communications match what we would do in person? We live on a great sea of information, and the default path is to be tossed about on the waves of unverifiable communication that wash over our society, and in the end have little to show but discontent and unhappiness. Technology is an excellent tool, but a poor friend, a useful servant, but an oppressive master. Technology is no substitute for the things that bring true joy and satisfaction in life – family, friends, giving, companionship, creativity, craftsmanship, mastery, accomplishment, nature, and faith. Like fire, information technology is only useful when used intentionally and constrained to some useful purpose. Otherwise, it will rob us of our time, relationships, humanity, skills, potential, and ultimately our soul.</p> Adventures with a ATSAMA5D27 board 2020-01-29T00:00:00+00:00 Recently, we brought up a new PCB (printed circuit board) a customer had designed based on the Microchip ATSAMA5D27C-D1G</a> processor. We encountered several interesting challenges.</p> </figure> The first issue was that the design only had serial ports (and possibly a USB Host port that could be used in device mode) for bootstrapping the system. Microchip provides a tool named SAM-BA to do this. There are two implementations of SAM-BA. The older implementation does not support this CPU. The newer implementation</a> does support this CPU, but the documentation and examples are sparse (at least when we were using it). The standard way to use SAM-BA is to directly program the bootstrap and bootloader into SPI flash. However, this design only has eMMC flash. So we came up with a way to bootstrap at91bootstrap and u-boot over the serial port, and then load a Linux kernel with an embedded initramfs from a USB flash disk. This initramfs contains an updater</a> that is used to program the eMMC flash. After the updater runs the first time, the system will then boot from eMMC. We developed and verified this process on a development board, and it is documented in a Github issue</a> (The SAM-BA team has since disabled Github issues on this repository)</em>, and as part of the Yoe Distro sama5d27 support</a>. Microchip has also since provided us with a procedure to program eMMC using SAM-BA, but the procedure is complex and we never had a need for it once we figured out the above method.</p> After receiving 10 prototypes of the new board, we learned pretty quickly that VCCBU must have voltage for the CPU to start. After that, the first board loaded u-boot and everything looked good, but things went south from there. U-boot would only load perhaps one out of twenty times. Basic checks of power supply voltage levels all looked good. The custom board used a 12MHz crystal for the main clock, where Microchip’s SOM1 module uses a 24MHz crystal. We decided to try a 24MHz crystal, and after that change, u-boot would reliably load over SAM-BA. We’re not sure why (12MHz is slower than 24MHz, so things should run better), but initializing a clock tree is complex and there is likely something that was not configured correctly in software. Lesson #1: Unless you want to spend a lot of time writing chip initialization software and debugging strange problems, follow a reference design as closely as possible.</strong> Working at the level of chip initialization often requires intimate knowledge of the MPU and likely the support of those who designed the chip. This is much more effort than most teams have time for and is best left to the MPU vendor.</p> Now that u-boot was loading reliably, we could turn our attention to the kernel. At this point, a couple boards would start the kernel, but most failed with no kernel messages after the “Starting kernel …” message on the serial console. The boards that would start the kernel would crash partway through the boot process. We started looking at power supplies and observed a 100mV sawtooth on the core power supply.</p> </figure> The SAMA5D27 datasheet specifies a maximum of 15mV of ripple for the core supply.</p> </figure> The ADP5023 power supply used for the core voltage had a mode pin that can be used to force it into PWM-only mode, instead of PWS/PWM mode. After forcing the supply into PWM-only mode, some boards would start Linux. However, there were still stability problems later in the boot cycle.</p> After trying many other things, we finally raised the voltage of the core supply slightly by changing one of the feedback resistors. Presto! Lesson #2: If a CPU has stability problems and everything else looks OK, trying raising the core voltage.</strong> The system was now stable. After communicating with Microchip, we learned that the CPU actually required a higher voltage than the documentation specified. Microchip provides a power supply that is designed to be used with the SAMA5D27 – if this supply would have been used, then we would have gotten the right voltage.</p> This bringup experience was harder and longer than most – especially for a board of this complexity. But, it illustrates an important point – full custom MPU design is hard – even with a part like the SAMA5D27-D1G which integrates SDRAM in the CPU package. It can require a significant amount of time. This design was done by a competent designer and was extensively reviewed, but it still contained serious problems that took weeks to resolve. In the end, we were still able to make the first PCB revision work with minor modifications. However, it is very possible that the first revision of a PCB like this might need to be scrapped and chalked up to a learning experience leading to months of project delays.</p> Lesson #3: Use a SOM</a> (System on Module) unless you have a good reason to do a full custom design, and you have the time and resources to pull it off.</strong> You are likely not Google or Apple</a>. Yes, you will pay slightly more for a SOM than the raw MPU system components, but the SOM cost may be offset by a lower cost baseboard. A full custom MPU design may require 8 layers and very small routing features where a baseboard for a SOM may only require 4 layers and use larger features, resulting in a less costly PCB. Additionally, a baseboard for a SOM may have a lot less parts on it, and requires less effort to set up for assembly – simpler ones can even be hand built to bypass weeks of assembly lead-time.</p> There are dozens of SOM vendors offering products today that fit about any price/quality/certification price point. Some SOMs contain pre-certified WiFi and Bluetooth radios. Others meet various mil-spec requirements. This situation today is much different than 10 years ago when there were only a few expensive SOM models available. SOM modules allow us to bring advanced technology to a problem, and implement a solution with a small team in a reasonable amount of time. Implementing a SOM based MPU design still requires a fair amount of knowledge, but the unknowns are drastically reduced, and chances of success are far better. Much of the design complexity is already handled, freeing you up to work on the value a product provides. Any time we can simplify our development effort, our chances of success are much greater.</p> Working with Microchip on this project was a good experience. Without their support, we would not likely have resolved the issues we faced in a timely fashion. This brings us to Lesson #4: When developing complex system, you are likely going to need some level of support.</strong> This may come from IC vendors, open source communities, consultants, etc. In this case, the MPU documentation was not up to date, and the chip designers were able to point out the problem. We could be critical of Microchip for this oversight, but with a 3000+ pages datasheet, this is a lot of detail, and there will be mistakes and oversights. The fact that most of the datasheet is available and accurate is a commendable accomplishment. Getting this level of documentation from some vendors is like pulling teeth. Before any of us cast the first stone, we must ask the question – is everything we do completely and accurately documented? Overall, the SAMA5D27 documentation is pretty good.</p> The support provided on the SAM-BA Github issues was also essential for us to figure out a way to program the device. The procedure we found for programming this system using only a serial port is relatively simple and easy to implement. Though it may violate some SDRAM refresh timing specs, in practice it works very well – and that is what matters – getting things done. It is unfortunate that Microchip has since closed the Github issues for the SAM-BA project, indicating they are not really interested in building a community around the project, but rather Github is only a way to distribute the tool that is developed in-house. As systems get more complex, Open Source communities are proving to be the only viable mechanism to develop and support all the various needs users might have. Large companies readily acknowledge this as many core technologies are now Open Source, and much effort is put into building communities around them. But, lack of Open Source vision aside, Microchip is a reliable partner – especially for smaller companies wanting to use advanced technology and needing support.</p> Drone for Continuous Integration 2019-11-26T00:00:00+00:00 Recently, we have been evaluating Continuous Integration</a> (CI) systems for a variety of projects (both OSS and customer). There are many OSS options to chose from</a>. Because we already use Docker containers for Yocto/OE builds</a>, Concourse</a> and Drone</a> made the short list. Both communities seem responsive</a> and helpful</a>.</p> https://github.com/YoeDistro/yoe-distro</a></figcaption></figure> Building OE in a container</a> is important, so naturally Drone and Concourse seem like good options as they are focused on containers. Containers make sense for most CI tasks as they allow you to easily control the environment without getting bogged down in setting up the host. Because Concourse does not seem to easily support bind mounting volumes</a> on the host system, we ended up implementing a prototype in Drone</a>. This was improved and merged</a> into the yoedistro master. Ironically, we are not currently building in a container, as the yoedistro CI machine is a 64-core ARM system and we had some problems with the container support. Hopefully we can get containers working again at some point. Since the initial evaluation, Drone has added an exec runner</a>, which allows us to execute a pipeline directly on a machine without a container.</p> Drone is an excellent fit for most of our projects. There are more capable systems (like Concourse, Jenkins, etc), but the simplicity of Drone is a big advantage in that it will more likely be used by developers. A tool is only useful if it gets used. With Drone, a developer can simply cd to a Git workspace that has a .drone.yml</code> file, and then run drone exec</code> and it will run the entire pipeline locally. Because drone typically does everything in containers, this works fairly well – it does not really matter what the host system is. This is powerful because developers can test the entire pipeline on their local machine before enabling it in the drone server/agents for automated testing/deployment. If there are problems, they can be debugged locally rather than the tedious process of commit, run the pipeline on the server, view the logs, repeat. Solutions that are simple to use make a big difference in productivity.</p> Below is a quote from a Drone user:</p> Today we have over 60 engineers interacting with Drone daily, and they really like it, due to its simplicity. We’ve always had a small DevOps team, so we tend to heavily favor solutions that don’t require a lot of maintenance overhead, and Drone fit the bill. Since most developers have at least a basic understanding of Docker, they hardly need any help maintaining their pipelines, and there have been a couple of instances of developers creating their own plugins! This way our team can focus on exciting upgrades to our infrastructure, even as teams regularly update their deployment pipelines and new projects are created every month.</p> https://blog.drone.io/why-quinto-andar-chose-drone/</a></p> </blockquote> Drone makes some interesting trade-offs in that it trades some flexibility for simplicity and developer ease of use. Initially, Concourse was my first choice as it is a more powerful tool, but because it does not support bind mounting host volumes or exec runners, we ended up using Drone. This tool makes sense for many teams because it is more likely to be used by the average developer without having a dedicated CI/Operations team.</p> This is a trend we are seeing at successful companies where developers are more responsible for everything related to their project, including testing, continuous integration, deployment, and operations. The quote below gives us some insight into how things are done at Amazon:</p> There is another lesson here: Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.</p> — Werner Vogels (CTO @ Amazon)</strong>, https://queue.acm.org/detail.cfm?id=1142065</a></p> </blockquote> Drone fits into this model well because it enables developers to become engaged in CI and deployment.</p> One potential problem with drone is the lack of support</a> for some Git systems like Assembla. One workaround would be to store pipelines in Github/Gitlab/Gitea, etc, and then fetch code from other repos. But it would quickly make sense to add auth/webhook support for whatever Git system is used if it does not already exist.</p> Reflections on KiCad and EDA Tools 2019-11-12T00:00:00+00:00 A recent interview</a> with a KiCad developer prompted some reflection on KiCad and EDA (electronic design automation) tools in general. Below are samples of several PCB (printed circuit board) designs, created with KiCAD, and implemented as part of the SimpleIoT project</a> in the last couple months.</p> </figure> </figure> The experience has been excellent. Above all, the tool is very fast, efficient to use, and stable. Schematic and PCB integration works well enough, and routing and copper pours are easy. Switching between inches and millimeters can be done on the fly. The KiCad library has many parts in it, and other organizations, such as DigiKey</a>, Seeed</a>, SnapEDA</a>, and Ultra Librarian</a> are also providing libraries. If a KiCad symbol/footprint for a part is not already available, it is relatively easy to create new symbols and footprints as needed. There is a good KiCad support forum</a>. KiCad is a pleasure to use and production-ready for standard PCB designs.</p> Choosing an EDA tool is an investment. There are several aspects to consider:</p> purchase/licensing cost of the tool</li> time invested in learning the tool</li> time spent implementing designs</li> time spent maintaining designs</li> </ol> #2, #3, and #4 swamp #1 for many projects – at least when low end EDA tools are used. Thus we want to be confident that the tool we choose will be around for a while and will be well maintained. With all the churn in the EDA industry in the last 10 years, it is hard to have a lot of confidence in the long term prospects of many tools. For KiCad, however, the development statistics indicate that momentum around the project is increasing and that this tool will likely be around for awhile.</p> https://github.com/KiCad/kicad-source-mirror/graphs/contributors</a></figcaption></figure> With 183 contributors, the KiCad project appears to be healthy – it is doing something right.</p> Some may fear that open source projects (like KiCad) are taking away the opportunity for companies to make money and are not sustainable. Has the advent of Linux decreased the amount of money being spent on operating systems? On the contrary, Linux has greatly increased the operating system ecosystem – much more so than if proprietary Unix were the only option. Redhat, Ubuntu, and many other companies seem to be doing well. Linux is used in everything from watches to supercomputers. Unix would never have enabled this breadth of products – it is simply too large a task for any one company or team. Collaboration is the only way to accomplish a project of this magnitude. When the commodity layers are taken care of, then people are free to work on higher level problems. If a commercial EDA tool does about what KiCad does, that product is likely not going to fare well long term. But there is plenty of space to add value higher up the stack. Examples might include tools that work with KiCad such as auto routers, advanced placement tools, library management tools, RF design, SI/EMC simulation, high voltage design rules, etc. There is endless opportunity. The “pie” of opportunity is not fixed – it can grow. Contribute code to KiCad that will enable proprietary tools to run in tandem. If companies build on top of KiCad, then they will not have to duplicate the task of building a basic schematic and PCB layout editor and can instead focus on other features, and as a result, there will overall be more value for society. Companies will already have a large market of people who are more likely to spend incremental amounts of money to add functionality as needed. The alternative is for every company to try to provide the entire EDA stack, which can be a tough sell, as it is expensive and customers will need to change everything they are currently doing. There is no reason EDA software cannot become modular like the software development ecosystem.</p> KiCad may displace a few commercial EDA products, but the value and opportunity KiCad will help create will far outweigh this loss. More companies will build more stuff, with less friction and effort. There will be more innovation. More problems will be solved (and likely a few created). There will be more collaboration. Even now, it is practical for developers, who only design PCBs occasionally, to use production grade tools.</p> EDA tools should not be evaluated merely on technical features and price, but also on the hassle factor, the capability to facilitate collaboration, and the ease of long term product maintenance. Who will maintain your product in two years when a part goes obsolete, the design needs tweaked, and the designer and the obscure tool used on the original design is long gone? If you contract out a PCB design, why not require that the design be implemented using a tool that anyone can use in the future? Part of the value of a product is its maintainability. Companies who can quickly adapt as things change will have an edge. The costs of not being able to quickly respond to some product need, or of having to learn new tools and re-implement a design are significant. Over the life of some products, maintenance costs may be much larger than the initial development costs.</p> In years past, I’ve used a number of different PCB design tools (Mentor Board Station, OrCad, P-Cad, gEDA, Eagle, etc). Some have been good experiences, others not so much. While there are no doubt EDA tools that have more features and are better than KiCad in some ways, KiCad is the low friction solution for your standard PCB design. It can be used easily by any one at any time on any computer. This flexibility is a big part of KiCad’s value. EDA tools are the “build system” for your hardware. They work best if they are open and easy to use by anyone on the team. There are many “free” PCB design tools available, but few of these are open. Some are cloud based, are very convenient to use, and technically a marvel that we can run a PCB design tool in a browser.. But the design is still locked up in a proprietary format. It is not easy to customize or extend these tools. And the sheer number of these commercial options available today suggests that long term they probably won’t all be around. “Open” is more important than “Free” in my opinion – especially for the long term.</p> KiCad is certainly a noteworthy project in the technology landscape, and deserves consideration. The availability of such a tool will change the world. The open source hardware movement will grow. Companies who learn to leverage this movement will benefit in the same way that most companies now benefit from the use of open source software. More wealth will be created for society. The traditional barriers between hardware engineers, software developers, and PCB designers will be reduced. Just as hardware engineers have had access to about any software development tool they want to use, now software developers have access to tools to develop and tweak hardware. The small amount of time required to learn KiCad is a worthwhile investment.</p> The Humble Programmer 2019-10-31T00:00:00+00:00 In 1972, Edsger W. Dijkstra</a> published a paper titled The Humble Programmer</em></a>. Dijkstra was trained in math and physics and was a university professor for much of his life. This paper is an interesting reflection on the history of computers and contains thoughts for the future. A few quotes are included below:</p> Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with. In other words: both goals point to the same change.</em></li> To put it in another way: as long as machines were the largest item on the budget, the programming profession could get away with its clumsy techniques, but that umbrella will fold rapidly.</em></li> The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.</strong></em></li> Another lesson we should have learned from the recent past is that the development of “richer” or “more powerful” programming languages was a mistake in the sense that these baroque monstrosities, these conglomerations of idiosyncrasies, are really unmanageable, both mechanically and mentally. I see a great future for very systematic and very modest programming languages.</em></li> The best way to learn to live with our limitations is to know them. By the time that we are sufficiently modest to try factored solutions only, because the other efforts escape our intellectual grip, we shall do our utmost best to avoid all those interfaces impairing our ability to factor the system in a helpful way.</em></li> We shall do a much better programming job, provided that we approach the task with a full appreciation of its tremendous difficulty, provided that we stick to modest and elegant programming languages, provided that we respect the intrinsic limitations of the human mind and approach the task as Very Humble Programmers.</em></li> </ul> Dijkstra was 42 years old when we wrote The Humble Programmer</em>. At a similar age of 47 years, I too have come to appreciate the value of simple programming languages and a careful approach to writing software. Two modern languages that embody some of these principles are Go</a> and Elm</a>. While on the surface these languages may seem very different (one is procedural and one is functional), the values of the languages are similar. Both are designed with a great respect for the finite capability of the human brain. Both strive for simplicity first, and features are only added to the language after careful consideration. Both have excellent tooling that helps ensure consistency and correctness. Both are not just a syntax, but rather a culture and mindset. Experience on recent projects indicates that Go and Elm are effective languages, especially when I compare to previous experiences on C++/Javascript projects. There are rarely run-time exceptions. Compiler errors are relatively easy to understand and fix. Tooling is simple and almost invisible. Program maintenance and refactoring is enjoyable. The combined cognitive load of the language and the tooling is small freeing up resources for solving the problem at hand.</p> Today, there is a huge amount of churn in the software development world. Especially in front-end development, new frameworks are being released at an astounding rate. Many have their place, and perhaps each addresses some need. But it seems there is something fundamentally wrong with so many solutions for the same problems. Thus, it is encouraging to see new approaches where the authors have carefully studied the past, tried to learn from everything that has been done, and have created solutions where simplicity is valued. This is key to developer effectiveness, and really to life in general.</p> Why are Go applications so reliable? 2019-10-10T00:00:00+00:00 Go does a lot of things well (good performance, easy to learn, very productive, extensive stdlib, excellent tooling, etc), but after programming with Go for three years (both embedded Linux and cloud applications), stability is the characteristic that really stands out.</p> </figure> In the past, I’ve observed the development of several fairly large C++ applications. Stability problems were more common than they should have been. Memory leaks, null pointers, crashes, etc. On one project (over 500,000 SLOC</a>) we built elaborate mechanisms to capture core dumps when things crashed, at times needed to run Gdb on the target system, and spent a fair amount of time in Valgrind tracking down memory leaks. The worst problems were the rare, intermittent ones which only happened in the field and were very difficult to instrument. It is possible to write reliable C/C++ code (one example is the Linux kernel), but it takes focused effort (code review, testing, coding standards, etc). My experience with Javascript has been similar. What if there was a programming language where basic program stability did not have to be a primary concern?</p> I’ve now participated in four decent-sized Go projects and stability problems are a thing of the past. I only recall one memory leak. Crashes are rare. Several attributes of Go contribute to this:</p> The pattern for returning multiple variables (including error) and aligning the happy path to the left</a> encourages error checking before doing anything with the data a function returns. Therefore you don’t end up trying to do things with nil variables. It is a little verbose, but it works and is a small price to pay for the problems it solves.</li> Garbage collection takes care of memory leaks.</li> The language encourages simplicity, and simple things are generally more reliable.</li> The type system catches many errors during compile time.</li> </ol> I’m sure there are others, but these are the ones that stand out to me. Go strikes a nice balance between safety and simplicity. There are safer languages (Rust, Haskell, etc), but they require more effort to learn, and probably require more coding effort (I don’t have a lot of experience, so I could be wrong about this). Every tool has its use and place. If I was developing an application on a space craft going to Mars, I’d probably use Rust. If I am writing a quick script to process some test data, I typically use Python. When writing MCU code, I still use C/C++, but someday may use Rust. But when I’m writing a reasonably complex cloud or edge (embedded Linux) application, I use Go. In Go, you can get things done quickly, and be reasonably confident that the application you deploy will not crash or leak memory.</p> Go’s type system and simplicity require a little more typing than Python, but likely less effort than more complex languages like Rust or Haskell. Go is a pragmatic language that gets you 98% there without a lot of overhead. In business, you generally don’t want to add a lot of process overhead on the entire company for the 2% of people who may be causing problems, as it imposes drag on the other 98% trying to get things done. Rather, it makes sense to address the 2% at a personal level. Likewise, with Go, the 2% of the time you have to fix a nil bug, or a unhandled case the type system did not catch, is acceptable as 98% of the time you are working in a language that is very productive with very little overhead. Go is ideal for many applications.</p> Auto-formatting/linting Go code 2019-04-22T00:00:00+00:00 Some things in life you just have to experience to truly appreciate the value. One of these is auto-formatting/linting source code. When I started programming in Go in Vim, I naturally looked for editor support, and found the excellent vim-go</a> project. Through this, I learned about gofmt</a>and then goimports</a>. These tools can be configured in your editor to automatically format your code when you save. goimports goes a step beyond and adds missing imports and removes unused ones.</p> Auto-formatting is quickly</a> becoming</a> the</a> norm</a>. The Javascript world also has an excellent formatter available named Prettier</a>. There are formatters for many other languages as well including C/C++</a>, shell</a>, Elm</a>, Rust</a>, etc. The neoformat</a> and ALE</a> plugins add auto-formatting functionality to Vim/Neovim. An example of how to enable these plugins in Neovim is included in my dotfiles</a>.</p> Below are instructions on how to set up Visual Studio Code</a> to auto-format code. Even though I use Neovim</a> for most of my code editing, I still try out VS code and IntelliJ IDEA</a> from time to time to learn what’s new in cutting-edge editors. I also recommend VS Code for most new Go developers – especially those coming from embedded systems development where an IDE is traditionally used. Give it a try – it will only take you a few minutes.</p> The instructions below should work with most systems (Windows, Linux, Mac). First, make sure Go and VS code are installed. Then install the Go VS code extension.</p> </figure> Next, type Ctrl-Shift-P to bring up the command palette and run Go: Install/Update Tools.</p> </figure> This will display a list of tools – select them all.</p> </figure> Next, configure VS Code to format on save. Open the settings by selecting File->Preferences->Settings from the menu. Select the “Format on Save” option and set the timeout to something larger than the default, like 5000ms.</p> </figure> Last, select goimports as the code formatting tool – this will automatically manage the imports in your file, as well as formatting the code.</p> </figure> Now, open a Go source file. Mess up the formatting of some code and then save – the code should be automatically formatted. Similarly, if you remove one of the import statements that is being used, and save, the imports will be corrected. Unused imports will be removed.</p> These tools allow you to focus on writing software and let the tools worry about formatting and imports. Prettier code with less effort – why not?</p> Microcontroller (MCU) or Microprocessor (MPU)? 2019-04-09T00:00:00+00:00 As technology advances, there are two basic processing platforms for implementing embedded systems</a>. The first is the Microcontroller Unit</a> (MCU). These devices have varying amounts of integrated Flash (<= 2MB) and RAM (<= 1MB), and are designed to run bare-metal code or a real-time operating system</a> (RTOS), like FreeRTOS. The second is the Linux-capable Microprocessor Unit</a> (MPU). An example of an MCU based system is most Arduinos, and an example of an MPU based system is the Raspberry PI. An MPU typically does not have embedded Flash and RAM – at least on the same die. The fundamental difference between MCU/RTOS and MPU/Linux systems is the memory architecture and the amount of memory in the system.</p> </figure> There are other differences as well, summarized in the table below.</p> Feature</strong></td> MCU</strong></td> MPU</strong></td></tr> # power supplies</td> 3.3V</td> VCORE, VIO, VDDR, etc.</td></tr> USB Host</td> limited</td> Full support many drivers</td></tr> Networking</td> limited</td> Options for GB and multiple interfaces </td></tr> Cost</td> lower $1-$20</td> higher, starts at $10 (cpu + power/mem) </td></tr> Programming Languages</td> C/C++ Micropython Rust</td> C/C++, Go, Python, Nodejs, Java, Erlang, Rust, about anything</td></tr> Realtime</td> hard</td> soft</td></tr> Data processing</td> limited</td> excellent</td></tr> Processing power</td> less</td> more</td></tr> Display size</td> small</td> any size </td></tr> Startup time</td> fast</td> slow</td></tr> Expandable over time</td> less so</td> more so</td></tr> On-chip Peripherals</td> more </td> less </td></tr> PCB real-estate</td> less</td> more (external flash/RAM)</td></tr></tbody></table> Performance is related to memory architecture. In most MCUs, the memory architecture is fairly simple. Code is executed directly from flash, and on-chip SRAM is accessed directly. Programs do not need to be loaded from flash into RAM before running them. This architecture is very simple, leading to predictable real-time response and code execution. It takes a consistent amount of time to load code from flash into the processor for execution, so the timing for every instruction is fairly predictable. However, with this architecture, clock rate is typically limited to around 190MHz (example STM32F4). Some of the newer STM32 parts, such as the STM32F7 and STM32H7, add a L1 cache, which allows the CPU to run at 216MHz and 400MHz respectively.</p> MPUs have a more complex memory architecture in that they page code from Flash into SDRAM (both external to the MPU), and then from SDRAM into two or more levels of cache memory located on the MPU. The local cache memory is very fast, allowing these processors to run at high clock rates (1GHz or more in some cases). A memory management unit</a> (MMU) implements a virtual memory system in which physical pages from RAM are mapped into a virtual address space. This is very efficient as physical pages are only mapped in as needed and can be discarded if RAM is needed elsewhere. But this memory management introduces delays that are relatively long for some hard real-time systems. The first access to a block of code that has not been run recently takes time, as code needs to be paged from a file in flash into SDRAM and then loaded into the respective caches. But subsequent accesses from cache are blazing fast. An MMU also provides protection against one user space process corrupting memory of another process, or corrupting kernel memory. For complex systems (server/desktop) running multiple processes, this protection increases the system reliability. For many Embedded systems, there is only one main application, and protecting processes from each other is less of a concern. However, this memory protection is still useful in that a bug in the application does not crash the system, and recovery mechanisms can be built in to restart the app, or debug it during development.</p> In an MCU, program flow is controlled through simple loops and state machines (no operating system) or a real-time operating system</a> (RTOS). In an MPU system, a fairly complex operating system like Linux is typically required to manage the multiple levels of memory and storage, schedule the multiple processes that are running, and provide drivers for the complex hardware systems found in an MPU (USB, large displays, networking, etc). There is a great gulf between these two systems. Even though a RTOS may run on an MPU, there are much longer delays in execution due to the memory architecture. You still have to load code from flash to SDRAM and then through several layers of cache memory before the CPU can execute it. Once it is in cache, the code executes much faster and can do more work overall, but the occasional delays are still there. You might be able to play tricks by locking lines of code in cache or by using the small amount of on chip SRAM that may be present, but by the time you do this, you are now back to the smaller memory sizes on an MCU, and may as well just use an MCU.</p> After understanding the differences in memory architectures, we can understand that MCUs and MPUs are optimized for different things. An MCU is optimized for simplicity, cost, and predictable (real-time) response times. An MPU is optimized for getting the maximum amount of work done over time. Cost and real-time response is a secondary concern. There is not really a lot of middle ground between an MCU and MPU, evidenced by the large gap in memory sizes. You can’t fit standard Linux in an MCU, as there is only so much NOR Flash and SRAM that will fit on an MCU die. Once you move to more dense memory technologies such as NAND Flash and SDRAM, these must be implemented as separate dies and typically separate integrated circuit packages. These technologies are so efficient at storing large amounts of data, that you see a huge jump in memory capacity once you switch to the off-chip memories.</p> MCUs and MPUs also have different origins. Today’s 32-bit MCUs have descended from simpler 8-bit MCUs and are scaled-up embedded controller technologies. MPUs have descended from desktop and server computer systems and are scaled-down technologies. The differing priorities of these two paradigms reflect their different uses.</p> Keeping these differences in mind, the most fundamental question to ask in selecting an MCU or an MPU is whether hard real-time performance and reliability is most critical (control-centric application</strong>), or whether data processing performance and connectivity is most important (data-centric application</strong>). If both of these are important in your application, you should consider having both an MCU and MPU in the system, saving yourself untold pain.</p> Reliability is an interesting topic. I have seen MPU/Linux systems operate very reliably, and have also experienced issues that were very difficult to solve. MPU systems are many times more complex and have more unknowns, which translates into more risk. There are many more physical components and solder connections required to implement the system, all of which can fail over time with environmental stress. There are many millions of lines of code in an MPU system that you did not write, but you are still responsible for all of it – it all has to work for the system to function. Although an MPU system can be made fairly reliable, simple statistics tell us that MCUs will generally be more reliable than an MPU because there is less hardware that can fail and less lines of code running that may contain bugs.</p> One example of a problem we experienced when implementing control in an MPU system is a product where we were collecting data, and the data was being transferred back to a cloud server over a cellular modem. Unfortunately, the cellular modem in the system was not 100% reliable, and the only way to recover in some instances was to reboot the system when we detected a network failure, which cycled power to the modem (this modem module itself ran Linux). At one point, we added some basic control functionality to the system to control plant blowers based on a schedule. The control worked fine until the system had to restart due to modem connectivity issues, then the blower control was inactive during the reboot cycle (perhaps 30s or so). This was not a fatal problem in this application but also not ideal. The obvious solution is to get a more reliable modem or develop a better recovery method, but with units in the field and other development priorities, it is not always so simple. This is a classic example of the tension between complex data/connectivity systems and reliable control.</p> In an MCU system, the hardware system is relatively simple, and you are using a relatively small RTOS, or none at all. You write a greater percentage of the code in the system yourself, so if you have a simple task to do, and write reliable code, there is a potential to have a more reliable system than a comparable MPU system. However, if you are trying to do complex data processing and connectivity tasks on an MCU (such as writing your own database, network, or USB stack), chances are there will be bugs in your code and you will have a less reliable system than if you would have chosen an MPU using proven technologies. If your application is large and complex, then writing it in a safer language like Go or Rust on an MPU may provide a more reliable and maintainable product than trying to implement the same functionality in C++. Again, if you need both reliable real-time control and advanced data processing, then put both an MPU and MCU in the system.</p> Another advantage of an MPU system is they tend to be more general purpose and more functionality can be added over time. There is little danger of running out of code space, which often happens in MCU environments. With interfaces like USB, additional peripherals can be added in the future as requirements change, and with Linux, drivers for a vast array of hardware are available.</p> In some cases these lines between MCUs and MPUs are blurred by technologies like Linux RT extensions, or uCLinux, but you still must keep the above principles in mind when selecting the building blocks for your product. MCUs and MPUs are very different devices designed to handle different tasks. Before choosing, you should understand what you are trying to do, and what is important for your product.</p> Reference:</p> Atmel MCU vs MPU paper</a></li> ST page referencing both MCU and MPU</a></li> </ul> Accepting Constraints in Build Systems 2019-04-01T00:00:00+00:00 As Embedded Systems become more complex, the complexity of the process to build the software for these systems also increases. As humans, our ability to deal with complexity is limited, so we develop tools and processes</strong> to manage the complexity. In the end, these tools and processes are about constraints and patterns</strong>. A well-designed tool or process encourages you to do things in a way that is consistent and maintainable</strong>, which leads to reliable and predictable</strong> results.</p> </figure> As an example, consider using CMake vs standard Makefiles for a C++ application build system. This is a trade-off I often encounter, as my customers are using Linux to build products, and they need to develop a relatively complex software application to implement the product functionality. There are usually a handful of library dependencies as well (such as Qt, OpenCV, etc) – after all, the reason you use Linux is it provides a large amount of advanced functionality that you cannot afford to write yourself.</p> For a simple application, CMake is somewhat constraining and annoying. Its syntax is a little clumsy, and it is definitely less flexible than a Makefile. Thus some developers conclude that a Makefile is a better choice for Linux application builds. As the application grows, the Makefile also grows, and the number of dependencies increases. Then comes the time to cross compile the application so it will run on the target system. With a little work, we can convince the Makefile to accomplish this. Tool-chain file names and paths are often hard-coded in the Makefile. As the variations of systems the application needs to build on increase, all these variations are coded in the Makefile.</p> Five years down the road, something major changes. The application developers need some new features in a new version of OpenCV, or the System on Module (SOM) the target system is using just went obsolete. The developer who wrote the original Makefiles is gone. The Linux host distributions have changed. At this point the application build system collapses. So much has changed, and so much is hard coded in the Makefiles that getting them working again is a painful and tedious process. With some effort, they are finally building on one developer’s machine. But on the next developer’s machine, they fail with a different error.</p> The same dynamic happens when developers hack together tools to build an operating system image for an embedded device. This typically involves things like building the application or Linux kernel manually outside the build system, then writing a script to unpack an image file, insert the new binary, then pack the image file back up. The application build process is heavily dependent on the configuration of the developer’s workstation.</p> There is a better way. Use tools like Yocto/OpenEmbedded and CMake as they are were meant to be used. These tools do have a learning curve, and require some investment to learn, but in the end they are well worth it. The reason tools like OpenEmbedded and CMake work is that they are consistent, and use the same patterns over and over. We can illustrate the basic architecture of OpenEmbedded in a simple sketch, as shown below. OpenEmbedded is made up of a relatively small number of primitives:layers, recipes, packages, classes,</em> conf files, and tasks</em>.</p> </figure> Dependencies can be specified at the layer, recipe, package, and task level. Once these basic concepts are understood, then it is easy to find things in the build system and make necessary modifications. The boundaries are precise and well-defined. These patterns are not as flexible as a Makefile or Bash script, but they abstract huge amounts of complexity. This allows our mind to see the big picture and not get bogged down in thousands of details and to reliably execute extremely complex tasks. The same patterns are used over and over and are easy to recognize. Dependencies are well-controlled. The cost is to understand and use the established patterns</strong> and to accept the constraints</strong>. These constraints may be a little inconvenient at times. But when you compare this to the work these tools do for you, these inconveniences are relatively minor.</p> If you are going to build complex products using complex technologies like Linux, then you need powerful tools to manage this complexity. There is a learning curve. You may need to invest in training or hire someone to help you get started. Most other industries recognize this. Car mechanics undergo continuous training to learn how to work on new vehicles/systems. Nurses and other professionals receive training on new techniques and equipment. Build systems also require a significant investment in learning. We routinely hire experts (doctors, dentists, plumbers, carpenters, etc) in our personal lives to do things we don’t know how or have time to do. But somehow many developers think the build system and associated infrastructure should not need attention, can be ignored, or is trivial and can be implemented from scratch. Never mind the fact that the underlying software in the system (bootloader, kernel, and libraries) are many times more complex than the custom application software written for the system. You must be able to maintain</strong> it all. You can’t ignore the stuff you did not write – it all has to work. It has to be reliable and predictable</strong>. A build system is largely invisible until it starts causing problems.</p> Getting something working initially is relatively easy; keeping it working for years is another matter. It is similar to product development – building the first prototype that somewhat works often goes fairly quick. Getting something into production is a long, arduous slog. The build system is the foundation of your software project. It must be reliable and predictable</strong> because you can’t do much without it. A build system must do its part well without a lot of fuss, so we can focus on the problem being solved. A well implemented build system produces an image ready for installation with a single command and no manual steps. You check source code into git, run the build command, and out pops an image ready to deploy. There is no copying around binaries, etc. The efficiency gained by the effort to implement such a system pays itself back many times over the life of a project – especially those that are produced for 5-10 years.</p> Yocto and CMake are only used as examples in this article and may not be the best solution for every project. If you have something that is working well, keep doing what you are doing. But if what you are doing is not working, then ask yourself some questions:</p> Is my current system the best way to build complex software?</li> Is the process</strong> to build your project’s software simple and easy to reproduce by any developer on any machine?</li> Is it well documented?</li> Is it maintainable</strong>?</li> Is it consistent</strong> across all your products?</li> Is it reliable and predictable</strong>?</li> </ul> SD Card Write Speed Tests 2019-03-13T00:00:00+00:00 </figure> During Embedded Linux development, we often need to write large operating system images to SD cards for testing. Recently, I purchased a USB 3.0 SD card reader</a> from Plugable. Before that, I used a USB 2.0 SD card reader from IOGear.</p> </figure> How much better does a USB 3.0 card reader perform compared to a 2.0 card reader? To test, I wrote a large (2.2GB) image to a Transcend v30 card using both readers.</p> IO Gear USB 2.0 Reader</strong></td> Plugable USB 3.0 Reader</strong></td></tr> 15.3 MB/s</td> 33.5 MB/s</td></tr></tbody></table> The USB 3.0 reader is over twice as fast with this same card!</p> I then used the USB 3.0 reader to compare a number of cards.</p> SD Card Brand</strong></td> Size</strong></td> Grade</strong></td> Tested Speed</strong></td></tr> Transcend</td> 8GB</td> C10 HC</td> 8.8 MB/s</td></tr> Sandisk</td> 32GB</td> C10 HC I</td> 16.8 MB/s</td></tr> Transcend</td> 16GB</td> C10 HC I V30</td> 33.5 MB/s</td></tr> Sandisk</td> 64GB</td> XC I V30 A1</td> 44.5 MB/s</td></tr></tbody></table> The markings on the cards mean various things. The Wikipedia SD Card page</a> provides a nice overview. A few details are summarized below:</p> Bus interface</strong></p> I: 50MB/s Half/Full duplex, 104MB/s Half Duplex</li> </ul> </li> Capacity</strong></p> HC: Secure Digital High Capacity (SDHC) format – supports cards 2GB to 32GB in size</li> XC: Secure Digital eXtended Capacity (SDXC) format – supports cards 32GB to 2TB in size</li> </ul> </li> Speed</strong></p> 10 with C around it: Minimal sequential writing speed is 10 MB/s</li> 3 with U around it: UHS Speed Class: 30MB/s</li> V30: Video Speed Class: 30MB/s</li> A1: Application Performance Class 1: Read 1500 IOPS, Write 500 IOPS</li> </ul> </li> </ul> If you have not upgraded to a USB 3.0 card reader, it is probably worth your time to do so.</p> One note of caution – there are counterfeit cards for sale that look like SanDisk or Transcend cards. They are not manufactured by these companies, and use lower quality components. If the price seems too good to be true, it probably is. Always buy from a reputable supplier.</p> Getting started with Embedded Linux 2018-10-04T00:00:00+00:00 </figure> Recently I was asked by a developer, who has done windows development for 10 years, how to get started with Embedded Linux. Embedded Linux covers a lot of ground and includes a broad range of components/skills to put together an entire system. Below are a few suggestions.</p> Do I have the right Mindset?</h3> There are many different types of people; there are those who are who are a good fit for Linux, and those who are not. There are those who think Open Source Software (OSS) and Linux are the craziest thing they ever heard of, and those who think it’s a great idea. There are those who thrive on openness and collaboration, and those who feel threatened by it. A few specific aspects of this mindset are described below.</p> Code Gardening</h4> Much of your time as an embedded Linux developer is spent figuring stuff out (vs writing new code). Figuring out stuff is harder than writing new code (at least initially). It is similar to weeding the garden – often not something you are raring to do, but after you are finished, it feels really good. Much of the truly valuable work in life is like this. It initially appears kind of boring, but once you get started there are many interesting and rewarding aspects.</p> You have to like build systems and automation. Linux systems are complex. You can’t throw millions of lines of kernel and user space code together with a simple Makefile or a fancy IDE. Proficiency with command line tools is required. Builds must be clean and completely automated if you want to keep your sanity. Once you allow one manual step in the build, the floodgate has been opened, and these manual steps will start to accumulate at a frightening pace until you have an unmaintainable mess.</p> Collaboration</h4> Because of the complexity, you have to be able to collaborate with others. This requires a generous mindset. How can I make this easy for someone else to pick up once I am gone? Most likely this someone will be you in a couple months or years when you have forgotten many of the project details. The same really applies to coding. Do I write my code to be clever, do I hack it together, or do I focus on making it maintainable and easily understand by others? Automation and simplicity reduce cognitive load, freeing up resources to think about the problems that actually need solved, and allow multiple people to work on a project without a lot of friction.</p> The flip side of this is humility. With a complex system like Linux, there will be a lot of things you don’t know, and some difficult problems to solve. There is no one “manual” for Linux where you can go look up an answer. So we rely on open source communities for help. There are many people willing to help, so the key is to have the wisdom to know where and when to ask for help, and then the humility to ask. In Linux communities (and OSS in general), you will be surrounded by developers who are much smarter and more knowledgeable than you. At the very least, there will always be someone who knows something you don’t. If you don’t have the humility to acknowledge this and be willing to learn, then you will likely never be comfortable or effective working in an OSS environment.</p> NiH</h4> This can all be a challenge for developers who are endowed with a healthy dose of NiH (Not Invented Here Syndrome). These are people who imagine themselves happiest writing and maintaining their own code, and who are not inclined to share it with others. Some imagine themselves as great software architects and coders (very few of us really are) who can make something better. They don’t want to spend long, arduous hours learning and debugging a complex system written by someone else. If this describes you, then perhaps Embedded Linux is not for you.</p> Learning</h4> Are you a learner? Do you enjoy learning about new technology, tools, techniques, etc? Everything is always changing in the Linux world. New and compelling solutions are coming out every year. The real power of Linux and Open Source is knowing when and how to leverage these new technologies to solve real problems. If you are more comfortable doing things the way they have always been done, then perhaps you should just keep doing what you are doing.</p> The Problem of Abundance</h4> Can you deal with a lot of choices? With Embedded Linux, there are many options for about every aspect of the system. There are multiple init systems. There are many programming languages. There are many libraries available for various tasks. There is no one, true way to implement Embedded Linux – you will have to make a lot of choices.</p> Ideas for Learning Embedded Linux</h3> Books/Presentations/Documentation/Blogs</h4> There are many books published on Linux, Embedded Linux, Yocto, etc. The Yocto documentation</a> is pretty good. Videos from the Embedded Linux Conference</a> are freely available. There are many blogs that cover Embedded Linux topics (such as this one). Subscribing to blog RSS feeds in feedly</a> is a good way to keep up with recent developments.</p> The list of learning resources is endless, but perhaps the key thing to remember is there is no one place you can go to learn everything you need to know, like you might for a commercial RTOS. The good news is there is information available in many different formats, so you can choose what best fits your learning style.</p> Run Linux on your computers</h4> One of the best ways to learn a lot about Linux is to simply use it for your personal and work computers. Ubuntu is very easy to get started with. If you go with something like Arch Linux, the install process will teach you a lot. I am very pleased with the combination of Arch Linux plus the XFCE desktop environment. This is a no-nonsense setup that does its job well, is very customizable, and stays out of my way. Any distribution will work – just pick something.</p> Embedded Linux aside, there is little to lose, and a lot to gain from running Linux. Linux systems are naturally much more immune to viruses, malware, and various other attacks than Windows. Linux systems are much more flexible than Apple computers. Some things might not look as pretty, or might take a little more work to set up. But the stability, flexibility, and consistency of a Linux system will save you time in the long run. As a software developer, most of our time is spent in an editor and browser – these work very well in Linux. Additionally, the terminal is a first class citizen in Linux rather than an afterthought.</p> Learn Git</h4> Git is used in almost all open source projects these days, and enables new levels of collaboration. You need to know Git</a>.</p> Do small projects with Raspberry PI, etc.</h4> There are a number of embedded computer boards out there that are very well supported and quite cheap such as the Raspberry PI. Find something in your home that you want to automate – there’s nothing like a real project to motivate learning. With Linux, you learn best by doing. This will also give you an opportunity to learn application programming under Linux.</p> Build and install OpenEmbedded/Yocto or Buildroot</h4> Take the previously discussed home automation project and create a custom build for the board using Yocto/OE</a> or Buildroot</a>. This will give you practice learning how to put together a custom embedded Linux distribution. Create recipes to package, and include your custom applications in the image at build time (instead of installing them afterwards).</p> These tools have a learning curve and may seem intimidating initially, but Linux is a complex system, and you need powerful tools.</p> We think the BEC OE template</a> (now yoe-distro</a>) is a good way to set up Yocto/OE for a project.</p> Modify the Linux Kernel</h4> Most embedded Linux projects require a small amount of kernel modification. This can be something as simple as changing the GPIO configuration on an ARM SOC (System on chip), or writing a driver for some custom hardware. There are many resources available for learning Linux kernel programming. Perhaps try creating a custom driver for your PC or embedded Linux board. This driver could do something trivial, like monitor several GPIOs, etc.</p> Modify and install U-boot</h4> U-boot is a common bootloader for Embedded Linux systems (but not the only one). Locate the source code for your embedded board and learn how to build, modify, and install a custom version of U-boot.</p> Explore the many Application Programming Options</h4> There are so many interesting application programming options with Linux. Of course you can stick with traditional C/C++. But today, there are many other languages and environments that may be better for various applications (Python, NodeJS, Golang, Qt/QML, etc).</p> Find something that is broken and fix it</h4> There are many opportunities to improve open source code. Find something that does not work quite right, fix it, and get the changes accepted in the upstream project. This will provide valuable experience working with OSS communities. When working with complex systems like Linux, you will run into problems that are very hard to solve. Being comfortable with asking, and getting help from various OSS communities will be key to your success. Again, your mindset is key here. People who don’t have the right mindset will create fake names and email aliases to hide their true identity when asking questions. People with the right mindset are willing to be up front and admit there is something they don’t know.</p> Helping Others</h4> One of the best ways to really learn something is to teach it or help others. Once you reach a certain level of proficiency, there are many opportunities in open source communities (stack overflow, mail lists, IRC, etc) to help others. In doing so, we usually receive back much more than we give.</p> Maintain Yocto or Buildroot support for an embedded board</h4> There are many embedded Linux platforms out there that are not actively maintained or tested in Yocto or Buildroot. Get involved and help keep the platform current and working in the build system.</p> Set up a server in the cloud</h4> It is very cheap (less than $10/mo) to set up a server in Amazon AWS, Digital Ocean, Linode, Vultr, or one of the many other cloud providers. Although this is technically not embedded Linux, the skills required are very similar. It will also open up interesting possibilities of creating systems that are a combination of Embedded Linux devices and cloud servers.</p> Publish an Open Source project on Github</h4> Publishing an open source project is a good experience as it helps you learn to create something that is generally useful. For example, I wrote a little app</a> to test serial ports and published it on Github. Over the past few years, other people have provided nice contributions to the application, making it much more useful, with little effort on my part. However, I had to spend a little more time up front making the app generally useful and documenting how to use it.</p> Summary</h3> Embedded Linux brings a level of functionality to systems that is unmatched by anything else out there, thus its popularity. With this broad scope of functionality comes complexity. Linux has a learning curve, and using it successfully requires some amount of discipline and interaction with various OSS communities. The resources are endless – time and motivation are the only limitations. In life, the rewards are proportional to the effort put forth. If you are committed, working with Embedded Linux is very rewarding, and brings a new level of functionality to products.</p> Git, Versioning, and Branching for Embedded Linux Development 2018-02-14T00:00:00+00:00 </p> When building a product using Linux, versioning and branching of your software is an important consideration. Everyone’s needs are different depending on the size of the team, culture, and testing requirements, so there is no one size that fits all. However, after working on a number of different projects for a dozen or so different companies, there are several practices that are often used.</p> The most fundamental is to use Git to manage your software development. It is surprising the number of small companies developing software who do not use Git, or any version control system yet. The many benefits of Git stand alone</a>, but the minute you enter into the Linux world (or the Open Source Software (OSS) world in general), there are thousands of projects available, and 99.9% of them are developed in Git. Having a good understanding of Git is essential in using OSS. This is not really specific to Linux development, but rather any modern software development (cloud, web, mobile, etc). Most foundational software technologies today are open source. There is no other development methodology that scales to the complexity of modern software systems, and no one company contains the resources needed to create and maintain these systems. Git does have a steep learning curve and usually requires 2-3 weeks of committed use for new users to become comfortable with it. However, it is well worth it.</p> An extension of using Git is to consider hosting your Git repositories in the cloud. There are many companies who provide this service at a reasonable cost (some like Gitlab</a> are free), or it is relatively simple to host your own git server (can be done for $2.50/mo at Vultr</a> using Gitea</a>). Some may balk at putting a company’s valuable IP (intellectual property) assets in the cloud (where hackers might have an easier time getting to them) vs behind the secure company firewall. However, which of the following is the main impediment to your business?</p> The danger of competitors hacking into your cloud server, stealing your IP, and then implementing a product with it.</li> Shipping products in a timely fashion, and then maintaining them.</li> </ol> In my observation, #2 seems to be the challenge for most – especially as systems become more complex. Unless you are a top secret government agency, or a cutting edge chip/algorithm company, your software IP is probably of marginal use to anyone else. The reason for this is that integration and execution is most often the challenge, not coding. There is plenty of free/OSS software out there to do about anything. Integrating this into something that will make a product is the hard part. It is difficult getting software and hardware to work together, debugging problems, communicating with cloud infrastructure, software updates, maintaining a product, adding features, etc. Modern products are typically not static entities where you design it, toss it over the wall to production, produce millions, make lots of money, and never see it again. Rather modern products are complex, dynamic beasts that have continually changing requirements, bugs to fix, features needed for new customers, manufacturing issues to solve, obsolete parts to replace, and are part of a larger complex system. Execution is the hard</a> and valuable</a> part. Increasing developer efficiency and collaboration is important, especially with a distributed team of developers. You can require your developers to use company issued Windows computers and fussy VPNs in an attempt to protect your IP, or you can simply host your Git repositories in the cloud secured by industry proven methods such as https and ssh. As systems become more complex, it is critical to be able to involve domain experts in the project (consultants, contractors, employees at a different office, manufacturing, etc). A Git server in the cloud is the foundation of distributed development collaboration. Emailing zip files may be temping as quick and easy, but it is not a sustainable way to develop products that have so many moving parts. As an extension of this, use a web based Git repository manager (like Gitea, Gitlab, or a hosted solution) to implement your Git server. These repository managers greatly simplify the management of Git repos (teams, users, permissions, creating repositories, issue tracking, etc).</p> Software should be released with a version number. Semantic Versioning</a> is a good place to start. Your needs may vary, but you may as well start with the good ideas of others. Additionally, a changelog</a> should be maintained that lists all changes for each software release. Being able to quickly understand what changed with each release is critical as multiple versions are being used. Keeping a changelog requires a little discipline, but is fairly easy to do if you keep an unreleased section at the top of your changelog file to track changes for the upcoming release, as described in the above link.</p> A discussion on branching only really makes sense if you are using Git. Source code branches greatly improve your development process flow. The ease with which Git allows you to create and merge source code branches enables development flows which were impractical before Git. With an Embedded Linux project you are typically building images that get loaded on some type of hardware device. In this case, it’s useful to have development</em> and production</em> releases. Production releases are sent to customers, and development releases are used internally to test new features under development. Every software component that is under active development (OE build system, custom applications, kernel, etc) should likewise have production and development branches that feed into their respective builds. For a more extensive discussion of this topic, consider this post</a>. Again, start with the good ideas of others, and then use what makes sense. An OE build can be configured to pull from the latest HEAD of your custom components, so that for each new build you get the latest from the respective branches for various components. To do this, you typically have something like the following in a bitbake recipe:</p> SRC_URI = "git://git.mycompany.com/myproject.git;branch=develop;protocol=ssh;user=git PV = "4.1.15+git${SRCPV}" SRCREV = "${AUTOREV}" </code></pre> In some cases, for production builds we’ll manually specify the Git hash or tag in the bitbake recipe so that versions of all components are explicitly locked down. To specify a git version, replace ${AUTOREV}</em> with a Git hash. Either way, if you include the Git version (SRCPV</em>) in the package version (PV</em>) variable, you can always figure out what Git version was used to build a package for a particular release. Below is an example of a package file name:</p> my-app_4.1.15+git0+a05d9b23b9-r0.10_var_som_mx6.ipk </code></pre> In this case, a05d9b23b9</em> tells you what Git version was used to build my-app</em>. This is often adequate if you don’t spend a lot of time building or patching old software releases. With each release of software, it is beneficial to tag the software components used in the build with a version. This allows a clear view of what changed between releases, and allows us to easily check out old versions of software.</p> With development and production branches, we should consider how to version the different branches. One simple way is to use a sequence like:</p> 1.0.0 (production)</li> 1.0.900 (develop)</li> 1.0.901 (develop)</li> 1.0.1 (production bug fix)</li> 1.0.902 (develop)</li> merge develop to production and do a new release</li> 1.1.0 (production)</li> 1.1.900 (continue development</li> 1.1.901 (develop)</li> </ul> In this case, the development version is always the last production release MAJOR.MINOR</em></a> and the PATCH</em> version starts at 900 and is incremented with each development release. If a version number ends in 9xx, you know it’s a development build. You can use more complex version schemes like 1.1.0-alpha1 (instead of 1.0.901), but then you might run into sorting problems where 1.1.0-alpha1 is listed after 1.1.0, even though 1.1.0 is a later release than 1.1.0-alpha1. Recent versions</a> of the BEC software updater</a> look at all update files on a USB disk and pick the latest one using a natural sort of version numbers within the text. In the example below, you can see how versions are sorted:</p> [cbrake@mars ver]$ ls -lv total 0 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.0 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.1 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.900 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.0.901 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.1.0 -rw-r--r-- 1 cbrake cbrake 0 Dec 7 10:23 1.1.0-alpha1 </code></pre> In this case, 1.0.9xx works much better for development builds if you want update systems to automatically pick the latest release from a list of options. You can also use versions like 1.0.0+1.1.0-alpha1 to get around the sorting problem if you want something a little more explicit.</p> Versioning and branching does not have to be complex, and in many cases the above is adequate for small teams. Your needs may vary, but the important thing is to have a source control, versioning, and branching process in place. These practices require some discipline to implement, and may seem unnecessary to those who have not used them, but with a little experience we soon realize that efforts to introduce appropriate amounts of organization and process into our development efforts clearly communicates what we did, what we are doing, and what gets released to production. This then frees us up to focus on what really adds value.</p> Separation of Source and Build Directories 2017-11-16T00:00:00+00:00 </p> As we work with larger and more complex systems (i.e. Linux), more and more of our time is spent on integration and pulling different pieces together. We often need to debug or understand code we did not write – especially in build systems. To work effectively in this scenario you must be able to quickly search through a lot of source code. Therefore, we are always looking for ways to make this more efficient.</p> One pattern that is very useful is separating source and build files into two separate directories at the top level of the project. There are several reasons for this:</p> It is easy to grep the source (or today we ripgrep</a>) if it is not littered with transient build data.</li> Its easy to reset the build by simply deleting the build directory.</li> </ul> This is especially important for large builds like OpenEmbedded, where a build directory can be on the order of 40GiB. If your grep tool has to process 40GiB of data every time it runs, it will be very slow.</p> This separation should occur at the top level of the project. Ideally you want something like this for an OpenEmbedded build:</p> ├── build/ │ ├── sstate-cache/ │ └── tmp/ ├── downloads/ ├── sources/ │ ├── bitbake/ │ ├── meta-bec/ │ ├── meta-browser/ │ ├── meta-freescale/ │ ├── meta-freescale-3rdparty/ │ ├── meta-freescale-distro/ │ ├── meta-openembedded/ │ ├── meta-qt5/ │ ├── meta-variscite-fslc/ │ ├── meta-myproject/ │ ├── meta-mycompany/ │ └── openembedded-core/ ├── conf/ └── etc ... </code></pre> The sources</em> directory contains OpenEmbedded metadata (core, 3rd party, and your own custom meta-myproject layers). The downloads</em> directory might contain 3rd party sources or tar balls that are downloaded by the build tool. And the build</em> directory contains transient build data that can be deleted and re-created at any time. This configuration is used in the Yoe distribution</a>. It should be noted that in this example, the desire is to cleanly separate OE metadata (source) from build data. Source code for individual packages is considered build data from an OE perspective. Most of the development work in this case is creating/customizing OE recipes, images, etc, thus the need is to quickly grep the various OE meta layers.</p> This idea can be contrasted with the Poky reference distribution directory layout:</p> ├── bitbake ├── build │ ├── cache │ ├── conf │ ├── downloads │ ├── sstate-cache │ └── tmp ├── documentation ├── meta ├── meta-poky ├── meta-selftest ├── meta-skeleton ├── meta-yocto-bsp └── scripts </code></pre> In this case, there are multiple meta directories in the top level. If you want to search all of the OE meta data, you need to instruct your search tool to exclude the build</em> directory. Additionally, the build directory is dynamically generated, which includes the conf directory. If you want to completely reset your build, you need to clean several directories in the build directory. For a production build, you need some of the conf files to be in revision control; therefore, they are more difficult to manage when they are dynamically generated, or copied from another location. Why not move conf to the top and use it there?</p> Most decisions are tradeoffs, so there may be good reasons for the Poky directory structure. For a reference (demo) distro, simplicity and ease of setup may be valued over reproducibility. For production builds (builds used in products), rock solid reproducability is important. To get that, you need:</p> all source and configuration is locked down in version control</li> a build should be a minimal number of steps: checkout source, a very simple setup/configuration step, and then a build command. There should be no complex scripts that copy files around that may be modified later, or manual steps.</li> the output is ready to program into the target. There are no manual post processing steps.</li> keep things as simple as possible.</li> </ul> The separation of source and build can also be used/observed with many other build systems. With CMake, if you have a project directory named myproject</em>, you can do something like:</p> mkdir myproject-build cd myproject-build cmake ../myproject make </code></pre> Qt Creator also sets up a separate build directory by default.</p> With the linux kernel, you can specify a separate build directory with the ‘O’ option:</p> cd linux mkdir ../linux-build make O=../linux-build </code></pre> Think about directory structure – it can make a difference.</p> Understanding the NXP i.MX6UL Pin Mux (Part 2) 2017-11-06T00:00:00+00:00 In the previous post</a>, it was noted that bit 30 needs to be set in the i.MX6UL pad config if you want to read the state of a GPIO output. Digging into this a bit more, we find the following text in the Documentation/devicetree/bindings/pinctrl/fsl,imx-pinctrl.txt</em> file:</p> SION(1 << 30): Software Input On Field.</em> Force the selected mux mode input path no matter of MUX_MODE functionality. By default the input path is determined by functionality of the selected mux mode (regular).</em></p> In the i.MX6UL developers manual, we also find the following:</p> </p> When we look at the SW_MUX_CTL</em> register, we notice the SION</em> bit is located at the bit 4 position.</p> </p> So the question is: How does bit 30 in the device tree file get mapped to bit 4 in the SW_MUX_CTL</em> register? In the Linux kernel source we find the following:</p> drivers/pinctrl/freescale/pinctrl-imx.c:</em></p> /* The bits in CONFIG cell defined in binding doc/ #define IMX_NO_PAD_CTL 0x80000000 / no pin config need / #define IMX_PAD_SION 0x40000000 / set SION / ... / SION bit is in mux register / config = be32_to_cpu(list++); if (config & IMX_PAD_SION) pin->mux_mode \|= IOMUXC_CONFIG_SION; pin->config = config & ~IMX_PAD_SION; </code></pre> drivers/pinctrl/freescale/pinctrl-imx.h</em></p> #define IOMUXC_CONFIG_SION (0x1 << 4) </code></pre> From this code, we can see that bit 30 is a virtual bit that gets translated into bit 4 in the mux mode register.</p> Hope this helps in understanding how to configure the i.MX6UL gpio. Once you understand the details; the process is very simple.</p> Understanding the NXP i.MX6UL Pin Mux 2017-09-14T00:00:00+00:00 (note, the article is also applicable to the i.MX6ULL as these processors are very similar)</em></p> The NXP i.MX6UL application processor has a very flexible pin multiplexer, that is somewhat difficult to understand at first glance. Most times when we’re configuring the pin mux in Linux, we modify Device Tree files, so perhaps that is the place to start. The pin mux options for the i.MX6UL are defined in the arch/arm/boot/dts/imx6ul-pinfunc.h</em></a> file. The arguments to the macros in this file are defined as:</p> /* * The pin function ID is a tuple of * <mux_reg conf_reg input_reg mux_mode input_val> / </code></pre> The first 5 are defined in the pinfunc.h file, where the config value is defined in a user’s DTS file. As an example, consider the following section from imx6ul-pinfunc.h.</em> Column headings are added for clarity.</p> // headings: mux_ conf_ input_ mux_ input // reg reg reg val _val #define MX6UL_PAD_GPIO1_IO09__PWM2_OUT 0x0080 0x030C 0x0000 0 0 #define MX6UL_PAD_GPIO1_IO09__WDOG1_WDOG_ANY 0x0080 0x030C 0x0000 1 0 #define MX6UL_PAD_GPIO1_IO09__SPDIF_IN 0x0080 0x030C 0x0618 2 0 #define MX6UL_PAD_GPIO1_IO09__CSI_HSYNC 0x0080 0x030C 0x0524 3 1 #define MX6UL_PAD_GPIO1_IO09__USDHC2_RESET_B 0x0080 0x030C 0x0000 4 0 #define MX6UL_PAD_GPIO1_IO09__GPIO1_IO09 0x0080 0x030C 0x0000 5 0 #define MX6UL_PAD_GPIO1_IO09__USDHC1_RESET_B 0x0080 0x030C 0x0000 6 0 #define MX6UL_PAD_GPIO1_IO09__UART5_DCE_CTS 0x0080 0x030C 0x0000 8 0 #define MX6UL_PAD_GPIO1_IO09__UART5_DTE_RTS 0x0080 0x030C 0x0640 8 2 </code></pre> From the i.MX6UL processor reference manual (IOMUX Controller chapter), we can determine the mux_reg is indeed located at offset 0x80. We can also see that the mux_val</em> values match MUX_MODE in the datasheet.</p> </a></p> The mux_val</em> in the imx6ul-pinfunc.h</em> file also match the MUX_MODE field values in the above register.</p> The config register can also be located in the datasheet. Again, we notice that the offset of 0x30C matches config_reg</em> setting.</p> </a></p> Lets consider the last option listed in the pinfunc file: UART5_DTE_RTS. The input register is used to mux a pin to an input function inside the SOC – in this case, the UART5_RTS_B input. We find the input register at offset 0x640 and can see that the input value of 0x2 matches what we would expect in the reference manual.</p> </a></p> So, how do we know if the pinmux is configured correctly for a pin? One way is to use devmem2 and dump the register contents directly from user space:</p> root@imx6ul-var-dart:/sys/class/gpio# devmem2 0x20e0080 /dev/mem opened. Memory mapped at address 0x76f02000. Read at address 0x020E0080 (0x76f02080): 0x00000005 </code></pre> The mux value is 0x5, which configures the pin for the default GPIO1_9 function. Next we can look at the config register:</p> root@imx6ul-var-dart:/sys/class/gpio# devmem2 0x20e030C /dev/mem opened. Memory mapped at address 0x76f0b000. Read at address 0x020E030C (0x76f0b30c): 0x000010B0 </code></pre> If we further decode the bits in this register we find:</p> Bit index: 12─╮ 8─╮ 4─╮ 0─╮ Bits: 1 0000 1011 0000 </code></pre> and can conclude that the following options are set:</p> bits 3-5 = 6: DSE_43ohm</li> bits 6-7 = 2: Medium speed (100MHz)</li> bit 12: PKE: Pull/Keeper Enabled</li> </ul> There are additional options that can be configured such as hysteresis, pull up/down, open drain, and slew rate. I/O is very configurable on this device!</p> If we want to use this pin as a GPIO, it appears that everything is configured correctly, so let’s give it a try. GPIO in the i.MX6UL is banked in 32 bit registers, so we need to map this to a sequential GPIO in Linux. The formula for this mapping is:</p> linux gpio number = (gpio_bank - 1) 32 + gpio_bit</em></p> For GPIO1_9, this would be GPIO9.</p> linux gpio number = (1 - 1) * 32 + 9 = 9</em></p> To read the GPIO, execute the following commands:</p> root@imx6ul-var-dart:~# cd /sys/class/gpio/ root@imx6ul-var-dart:~# echo 9 > export root@imx6ul-var-dart:~# cat gpio9/value 0 (current gpio value) </code></pre> Another nuance of configuring the i.MX6UL pin mux in Linux is that for GPIO outputs, you need to set a magical bit 30 in the Device Tree configuration as show below. If you don’t set bit 30, it will always read back as 0.</p> MX6UL_PAD_JTAG_TDO__GPIO1_IO12 0x4001b0b0 /* 2_nSLEEP / </code></pre> The reason for this will be explored in a future post.</p> Why Git? 2016-08-18T00:00:00+00:00 Some time back, I gave a presentation</a> that included an overview of the Git version control system. I still occasionally get asked why Git should be used instead of Subversion, as it seems harder at first. Most developers don’t really understand Git until they have used it for awhile, and then they will have an “aha moment.” There are 3 features of Git that are especially interesting to me:</p> many repositories (vs. one large repository)</li> distributed development</li> cheap branches</li> </ol> The fundamental driver for better tools is increasing system complexity. Developers are required to manage and integrate more third party software, work in distributed teams, and more effectively re-use the software we do have. Git is a tool that helps you accomplish these goals.</p> Git is a fairly new project, but it is already nearly universal among open source software projects. Its uptake and acceptance speaks for itself.</p> Many Repositories</h2> In the past when most companies used Subversion, there was one huge company repository (I’ve seen them in the 10’s of GB in size) that held a hierarchical tree of all the company source code. This worked fairly well, and is comfortable in that it is very similar to a file system directory structure. However, this model is not very flexible in that it does not have a consistent way to re-use components between different projects. Some people simple copy source code. Some have a “common” project that is included in all their other projects. Subversion externals can be used. But with Git, typically a separate repository is created for each software component. There are perhaps several reasons for this, but one reason is that Git simply does not scale to huge multi-GByte repositories. However, this turns out to be a blessing in disguise as I think it is a better model in many cases. What we end up with is more of a catalog of software components rather than a rigid hierarchy.</p> </p> There is much emphasis these days made on modular, re-usable software components (Object Oriented Programming, plugins, etc.). Why not keep things modular at the repository level? Another example of this type of organization is the Internet itself. It is not a hierarchy of information, but rather a flat system that is organized by hyperlinks.</p> One of the benefits of organizing your source code this way is that it encourages clean boundaries between software components. Each software component needs to stand on its own fairly well without being propped up by header files located in an unrelated source tree 3 levels up in the directory hierarchy. This type of organization forces us to make better use of standard build system practices.</p> Distributed Development</h2> The “many repository” paradigm has been partly driven by the distributed development paradigm. Git solves the problem of multiple developers working in multiple repositories very well. Because we want to use and customize projects like the Linux kernel, U-boot, and OpenEmbedded in our projects, then we naturally find ourself in the situation where we need to manage multiple repositories. Yes, you can check the Linux kernel into your company Subversion repository, but you are much better off long term if you bite the bullet and implement your own Git infrastructure.</p> As we consider the product development process, we need to consider the life cycle of a product. Most products live for at least several years, and will go through several software iterations. If we can update the software components we use, then we can add value to the product in the form of new or updated drivers to support new peripherals, new libraries, performance improvements, etc. But we are dealing with millions of lines of source code, so we must have an efficient way to deal with software projects of this size. The below Figure 2, below, illustrates how you might organize a typical project. Notice we can pull updates from the U-boot and Kernel source trees at any time in the development process. We might have an outside team working an application, and we then easily synchronize the repositories when it makes sense.</p> </p> There are many other design flows possible. Once you have the ability to support multiple branches and repositories easily, it becomes trivial to implement a staging/testing repository for QA processes, maintenance repositories for supporting old releases, etc.</p> Cheap Branches</h2> The last feature discussed in this article is that of cheap branches. Just as Git excels in allowing us easily integrate changes from multiple repositories at a team level, Git also allows us to easily work with multiple branches at the personal developer level. This is a revolutionary feature, and perhaps has to be experienced to be fully appreciated. With SVN, branching and merging is painful; therefore branches are not used a lot. In Git, branches are easy, therefore branching is naturally encouraged. Below is a typical flow of development.</p> </a></p> In this case, a developer is working on a performance improvement that is largely experimental – depending how it works out, it may or may not be used. So he starts a branch and experiments with various changes. As he goes along, he makes nice granular commits that are easy to follow. He may reach a dead-end, delete the branch and start a new one. In this middle of this development, his boss asks him to make a quick bug fix. He checks out master, makes the change, switches back to the branch and keeps working. The switching between branches takes seconds and is very easy. Finally, he has some performance changes he is happy with. He merges master to his perf-improvement branch and tests his changes with the latest master changes. Everything looks good, so he then merges his branch to master. Git makes this type of development flow natural and easy. Because branching is painful in SVN, changes tend to be made in large commits that are difficult to understand.</p> The ability to keep separate tasks isolated is analogous to having a clean, neat desk with only one thing on it, or having a messy desk with a half a dozen different projects going at once. Git is a tool to help maintain different working contexts, and it allows work to be staged in a natural flow. In the end, Git will change how you work.</p> Using Go in place of a Spreadsheet 2016-06-28T00:00:00+00:00 Recently I needed to calculate NAND partition tables for a project where we will be supporting a number of different flash parts from 500MB to 2GB. I first tried this in a spreadsheet, but found it difficult to work easily with hex numbers and do the calculations I needed. I then looked into options for formatting text in columns from a program and found the nice text/tabwriter</a> Go library. With a few lines of code</a>, I was then able to get the below output, which is quite easy to read. The only tricky part was figuring out that for right justified data, you need to:</p> not use tabs for the padding character</li> add a trailing \t in the input data</li> </ol> The Go stdlib is quite extensive, and often has everything you need. This makes “go run” a nice solution for scripting tasks.</p> Old Partitions Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x1c0000 1.75 14 mtd2 uboot env 0x240000 0x40000 0.25 2 mtd3 linux 0x280000 0xa00000 10 80 mtd4 rootfs 0xc80000 0x186a0000 390.625 3125 mtd5 data 0x19320000 0x26ce0000 620.875 4967 total size = 0x40000000, 1024MiB New Partitions 500MB Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x500000 5 40 mtd2 uboot env 0x580000 0x40000 0.25 2 mtd3 linux1 0x5c0000 0x1400000 20 160 mtd4 linux2 0x19c0000 0x1400000 20 160 mtd5 rootfs1 0x2dc0000 0xa000000 160 1280 mtd6 rootfs2 0xcdc0000 0xa000000 160 1280 mtd7 log 0x16dc0000 0xa00000 10 80 mtd8 data 0x177c0000 0x7c40000 124.25 994 total size = 0x1f400000, 500MiB size of boot partitions: 45.75 MiB New Partitions 1G Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x500000 5 40 mtd2 uboot env 0x580000 0x40000 0.25 2 mtd3 linux1 0x5c0000 0x1400000 20 160 mtd4 linux2 0x19c0000 0x1400000 20 160 mtd5 rootfs1 0x2dc0000 0xfa00000 250 2000 mtd6 rootfs2 0x127c0000 0xfa00000 250 2000 mtd7 log 0x221c0000 0x3200000 50 400 mtd8 data 0x253c0000 0x1ac40000 428.25 3426 total size = 0x40000000, 1024MiB size of boot partitions: 45.75 MiB New Partitions 2G Device Name Start Size(B) Size(MiB) Size(Blks) mtd0 xloader 0x0 0x80000 0.5 4 mtd1 uboot 0x80000 0x500000 5 40 mtd2 uboot env 0x580000 0x40000 0.25 2 mtd3 linux1 0x5c0000 0x1400000 20 160 mtd4 linux2 0x19c0000 0x1400000 20 160 mtd5 rootfs1 0x2dc0000 0x1f400000 500 4000 mtd6 rootfs2 0x221c0000 0x1f400000 500 4000 mtd7 log 0x415c0000 0x6400000 100 800 mtd8 data 0x479c0000 0x38640000 902.25 7218 total size = 0x80000000, 2048MiB size of boot partitions: 45.75 MiB </code></pre> </code></p> Using Docker for OE/Yocto builds 2016-04-21T00:00:00+00:00 </a></p> Why Docker? When using OE to build software for products, we often run into the scenario where software needs to be built using the same version of OpenEmbedded over the course of several years. Production builds need to be predictable. We’ve also observed that old versions of OE often break as new Linux distros come out. This is just the result of the complexity of building tool chains. Additionally, for predictable builds you really don’t want to be changing the build OS. This requirement automatically rules out Arch Linux, Debian Unstable, Gentoo, etc as production build machines. Additionally, having developers debug OE build issues on varying workstation distributions is frustrating and time consuming.</p> There are several options:</p> keep a golden build machine around with a fixed OS</li> do builds in cloud on a machine with a fixed OS</li> build in a systemd-nspawn, or chroot</li> build in a virtual machine (Virtualbox, etc)</li> use Docker</li> </ul> I have used all of the above and recently started experimenting with Docker. As an example, I set up a container named cbrake/oe-build</a> and provided an example of how to use a Debian based container for OE builds.</p> Docker requires a bit of a mind-shift from virtual machines. The 1st aha moment with Docker comes when you realize data and build directories should be mapped from the host into the container such that the container does not really have any state. There are no ssh keys, etc in the container. This allows the container to be immutable and easily replicated across any number of machines. Stuff that needs to change goes in your build directory and is managed using Git. Users specific information lives on your host machine.</p> The 2nd aha moment with Docker is running commands with Docker instead of working in the context of Docker. For years, I have run OE builds in a systemd-nspawn. I’d always have two terminal windows open – one for the container context, and one for the host. With Docker, you typically just work in your host context, and spin up a Docker container every time you need to run a command in the container (bitbake, etc). If you define a few shell functions to wrap the docker commands, you hardly even know that commands are executing in the container. This might seem impractical, but it actually works quite well. Of course you can still run a container shell and work in the context of the container if you want, but its not really necessary or desired once you get the container set up.</p> Docker is a neat tool and allows you to have a consistent environment for developer and production build machines. It also encourages a natural separation of immutable containers and transient data that makes it much simpler to manage and distribute containers.</p> Setting up a Go development environment 2016-04-15T00:00:00+00:00 Go has a pretty neat development environment, and its helpful to set up a standard GOPATH on your workstation up front. This is what I do:</p> mkdir ~/go</p> </li> add the following to .bashrc (or some file that configures your env on login)</p> export GOPATH=~/go</li> export PATH=$GOPATH/bin:$PATH</li> </ul> </li> </ul> Now, after you log in, you can do things like:</p> go get -u github.com/monochromegane/the_platinum_searcher/... </code></pre> </code></pre> go get -u github.com/schachmat/wego </code></pre> (set up wego with forcast developer key) </code></pre> </code></pre> There are lots of</a> Go apps</a> out there and if you just set up a global GOPATH, they are trivial to install and use anywhere on your system! Generally your ~/go directory is also used for Go development projects, but that is a topic for another day.</p> </code></pre> Go Language for Embedded ARM Linux Systems 2016-03-08T00:00:00+00:00 In the quest for technologies that work well for embedded Linux systems, I recently gave Go another try. The last time</a>I tried this was very early on and there were some floating point issues on ARM that appear to be fixed now. Having spent a few days porting an existing application to Go, there is a lot to like about Go.</p> Compiles very fast</li> Nice set of default libraries that are nice to use</li> Binaries are reasonable size</li> Memory usage is reasonable</li> Very simple to deploy (run time and libs are bundled with a single executable)</li> many 3rd party libraries available</li> Tooling is good (build, unit testing, etc)</li> </ul> As an example of running a Go app on an embedded Linux system, enter the following in hello.go:</p> package main</p> import (</p> “fmt”</p> )</p> func main() {</p> fmt.Println(“Hello”);</p> }</p> To run on x86:</p> go run Hello.go </code></pre> To build for ARM Linux:</p> $ GOARCH=arm GOARM=5 go build hello.go $ file hello hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped </code></pre> As you can see, cross compiling is very easy with Go.</p> Now, to try this exe on a Raspberry PI 3 running Arch Linux:</p> [root@alarmpi ~]# ./hello Hello </code></pre> I did notice the time to build for ARM is a bit longer than building for x86:</p> $ time go build hello.go real 0m0.262s user 0m0.337s sys 0m0.030s </code></pre> $ time GOARCH=arm GOARM=5 go build hello.go </code></pre> real 0m2.533s user 0m4.930s sys 0m0.183s </code></pre> But, still pretty reasonable considering what all it is doing. The simplicity of all this is very attractive. There is no runtime to cross compile and deploy. Just one executable and you’re good to Go.</p> IOT Protocols: MQTT vs CoAP vs HTTP 2015-12-11T00:00:00+00:00 With the explosion of IOT (Internet of things), there are now more technologies we can use to build systems. In reality, we’ve been doing IOT for years. We’ve been networking devices for a long time. We’ve been collecting data from remote nodes. This is nothing new, but what the IOT movement brings to the table is technologies that are much lower cost, and more standardized. Two of these technologies are MQTT and CoAP. Both very interesting, and very useful. Recently, I helped a system manufacturer think through the architecture of a system with the following requirements:</p> They plan to sell a relatively small number (200-1000) of fairly expensive machines ($5,000-$10,000). The point here is we are not dealing with millions of $10 nodes, which is really the vision of IOT.</li> There might be as many as a dozen machines installed in each location.</li> Users of these machines need to monitor them remotely. The manufacturer would also like to monitor them for diagnostics and customer support.</li> The machines can be connected to the Internet at the install sites.</li> These machines are installed at various locations where the manufacturer doesn’t have much control over the network (firewall, filtering, which ports are blocked, etc).</li> The machines don’t need to talk to each other.</li> It should be relatively difficult to hack into a machine and mess up what it is doing.</li> Data will be mostly going from machines to the cloud.</li> </ol> There are several ways to address these requirements, but one of the simplest today is for each machine to talk to a server in the cloud, and then users monitor and control the machines through a web application running in the cloud. The web application can be designed to be viewed on both personal computers and mobile devices.</p> </a></p> So the question is which protocol do we use to communicate from the installed machines to the cloud? In this case, we really do not need a lot of efficiency, as the number of nodes is relatively small. Bandwidth requirements are not really a huge concern because a) the number of nodes is relatively small, and b) we can use the existing broadband infrastructure at the install sites. While it might be interesting to use one of the new protocols like MQTT or CoAP, we need to carefully consider them in the context of requirement #5 – we don’t have control over what ports are blocked or what type of filtering/proxying happens at the install site. Support costs can run very high debugging networking problems at remote locations. Because MQTT and CoAP typically use non-standard ports, they are less attractive for this application. Even if we did run MQTT or CoAP over port 80, the install site might have internet filters or proxies that would reject non HTTP data. So this puts us back to HTTP, perhaps with a REST implementation. As it turns out, this is not a bad way to go. Even though HTTP is old, it is still a perfectly valid protocol for this application. Some of the advantages include:</p> HTTP is the most compatible with existing network infrastructure, and you are pretty much guaranteed HTTP will work. If a web browser works, then machines in this scenario should be able to send data to the cloud.</li> Stateless HTTP/REST will traverse firewalls and NAT systems easily. Again, web browsers work.</li> Because the Machines initiate the connection to the server (users are not connecting directly to a browser on the machine), the machines are much easier to secure. Thy don’t even need any open ports. Most of the security effort in this system only needs to be focused in one place (cloud server).</li> HTTP provides well understood authentication and encryption mechanisms.</li> </ul> There are other applications where MQTT or CoAP might make more sense.</p> In a system connected via cellular modems where data is very expensive, you might want to consider CoAP, because it can use UDP which eliminates the overhead of TCP/IP, which is significant (both MQTT and HTTP require TCP/IP). Especially if you have 1000’s of nodes, and are trying to keep the monthly usage of each one under 5MB.</li> If the various machines need to talk to each other, then MQTT would be a good choice.</li> </ul> While I’d like to use MQTT or CoAP, in this application I’m going to recommend they stick with HTTP(S). Even though MQTT might be more convenient to work with, the support costs of debugging network issues at install sites will probably swamp any advantages during development. This project can benefit from the host of</a> Wifi enabled</a> microcontroller systems</a>, low cost embedded Linux</a> technologies, or cloud technologies</a> aimed at IOT systems. All of these support HTTP just fine.</p> OpenEmbedded / Yocto Usage Survey 2015-11-18T00:00:00+00:00 Recently, I sent a survey to the OpenEmbedded and Yocto mail lists asking a series of questions about how they use OE. 38 people responded. The average number of years using OE is 4.8 years.</p> Below are the results of the questions. (update 2015-11-23 – the below numbers are absolute numbers, not percentage)</p> </a></p> </a></a></a></a></a></a></a></a></a></p> On the question about tooling, 4 people responded they use Toaster (two specified toaster analysis mode). Two use devtool. One uses ADT.</p> I plan to follow up with additional articles discussing some of these results.</p> Thanks to all who responded to this survey.</p> Rebasing a set of changes with Git 2015-11-11T00:00:00+00:00 One of the common things we do during Linux kernel development is move a series of patches from one kernel version to a similar version (say Linux 4.1 to 4.1.12). This is required as new stable versions of particular kernel version are released. One approach is to merge, but then your changes are mixed in with upstream commits and are more difficult to manage. Git rebase offers a convenient way to move a set of patches. In the following example we have a series of changes we made (or patches we applied) on top of the 4.1 kernel.</p> </a></p> Now we want to move the same patchset to the 4.1.12 stable kernel. To do this, we can use the following commands:</p> git checkout -b 4.1.12-armv7-x0-dugen2 4.1-armv7-x0-dugen2 (create a new branch that will be rebased)</li> git rebase --onto v4.1.12 v4.1 4.1.12-armv7-x0-dugen2</li> </ul> A few details on the git rebase arguments:</p> --onto v4.1.12 (this tells git what version we want to move our changes to)</li> v4.1 (upstream – tells git where our patchset starts)</li> 4.1.12-armv7-x0-dugen2 (branch – is the branch to do the operation on)</li> </ul> After the rebase, our patchset looks very similar, only on top of the 4.1.12 kernel version.</p> </a></p> It this example (which included many more patches than shown above), I only encountered one conflict which was easy to resolve. This process can also be used to move a patchset to a new kernel version (say 4.2, or 4.3), but you would obviously expect more conflicts.</p> Modifying the BusyBox config in OpenEmbedded 2015-09-30T00:00:00+00:00 Recently, I needed to enable the eject</em> command in BusyBox for an OpenEmbedded (Yocto) based project. Below is a way to do this in a fairly painless way:</p> bitbake -c menuconfig busybox (enable the eject command in the config and save)</li> bitbake -c diffconfig busybox (this generates a config fragment, note the fragment file location)</li> recipetool appendsrcfile -w [path to layer] busybox [path to fragment generated in step #2]</li> </ol> Now, in my project layer, I have the following files:</p> recipes-core/busybox/busybox_%.bbappend:</em></p> SRC_URI += "file://fragment.cfg" FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:" </code></pre> recipes-core/busybox/busybox/fragment.cfg:</em></p> # Wed Sep 30 15:23:49 2015 CONFIG_EJECT=y CONFIG_FEATURE_EJECT_SCSI=y </code></pre> Wow, that is pretty neat! The same procedure should work on kernel recipes as well.</p> Notes:</p> If you have trouble running bitbake -c menuconfig</em> in a remote ssh shell, trying installing screen on the build machine.</li> recipetool is located in openembedded-core/scripts/recipetool</em></li> Chris L.</a> noted the -w option in recipetool would have simplified the append slightly (would avoid the subdir= on the SRC_URI, since it’d install it to workdir instead of the source tree) Great idea – example above modified to include -w.</li> </ul> Best practices for using CMake in OpenEmbedded/Yocto 2015-03-10T00:00:00+00:00 I’ve already written about using autotools</a> and qmake</a> in OE. With recent projects, we’re using CMake to build most C/C++ components. Like any good tool, CMake has a learning curve, but it seems to do its job quite well. Below are examples of a CMake build file, and corresponding OE recipe.</p> CMakeLists.txt</em></p> cmake_minimum_required(VERSION 2.8) project(MyApp) file(GLOB_RECURSE src/.cpp) add_executable(MyAPP ${src_files}) target_link_libraries(MyApp pthread) install(TARGETS MyApp DESTINATION bin) </code></pre> myapp.bb (OE recipe)</em></p> DESCRIPTION = “My xyz app” LICENSE = “CLOSED” inherit cmake SRCREV = “${AUTOREV}” PV = “1.1+gitr${SRCPV}” SRC_URI = “git://my-git-server/myapp.git;branch=master;protocol=ssh;user=git” S = “${WORKDIR}/git” BBCLASSEXTEND = “native”</p> Notice the OE recipe above has no build/install logic other than “inherit cmake”.</p> Below are some tips on using CMake.</p> put your source files in a src directory. This allows you to easily build a list by using GLOB patterns. If you put the source files in the root directory of your application source, then the GLOB pattern will also find temporary C files that CMake generates, and you’ll get multiple “main” errors. Some recommend against using GLOB patterns for finding source files because it may miss additions, but in practice I find the benefits of this pattern outweigh the negatives.</li> do all install operations in CMake if possible (vs do_install in the OE recipe). The reason for this is its much</strong> easier to debug install in CMake then OE. You can build the app for x86 on your workstation, run “make install” and easily verify the files get installed correctly. Typically, things get installed in /usr/local when building on your workstation, and /usr when built in the context of OE.</li> do most of your development on your workstation. Most of the time, you can test 99% of an application’s functionality on your workstation. Do what you can there with unit tests where development is much faster and easier.</li> change ownership of /usr/local by running “sudo chown -R $USER /usr/local</a></em>”. This allows you to run “make install” of the CMake project during development without sudo. It also helps ensure you are not accidentally installing stuff outside /usr/local.</li> you don’t need to add header files to the target sources. CMake automatically figures out include dependencies.</li> you can run “make VERBOSE=1”</em> to see the commands make is running during a compile.</li> cmake -DCMAKE_BUILD_TYPE=Debug can be used to add debugging information (essentially -g).</li> When developing on your workstation, do the following to build: mkdir build; cd build; cmake ../; make</em> The advantage of running cmake inside a build</em> directory is you can simply wipe the entire build dir to completely reset the build. Otherwise, the cmake files are mixed up with your source files.</li> The following is the CMake version of printf – very useful for debugging: message(some_var: ${some_var})</em></li> ccmake</em> and cmake-gui</em> are useful tools for examining the state of various CMake variables.</li> </ol> Most of these tips are general and have nothing to do with OE, but that is the entire point – using CMake with OE rarely requires you to do anything beyond what you normally do with CMake. So, if you use CMake correctly, there is usually not a lot more required to cross compile and package your apps in OE.</p> PCB Schematics Style Guide 2014-08-25T00:00:00+00:00 As I work at the intersection of hardware and software, I spend a fair amount of time reading schematics for printed circuit boards (PCBs). I’m astounded at how sloppy most schematics are, and have come up with a few suggestions to consider.</p> Schematics are not only design entry – schematics are documentation</strong>. Unless you are writing all the software, handling all the manufacturing, and debugging all field issues yourself, there will be others who will be using them. Be considerate.</li> Not everyone has a D size plotter (even B size printers are fairly rare). I have a rather large (27“, 2560x1140) monitor, and have seen schematics that I have to zoom in to read. Schematics should be created such that they can be printed on letter size paper</strong> (everyone has a letter size printer), and easily read – even by folks in their 60’s. This is fairly simple – just spend some time breaking large symbols into smaller sections, and use a few more pages. Even for viewing on a screen, putting less stuff on each page</strong> makes it easier to scroll through pages. Most people print out schematics for review or bench work – so having letter size pages makes this possible.</li> Schematics should be organized to have one function per page</strong>, or group similar functions on a page. Don’t intermix power supplies and an audio codec unless they are related. We realize the importance of organizing software in modules, and a large schematic is really no different.</li> Keep all pages the same size</strong>. When every page is a different size, you can’t easily scroll through pages, or print them out.</li> Avoid hierarchical schematics</strong>. For most PCB designs, hierarchical schematics offer no advantage over a flat schematic logically organized on pages. PCB’s can only be so big, therefore even if you have 20 pages for a schematic, 20 pages is really not all that much, and adding hierarchy to 20 pages just adds complexity and makes them harder to read. In a sense, your schematic page becomes a “hierarchical block.”</li> Off page references</strong> are the key to making smaller schematic pages work.</li> Spend a little more time putting a little art</strong> into your schematic symbols. If you have a DB-9 connector, and you draw a connector in the shape of a DB-9 (only takes a few more minutes), then you can instantly recognize what it is in the schematic, and it is much quicker to find the circuit you are looking for during debugging sessions. The same can be said for USB connectors, etc. In some cases, it may make sense to draw a connector symbol to match how it looks physically, so you can quickly find a pin on the bench without digging out a datasheet.</li> Schematic text must be easily searchable</strong>. I don’t want to install a clunky schematic viewer that only runs on X.XX version of windows just to search for text. I want to use the PDF reader I already have installed. Make sure schematics are cleanly exported to PDF, and are easy to search.</li> </ol> I’m sure there are many other ideas – in a brief search, I turned up the following:</p> http://blog.upverter.com/schematic-style-guide</a></li> http://www.k-state.edu/ksuedl/publications/Technote%208%20-%20Guidelines%20for%20Drawing%20Schematics.pdf</a></li> http://electronics.stackexchange.com/questions/28251/rules-and-guidelines-for-drawing-good-schematics</a></li> </ul> Library sizes for C vs C++ in an embedded Linux system 2013-12-18T00:00:00+00:00 Is the size of the libraries required for C++ (vs C) a concern in Embedded Linux systems? Most Embedded Linux systems likely include some C++ code, so this is probably not even a decision we need to make in many cases. However, often there is a need for a small initramfs that is used as part of the boot process (perhaps for software updates) before switching to the main root file system. In this case, it makes sense to keep the initramfs as small as possible, and we might be concerned here with the size of C++ libraries.</p> To find out what additional libraries are required when compiling with a C++ compiler vs C, a simple hello-world app was created. It was then compiled with gcc and g++. The outputs were then parsed with readelf to determine the required dynamic libraries. The results are shown below:</p> C:</p> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]</p> C++:</p> 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]</p> 0x0000000000000001 (NEEDED) Shared library: [libm.so.6]</p> 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]</p> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]</p> The sizes of packages for these libraries in an OpenEmbedded build are:</p> libc: 1.4KiB (libm is inlcuded with libc)</li> libstdc++6: 256KiB</li> libgcc1: 204KiB</li> </ul> (The above sizes are compressed, so the actually will be some larger, but are useful for comparison.)</p> The installed sizes of libstdc++6 and libgcc1 on a ext3 file system are 1.4MiB and 750KiB.</p> It may also take slightly longer to load a C++ application as there is time required to dynamically load and link additional libraries. Is an extra 2MiB of libraries a big deal? It depends on your situation, but should be considered any time you are trying to create a small image.</p> Setting up an OpenEmbedded Package Feed Server 2013-12-11T00:00:00+00:00 With the BEC OE build template</a>, you can easily set up an opkg feed server that serves up packages from your build directory. This allows you to easily install new packages during development, without generating and reflashing an entire image. To use:</p> On workstation:</p> edit local.sh, and define MACHINE_IP to point to your target machine, and re-run “source envsetup.sh”. Alternatively, you can export the MACHINE_IP variable in your environment.</li> run: oe_setup_feed_server</li> run: oe_feed_server</li> </ul> On target system:</p> opkg update</li> opkg install </li> </ul> Opkg will install dependencies, so if you have a complex package, this is so much easier than copying over an opkg file manually, and figuring out you need 6 other packages as dependencies.</p> Why systemd in Embedded Linux Systems? 2013-12-11T00:00:00+00:00 Recently I was asked why use systemd vs sysvinit in embedded systems? There are many discussions on this, and really most of the reasons people use it for servers and desktops are also valid for embedded systems. Lennart Poettering’s articles</a> explain very well why you might want to consider systemd. A few things that rank high on my list:</p> systemd systems generally boot must faster than sysvinit systems.</li> systemd monitors every process it starts, so you know conclusively if your app is still running.</li> systemd has flexible mechanisms for restarting apps that crash, and rebooting if you get so many restarts in X amount of time. This is very important in embedded systems where you need to keep things running.</li> systemd logging (journald) is very flexible. The stdout of every process that is started by systemd is captured in the journal. You can also capture kernel messages in the journal. The end result is you have one place to look for system logs.</li> systemd has support for watchdog chaining</a> – again, important for many embedded systems.</li> The systemd dependency mechanism is very flexible.</li> systemd has tools to debug the init process: systemd-analyze, systemd-cgls, systemd-cgtop, bootchart, pybootchargui, etc.</li> on demand launch of services can improve boot time and conserve resources.</li> </ol> In summary, with systemd I have more control over the system startup and shutdown, and have better mechanisms to monitor things to make sure they stay running.</p> OpenEmbedded Build Template 2013-06-28T00:00:00+00:00 How does one set an OpenEmbedded/Yocto/Poky/Angstrom build? There are many options. Some include:</p> Angstrom setup scripts</a></li> Poky</a></li> Freescale Community BSP</a></li> OpenEmbedded core standalone</a></li> </ul> (I’m sure there are many others, feel free to add in comments …)</p> Over the past years, we’ve supported a number of customers using OpenEmbedded to develop products using various SOC’s. We also try to keep builds going for a number of different systems so that we can continuously evaluate the state of OpenEmbedded and related Embedded Linux technologies. We also needed a standard way to set up builds for customers (some don’t have a lot of experience with OE and Git) that is simple and consistent. What we ended up with is the BEC OpenEmbedded build template</a>.</p> The goal is to have a quick entry point into OpenEmbedded that includes the necessary layers for a number of different machines, and automates a number of routine tasks such as installing images to a SD card, setting up a development feed server, etc. The build template is only updated when the build is stable and tested on a number of machines, so it provides a series of stable snapshots of OpenEmbedded and associated layers.</p> This build template currently tracks the master branches for all the layers used. This gives us a platform to track the latest OE changes. With most projects, that ability to use the features in the latest versions of software outweighs the stability benefits of OpenEmbedded release branches. There are times when the OpenEmbedded project goes through invasive changes (such as the systemd integration), and using the master branches is not practical, so in this case we simply use the last stable snapshot that builds and works. In most cases if there are issues, simply report or fix the issue and wait a week.</p> Perhaps the most controversial decision is the use of Git submodules for including OpenEmbedded layers. The Internet is full</a> of rants</a> against Git submodules. For heavy developer use, submodules may not be optimal. However, from a user perspective, Git submodules provide a simple mechanism for including external repositories in a project. If most of the submodules (OE layers) won’t be touched (typical OE user scenario), submodules work very well. The fact that Git locks down submodules to a specific commit ensures you are getting exactly what you think you are getting (vs a branch that may have been rebased, modified etc). If the git hash matches, you can be pretty sure it is the same as the last time you built it. This is an important factor in production build systems where you want to be sure of what you are building. Google repo is another option under consideration, but there are still some trade-offs to work through.</p> In the end, build systems are very personal, and must be customized for your product and development team. The number one requirement for a an Embedded Linux build system is that you can get repeatable builds with a single command. There must be no manual steps where human error can be introduced. This is just one way to accomplish this goal.</p> (2023-09-05 update: the BEC OE Build template has been replaced by the Yoe Distribution</a>)</p> OS Containers for Build Systems 2013-05-08T00:00:00+00:00 Since I’ve been running archlinux on some of my systems, one thing I’ve found useful is systemd-nspawn</a>. systemd-nspawn containers (or chroots on non-systemd systems) give you a quick way to install a Linux distribution, that can run inside an existing Linux system.</p> Some cases where systemd-nspawn containers (referred to as containers in this document) are useful:</p> At one point, OpenEmbedded would not build with GCC 4.8 (this is no longer the case with recent versions of OE). So a Debian or Ubuntu OS container was a quick way to get builds going again.</li> For product build systems (may live for many years), typically OE will eventually break as you upgrade the workstation distribution. For projects that need a long-lived OpenEmbedded build system, setting it up in a chroot makes a lot of sense.</li> Someone might be having a compile or build problem with a distribution you don’t currently have installed. With containers, you can quickly set up a test distribution to reproduce problems.</li> I’ve had cases where I need an older version of Qt for a project, but my workstation includes a newer version. Again, setting up a OS container is sometimes simpler than getting two versions of Qt to dwell together peaceably in the same distribution.</li> Backing up or replicating your entire build system is very easy – simply rsync the OS container directory to another machine.</li> </ol> So the solution is to select a relatively stable, long-lived distribution to host your product builds. Debian is good choice. Because the container is simply a directory in the host workstation filesystem, you can use host workstation tools (editors, git, etc) directly in the container filesystem. The only thing you need to use the chroot for is the actual building. If you make sure the user ID is the same between your workstation and nspawn container, then permissions are seamless – you can easily access files in the container from the context of your host workstation.</p> To set up a nspawn-container:</p> Install debootstrap. On arch systems, this needs to be obtained for the AUR.</li> host: sudo debootstrap --arch=amd64 wheezy ~/debian-wheezy/</em></li> host: sudo systemd-nspawn -D ~/debian-wheezy/</em></li> container: apt-get update && apt-get install ssh</em></li> container: edit /etc/ssh/sshd_config, and set port to something other than 22 (23 in this example)</em></li> container: /etc/init.d/ssh start</em></li> </ol> (This systemd-nspawn</a> man page gives examples for setting up other distributions.)</p> To set up a user in your container:</p> host: id</li> will return something like: uid=1000(cbrake) gid=100(users) …</li> container: adduser --uid 1000 -gid 100 cbrake</li> host: ssh-copy-id -p 23 localhost (will copy public key to container)</li> </ol> Now, on the host system, you can simply “ssh -p23 localhost”</em> any time you want to log into the container. Soft links between the project workspace on the host system, and the container can also make shifting between the two easier.</p> An alternative way to start the container once its set up is:</p> sudo systemd-nspawn -D ~/debian-wheezy /sbin/init</em></p> Its also handy to make the shell prompt in the container slightly different than the host OS so that you can easily tell the difference. To accomplish this, add the following to ~/.profile in the container OS:</p> export PS1=“[\u@wheezy \w]\$ “</em></p> To create a service that starts your container, put something like the following in /lib/systemd/system/debian-wheezy.service</p> [Unit] Description=Debian Wheezy [Service] ExecStart=/usr/bin/systemd-nspawn -D /scratch/debian-wheezy/ /sbin/init 3 KillMode=process [Install] WantedBy=multi-user.target </code></pre> Hopefully this gives you a quick overview of how OS containers can be set up, and used in your OpenEmbedded build systems.</p> Git submodules can now track branches 2013-04-22T00:00:00+00:00 As of version 1.8.2, Git submodules can now track branches instead of specific commits. This is good news as in many cases, this is exactly the behavior we want. However, Git submodules are still not as flexible as Google repo</a>, but since submodules are built into Git, the submodule command is a good solution in many cases.</p> The “git submodule update --remote</em>” command is the key to tracking branches with submodules. The following is from the Git man pages:</p> --remote This option is only valid for the update command. Instead of using the superproject’s recorded SHA-1 to update the submodule, use the status of the submodule’s remote tracking branch. The remote used is branch’s remote (branch..remote), defaulting to origin. The remote branch used defaults to master, but the branch name may be overridden by setting the submodule..branch option in either .gitmodules</strong> or .git/config (with .git/config taking precedence).</em></p> This works for any of the supported update procedures (--checkout, --rebase, etc.). The only change is the source of the target SHA-1. For example, submodule update --remote --merge will merge upstream submodule changes into the submodules, while submodule update --merge will merge superproject gitlink changes into the submodules.</em></p> In order to ensure a current tracking branch state, update --remote fetches the submodule’s remote repository before calculating the SHA-1. If you don’t want to fetch, you should use submodule update --remote --no-fetch.</em></p> So, if you already have a Git submodule set up, its a simple matter to run git submodule update --remote</em> to update the submodule to the latest master branch. If you want a different branch, simple edit .gitconfig</em>.</p> [submodule "meta-bec"] path = meta-bec url = git@github.com:cbrake/meta-bec.git branch = test </code></pre> Now, if you run git submodule update --remote,Git will update the meta-bec</em> submodule to the latest on the test</em> branch.</p> This functionality is purely a convenience feature in the submodule update</em> command. In the actual repository, Git still stores submodules pointed to a particular commit. The same thing could be accomplished with something like git foreach “git fetch && git checkout test”</em>. The branch option in .gitmodules functions more as documentation and convenience. It is very handy to be able to look at .gitmodules and quickly determine that submodule X is tracking branch Y. Normally, this would have to be documented elsewhere, or figured out in some other way. Also, for build systems where you want the build to always track the production branches of various projects, update --remote</em> gives you a convenient way to update the build tree.</p> A quick way to share files from any directory 2013-01-22T00:00:00+00:00 Did you ever need a quick way to share files from a directory on your computer? Or perhaps transfer a large file to another person? With nodejs</a> and express</a>, you can easily set up a temporary web server that allows users to browse and access a list of files in a directory. For convenience, I created a simple github project</a> that can be cloned into any directory, and then a server started in a matter of seconds. Yes, you could upload files to a server, or share them with a file sharing service, but if you can expose a random port on your computer to the person who needs the files, then this is faster, and does not require any intermediate steps. Check out https://github.com/cbrake/http-file-server</a> for more information.</p> OpenEmbedded Source Mirrors 2013-01-07T00:00:00+00:00 When using OpenEmbedded for product development, there are several reasons you may want to consider setting up a source mirror for your OpenEmbedded build:</p> over time, sources disappear from download locations</li> various servers for source packages may be off-line at the time a build is run</li> some servers may be very slow, which slows down your build</li> occasionally the checksums of a source package will change</li> </ul> For a production build system, you want the build to be reliable and consistent, so this means not depending on 3rd party web sites/servers for a clean build to complete. Fortunately, OpenEmbedded makes it easy to set up a source mirror with the PREMIRRORS</a> variable. When bitbake tries to fetch source code, it tries PREMIRRORS, the upstream source, and then MIRRORS. There are several advantages to using a PREMIRROR variable over a MIRROR for your source mirror:</p> your source mirror will be used first, thus slow web sites are not an issue</li> if the checksums of the package change, the build will not fail because its still using the original source package from the mirror. You are guaranteed to be always using the same source package.</li> </ul> Setting up a source mirror is as simple as copying the contents of your downloads directory to a web server, and then populating the following variable in local.conf:</p> PREMIRRORS_prepend = "\ git://./. http://my-server/sources/ \n \ ftp://./. http://my-server/sources/ \n \ http://./. http://my-server/sources/ \n \ https://./. http://my-server/sources/ \n" </code></pre> The Poky reference manual has more details</a>.</p> Perisistent device names for USB serial ports 2012-12-19T00:00:00+00:00 Currently, my workstation has two 8-port USB<->RS232 devices</a>, one dual port USB<->RS422/RS485 adapter</a>, and several single port adapters such as the very useful BUBII</a>. So with around 20 USB->serial devices, figuring out which /dev/ttyUSBx entry corresponds to which port is not really practical. However, with udev in Linux, you can easily give static names to each device. This is especially convenient to do with FTDI devices because each FTDI device has a serial number. In devices such as the 8-port RS232 adapter, there are 4, 2-port FTDI chips. I could not find a serial number available in the adapter with a Prolific IC, so I would avoid those until that is sorted.</p> udevadm can be used to discover the serial number for a FTDI device:</p> udevadm info --attribute-walk -n /dev/ttyUSB0\|grep serial </code></pre> After the serial number is known, rules can be created in the /etc/udev/rules.d/99-usb-serial.rules</em> as shown in this example</a>. Now, serial ports can be accessed using convenient names such as /dev/ttyUSB_beagle</em>.</p> A quick way to set up an OpenEmbedded feed server 2012-12-08T00:00:00+00:00 During development with OpenEmbedded (oe-core, meta-oe, meta-angstrom), I often find it useful to set up a feed server so that packages can quickly be installed on the target system without manually copying them over or building a new image. One way to do this is copy your deploy/ipk directory to an existing web server (perhaps Apache running on your workstation), or configure Apache to point at your OE build directory, etc. But, it might be more convenient if your build system could directly create an opkg feed with no extra configuration. In the below example, we use node.js</a> + express.js</a> to create a feed server (basically just a web server that serves up the ipk files).</p> tools/feed-server/app.js:</em></p> // nodejs script to start a http server with the feed from this directory // default is port 4000 var express = require('express') var app = express() app.use('/', express.static(__dirname + '/../../build/tmp-angstrom_next-eglibc/deploy/ipk/')) app.use('/', express.directory(__dirname + '/../../build/tmp-angstrom_next-eglibc/deploy/ipk/')) console.log("feed server started on port 4000") app.listen(4000) </code></pre> The express.directory</em> function is used to create a directory listing that can be browsed. With most other web servers, at least this much code is required just for configuration. With node.js, this is the code to create an entire server from scratch! This node.js app can be started with a bash function in the environment:</p> function oe_feed_server() { cd $OE_BASE bitbake package-index node tools/feed-server/app.js cd - } </code></pre> The package-index</em> target is used to rebuild the index files that list the available packages.</p> On the target system, the /etc/opkg configuration files must be modified to point to the feed server, and then you can run:</p> opkg update; opkg install <package> </code></pre> An exmaple build environement with this integrated is located:</p> https://github.com/cbrake/oe-build-core</a></p> https://github.com/cbrake/oe-build-core/commit/23352b9a43c60d67070abe5ac001aba9a9ac5cc4</a></p> https://github.com/cbrake/oe-build-core/commit/6baceb8b1e4477ccfd03aa553a5fcf501b398196</a></p> It might be argued that it is just as easy or easier to set up a more conventional web server. However, the benefit of node.js is that it is a full blown programming environment. You can quickly extend it to provide a web interface, perhaps add functionality to automatically push updated opkg configuration files to the target system that point to your feed server, etc. It is much more than just a web server.</p> Setting the root password in an OpenEmbedded image 2012-12-05T00:00:00+00:00 During development, often a blank root password is used for the embedded Linux target system. However, when deploying an embedded Linux system, often there is a requirement to set the root password to something non-obvious. One way to do this is boot the system, and change the password using the passwd command. Then copy the password hash from the /etc/shadow file into the below line in your image recipe.</p> ROOTFS_POSTPROCESS_COMMAND += "\ sed 's%^root:[^:]:%root:password_hash_from_etc_shadow:%' \ < ${IMAGE_ROOTFS}/etc/shadow \ > ${IMAGE_ROOTFS}/etc/shadow.new;\ mv ${IMAGE_ROOTFS}/etc/shadow.new ${IMAGE_ROOTFS}/etc/shadow ;" </code></pre> The ROOTFS_POSTPROCESS_COMMAND is useful for simple modifications like this to the rootfs image.</p> OpenEmbedded: configuring openssh to allow a blank password 2012-12-05T00:00:00+00:00 Noticed the following when browsing around in the OpenEmbedded sources the other day:</p> ROOTFS_POSTPROCESS_COMMAND += "openssh_allow_empty_password ;" </code></pre> This allows a blank password for development, which is conveient for running ssh/scp commands to the device. The above can be placed in an image recipe.</p> The command modifies the PermitEmptyPasswords config option in /etc/ssh/sshd_config or /etc/default/dropbear.</p> Running a reboot cycle test shell script with systemd 2012-08-29T00:00:00+00:00 One of the easiest ways to stress test an embedded Linux system is to continuously reboot the system. Booting is a difficult activity for a Linux system (similar to waking up in the morning). The CPU is maxed out. There are a lot of things happening in parallel. Software is initializing. There is a lot of filesystem activity. Often, if there is an instability in the system (and especially the filesystem), continuously rebooting will expose it.</p> Now that we use systemd for most new systems, we need to run a simple script X seconds after the system boots to increment the boot count, and reboot. This is the classic domain of a shell script. However, for a simple task like this, it’s a little cumbersome to create a seperate shell script and then call this script from a systemd unit. The following is a way to embed the script directly in a systemd unit.</p> Put the following in: /lib/systemd/system/cycletest.service</em></p> [Unit] Description=Reboots unit after 30s [Service] StandardOutput=syslog+console ExecStart=/bin/sh -c "\ test -f /cycle-count \|\| echo 0 > /cycle-count;\ echo 'starting cycletest';\ sleep 30;\ expr `cat /cycle-count` + 1 > /cycle-count;\ systemctl reboot;\ " [Install] WantedBy=multi-user.target </code></pre> To install and start the script:</p> systemctl daemon-reload</li> systemctl enable cycletest.service (enable the service to start on reboot)</li> systemctl start cycletest.service (start the service, should reboot in 30s)</li> </ul> Bitbake has a new way of displaying build status 2012-08-29T00:00:00+00:00 Now instead of displaying a scrolling log, bitbake will display a simple output that lists which tasks it is working on at the moment:</p> Currently 4 running tasks (185 of 3093): 0: gmp-native-5.0.5-r0 do_configure (pid 22919) 1: lzo-native-2.06-r1 do_configure (pid 27103) 2: expat-native-2.1.0-r0 do_compile (pid 7463) 3: ncurses-native-5.9-r10.1 do_compile (pid 9820) </code></pre> This really allows for a clear view of how the parallel threads option (BB_NUMBER_THEADS) in bitbake works.</p> Mounting a UBIFS partition using systemd 2012-08-16T00:00:00+00:00 Systemd</a> is becoming the defacto system and service manager for Linux, replacing the SysV init scripts. The Angstrom distribution</a> has supported systemd for some time now. Recently, I needed to mount a UBIFS filesystem in one of my projects. The main application is being started with systemd, so it seemed like a good fit to also use systemd to mount a data partition needed by the application. Systemd can use entries from /etc/fstab, but one additional wrinkle in this system is that I also wanted to run the UBI attach in a controlled way. This can be done with a kernel command line argument, but there are times in this system where we will want to format the data partition, so this requires a detach/attach operation.</p> The resulting systemd units are:</p> data.mount</strong></em></p> [Unit] Description=Mount data partition Requires=data-attach.service After=data-attach.service </code></pre> [Mount] What=ubi1:data Where=/data Type=ubifs </code></pre> data-attach.service</strong></em></p> [Unit] Description=Attach data ubi partition [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/sbin/ubiattach /dev/ubi_ctrl -m 6 ExecStop=/usr/sbin/ubidetach /dev/ubi_ctrl -m 6 </code></pre> add the following to my-application.service</strong></em></p> Wants=data.mount </code></pre> The only real problem I ran into is the “After” statement in the data.mount</em> unit. It turns out that the mount will run before the attach operation is finished unless this is included.</p> The RemainAfterExit</em> seems to be required so that ExecStop</em> can be run when the unit is stopped.</p> (There may be a better ways to do all this, so comments are welcome!)</p> One of the benefits of systemd is that everything is very controlled. If the dependencies are specified properly, there are no race conditions. Additionally, if you need to manage units (services, mounts, etc) from a program, it is much easier to check the states as everything is very consistent. For example, if you want to query the state of a unit, the systemctl show command will return easily parsed output (as shown below):</p> ... Before=umount.target After=data-attach.service systemd-journald.socket -.mount Description=Mount data partition LoadState=loaded ActiveState=active SubState=mounted FragmentPath=/lib/systemd/system/data.mount UnitFileState=static InactiveExitTimestamp=Thu, 01 Jan 1970 00:00:17 +0000 ... </code></pre> If the data-attach service is stopped, systemd automatically unmounts the data partition first – very nice!</p> Hopefully this example illustrates how to do simple tasks in systemd. It appears that instead of having a complex script or program to initialize a system, the systemd “way” is to create a number of small units that get connected with dependencies. This seems like a much more controlled and flexible approach.</p> A Review of Graphical Application Solutions for Embedded Linux Systems 2012-08-07T00:00:00+00:00 One of the decisions we face when building Embedded Linux systems is what components to use. With Open Source software, there is often more than one good option. Graphical libraries are no exception. In this article, we’ll examine GTK+, Qt, EFL, Android, and HTML/Javascript.</p> There are many factors that go into a choice like this, but some of them are:</p> Does the application need to run on Windows or MacOS?</li> Does the GUI need to be viewed remotely over a network?</li> Are dynamic effects (think iPhone) desired?</li> Does the application need to run on low end CPU’s (ones without a GPU)?</li> </ul> Putting some effort into selecting the right GUI technology is important for the following reasons:</p> Most of the development effort for the product will likely be in the graphical application, so it makes sense to maximize the productivity of the application developers. Typically there might be 3-10 application developers on a project for every system software developer.</li> You want a technology that will scale with new revisions and improvements of the product.</li> You want a technology that will be supported and improved long term.</li> </ul> With Open Source software, things are always changing. Leading technologies change. This can be illustrated by following Intel’s support for their open source software platforms. While some of this may be driven by politics, we can also see clear technical and licensing reasons why these shifts were made.</p> Moblin (GTK+), became public around 2007, merged into MeeGo in 2010</li> MeeGo (Qt), announced in 2010-02, cancelled 2011-09</li> Tizen (HTML/Javascript, EFL), announced in 2011-09</li> </ul> We can see similar shifts in other companies:</p> Nokia started with GTK+ as the GUI technology for their tablet computers, and then shifted to Qt</li> Samsung has been using GTK+ on top of DirectFB, and now is moving toward EFL/HTML/Javascript for some phones</li> Many phone manufactures are producing Android products</li> Palm moved from a proprietary GUI to HTML in webOS</li> </ul> GTK+</h2> GTK+ is part of the GNOME desktop project and is perhaps the most used graphical library for desktop Linux applications, and in the past has been very popular in Embedded systems. Nokia has invested heavily in GTK+ in the past with its early tablet products (N770, N800, N900, etc). However, with the advent of the iPhone and faster processors with GPUs, everything has changed. The standard is now dynamic GUI’s with sliding effects, etc. The Clutter project is a library that can be used to build dynamic GUIs and fits in with the GNOME stack. GTK+ supports Windows and MacOS, but probably not as well as Qt.</p> Qt</h2> Qt is a very mature project that is also extensively used for desktop projects, and recently is being used on some of Nokia’s phones. Qt was originally developed by the Norwegian company Trolltech. Originally, Qt was only offered with either a proprietary license or the GPL license. This meant if you wanted to write a proprietary application using Qt, you had to purchase Trolltech’s commercial license. This factor alone probably made GTK+ a much more popular solution for many years for Embedded Linux GUI’s, including most cell phone stacks. In 2008, Trolltech was acquired by Nokia, and shortly after that, Qt was offered under the LPGL license, giving it the same license as GTK+.</p> One of Qt’s compelling features introduced in Qt 4.7 is its QML (or Qt Quick) technology. This allows you to write declarative GUI’s in a Javascript like syntax with many automatic bindings. There is good support for dynamic operations like sliding effects, and the performance is reasonable, even on low-end systems without a GPU.</p> In the future, Qt 5.0 will require OpenGL, and hence be relegated to high end ARM CPU’s with a GPU, or desktop systems.</p> Qt’s cross platform support is excellent, and provides good native support for Linux, MacOS, and Windows.</p> Recently, Nokia has made efforts to set up Qt as more of a community project, instead of retaining exclusive control over it.</p> EFL</h2> EFL (Enlightenment Foundation Libraries) is a project that started out as the Enlightenment window manager, and grew into a set of general purpose libraries. It claims to be to be more efficient than GTK+ and Qt, and work with or without hardware acceleration. Recently, EFL seems to have garnered the commercial interest of Samsung, Intel, and others involved in the Tizen project. According to a presentation by Carsten Haitzler (one of EFL’s founders and developers), Samsung was using GTK+ and DirectFB, but switched to EFL after seeing the performance. Perhaps the most compelling story for EFL is the high performance over a range of hardware capabilities (from simple phones with low end processors, to high end smart-phones running OpenGL). Parts of EFL are also used in the commercial GUI FancyPants.</p> Android</h2> Android is an interesting GUI solution, especially as many developers have experience working on Android applications. Android now seems to be used in many applications where Windows CE was used in the past, probably due to its polished application development tool-set.</p> HTML/Javascript</h2> Application development with HTML and Javascript is one of the more interesting developments because many embedded systems are headless (don’t have a local display). Couple this with the fact that many users now have smartphones or tablets readily available, and it may make sense to simply use an external device for displaying the UI. There is often a requirement for accessing the UI of a device remotely, and in this case, HTML/Javascript works very well. If the UI needs to be displayed locally, then a fairly powerful CPU is required (an ARM Cortex-A8, etc) to run a modern web browser. If there is no physical display on the device, and the UI is accessed remotely on a computer or mobile device, then a less powerful CPU is required because the embedded device does not actually have to do any of the rendering. HTML/Javascript also has the benefit that it is a very popular technology, thus there are many experienced developers.</p> Summary</h2> Each of the above technologies has benefits and drawbacks. Understanding your project’s requirements, and what each solution offers is key to making the best decision.</p> A Linux Kernel Tracing Tutorial 2012-01-31T00:00:00+00:00 The Linux kernel has a fairly extensive tracing infrastructure that is quite useful for debugging. There are a number of things you can do with tracing, but the focus of this article will be the traditional printk type debugging we often end up doing to trace initialization issues with a driver. The following links provide additional information on the linux kernel tracing infrastructure:</p> </a>http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=tree;f=Documentation/trace</a></li> Debugging the kernel using Ftrace - part 1: http://lwn.net/Articles/365835/</a></li> Debugging the kernel using Ftrace - part 2: http://lwn.net/Articles/366796/</a></li> Secrets of the Ftrace function tracer: http://lwn.net/Articles/370423/</a></li> http://elinux.org/Kernel_Trace_Systems</a></li> </ul> In this example, I am working on a new audio driver. The typical experience with a new driver is that you install it and nothing happens because something is not registered correctly with the Linux driver model. So, the first thing I do is start with with the platform_device_add() function in my drivers init function. To observe the kernel activity around the kernel platform code, I can do the following:</p> cd /sys/kernel/debug/tracing/ echo 0 > tracing_on (keep trace from filling up until we set filter) echo function_graph > current_tracer echo platform > set_ftrace_filter echo 1 > tracing_on cat trace_pipe (leave running in a different shell) </p> After executing the above, we see the following. For this example, trace_pipe is preferred because the trace is then emptied and only new information is shown.</p> 0) + 30.518 us \| platform_device_alloc(); 0) \| platform_device_add() { 0) 0.000 us \| platform_uevent(); 0) + 30.518 us \| platform_uevent(); 0) 0.000 us \| platform_uevent(); 0) + 30.518 us \| platform_match(); 0) + 30.518 us \| platform_match(); 0) 0.000 us \| platform_match(); 0) 0.000 us \| platform_match(); ... 0) + 30.518 us \| platform_match(); 0) 0.000 us \| platform_match(); 0) 0.000 us \| platform_match(); 0) 0.000 us \| platform_match(); 0) 0.000 us \| platform_match(); 0) ! 3936.767 us \| } 0) + 30.518 us \| platform_uevent(); 0) + 30.518 us \| platform_device_alloc(); </code></pre> From the above, I can conclude that the platform_match() is not succeeding, because I would expect some more activity. At this point I chose to add a printk:</p> diff --git a/drivers/base/platform.c b/drivers/base/platform.c index 7a24895..f9ce0c7 100644 --- a/drivers/base/platform.c +++ b/drivers/base/platform.c @@ -662,6 +662,8 @@ static int platform_match(struct device dev, struct device_driver drv) struct platform_device pdev = to_platform_device(dev); struct platform_driver pdrv = to_platform_driver(drv); + trace_printk("pdev->name = %s, drv->name = %s", pdev->name, drv->name); + /* Attempt an OF style match first / if (of_driver_match_device(dev, drv)) return 1; </code></pre> Now, if I re-run the trace, I see the following:</p> 0) \| / pdev->name = soc_audio, drv->name = davinci_emac / 0) 0.000 us \| } 0) \| platform_match() { 0) \| / pdev->name = soc_audio, drv->name = snd-soc-dummy / 0) 0.000 us \| } 0) \| platform_match() { 0) \| / pdev->name = soc_audio, drv->name = soc-audio / 0) 0.000 us \| } 0) \| platform_match() { 0) \| / pdev->name = soc_audio, drv->name = omap-pcm-audio / 0) 0.000 us \| } 0) ! 4241.943 us \| } / platform_device_add / </code></pre> From the above, it looks like we have a simple mismatch between “soc_audio” and “soc-audio.” Fixing this problem, and re-installing the module, we now have:</p> 0) \| platform_match() { 0) \| / pdev->name = soc-audio, drv->name = snd-soc-dummy / 0) 0.000 us \| } 0) \| platform_match() { 0) \| / pdev->name = soc-audio, drv->name = soc-audio / 0) 0.000 us \| } 0) + 91.553 us \| platform_drv_probe(); 0) ! 4241.943 us \| } / platform_device_add / </code></pre> Now we can see that the names match, and the probe function is now being called. At this point, we may want to turn on tracing of some additional functions to try to determine what is happening next.</p> echo "platform snd* mydriver" > set_ftrace_filter </code></pre> And the result:</p> 0) \| / pdev->name = soc-audio, drv->name = snd-soc-dummy / 0) 0.000 us \| } 0) \| platform_match() { 0) \| / pdev->name = soc-audio, drv->name = soc-audio / 0) + 30.517 us \| } 0) \| platform_drv_probe() { 0) \| snd_soc_register_card() { 0) + 30.518 us \| snd_soc_instantiate_cards(); 0) ! 17852.78 us \| } 0) ! 17883.30 us \| } 0) ! 22125.24 us \| } / platform_device_add / </code></pre> With the above additional information, we can continue to learn more about the flow through the kernel.</p> While all of the above could have been done with printk’s, it would have been more time consuming. The kernel function tracing capabilities allow us to quickly get a high level view of the flow through the kernel without manually adding a bunch of printk statements. The kernel tracing features are completely contained in the kernel without requiring additional user space utilities which makes it very convenient to use in embedded systems. The low overhead is also important in resource constrained embedded systems.</p> The easy way to get serial terminal in Linux 2012-01-26T00:00:00+00:00 When doing embedded Linux development, most of us spend out time tethered to a target system with a serial cable, which is used for a serial console. Minicom is the defacto serial terminal software for Linux. However, Minicom is a little fussy in that you typically have to set it up for each port you want to use. This is no big deal, but is generally difficult for new users to understand, and yet another hurdle. And with 8-port USB->serial adapters, I have a lot of ports to set up.</p> Just recently, I discovered that screen can be used as a serial terminal program:</p> screen /dev/ttyUSB0 115200 </code></pre> A few notes on using:</p> to exit screen: Ctrl-a k</li> to write a hardcopy of the screen image: Ctrl-a h</li> to get help: Ctrl-?</li> </ul> All the neat features of screen are two numerous to list here, but one more that I’ll point out is the scrollback/copy feature (activated by Ctrl-a [ ). This allows you to scroll back and the navigation works much like VI – what could be nicer?</p> Verizon UML290 and Sprint U600 USB Modems in Embedded Systems 2011-03-31T00:00:00+00:00 Recently I tested support for the Verizon UML290 and Sprint U600 USB Cellular modems in an embedded Linux system. Both modems support 3G and 4G networks, but only the 3G modes were tested due to lack of 4G coverage at the testing location.</p> </a></p> Fortunately, both modems function very similar to previous modems, so with the drivers available in the Linux kernel, and standard pppd support in OpenEmbedded</a>, they worked fine.</p> The Verizon UML290 modem provides a challenge in that it must be manually switched between 4G and 3G modes. Typically this is done automatically by the vzaccess program Verizon supplies with the modem that runs on Windows. The solution for this system was to manually set the modem to 3G mode as detailed on the following page:</p> http://www.evdoinfo.com/content/view/3492/64/</a></p> It appears that some embedded systems such as the Cradlepoint routers have implemented automatic 3G/4G switching support for the UML290, so this is no doubt possible with a little effort.</p> The Sprint U600 modem appears to default to 3G, or automatically switch inside the modem.</p> The same pppd scripts can be used with both modems:</p> # /etc/ppp/peers/verizon_um290 user a password v connect "/usr/sbin/chat -v -f /etc/ppp/peers/verizon_um290_chat" defaultroute usepeerdns ttyACM0 921600 local usepeerdns debug -detach </code></pre> # /etc/ppp/peers/verizon_um290_chat '' 'ATZ' 'OK' 'ATDT#777' 'CONNECT' '' </code></pre> To initiate a connection:</p> pppd call verizon_um290 </code></pre> With Verizon cellular modems, it appears that port 22 is often blocked, so if you need to access a remote device via ssh, you may need to run ssh on a higher port number. With 4G networks, it appears that the networking setup may be different in that a public IP address may not be assigned. From the above evdoinfo.com page, we find the following text:</p> This fix will also work for users looking to use their device for remote based applications because it assigns a public facing IP address (3G ONLY). With eHRPD you’re assigned a private IP in either 3G or 4G mode, which has prevents UML290 users from accessing remote applications.</em></p> Perhaps the Rev A HDR Service mode will also work in 4G mode, but its seems as cellular networks become more complicated, there will be more issues to deal with in using USB Cellular modems for remote access in embedded systems.</p> Past articles on USB Cellular modems:</p> /353/sprint-598u-usb-broadband-modem-in-embedded-systems/</a></li> /203/using-a-verizon-usb720-modem-in-an-embedded-linux-system/</a></li> </ul> Git and Distributed Development 2011-03-29T00:00:00+00:00 This is part of an ongoing series of articles</a> on the Git version control system.</p> The “many repository</a>” paradigm has been partly driven by the distributed development paradigm. Git solves the problem of multiple developers working in multiple repositories very well. Because we want to use and customize projects like the Linux kernel, U-boot, and OpenEmbedded in our projects, then we naturally find ourself in the situation where we need to manage multiple repositories. Yes, you can check the Linux kernel into your company Subversion repository, but you are much better off long term if you bite the bullet and implement your own Git infrastructure.</p> As we consider the product development process, we need to consider the life cycle of a product. Most products live for at least several years, and will go through several software iterations. If we can update the software components we use, then we can add value to the product in the form of new or updated drivers to support new peripherals, new libraries, performance improvements, etc. But, we are dealing with millions of lines of source code, so we must have an efficient way to deal with software projects of this size. The below Figure 2 illustrates how you might organize a typical project. Notice we can pull updates from the U-boot and Kernel source trees at any time in the development process, and merge the changes with our own modifications. We might have an outside team working an application, and we then easily synchronize the repositories when it makes sense.</p> </a></p> There are many other design flows possible. Once you have the ability to support multiple branches and repositories easily, it becomes trivial to implement a staging/testing repository for a QA processes, maintenance repositories for supporting old releases, etc.</p> Even at a personal developer level, Git’s distributed capabilities offers many advantages. Each Git workspace is actually a full Git repository. This means you can check changes in locally, re-organize your changes, be able to track changes when off-line, etc. For this reason, many developers are now using git-svn</a> when they need to work with Subversion repositories. With git-svn, you have all the benefits of using git locally, and yet you can easily synchronize with Subversion repositories. And this leads us to our next topic: cheap branches (coming soon).</p> How do modern USB chargers work 2011-03-19T00:00:00+00:00 As we help customers design products, we often try to leverage the latest cell phone practices and technologies. One of these is USB charging. There has been a push in recent years to standardize on USB chargers for cell-phones. There are a number of organizations involved including the USB-IF</a> and OMTP</a>. Two of the specification available include:</p> [http://internal.omtp.org/Lists/ReqPublications/Attachments/59/OMTP CCLDC V1.1 Final.pdf](http://internal.omtp.org/Lists/ReqPublications/Attachments/59/OMTP CCLDC V1.1 Final.pdf)</li> http://www.usb.org/developers/devclass_docs/Battery_Charging_V1_2.zip</a></li> </ul> There is also a Chinese standard titled the “Telecommunications Industry Standard of the PRC“ that has been instrumental in influencing these standards.</p> The fundamental problem is a battery powered device needs to know if it is plugged into a computer USB port, or a dedicated charger. A standard USB port on a computer is only specified to provide 500mA. Typically a dedicated charger that plugs into a wall outlet provides more current than 500mA so that the battery can be charged quicker. To differentiate between a computer USB port, and a dedicated charger, the dedicated charger shorts the D+ and D- USB signals together with a resistance of less than 200 ohms (specified by USB-IF).</p> One of the goals of these standards is that any charger can be used with any battery powered device. The next question is how do we handle the case where every USB charger has a different rated current? How can a device that charges its battery at 1.7A be compatible with a charger that only outputs 0.7A? One theoretical solution would be for the device to query the current capacity of the charger and then only use that much current. It turns out a much simpler approach is used. The USB-IF provides the following chart in the above specification:</p> </figcaption></a></figure> As long as the charger outputs power in the Required Operating Range, the battery powered device must be able to use whatever power is available to charge its battery. If the device uses more current than the charger can supply, the charger simply goes into a constant current mode and the battery charges slower than it would with a higher capacity charger. Thus we have a very simple scheme where we can theoretically use any charger with any device.</p> As a simple test, I charged a Nokia N900 phone (which came with a 1.2A charger) with a smaller LG charger that is rated for 0.7A. I monitored the LG charger a couple times during charging to make sure it was not getting hot. It seemed to charge the N900 battery just fine.</p> When purchasing after-market USB chargers, it is sometimes difficult to determine if they have the USB D+ and D- lines shorted together. In once case I purchased a car USB charger that did not work with my phones, but was able to “fix” it by disassembling it, and putting a solder blob between the D+ and D- signals on the USB connector.</p> </a></p> </a></p> </a></p> While shorting the D+ and D- pins usually makes a charger work, it is nice to know if the charger is compliant with the USB-IF specification and is designed to work in the constant current mode without shutting down, catching fire, etc.</p> Git Overview Screencast 2011-03-17T00:00:00+00:00 This screencast</a> (use Firefox to view the screencast) provides an overview of the Git version control system. There are 3 features of Git that are especially interesting:</p> many repositories (vs. one large repository)</li> distributed development</li> cheap branches</li> </ol> The fundamental driver for better tools is increasing system complexity. More and more we are required to manage and integrate more third party software, work in distributed teams, and more effectively re-use the software we do have. Git is a tool that helps you accomplish these goals.</p> Git and Why Multiple Repositories 2011-03-17T00:00:00+00:00 This is part of an ongoing series of articles</a> on the Git version control system.</p> This article discusses the trend in software configuration management toward multiple repositories, rather than one large repository. In the past when many companies used Subversion or comparable systems, there was typically one huge company repository (I’ve seen them in the 10’s of GB in size) that held a hierarchical tree of all the company source code. This worked fairly well, and is comfortable in that the organization is very similar to a file system directory structure. However, this model is not very flexible in that it does not have a consistent way to re-use components between different projects. Some people simply copy source code. Some have a “common” project that is included in all their other projects. Subversion externals can be used. With Git, typically a separate repository is created for each software component. There are perhaps several reasons for this, but one reason is that Git simply does not scale to huge multi-GByte repositories. However, this turns out to be a blessing in disguise as I think it is a better model in many cases. What we end up with is more of a catalog of software components rather than a rigid hierarchy.</p> </a></p> There is much emphasis these days on modular, re-usable software components (Object Oriented Programming, plugins, etc.). Why not keep things modular at the repository level? Another example of this type of organization is the Internet itself. It is not a hierarchy of information, but rather a flat system that is organized by hyperlinks.</p> One of the benefits of organizing your source code this way is that it encourages clean boundaries between software components. Each software component needs to stand on its own without being propped up by header files located in an unrelated source tree 3 levels up in the directory hierarchy. This type of organization forces us to make better use of standard build system practices.</p> How do you implement this type of repository infrastructure? Build systems such as OpenEmbedded, Gentoo, and the Android build system manage this fairly well. However, Git also includes a feature named “submodules” that provides a mechanism for one repository to reference source code from another. What you end up using really depends on your needs, and what you are trying to accomplish.</p> A screencast</a> is also available that covers this topic.</p> C++ callbacks to member functions 2010-12-16T00:00:00+00:00 How to properly do callbacks to C++ member functions is something that has intrigued me for some time now. There are a number of solutions, none of which I really liked. But now with the std::tr1::function</strong> and std::tr1::bind</strong> functions, there appears to be a clean solution.</p> In the past, if you wanted a callback to a C++ member function a typical approach was to create a static function wrapper as described here</a>. However, this is ugly, and a step backwards from the capabilities of plain old C.</p> std::tr1::function</strong> is a function wrapper that can be used to create function objects for a number of scenarios. We are interested in representing class member functions.</p> #include <tr1/functional> struct X { int foo(int); }; std::tr1::function<int (X, int)> f; f = &X::foo; X x; f(&x, 6); </code></pre> f() is now a simple function representation that can be passed around, etc. However, what we don’t like is the reference to the struct X type, which makes it much less generic. This is where the std::tr1::bind</strong> function comes in. With bind, we can dynamically bind one function to another, and re-arrange the arguments, etc. So to make a generic portable function that points to X::foo(), we can do something like the following.</p> #include <tr1/functional> struct X { int foo(int); }; std::tr1::function<int (int)> f; X x; f = std::tr1::bind(&X::foo, &x, _1); f(6); </code></pre> Now, f() is a very generic function that could be used in various interfaces without specifically referencing struct X.</p> A more complete example of how callbacks might work is given below:</p> #include <iostream> #include <tr1/functional> struct B { std::tr1::function <void ()> _callback; void reg_callback(std::tr1::function <void ()> callback) { _callback = callback; } void unreg_callback() { _callback = NULL; } void process() { // simply calls callback if (_callback) _callback(); } }; struct A { B * _b; A(B * b) : _b(b) { std::tr1::function<void()> callback; callback = std::tr1::bind(&A::callback, this); b->reg_callback(callback); } ~A() { _b->unreg_callback(); } void callback() { std::cout << "A::callback called\n"; } void process() { _b->process(); } }; int main() { B * b = new B(); A * a = new A(b); // the following is example where object b calls // back into a method // in object a a->process(); // now delete object a, and verify the callback // in B does not crash or still get called delete a; // now b should not execute the callback b->process(); } </code></pre> In the above example, the callback in the object a</em> will be called once.</p> Yocto and OpenEmbedded 2010-12-10T00:00:00+00:00 Recently, I attended an Embedded Linux summit hosted by the Linux Foundation to discuss the Yocto project. Overall, I thought the meeting was very positive and constructive. Having met and discussed at length the methods and goals of the Linux Foundation with some of their people, I’m impressed with their approach. They are there to help Linux succeed in the Embedded space in any way they can.</p> It is my understanding that the Yocto project has the goal of making Linux much more accessible and usable in embedded systems, and improve efficiencies. While the OpenEmbedded project has served many of us well for many years, we all can readily understand there are some deficiencies. These can be overcome by experienced developers, but there are a number of areas that can obviously be improved. Not only are we concerned with making it easier for new users to use Embedded Linux, I think there are areas where we can drastically improve the efficiency of implementing embedded Linux systems for those who are experienced. It was stated once that tools implemented must be useful to both the new developer as well as the experienced developer.</p> It should be noted that building an embedded Linux system is an inherently complex undertaking, and although we can improve tools and processes to make it more efficient, and somewhat easier, in the end it is still a very complex problem, and will require considerable skill to be an effective embedded Linux developer. There is no substitute for experience, and developer skill. Just as we would not slap a fancy GUI on top of a robotic surgery instrument, and tell a novice to have at it, likewise it is still going to require considerable engineering skill to be effective in developing Embedded Linux systems. But, if we improve the base technologies, and tools, we will spend less time messing with these and doing the same things over and over, and will have more resources toward implementing new things.</p> One example of the pain experienced in a typical OpenEmbedded system is getting Oprofile to run. Oprofile requires that ARM system be built with frame pointers (OE defaults to not), and that you have symbols in your binaries. Figuring out how to build a reasonably sized image with the needed symbols might be a 1-2 day process for me. Then there is the issue of kernel symbols, etc. I’m sure many other developers go through the same steps, but as many of us are paid to develop products on tight schedules, we don’t have the time to polish the Oprofile integration and tools.</p> As an extension of this, the application development story with OpenEmbedded is not all that great. Yes, we can generate a binary SDK, but again it may take a couple days of messing around to figure out the process, and get all the necessary libraries, etc. Then you still have the problem that application developers want Eclipse/Qt Creator integration, etc. Again, this can all be done, but takes time, and many people are doing the same things over and over again.</p> The Yocto project seems to have two primary thrusts:</p> stabilize a subset of the OpenEmbedded (called Poky) metadata and make it very high quality.</li> improve tools associated with the build process, and general embedded Linux development.</li> </ol> One thing the OpenEmbedded project has historically lacked in the past is industry funding and ongoing developer commitment. The OE project is characterized by a rotating group of developers. When someone needs OE for a project, they jump in and do a lot of work on OE, then they disappear again once their need is met. We have very few developers who work on the project consistently for long periods of time. This has been positive in some ways in that we have a very large number of contributors, and a very open commit policy. There are so many patches coming in right now, we can’t even keep up with processing them. Week after week, we have a healthy number of committers and changesets. The community behind OE is rather astounding, and it is amazing how OE succeeds almost in spite of itself as a self organizing project without much organization.</p> In the past the OpenEmbedded and Poky projects have existed as two independent trees, and things were shared back and forth manually. This works fairly well as recipes are self contained, and generally can simply be copied from one tree to another. However, some of the core recipes are fairly complex, and if they start to diverge, then sharing information can get more difficult.</p> It seems the vision that is emerging is that Poky could become the core component of OpenEmbedded. Poky would contain a very controlled subset of core recipes that must remain at a very high quality level. OpenEmbedded could then expand on this (through the use of layers) to provide the breadth and scope it has traditionally provided.</p> We may bemoan the lack of quality in the OpenEmbedded project as it has 1000’s of recipes, and many of them have rotted, etc. But as a consultant helping people implement products, I still find considerable value in this. For example, one of the products I support uses the HPLIP drivers from HP. Yes, the recipe is broke every time I go to use the latest version, but with a little work I can get it working again. Having something to start with provides value. The same is true for the geos mapping libraries. Very few people are going to use geos, so it will never be in a Poky type repository, but some of us do use geos, so having a common place like OpenEmbedded to put recipes like this is very important. Using Poky as the core of OpenEmbedded seems like a win-win. We are relieved of some of the burden of maintaining core components (like compilers, C library, tool integration, sdk generation, etc), but we can still have a very open culture, and provide a wide scope of platform, library, and application support that we have historically provided.</p> Richard Purdie is poised to become the Linus Torvalds of Yocto, and if OpenEmbedded choses to base on Poky, then the Torvalds of the OpenEmbedded core. I am personally fine with this as I don’t know anyone else who has contributed so much time and has the same level of skill in the OE/Poky area. He has proven over time to be levelheaded and able to deal with difficult situations. Also, as a Linux Foundation fellow, he is positioned to be company neutral, which is very important for someone in this position.</p> Yocto is using a “pull” model similar to the Linux kernel for accepting changes. It is planned to put maintainers in place for various subsystems. With the goal of providing a very high quality subset of embedded Linux technologies, it seems this makes a lot of sense. If OpenEmbedded choses to base on Poky, there is no reason OpenEmbedded can’t continue to still use the push model for its layer that has worked well in the past. But, as we see patches languishing in the patchwork, perhaps a pull model might actually be more efficient at processing incoming patches so they don’t get lost. This is also an area where layers might actually help in that we have dedicated people committed to processing patches for each layer. One of the problems with the OpenEmbedded project with its 1000’s of recipes, the structure is very flat, and its fairly difficult to divide it up into areas of responsibility.</p> So going forward, it seems that if OpenEmbedded can somehow use Poky as a core technology (just like it uses bitbake, etc), then we could have the best of both worlds. With Poky, we get strong industry commitment and dedicated people working on making the core pieces better, and the tools better (50 engineers are already dedicated). This is something those of us developing products don’t really have time for and we are not currently funded to do this. With OpenEmbedded we can still keep our broad involvement, and vast recipe and target collection. Yes, the quality of OpenEmbedded will be lower, but it still provides a different type of value. Poky provides the known stable core, and OpenEmbedded is just that–it is “open”, and provides the broad menu of components that can be used. Over time this could evolve into something different, but for now it seems a good place to start.</p> The last thing to consider is the OpenEmbedded brand. It is recognizable from the name what it is (unlike Yocto, and Poky). It has had broad exposure for many years. It has historically been very vendor neutral with very little corporate direction or influence. From this perspective, it seems the Yocto project can benefit greatly from having an association with the OpenEmbedded project. This topic was discussed at the summit, and there was general consensus on the strong awareness of the OpenEmbedded brand, as well as an openness to how branding for the Yocto core build system might look.</p> Qt Developer needed 2010-11-05T00:00:00+00:00 We are currently in need of a Qt developer to help on a project. If you are interested, please contact us</a>. Thanks.</p> OpenEmbedded srctree and gitver 2010-10-11T00:00:00+00:00 Recently an OpenEmbedded</a> class name srctree</em> became usable. The srctree.bbclass</a> enables operation inside of an existing source tree for a project, rather than using the fetch/unpack/patch idiom. The srctree.bbclass in combination with the OpenEmbedded gitver.bbclass</a> and git submodules provides a very interesting way to build custom software with OpenEmbedded.</p> One of the classic problems with OpenEmbedded is how do application and kernel developers use it. While OpenEmbedded excels at automating image builds, it is less friendly when used as a a cross development tool. Historically there are several options for iterative development:</p> develop in the working directory: cd /work/arm…/; ../temp/run.do_compile …</li> a variation of #1: bitbake -c devshell </li> manually set up an environment that uses the toolchain generated by OE. As example see this script</a> in the BEC OE template</a>.</li> a variation of #3: export a SDK that includes a toolchain and libs</li> </ol> While the above solutions work OK, the process can be a little cumbersome. Unless your OE recipe pulls software directly from the TIP of a SVN repository, you may have to manually update the recipe after you make changes, create patch files, etc. There is also the problem that if your recipe fetches the latest from SVN, it drastically slows down the recipe parsing as it has to check the repository for a new version every time the recipe is processed.</p> The optimal solution would to be to simply check a software component out of a version control system, and build it directly using OpenEmbedded. Icing on the cake would be if the package generated would automatically extract version information from the version control system. This would facilitate iterative development for software components that need to be cross compiled.</p> Although srctree can be used with any directory of source code, it really works best with a git repository. The gitver.bbclass provides a GITVER variable which is a (fairly) sane version, for use in ${PV}, extracted from the ${S} git checkout, assuming it is one (text from recipe). gitver uses the ‘git describe’ command to extract the last tag and uses that for the version.</p> The best way to illustrate the use of these tools is an example:</p> https://github.com/cbrake/autotools-demo</a></li> </ul> The easiest way to try this is clone the above project in your openembedded/recipes directory:</p> $ cd openembedded/recipes $ git clone https://github.com/cbrake/autotools-demo.git $ cd autotools-demo $ git describe 1.1 $ git tag -l 1.0 1.1 </code></pre> Notice that git describe</em> simple returns the latest tag. The recipe</a> can be located in the same directory as the source code and has the following contents:</p> # recipe to build the autotools-demo in the working tree inherit srctree autotools gitver PV = "${GITVER}" </code></pre> Can’t get much easier than that! If you build the recipe, you end up with a package:</p> $ bitbake autotools-demo $ ls tmp/deploy/glibc/ipk/armv5te/autotools-demo_1.1-r0.6_armv5te.ipk </code></pre> Now, what happens if you make changes and commit them?</p> $ cd .../autotools-demo $ (make a change and commit) $ git describe 1.1-1-gfbc1ecc (notice the count and hash automatically appended) $ (make another change and commit) $ git describe 1.1-2-g7ad3715 (notice the count is now 2) </code></pre> If we bitbake the recipe now, we end up with a packaged named:</p> tmp/deploy/glibc/ipk/armv5te/autotools-demo_1.1-2-g7ad3715-r0.6_armv5te.ipk </code></pre> The gitver</em> class in OpenEmbedded automatically takes care of creating a usable PV (package version) that always increments.</p> So in summary, srctree</em> and gitver</em> give developers a convenient way to handle custom components that change often in an Embedded Linux build without increasing parse times, requiring manual tweaks to version numbers, or creating a separate workspace for each version of the application that is built. As practices such as continuous integration become more common, OpenEmbedded features like this are increasingly needed. An added benefit is that the OpenEmbedded recipe can now be stored in the same location as the source code. Perhaps in the future, most applications will include an OpenEmbedded recipe as part of their source code and git submodules could be used to simple populate the components you want to use.</p> 2017-11-08 update: A tool named devtool</a> is now the preferred way to do much of the above.</em></p> Qt Creator for C/C++ development 2010-08-07T00:00:00+00:00 Recently, I’ve been evaluating Qt Creator</a> for general C/C++ development. I’m currently involved in the development of a rather large C++ application that is approaching 200,000 lines of code and 1000 source modules. In the past, I’ve typically used Vim for editing, and Eclipse as a gdb front-end when needed. Qt Creator is a rather new tool developed by Nokia’s Qt team. What initially attracted my attention was that one of the reasons the project was started was no existing tools effectively handled the Qt codebase, which is quite large. Things I like about Qt Creator:</p> it works fairly well with any C++ Make based project. This includes projects built with autotools as well as the Qt Qmake build system.</li> easy to import existing projects</li> it is very fast. Indexing 200,000 lines of code happens in around a minute or so.</li> Provides a Vim compatibility mode that actually works</li> provides fast methods for finding files, symbols, and seeing where symbols are used</li> did I mention yet it is fast</strong></em>?</li> </ul> I also recorded a screencast that demos Qt Creator with a large project</a> (can be viewed in firefox). As always, I’m interested in what others find interesting in this or other tools. Future efforts will be to use Qt Creator to build and remotely debug ARM binaries – I am interested in what others have done in this regard.</p> If you do try Qt Creator, I recommend the latest pre-release snapshot</a>.</p> Embedded Linux Presentation in Cleveland 2010-07-26T00:00:00+00:00 Chris Cole</a> and I will be giving a presentation</a> in Cleveland Tues evening (2010-07-26). Stop by if you are interested.</p> Git submodules: what to do when you commit to (no branch) 2010-06-17T00:00:00+00:00 When using git submodules, eventually you’ll run into a situation where a workspace has checked out a submodule as (no branch), and you unknownly make modifications and commits to (no branch). Now what? It turns out Git has a very useful feature named reflog. Reflog keeps a local time based log of all activity. That means that as long as you do not garbage collect your workspace, you can go back to any point in time and get any commit that was made. This is useful in a number of scenarios, but lets consider the case where we made two commits to a submodule in the (no branch) state. We are then ready to push our changes to the upstream repo and we get:</p> cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git push bec HEAD error: unable to push to unqualified destination: HEAD The destination refspec neither matches an existing ref on the remote nor begins with refs/, and we are unable to guess a prefix based on the source ref. error: failed to push some refs to 'git@git.bec-systems.com:openembedded.custom' </code></pre> If we do a git log, we find we are on the infamous (no branch).</p> cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git branch * (no branch) master </code></pre> At this point we may wonder what to do. How do we get these changes onto the master branch? Using git reflog, we can view all commits and changes to our local repository based on time.</p> cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git reflog 6d195b3 HEAD@{0}: commit: add recipe for qt-embedded image 4cf1bd6 HEAD@{1}: commit: add touchscreen utilities bcc673c HEAD@{2}: checkout: moving from master to bcc673c9824e916a7ec6c04f60a17116af539423 56d9def HEAD@{3}: commit: add more features 5286f87 HEAD@{4}: commit: rename debug image 90896de HEAD@{5}: pull : Fast-forward 5cc3f3b HEAD@{6}: checkout: moving from 90896deb2a25f0d40b1e6305a0ad994f932f22d8 to master 90896de HEAD@{7}: checkout: moving from master to 90896deb2a25f0d40b1e6305a0ad994f932f22d8 </code></pre> Git will keep our keeps commits to (no branch) even though we might switch to another name branch. And using git reflog, we can easily find these commits and merge them to the master branch.</p> cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git checkout master Previous HEAD position was 6d195b3... add recipe for qt-embedded image Switched to branch 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git reflog 56d9def HEAD@{0}: checkout: moving from 6d195b3909553b04cba3d3ac182076a2c6b866b2 to master 6d195b3 HEAD@{1}: commit: add recipe for qt-embedded image 4cf1bd6 HEAD@{2}: commit: add touchscreen utilities bcc673c HEAD@{3}: checkout: moving from master to bcc673c9824e916a7ec6c04f60a17116af539423 56d9def HEAD@{4}: commit: add more features 5286f87 HEAD@{5}: commit: rename debug image 90896de HEAD@{6}: pull : Fast-forward 5cc3f3b HEAD@{7}: checkout: moving from 90896deb2a25f0d40b1e6305a0ad994f932f22d8 to master 90896de HEAD@{8}: checkout: moving from master to 90896deb2a25f0d40b1e6305a0ad994f932f22d8 cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git merge 6d195b3 Updating 56d9def..6d195b3 Fast-forward recipes/images/bec-debug-image.bb \| 38 +++++++++++++++++++----------- recipes/images/bec-qt-embedded-image.bb \| 8 ++++++ 2 files changed, 32 insertions(+), 14 deletions(-) create mode 100644 recipes/images/bec-qt-embedded-image.bb </code></pre> At this point, the changes have been merged to the master branch, and we can now push to our public repository:</p> cbrake@happy:/scratch/oe/oe-build-overo/openembedded.custom$ git push bec HEAD Counting objects: 14, done. Delta compression using up to 8 threads. Compressing objects: 100% (9/9), done. Writing objects: 100% (10/10), 1.13 KiB, done. Total 10 (delta 3), reused 0 (delta 0) To git@git.bec-systems.com:openembedded.custom bcc673c..6d195b3 HEAD -> master </code></pre> Git reflog can also be used to restore commits that you decided at one point to throw away with “git reset --hard HEAD~2”, etc. As long as you don’t garbage collect the repo, they are simply dangling state and always still exist. This is just another example of the ways git can be very confusing, but also very powerful if you understand how it works.</p> This screencast</a> covers the operations in this article.</p> Installing OMAP3 images on a SD card 2010-06-14T00:00:00+00:00 This article and screen-cast is a continuation of that last couple posts describing the BEC OE build template</a>. The purpose again for a build system is to automate tedious manual tasks, and in doing so, we end up documenting how the build system works. Having a good build system is important during product development so that you have an easy, repeatable process for building and deploying images. One of these tasks is setting up a SD card and installing images to the SD card that an OMAP system can boot from. This screencast</a> goes over the script and makefile targets that are used to automate this process.</p> In summary, the steps to set up a SD card and install an image using the build template are:</p> cd </li> determine the SD card device name (/dev/sdX)</li> unmount any file systems mounted on the SD card</li> sudo ./scripts/omap-sd-partitions.sh /dev/sdX (make sure you have the right device!!!)</li> make omap-install-boot</li> make omap-install--rootfs</li> </ol> Previous articles in this series:</p> OpenEmbedded build template</a></li> Creating a Custom OpenEmbedded Image</a></li> </ul> OpenEmbedded Build Template posted 2010-06-08T00:00:00+00:00 A password protected version of this article accidently got pushed into the feed before it was finished, so this post is simply to indicate the following article is now available. Sorry for the confusion.</p> /637/oe-build-template/</a></p> Creating a Custom OpenEmbedded Image 2010-06-08T00:00:00+00:00 In this article screencast, we’ll demonstrate how to create a custom Linux OS image using the OpenEmbedded build system. This demonstration builds on the earlier article about using the BEC OE build template</a>. The OpenEmbedded build system is similar to Linux distributions in that you can select from a wide array of components to install. One of the big differences is you can select these components when building the OS image, instead of after you have installed the “standard” image. This screencast</a> demonstrates how to set up a custom image (can be viewed with Firefox).</p> A related article</a> also covers this procedure.</p> In summary, the steps include:</p> Create an image recipe (typically in your meta data overlay). An example is located here</a>. An existing image can be referenced for simplicity.</li> Figure out what packages need to be added. This can be done by browsing the recipe tree</a> to see what is available.</li> Bitbake the recipe.</li> Look at what packages were generated. In this example they are found in the build/angstrom-2008.1/tmp/deploy/glibc/ipk/armv7a</em> directory.</li> Add the relevant package names to your image recipe.</li> </ol> As the OpenEmbedded project includes over 6000 recipes, it gives you a big head start in including standard components in your Embedded Linux build. Leveraging these existing components is why we use Linux, and having a way to easily build and deploy these components to your target hardware is what OpenEmbedded is all about.</p> OpenEmbedded Build Template 2010-06-05T00:00:00+00:00 Setting up an OpenEmbedded build is a fairly simple process if you carefully follow the instructions</a>. There are also a number of scripts available that automate the setup such as the OpenEmbedded Tools for Eclipse</a>, the Angstrom setup scripts</a>, the KaeilOS Openembedded Manager</a>, and I’m sure there are many more (feel free to add in comments section). As we have helped a number of clients use OpenEmbedded in commercial projects, we have the following requirements for a build template:</p> The build needs to be completely automated such that only one step is required to build images.</li> We need to be able to lock down versions of various components.</li> Needs to be as simple as possible.</li> </ol> As we continue to think about creating a build template, the primary reason for creating one is to get repeatable builds with very little effort. The build template serves as documentation; how did we build the 1.10 release. A good build system allows any developer to create any release on about any machine at any time. A good build system does not depend on the “golden build machine.” We have been using makefile wrappers around OpenEmbedded for several years now to accomplish much of this. These wrappers were typically stored in a Subversion repository with the rest of the project files. However, now that most open source projects have moved to Git repositories, there are significant advantages to using Git for the build system. Git has a feature called submodules which allows you to embed a git repository inside a source tree, always pointed at a particular commit. In concept, this is perfect. In practice, Git submodules are sometimes difficult to use, but considering the functionality provided, I think it is a reasonable trade-off.</p> The use of Git encourages a different development paradigm. Typically when people use Subversion within a company, there is one repository that stores everything. Subversion is actually quite good at this. I’ve run into a number of companies who have repositories many GB in size that have been running for many years. The stability is remarkable. Git does not scale in this way, so people tend to set up a separate repository for each component, application, etc. While there may be some disadvantages to this model, there are also significant advantages, and we can learn some of these by observing Open Source Software (OSS) development. Just as we strive for modularity in software development where each class can stand on its own and be generally useful, OSS maintains this dynamic at the project level. Each OSS project moves at its own pace, largely decoupled from other projects. This dynamic forces a level of organization that might be neglected otherwise. Likewise, in any project, if each major component is stored in its own source code repository, there will also tend to be less coupling, willy-nilly including of header files from other components, etc. It helps people think in a more modular manner at the component level. A component may be the Linux kernel, bootloader, each application in the system, etc. A similar concept may be thinking about storing documents in hierarchical filesystem directory structure versus documents in a Wiki, which are stored in a flat structure and then organized by cross linking. Yes, there are times to use a directory structure, but flexibility of a linked organization system (the essence of the Internet) is hard to beat. Git submodules (or similar concepts) provides a similar linking method of organization.</p> With this in mind, we look at a build system as a way to collect a number of independent components and build them in a repeatable fashion. While this may seem obvious to anyone somewhat familiar with OpenEmbedded, our vision is to extend this concept to easily facilitate product development so that we can move quickly and efficiently, but yet in a controlled and reliable way. The first step is a new build template that is available here</a>. The following video</a> provides a short overview of how to use the build template (can be viewed in Firefox).</p> </video></p> The functionality provided so far is fairly simple and provides a way to quickly set up a build environment. With Git submodules, the versions of Bitbake and the OpenEmbedded are locked down. Submodules are also used to provide an array of components to choose from, and the user initializes only the submodules they are interested in using. The first example of this is providing both the recipe meta data from the OpenEmbedded project or the modified version from Gumstix. In the future, examples will be provided for doing custom application and kernel development using the OE srctree and gitver classes. Feedback is welcome; please let me know if you have questions or suggestions.</p> MeeGo Review 2010-06-02T00:00:00+00:00 As we evaluate various technologies that might be applicable in embedded systems, MeeGo</a> is the subject of this article. MeeGo is a collaboration between Intel and Nokia, and is replacing the Moblin and Maemo efforts. For this review, MeeGo was installed to a USB flash disk and booted on a Asus EEPC. This was quite trivial to do, and the instructions on the MeeGo website are very good</a>. The following video</a> (can be viewed in Firefox) provides a quick overview of the MeeGo Netbook UI.</p> </video></p> Overall, MeeGo looks interesting. Hopefully with the collaboration between Intel and Nokia, there will emerge a number of components that become “standards” for embedded systems such as ways to manage wireless networks, cell phone radios, etc. It would also be nice to have better options for implementing GUI’s on embedded devices with small screens. While the MeeGo user interface is nice, it is not a radical departure from desktop applications. Maemo (used on the Nokia N900) is more advanced in that it has optimized widgets for a small screen. With Maemo, all applications run in a true full screen mode, which is generally desired for devices with smaller screens, or industrial devices where we want to keep user confusion to a minimum. MeeGo applications seem to run full screen, but you can still drag them around, and some applications like the terminal do not start full screen.</p> It seems there is considerable interest in providing distributions based on MeeGo. Novell has announced</a> it plans to provide MeeGo based distributions for netbooks, and Intel claims</a> there are a number of other companies planning to build on top of MeeGo.</p> Gumstix Overo review 2010-02-08T00:00:00+00:00 Based on the interest and number of embedded modules currently available, it appears that the OMAP3 CPU from TI will be very popular in the general purpose embedded Linux market. One of the OMAP3 modules available is the Overo from Gumstix</a>. As the company name suggests, this module looks about like a stick of gum, but smaller, as shown in the photo below. The Gumstix module provides a lot of functionality in a very small package. It is reasonably priced, and is an excellent way to quickly create a product that has advanced functionality such as a high powered CPU (OMAP3), high speed USB, DSP, 3-D graphics acceleration, etc. For an example of the OMAP3 performance compared to previous generations of ARM processors, see this article</a>.</p> </p> The module contains all the core components, and can then be attached to a custom baseboard using the two high density connectors shown in the above photo. There are 70 signals on each of the two connectors. BEC provides a spreadsheet</a> of the Overo signals to assist in designing a custom baseboard.</p> Due to the high level of component integration, the Overo has very few components on the module. As shown in the below photo, there are really only 3 major visible components. The flash and RAM is stacked on top of the CPU. The PMIC contains all the required power managment circuitry, as well as Audio and USB interfacing functionality.</p> </p> The Overo module must be used with a baseboard. Gumstix provides several development baseboards. For production, a custom baseboard is typically created. The Gumstix baseboards are reasonable cost, so they could be used for prototyping or low volume production if they have the required functionality. Below is a photo of the Summit baseboard that provides DVI video out, audio, USB host and OTG, and an expansion connector with a number of signals.</p> </p> Below is the Palo baseboard from Gumstix that can be used to interface with a LCD display. The Palo board contains a resistive touch screen controller, and circuitry required to interface with a LCD.</p> </p> </p> The Overo does not include an Ethernet controller, so you typically use a USB-Ethernet adapter for development. Below shows a typical development setup.</p> </p> One of the things about the OMAP3 that makes it very attractive for embedded development is the amount of software support, and the number of development and production solutions available. The OMAP3 is very well supported by the OpenEmbedded project, and TI is very active in making contributions to various open source projects. This greatly increases the quality and availability of advanced software functionality needed to support a complex system like the OMAP3.</p> </p> </p> There are many options for software development with the OMAP3 systems. Below is a photo of a very simple Qt application. GTK+ and Enlightenment are other popular GUI toolkits. With 256MB of both flash and RAM, this is more than enough memory for most embedded applications, and provides plenty of headroom for adding features for future product revisions. The Overo has the CPU processing capability to run advanced GUI applications with 3-D affects, as well as advanced web application frameworks like web2py that previous generations of ARM CPUs could not effectively run.</p> </p> The Gumstix Overo provides an excellent value for developing advanced products. Especially for low volume products, when you compare the effort and time required to develop a full custom OMAP3 solution versus a simple Overo baseboard, using a module can provide significant advantages in terms of time to market, and development cost.</p> The Go language for embedded systems 2010-01-06T00:00:00+00:00 As one of the things I do is evaluate new technologies for embedded systems, the Go language</a> from Google is an interesting development. I have been looking for a better “system” language for some time. What I mean by a better system language is one that is nearly as efficient as C, does not require a large runtime, has a fast start-up time, and yet supports modern language features. It is interesting that the constraints for very high performance server applications are often similar to those found in embedded systems in that applications must make very efficient use of memory and CPU cycles. This is why C++ is still used a lot by Google and is also popular in embedded systems, but is not as common in applications where CPU power is plentiful compared to the resources to implement the application such as business applications. One area of divergence is the focus on multi-core for high end server applications, but in the next few years, multi-core CPU’s such as the Atom and ARM Cortex-A9 will likely be common in low power embedded systems.</p> There are other options. Vala</a> is very nice to use, is efficient, offers advanced language features, and extensive language bindings. Vala is being used to implement some of the http://freesmartphone.org</a> software stack. C/C++ are the old standby languages, but are rather tedious to program in compared to some of the newer languages in that source and header files are separate, and memory management is manual.</p> My interest in Go at this time is for ARM systems, so I ran a few experiments. From what I can tell, there does exist an ARM Go compiler that is able to produce ARM code, but it currently lacks support for EABI soft floating point</a>. The lack of floating point support is pretty much a show stopper for now, but it is at least a start. The following is an example:</p> export GOARCH=arm cd $GOROOT/src/ ./make-arm.bash cd (create the following source file: hello.go) ================== package main import "fmt" func main() { fmt.Printf("Hello, world\n"); } ================= 5g hello.go (5g is the ARM compiler) 5l hello.5 scp 5.out root@n900: (copy to ARM system) ssh root@n900 ?:~# ./5.out Hello, world (modify with floating point code) ================== package main import "fmt" func main() { var f = 1.5; fmt.Printf("Hello, world\n"); fmt.Printf("float f = %f\n", f); } ================= (now run on ARM system) ?:~# ./5.out Segmentation fault </code></pre> As, you can see, floating point does not work as expected. I’m not sure yet if gogcc can be configured to compile ARM binaries. The other area where Go appears to need a lot of work is bindings to various libraries. Though it does not appear to be ready for embedded ARM systems at this time, Go will be an interesting language to watch over the next few years.</p> OpenEmbedded development activity 2009-12-29T00:00:00+00:00 Ever since I have been sending out weekly change logs</a>, I have been impressed by the consistent amount of development activity in the OpenEmbedded project. Every week there are consistently over a dozen developers making changes. Developers come and go, but the contribution level always seems healthy. While this amount of development leads to some amount of churn and issues, this amount of development is also required to keep pace with all the new developments in the OSS space. Not having to wait 2 years for the next big release is one of the big advantages of using Linux and OpenEmbedded. Having access to the latest technology provides a competitive advantage for many products, and makes dealing with the occasional issues that come up an acceptable trade-off. Coupled with a flexible and consistent build system, OpenEmbedded is the vehicle to build advanced products. Yes, its hard. Yes, you better budget for some system software development time. Yes, you should get some help if you’re not an embedded Linux expert. Yes, you can start with some canned vendor “BSP” that seems to work and is seemingly the low-cost way to go, but eventually you’ll hit a brick wall where you need some bit of functionality that is not there, and then you’ll start up the exponential curve of dumping lots of time into trying to make something work with little progress forward. It is much better to count the cost up front, and do things right.</p> OMAP3 Resume Timing 2009-12-07T00:00:00+00:00 One of the most common power management modes for ARM processors is the suspend mode. In this mode, peripherals are shut down when possible, the SDRAM is put into self-refresh, and the CPU is placed in a low power mode. A useful bit of information is to know how soon the system can respond to a resume (wake) event.</p> It turns out the answer to this question depends on where you want to do the work. In the first test, I created a simple application that continuously toggled a GPIO. This is an easy way to tell determine with a scope when the application was running. I then measured the time between the wake event, and when this application signal started toggling. It was a consistent 180ms.</p> For the next test I simply toggled a gpio in the omap3_pm_suspend()</em> first thing after resume. This tells me roughly how fast I can execute code in the kernel after a wake event. This turned out to be 250uS.</p> So the good news is we can run kernel code very quickly after a resume event (250uS), but it currently takes a long time until user space processes start running again (180mS). The 180ms can likely be reduced with some work, but this at least gives us a baseline. 180ms is fairly quick from a human perspective (seems pretty much instant), but for industrial devices that are responding to real-world events, then 180mS can be a long time.</p> Linux kernel stats, and long term advantages 2009-12-01T00:00:00+00:00 I just read an interesting interview</a> with Greg Kroah Hartman. According to Greg, we add 11,000 lines, remove 5500 lines, and modify 2200 lines every single day. This rate of change is something that few organizations have the resources to match. It is interesting that Google chose to use the Linux kernel in the Android project even though they implemented most other parts of the system. Another interesting thing to consider is the consequences of the Linux kernel license, which requires many drivers to be released as GPL. While initially, this may be viewed as a disadvantage, it leads to some interesting long-term implications. The obvious effect is that most Linux driver code is maintained in the mainline kernel source. This leads to other effects. One is that things are allowed to change, and get better. The USB stack in both Linux and Windows has been re-written several times, but the difference is that in Linux the old software does not need to be maintained, as most drivers are maintained in the kernel, and can simply be modified along with core changes. Another advantage is that drivers tend to be much simpler as common code is merged. As a result, a Linux driver tends to be about 1/3 the size of a driver in other operating systems. Yes, getting driver code accepted in the mainline kernel can be difficult at times, but it is clearly the best long-term option.</p> Linux PM: OMAP3 Suspend Support 2009-11-23T00:00:00+00:00 This article provides an overview of the Linux kernel support for the suspend state in the TI OMAP3. Power management has always been one of the more difficult parts of a system to get right. The OMAP3 power management is quite extensive. There are many levels of very granular control over the entire system. Initially, we are going to focus on a simple power state where we want to put the CPU in a low power mode (basically not doing anything), but resume very quickly once an event occurs and continue running applications where they left off. This is typically called “Suspend”.</p> Power States</h2> We must first match up terminology between the Linux kernel and the OMAP documentation. The Linux kernel supports several suspend power states:</p> PM_SUSPEND_ON</strong>: system is running</li> PM_SUSPEND_STANDBY</strong>: system is in a standby state. This is typically implemented where the CPU is in a somewhat static state. After all the peripherals have been shut down, and the memory put into self refresh, the CPU executes an instruction that puts the CPU into a low power state. After the next wake-event occurs, the CPU resumes operating at the next instruction. The CPU state is static in this case.</li> PM_SUSPEND_MEM</strong>: This goes one step beyond STANDBY in that the CPU is completely powered down and loses its state. Any CPU state that needs to be saved is stored to SDRAM (which is in self refresh), or some other static memory. When the system resumes, it has to boot through the reset vector. The bootloader determines that the system was sleeping, and re-initializes the system, enables the SDRAM, restores whatever state is needed, and then jumps to a vector in the kernel to finish the resume sequence. This mode is fairly difficult to implement as it involves the bootloader, and a lot of complex initialization code that is difficult to debug.</li> PM_SUSPEND_MAX</strong>: From browsing the kernel source code, it appears this state is used for “Off” or perhaps suspend to disk.</li> </ul> The OMAP3 documentation uses the following terms:</p> SLM</strong>: Static Leakage Management. Includes “suspend” functionality.</li> Standby</strong>: OMAP3x retains internal memory and logic. (uses 7mW)</li> Device Off</strong>: System state is saved to external memory(0.590mW). This could theoretically map to the PM_SUSPEND_MEM state.</li> WFI</strong>: Wait for Interrupt. This is an ARM instruction that puts the CPU in the standby state.</li> </ul> Based on the source code in the Linux kernel, there is currently only support for the Standby state:</p> static int omap3_pm_enter(suspend_state_t unused) { int ret = 0; switch (suspend_state) { case PM_SUSPEND_STANDBY: case PM_SUSPEND_MEM: ret = omap3_pm_suspend(); break; default: ret = -EINVAL; } return ret; } </code></pre> What kernel branches/versions support PM</h2> Kevin Hilman has been maintaining an OMAP3 power management branch at http://git.kernel.org/?p=linux/kernel/git/khilman/linux-omap-pm.git</a>. Bits are being merged upstream. Some of the things that don’t seem to be merged yet are SmartReflex support, and a number of other details. But, it seems that basic suspend/resume support is available in the upcoming 2.6.32 kernel.</p> More Details on the Standby Mechanism</h2> Due to the granularity and complexity of the the OMAP3 power management, relevant pieces of documentation are scattered throughout the OMAP35x Technical Reference Manual. The following were helpful:</p> Table 3-17: details the MPU subsystem operation power modes</li> Table 3-18: Power Mode Allowable Transitions. Standby mode can enter from active mode only by executing the wait for interruption (WFI) instruction.</li> SDRC, “Self Refresh Management”. The OMAP3 can automatically put the SDRAM in self refresh mode when the CPU enters the idle state.</li> MPU Power Mode Transistions: The ARM core initiates entering into standby via software only (CP15 - WFI).</li> </ul> The omap34xx_cpu_suspend</em> function is called to put the CPU into standby mode. The actual instruction that stops the CPU is the ARM WFI (wait for interrupt) instruction. This is a little different than previous generations of ARM CPUs that used coprocessor registers.</p> Testing</h2> With the 2.6.32-rcX kernels, suspend/resume appears to work at least from the console:</p> root@overo:~# echo mem > /sys/power/state PM: Syncing filesystems ... done. Freezing user space processes ... (elapsed 0.00 seconds) done. Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done. Suspending console(s) (use no_console_suspend to debug) <after key press on serial console ...> omapfb omapfb: timeout waiting for FRAME DONE CLIFF: going into suspend Wake up daisy chain activation failed. CLIFF: returning from suspend Successfully put all powerdomains to target state Restarting tasks ... done. root@overo:~# </code></pre> The CLIFF messages are debug statements placed in the low-level suspend code to make sure we were getting to that point. The console is disabled early in the suspend process, so we don’t get the debug messages until after resume. The no_console_suspend</em> kernel option can be used to leave the console enabled during suspend, but that feature does not seem to work.</p> Summary/Future work</h2> This covers the basics of Linux Suspend/Resume support for the OMAP3 CPU. There are many more options and details such as configuring wake sources, dynamic power management while running, etc.</p> Integrated CAN solutions for Linux 2009-11-19T00:00:00+00:00 I just received an email notification from EMS (http://ems-wuensche.com/</a>) that support for their CAN controllers is now in mainline Linux kernels. The EMS PCI products are supported in 2.6.31, and the CPC-USB product will be supported in 2.6.32. I’ve used various Linux CAN stacks in the past, but none were as well integrated as the SocketCAN</a> solution that has been merged into the mainline Linux kernel. Though I’ve not used the EMS products yet, the fact that support is in the mainline kernel source provides a good indication that they know how to do things correctly, and that the software is of good quality.</p> The Microchip MCP2515 is another option for interfacing CAN with a Linux system, but uses the SPI bus so it typically requires a custom hardware design. The availability of a USB adapter provides an easy way for CAN to be used with about any existing Linux system including small embedded systems. From looking at the kernel source code, it appears that EMS solutions use the SJA1000 CAN controller, which has been an industry standard controller for many years.</p> Message buffering is an important consideration when selecting a Linux CAN solution. Often in a system, the Linux computer is responsible for data processing, user interface, and other CPU intensive operations. The real-time control is often relegated to a separate microcontroller. The CAN bus is then just a convenient and robust way to get data from the control part of the system to the data processing part. The amount of hardware buffering in the CAN controller determines how quickly the system needs to respond to process the CAN data, so the CAN controller has space to receive new data. The SJA1000 provides a 64-byte receive fifo, or roughly space for 4 CAN messages. This provides a little more buffering than the 2 messages in the MCP2515. Dealing with full bandwidth CAN data in a Linux system can be a challenge with small amounts of buffering as you typically have to implement the Linux RT patch, and spend some time tuning it. Another upcoming CAN solution that looks nice is the TI High End CAN Controller (HECC) found in the new Sitara CPUs. This controller provides 32 hardware mailboxes which should provide even more buffering, which makes it even better suited for operating systems like Linux. As with any technology, there are tradeoffs, so its nice to have options.</p> One area the EMS solution may be very useful is for prototyping and development. The ideal development flow when developing an Embedded Linux product is to development as much as possible on a PC. The EMS product allows you to easily implement SocketCAN functionality on a PC so you can talk to the rest of your system directly from a PC. The final embedded solution may be based on a more cost effective solution like the MCP2515 or the TI HECC, but being able to interface to the rest of the system directly from a PC has the potential to improve development efficiency. This is part of the “Big Win” with Embedded Linux where the same technology scales from the embedded system, to the PC, to the server.</p> Notification at the end of builds 2009-11-05T00:00:00+00:00 I do quite a few OpenEmbedded project builds during the course of a week. This process usually takes 3-5 minutes. That is just enough time to get distracted doing something else and forget about the build until an hour later when you realize – oops, I was supposed to send out a release email once the build was finished and uploaded. It occured to me that it would be nice if my computer played a distinct sound at the end of the build. With Linux this is incredibly easy:</p> wget http://upload.wikimedia.org/wikipedia/commons/7/76/Ding_Dong_Bell.ogg</li> sudo mv Ding_Dong_Bell.ogg /usr/local/</li> sudo aptitude install cplay</li> sudo echo “cplay /usr/local/Ding_Dong_Bell.ogg” > /usr/local/bin/bell</li> sudo chmod 775 /usr/local/bin/bell</li> </ul> Now, when I have a long build, I simply do something like:</p> bitbake my-image; bell </code></pre> There is probably something better than cplay that does not start a UI – any suggestions?</p> A quick serial logging application 2009-10-21T00:00:00+00:00 Recently when debugging a device connected to a rs485 bus, I needed a simple application to dump the raw data coming over the rs485 bus. Minicom does all kinds of weird terminal stuff, plus it will not display binary data, so that was out. While looking for serial analyzer programs for Linux, and pondering writing my own simple C application, it occurred to me to use pyserial. With a few lines of code, I got exactly what I needed:</p> import serial import sys import struct if len(sys.argv) != 3: print "Usage: serial-dump.py <serial port> <baud rate>" sys.exit(-1) print "Dumping data on %s" % (sys.argv[1]) ser = serial.Serial(sys.argv[1], sys.argv[2], timeout=0.3) while(1): buf = ser.read(500) if len(buf) > 0: print "Read %i bytes:" % (len(buf)), for i in buf: value = struct.unpack('B', i)[0] print "%02x" % (value), print "" </code></pre> The full source is available here</a>. This could obviously be improved to do true blocking reads, automatically parse protocols, implement a gui, etc. Ideas for efficiently reading data using pyserial is available in a previous post</a>. Another example of using a scripting language for quick tasks is /226/a-really-nice-hex-calculator/</a>.</p> Apache and how to correctly use NameVirtualHost 2009-10-16T00:00:00+00:00 As I often get involved in server administration (SVN, git, redmine, etc setup), I deal with Apache on a regular basis. For simple configurations, the default Ubuntu/debian config works well. However, for a more complex setup with virtual hosts, multiple IP addresses, and SSL support, it is common to run into the following message:</p> mixing * ports and non-* ports with a NameVirtualHost address is not supported, proceeding with undefined results</em></p> Googling does not return a clear explanation to this problem, so that is why I’m writing this article. The impulsive approach of randomly changing things does not get you too far – especially with the Debian/Ubuntu Apache configuration that is composed of seperate files for each virtual host. After reading the documentation, things become clearer:</p> Note that the argument to the <VirtualHost></code> directive must exactly match the argument to the NameVirtualHost</code> directive.</em></p> And, by considering the fact that the seperate virtual host config files really just get concatenated into one big file, it makes sense that you can’t have more than one NameVirtualHost entry. Ubuntu by default puts the NameVirtualHost in the default virtual host file, which gets soft linked to sites-enabled/000-default. I presume the ‘000’ means it gets loaded first. I find it much more intuitive to put the NameVitualHost directives in the /etc/apache2/httpd.conf file. This way it is clear that these entries are common to all virtual host entries. So what I end up with is:</p> /etc/apache2/httpd.conf</p> NameVirtualHost 74.208.184.45:80 NameVirtualHost 74.208.184.46:80 NameVirtualHost 74.208.184.45:443 </code></pre> /etc/apache2/sites-available/bec-systems.com</p> <VirtualHost 74.208.184.45:80> ServerName bec-systems.com ... </code></pre> /etc/apach2/sites-available/bec-systems.com.ssl</p> <VirtualHost 74.208.184.45:443> ServerName bec-systems.com ... </code></pre> /etc/apach2/sites-available/svn.bec-systems.com</p> <VirtualHost 74.208.184.45:80> ServerName svn.bec-systems.com ... </code></pre> </code></pre> /etc/apach2/sites-available/site2.com (with different IP address)</p> <VirtualHost 74.208.184.46:80> ServerName site2.com ... </code></pre> Again, the points to remember are:</p> The argument to VirtualHost must match a NameVirtualHost directive</li> the NameVirtualHost directive is common, so there should only be one entry</li> </ol> Best practices for kernel development with OpenEmbedded 2009-10-13T00:00:00+00:00 A common question is how do you do kernel development with OpenEmbedded? Typically, OpenEmbedded builds a kernel by checking the source out of a git repository, or by applying patch files to a released version of the kernel. See the many recipes</a> for examples. This works very well for a Linux distribution build system, but is less than ideal for ongoing kernel development where you will have many iterations over a period of time. The OE work directory is considered transient, and should not be used for any ongoing development. Fortunately, the kernel build system is self contained, so it is a very simple matter to reference the toolchain built by OpenEmbedded and work outside the OpenEmbedded build system.</p> Use Git</h2> Trust me on this one, just use it! It will make your life easier. There is a learning curve to any powerful tool. Another way to state this is if a tool does not do much, then it would be easy to learn. So expect to spend a little time learning to use git. With modern Linux kernel development, much of your time is spent integrating work that others are doing. For example, if you are using the OMAP3 CPU, you might want to start with linux-omap branch, integrate some patches from the OE beagleboard build, merge the OMAP PM tree, and then merge some bits from the Linux-wireless tree. This is the way embedded Linux development is done today. Git makes it possible to have many parallel developments going at their own pace, including your own. Even in your own development, you will find significant advantages to being able to quickly create and merge branches, revise commits, etc. There are several articles about git</a> on this site.</p> Importing an OE kernel into git</h2> If you are starting with an existing OE kernel tree, then you typically need to set up an equivalent tree in your local git workspace. Typically you just look at the recipe to see how to manually reconstruct it. For example, lets consider the SRC_URI from the palm-omap1</a> recipe:</p> SRC_URI = "${KERNELORG_MIRROR}/pub/linux/kernel/v2.6/linux-2.6.22.tar.bz2 \ http://www.muru.com/linux/omap/patches/patch-2.6.22-omap1.bz2;patch=1 \ file://defconfig" </code></pre> To set this kernel up in git, we might do the following:</p> git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git</li> cd linux-2.6</li> git checkout -b my_branch v2.6.22</li> wget http://www.muru.com/linux/omap/patches/patch-2.6.22-omap1.bz2</code></li> bzip2 -d patch-2.6.22-omap1.bz2</li> git am patch-2.6.22-omap1</li> </ol> If the git am fails, then you might need to add a subject and author line to the patch in the following form and then retry git am.</p> From: Name <name@company.com> Subject: omap1 patch </code></pre> Now when you do a git log, you will see the above patch applied. git apply can also be used to apply patches, but I prefer git-am as it automatically handles new files and fills in the commit message and author.</p> Setting up the toolchain</h2> Now that you have a kernel, you must set up a toolchain so you can build it. I typically set up my kernel directory next to my OE build directory in my project space, so I can then source the following script</a> to set up the toolchain:</p> CROSS_COMPILER_PATH=`cd ../oe/build/angstrom-2008.1/tmp/cross/armv7a/bin; pwd` BUILD_ARCH=`uname -m` OE_STAGING_PATH=`cd ../oe/build/angstrom-2008.1/tmp/staging/${BUILD_ARCH}-linux/usr/bin; pwd` STAGING_KERNEL_DIR=`cd linux-2.6; pwd` export PATH=$PATH:$CROSS_COMPILER_PATH:$OE_STAGING_PATH export ARCH=arm export CROSS_COMPILE=arm-angstrom-linux-gnueabi- </code></pre> Adjust to whatever you are building for. Now you can simply type make. The reason the OE_STAGING_PATH is added to the PATH in the above example is so that the uboot mkimage utility can be used directly from the OE build if your build target is make uImage.</p> Installing modules</h2> Sometimes you need to re-install all modules on the target because you changed kernel version, etc. Typically OE packages up all modules and adds the one you specify to the rootfs. Because we are building the kernel outside the OE directory, OE can no longer do this. However, its still very easy to install modules from your kernel development directory:</p> rm -rf modules_install; INSTALL_MOD_PATH=modules_install make modules_install rsync -av modules_install/lib/modules/2.6.22 root@192.168.1.115:/lib/modules/ </code></pre> The first command installs the modules to a temporary directory, and the second command rsync’s the modules to your target system.</p> Creating a kernel recipe with your changes</h2> Once you get to the point where you want to create a recipe with your changes, you can easily export a patch, or series of patches using git diff or git format-patch. Simply add the patch files to your kernel recipe SRC_URI. You can also teach bitbake to fetch your kernel source directly from a git repository, but I’ve generally found the patch mechanism to be adequate and easier in most cases. As an example, you can create one large patch with all your changes that can be applied on the 2.6.22 released kernel:</p> git diff HEAD..v2.6.22 > my_kernel_changes.patch </code></pre> Other ways …</h2> This is just one flow and there are many variations. There is also a new mechanism in OpenEmbedded called srctree that allows you to use OpenEmbedded in an external source tree. srctree has the potential to further streamline this type of development.</p> As with any task, you want to use the tools that make sense for the task. OpenEmbedded makes sense for building Linux distributions, but it is not a kernel development tool. Git and the kernel build system make sense for kernel development.</p> converting digital camera movies to ogg format 2009-09-09T00:00:00+00:00 Now that firefox (v3.5) has built in support for ogg videos, ogg is now the most convenient format for “easy to view” video files (assuming you don’t want to use a service like youtube). Some digital cameras (like my Canon) records movies in AVI format. I created a script that uses gstreamer to convert the AVI file to an ogg file for easy viewing.</p> http://dev.bec-systems.com/svn/pub/avi_to_ogg/avi_to_ogg</a></p> And some of the results:</p> /video/</a></p> (hint, download firefox 3.5 to view these videos)</p> Best practices for building Gtk+ applications with OpenEmbedded 2009-09-08T00:00:00+00:00 I recentlywrote an article about best practices building Qt applications with OpenEmbedded</a>, and it occured to me that I should write an equivalent article for Gtk+ applications. The same points apply – put your application source in a SCM system, and put the install logic in the application source (read the above article). The difference is that Gtk applications typically use autotools where Qt applications use qmake to build the application. This article details how a minimal GTK+ application should be set up and built using OpenEmbedded.</p> Application Source</h2> I created a sample GTK hello application located at: http://dev.bec-systems.com/svn/pub/gtk_hello</a>. This is a SVN repository, so you can simply “svn co” the above URI to check out the code. If you are running Ubuntu, you can install the necessary tools to build a native Gtk+ application by:</p> sudo apt-get install libgtk2.0-dev build-essential autoconf automake pkg-config</li> </ul> To build on your x86 Linux PC, run the following steps:</p> svn co http://dev.bec-systems.com/svn/pub/gtk_hello</a></li> cd gtk_hello</li> autoreconf -i</li> ./configure --prefix=`pwd`/install</li> make install</li> </ul> This compiles and installs the application in to the “install” directory. If you look in this directory, you will notice the application binary is installed in the “install/bin” directory. Typically, the install directory is set to /usr/bin, but in this example we set it to install so we don’t need to run “make install” as root, but yet we can verify the install logic works properly.</p> OpenEmbedded Recipe</h2> Now that you have verified the application builds and installs properly on a x86 PC, it is trivial to build the application in OpenEmbedded. Create a recipe in your OE recipes directory named gtk-hello_svn.bb</em> with the following contents:</p> DESCRIPTION = "Sample Gtk+ Hello application, used to demonstrate build system" AUTHOR = "Cliff Brake <cbrake@bec-systems.com>" SRCREV = "17" PV = "0.0+svn${SRCREV}" PR = "r0" DEPENDS = "gtk+" SRC_URI = "svn://dev.bec-systems.com/svn/pub;module=gtk_hello;proto=http;rev=17" S = "${WORKDIR}/gtk_hello/" inherit autotools </code></pre> Now, run: bitbake gtk-hello. That is it! Building Linux applications is easy if you simply use the tools, whether it be autotools, qmake, etc. Too often there is the tendency to set up your own compile steps with ${CC} variables, etc. While this seems to be the simple approach at first glance (autotools is too hard), it quickly becomes unmaintainable and in the end is a lot more work than simply learning the basics of the industry standard tools. See previous autotools article</a> for more information.</p> Best practices for building Qt applications with OpenEmbedded 2009-08-06T00:00:00+00:00 This article describes how to cross compile a Qt application (named qt_tutorial) with OpenEmbedded, and several best practices you should consider. OpenEmbedded currently includes fairly good support for building Qt</a> – both Qt Embedded and Qt X11. OE also includes a number of qt classes</a> that make building Qt applications easy. One of the main considerations with embedded Linux application development is that you keep the build system flexible so that you can easily build on a PC or for your embedded hardware. This means that hand crafted Makefiles with hardcoded paths to cross compilers do not quality.</p> Put Your Application in a SCM system</h2> No matter what type of application you are building, it is a good idea to put your source in a SCM or revision control system and fetch it directly using OE. OpenEmbedded supports fetching sources from a number of revision control systems including Subversion, Git, Bazaar, Mercurial, CVS, etc. The not so obvious reason we do this is so you can easily check out the source code and build it on a PC as well as your target system in OE. In this example we fetch the application source from a SVN repository:</p> SRC_URI = "svn://dev.bec-systems.com/svn/pub;module=qt_tutorial;proto=http" </code></pre> For the above repository, the direct URI to the source would be: http://dev.bec-systems.com/svn/pub/qt_tutorial/</a></p> Put the Install logic in the Application Source</h2> Most Linux applications support “make install”, and this is the case with autotools, and qmake (Qt’s build tool). We could put logic in the OE recipe to install the application something like the following:</p> do_install() { install -d ${D}/${bindir} install -m 0755 ${S}/qt_tutorial ${D}/${bindir} } </code></pre> But, the problem with this approach is you can’t install the application in other environments (like a native x86 PC build) unless you are building with OE. So a better approach is to put the logic to install the application in the application source, so that in can be installed in both your PC environment, and your OpenEmbedded build. To accomplish this, you can set up the project as follows</p> application qmake project file (qt_tutorial.pro):</strong></p> TEMPLATE = app TARGET = qt_tutorial DEPENDPATH += . INCLUDEPATH += . # Input HEADERS += lcdrange.h SOURCES += lcdrange.cpp main.cpp target.path = /usr/bin INSTALLS += target </code></pre> OpenEmbedded recipe (qt-tutorial_svn.bb):</strong></p> inherit qt4e SRCREV = "${AUTOREV}" PV = "1.0+svnr${SRCREV}" PR = "r1" SRC_URI = "svn://dev.bec-systems.com/svn/pub;module=qt_tutorial;proto=http" S = ${WORKDIR}/qt_tutorial FILES_${PN}-dbg += "${bindir}/.debug" do_install() { export INSTALL_ROOT=${D} make install } </code></pre> Now, the same mechanism is used to install the application on both a PC native build, as well an OpenEmbedded build. If you look in the Makefile generated by qmake, you see the following:</p> install_target: first FORCE @$(CHK_DIR_EXISTS) $(INSTALL_ROOT)/usr/bin/ \|\| $(MKDIR) $(INSTALL_ROOT)/usr/bin/ -$(INSTALL_PROGRAM) "$(QMAKE_TARGET)" "$(INSTALL_ROOT)/usr/bin/$(QMAKE_TARGET)" install: install_target FORCE </code></pre> INSTALL_ROOT can be set to force the entire system to be installed in a subdirectory. This is required for build systems that generate packages, like OpenEmbedded. To build this example, put the qt-tutorial_svn.bb file in your OE recipes tree, and run: bitbake qt-tutorial. This will fetch the source code and build a package. To run the tutorial, install the package on a system that includes Qt Embedded, and then run: qt_tutorial -qws.</p> Gumstix Overo Pinout Spreadsheet updated 2009-07-30T00:00:00+00:00 The Gumstix Overo Pinout Spreadsheet</a> has been updated with the Palo board connector pinouts, and a few mistakes have been fixed.</p> </p> </p> How to capture source changes to an OpenEmbedded package 2009-07-14T00:00:00+00:00 One task that is a often confusing to new OpenEmbedded users is how to capture changes to the source code for a package/recipe. First, lets review the progression in tools use to capture source code changes.</p> Figure 1: Progression of tools</figcaption></figure> In days of yore, developers often used gnu diff to capture modifications to a source tree. The typical practice was to create two source trees: a pristine, and a dirty tree. Modifications were made in the dirty tree, and periodically gnu diff was run between the two trees to capture the changes. Then came quilt which allowed you to capture changes without a pristine tree – kind of a cheap version control system. The typical flow was:</p> cd </li> quilt new my_changes.patch</li> quilt add file1.c</li> edit file1.c</li> quilt refresh (now my_changes.patch records the edits to file1.c)</li> debug</li> edit file2.c</li> (oops forget to quilt add file2.c – now what ….)</li> restore file2.c from pristine source</li> quilt add file2.c</li> edit file2.c</li> quilt refresh</li> (now my_changes.patch records edits to file1.c and file2.c)</li> </ul> As you can see, things are much better than plain diff, but you still have to keep track of the files you edit. With git, we can do essentially the same things, only git keeps track of the files you edit, plus it is easier to organize patch files. The flow with git is something like this:</p> cd </li> git init</li> git add </li> git commit (now the entire source tree is recorded by git)</li> edit file1.c</li> edit file2.c ….</li> git status (shows you files that were modified)</li> git add file1.c file2.c</li> git commit</li> git log (shows you changes you have made)</li> git format-patch HEAD~1 (outputs the patch created by the last commit)</li> </ul> Once your source code is in git, then you can do all kinds of fancy things like merging, re-ordering, and re-naming patch files with the git-rebase --interactive</a> command.</p> One of the uses of this technique is when you need to record a small change to a package source in a build system such as OpenEmbedded, and apply that change for subsequent builds. In this case you might do something like the following:</p> bitbake -c devshell <recipe_to_patch></p> </li> in devshell capture changes in git as shown above to create my_changes.patch</em></p> </li> cp my_changes.patch <oe_tree>/recipes/<recipe_to_change>/files/</p> </li> edit <oe_tree>/recipes/<recipe_to_change>/<recipe_to_change>.bb</p> tell bitbake to apply the patch when building the recipe: SRC_URI += “file://my_changes.patch;patch=1”</li> increment the PR variable so that bitbake knows to rebuild the recipe.</li> </ul> </li> </ul> Figure 2 illustrates the above steps.</p> Figure 2: Flow to capture package source changes in OpenEmbedded</figcaption></figure> While the above steps might seem overly complex, you must consider what you are trying to do. Building a Linux distribution is not a simple task. We are dealing with large amounts of source code from many different projects. Putting all this source code in one tree is not an option, because then we would be defeating the very reasons open source is attractive in the first place: flexibility, largly decoupled projects, advanced technology, fast bug fixes, many options to choose from, developer support, etc. Each open source project must remain a self contained project that is largly decoupled from other projects so it can develop at its own pace. It is for these reasons that open source technologies are used to give products a competitive edge. So, like any complex process or task, there must be some understanding of how it works, an appreciation for what it can do for you, and an investment in tools and techniques required to be effective.</p> update 2023-09-23</strong>: recipetool</code> and devtool</code> are updated tools that can be used to make and capture changes.</p> Embedded Linux versus Windows CE 2009-06-20T00:00:00+00:00 Occasionally I am asked how Embedded Linux compares with Windows CE. I have spent the past 5 years doing mostly embedded Linux development, and the previous 5 years doing mostly WinCE development with a few exceptions, so my thoughts are no doubt a little biased toward what I understand best. So take this with a grain of salt :-) In my experience, the choice is often made largely on perception and culture, rather than concrete data. And, making a choice based on concrete data is difficult when you consider the complexity of a modern OS, all the issues associated with porting it to custom hardware, and unknown future requirements. Even from an application perspective, things change over the life of a project. Requirements come and go. You find yourself doing things you never thought you would, especially if they are possible. The ubiquitous USB and network ports open a lot of possibilities – for example adding Cell modem support</a> or printer support</a>. Flash based storage makes in-field software updates</a> the standard mode of operation. And in the end, each solution has its strengths and weaknesses – there is no magic bullet that is the best in all cases.</p> When considering Embedded Linux development, I often use the iceberg analogy; what you see going into a project is the part above the water. These are the pieces your application interacts with, drivers you need to customize, the part you understand. The other 90% is under water, and herein lies a great deal of variability. Quality issues with drivers or not being able to find a driver for something you may want to support in the future can easily swamp known parts of the project. There are very few people who have a lot of experience with both WinCE and Linux solutions, hence the tendency to go with what is comfortable (or what managers are comfortable with), or what we have experience with. Below are thoughts on a number of aspects to consider:</p> SYSTEM SOFTWARE DEVELOPMENT</h2> Questions in this realm include CPU support, driver quality, in field software updates, filesystem support, driver availability, etc. One of the changes that has happened in the past two years, is CPU vendors are now porting Linux to their new chips as the first OS. Before, the OS porting was typically done by Linux software companies such as MontaVista, or community efforts. As a result, the Linux kernel now supports most mainstream embedded cpus with few additional patches. This is radically different than the situation 5 years ago. Because many people are using the same source code, issues get fixed, and often are contributed back to the mainstream source. With WinCE, the BSP/driver support tends to be more of a reference implementation, and then OEM/users take it, fix any issues, and that is where the fixes tend to stay.</p> From a system perspective, it is very important to consider flexibility for future needs. Just because it is not a requirement now does not mean it will not be a requirement in the future. Obtaining driver support for a peripheral may be nearly impossible, or be too large an effort to make it practical.</p> Most people give very little thought to the build system, or never look much beyond the thought that “if there is a nice gui wrapped around the tool, it must be easy”. OpenEmbedded is very popular way to build embedded Linux products, and has recently been endorsed as the technology base of MontaVista’s Linux 6 product, and is generally considered “hard to use” by new users. While WinCE build tools look simpler on the surface (the 10% above water), you still have the problem of what happens when I need to customize something, implement complex features such as software updates, etc. To build a production system with production grade features, you still need someone on your team who understands the OS and can work at the detail level of both the operating system, and the build system. With either WinCE or Embedded Linux, this generally means companies either need to have experienced developers in house, or hire experts to do portions of the system software development. System software development is not the same as application development, and is generally not something you want to take on with no experience unless you have a lot of time. It is quite common for companies to hire expert help for the first couple projects, and then do follow-on projects in-house. Another feature to consider is parallel build support. With quad core workstations becoming the standard, is it a big deal that a full build can be done in 1.2 hours versus 8? How flexible is the build system at pulling and building source code from various sources such as diverse revision control systems, etc.</p> Embedded processors are becoming increasingly complex. It is no longer good enough to just have the cpu running. If you consider the OMAP3 cpu family from TI, then you have to ask the following questions: are there libraries available for the 3D acceleration engine, and can I even get them without committing to millions of units per year? Is there support for the DSP bridge? What is the cost of all this? On a recent project I was involved in, a basic WinCE BSP for the Atmel AT91SAM9260 cost $7000. In terms of developer time, this is not much, but you have to also consider the on-going costs of maintenance, upgrading to new versions of the operating system, etc.</p> APPLICATION DEVELOPMENT</h2> Both Embedded Linux and WinCE support a range of application libraries and programming languages. C and C++ are well supported. Most business type applications are moving to C# in the WinCE world. Linux has Mono, which provides extensive support for .NET technologies and runs very well in embedded Linux systems. There are numerous Java development environments available for Embedded Linux. One area where you do run into differences is graphics libraries. Generally the Microsoft graphical APIs are not well supported on Linux, so if you have a large application team that are die-hard windows GUI programmers, then perhaps WinCE makes sense. However, there are many options for GUI toolkits that run on both Windows PCs and Embedded Linux devices. Some examples include GTK+, Qt, wxWidgets, etc. The Gimp is an example of a GTK+ application that runs on windows, plus there are many others. The are C# bindings to GTK+ and Qt. Another feature that seems to be coming on strong in the WinCE space is the Windows Communication Foundation (WCF). But again, there are projects to bring WCF to Mono, depending what portions you need. Embedded Linux support for scripting languages like Python is very good, and Python runs very well on 200MHz ARM processors.</p> There is often the perception that WinCE is realtime, and Linux is not. Linux realtime support is decent in the stock kernels with the CONFIG_PREEMPT option, and real-time support is excellent with the addition of a relatively small real-time patch</a>. You can easily attain sub millisecond timing with Linux. This is something that has changed in the past couple years with the merging of real-time functionality into the stock kernel.</p> DEVELOPMENT FLOW</h2> In a productive environment, most advanced embedded applications are developed and debugged on a PC, not the target hardware. Even in setups where remote debugging on a target system works well, debugging an application on a workstation works better. So the fact that one solution has nice on-target debugging, where the other does not is not really relevant. For data centric systems, it is common to have simulation modes where the application can be tested without connection to real I/O. With both Linux and WinCE applications, application programing for an embedded device is similar to programming for a PC. Embedded Linux takes this a step further. Because embedded Linux technology is the same as desktop, and server Linux technology, almost everything developed for desktop/server (including system software) is available for embedded for free. This means very complete driver support (see USB cell modem and printer examples above), robust file system support, memory management, etc. The breadth of options for Linux is astounding, but some may consider this a negative point, and would prefer a more integrated solution like Windows CE where everything comes from one place. There is a loss of flexibility, but in some cases, the tradeoff might be worth it. For an example of the number of packages that can be build for Embedded Linux systems using Openembedded, see http://cgit.openembedded.org/cgit.cgi/openembedded/tree/recipes</a>.</p> GUI TRENDS</h2> It is important to consider trends for embedded devices with small displays being driven by Cell Phones (iPhone, Palm Pre, etc). Standard GUI widgets that are common in desktop systems (dialog boxes, check boxes, pull down lists, etc) do not cut it for modern embedded systems. So, it will be important to consider support for 3D effects, and widget libraries designed to be used by touch screen devices. The Clutter library</a> is an example of this.</p> REMOTE SUPPORT</h2> Going back to the issue of debugging tools, most people stop at the scenario where the device is setting next to a workstation in the lab. But what about when you need to troubleshoot a device that is being beta-tested half-way around the world? That is where a command-line debugger like Gdb is an advantage, and not a disadvantage. And how do you connect to the device if you don’t have support for cell modems in New Zealand, or an efficient connection mechanism like ssh for shell access and transferring files?</p> SUMMARY</h2> Selecting any advanced technology is not a simple task, and is fairly difficult to do even with experience. So it is important to be asking the right questions, and looking at the decision from many angles. Hopefully this article can help in that. For additional assistance, please do not hesitate to contact BEC Systems</a> – we’re here to help.</p> Dealing with large data structures efficiently in embedded systems 2009-05-26T00:00:00+00:00 I’m currently dealing with a programming problem where I need access to several 64MB, file-backed data structures concurrently on an Embedded Linux system that only has 64MB of RAM. The data structures are fairly sparse (mostly zero data), and I typically only need to access small portions of them at any particular time. There is always the brute-force approach where you write code to manually load sections of the file as you need them. But with a little thought, the realisation hits “this is what operating systems do.” This article explores using memory mapping, and the sparse file</a> support in file systems to solve this problem in a very efficient manner.</p> Sparse File Support</h2> Most Unix file systems support sparse files</a>. This means that sections of data that is zeros is not stored on the disk. Consider the following example where we create a 200MB file that is all zeros:</p> root@cm-x270:/media/mmcblk0p1# df Filesystem 1k-blocks Used Available Use% Mounted on /dev/mmcblk0p1 3917212 202448 3515776 5% /media/mmcblk0p1 root@cm-x270:/media/mmcblk0p1# dd if=/dev/zero of=sparse-file bs=1 \ count=1 seek=200M root@cm-x270:/media/mmcblk0p1# df Filesystem 1k-blocks Used Available Use% Mounted on /dev/mmcblk0p1 3917212 202456 3515768 5% /media/mmcblk0p1 root@cm-x270:/media/mmcblk0p1# ls -l -rw-r--r-- 1 root root 209715201 May 22 08:54 sparse-file root@cm-x270:/media/mmcblk0p1# time od sparse-file 0000000 000000 000000 000000 000000 000000 000000 000000 000000 1440000000 real 0m 17.23s user 0m 14.21s sys 0m 2.66s root@cm-x270:/media/mmcblk0p1# du sparse-file 68 sparse-file </code></pre> It this case we created a 200MiB file on a SD card formatted as ext3. Even though the file is 200M in size, it only uses a few KiB of disk space. This same test also worked fine with a JFFS2 filesystem. With sparse file support, we get a cheap form of run length compression (at least for zeros) with no effort.</p> mmap()</h2> The mmap() system call is used to map a file, or portions of a file into memory. The data in the file can then be accessed directly in memory. Because Linux is a demand paged system, portions of the file are paged in as needed, so the entire file does not need to be present in RAM at one time. There are several advantages to using mmap():</p> mmap() avoids the extraneous copy that occurs when using read() or write() as the data does not need to be copied to a user space buffer.</li> there is very little overhead</li> you can directly access any part of the file without doing a lseek() and keeping track of where you are.</li> the operating system takes care of paging in sections of the file you are using, and discarding sections that are not in use when memory is low. This includes flushing dirty portions to disk.</li> </ol> What this means, is mmap gives you an easy way to work on large, file backed data structures, and the OS takes care of loading the portions you need, and saving the modified portions back to disk. As most embedded systems are 32-bit, there is a limit to the file size you can use as virtual memory space is limited.</p> Test Application</h2> Next I wrote a small application that is used to create, and modify files using mmap(). I wanted to experiment with creating files of various sizes, writing non-zero data at various intervals, and test the performance of this. The test application source is located here</a>.</p> Usage: sparse_file_mmap_test [OPTION] -s, --filesize File size to allocate -o, --offset Will write a few bytes every offset bytes -m, --modify Modify existing file -d, --data Data to write to file at offset location (0-255) </code></pre> The application creates a file of size filesize</em>, and writes the value of data</em> to the file every offset</em> bytes. There is also an option to modify existing files, so we can test the performance of opening a large file, making a small change, and then closing it.</p> Test Results</h2> The results are pretty amazing, and the performance is beyond my expectations. This type of problem is where you learn to appreciate the performance of an advanced operating system, and file system. There is a reason for the complexity! There are 3 things I wanted to test: 1) does the sparse file support work as expected? 2) can mmap be used to easily modify large files? 3) can mmap be used to work on data structures that are larger than physical RAM?</p> 1. Sparse File Support</h2> The basic tests above confirm that sparse file support works for an empty file, but what about a file that has some data every X bytes? Below are the test results for creating a 10MB file, and writing data at various offset intervals.</p> Offset (bytes)</strong></td> File Size (KB)</strong></td> </tr> 1024</td> 9784</td> </tr> 2048</td> 9784</td> </tr> 4096</td> 9784</td> </tr> 8192</td> 4904</td> </tr> 16384</td> 2464</td> </tr> 32768</td> 1244</td> </tr> 65536</td> 632</td> </tr> 1048576</td> 60</td> </tr> 2097152</td> 40</td> </tr> 4194304</td> 32</td> </tr> </tbody></table> As soon as the offset was greater than the MMU page size (4KB), then the sparse file effect started to kick in and worked as expected.</p> 2. Can mmap() be used to efficiently modify large files?</h2> The test here was to open a large file, make a small change in the middle, and then close it. In this case a 100MB file was created with a data value of 1 written every 1MB, and then re-opened the same file and wrote a data value of 2 every 2MB. The od utility was used to examine the file to verify the contents were correct.</p> root@cm-x270:/media/mmcblk0p1# time sparse_file_mmap_test_arm -s104857600 \ -o1048576 -d1 Sparse File mmap() test Filesize = 104857600, offset = 1048576, data = 1 size = 102400 KB size on disk (KB): 508 sparse-file real 0m 1.10s user 0m 0.00s sys 0m 0.43s root@cm-x270:/media/mmcblk0p1# time sparse_file_mmap_test_arm -s104857600 \ -o2097152 -d2 -m Sparse File mmap() test Filesize = 104857600, offset = 2097152, data = 2 size = 102400 KB size on disk (KB): 508 sparse-file real 0m 0.37s user 0m 0.01s sys 0m 0.05s root@cm-x270:/media/mmcblk0p1# time od -x sparse-file 0000000 0002 0000 0000 0000 0000 0000 0000 0000 0000020 0000 0000 0000 0000 0000 0000 0000 0000 * 4000000 0001 0000 0000 0000 0000 0000 0000 0000 4000020 0000 0000 0000 0000 0000 0000 0000 0000 .... * 614000000 0001 0000 0000 0000 0000 0000 0000 0000 614000020 0000 0000 0000 0000 0000 0000 0000 0000 * 620000000 real 0m 12.93s user 0m 7.33s sys 0m 1.70s </code></pre> Creating and modifying large file-backed data structures is very fast and efficient, and takes on the order of 1 second for a 100MB file that contains data every 1MB. Dumping the data with od took considerably longer (13 seconds) as 100MB of data needed to be processed.</p> Possible Issues</h2> There are several things to watch out for when using sparse files and mmap()</p> With sparse files, there is the potential to run out of disk space as the files are using less space on disk than the file size. It is a good idea to monitor disk space when working with sparse files, so you don’t use up all of the disk space.</li> mmap() requires virtual memory space for the size file it maps. With embedded systems, this is less of an issue, because the physical RAM size tends to be much less than the 4GB virtual address space. With a system that only has 64MB of RAM, mmap()’ing files in the 10’s of MB in size makes a lot of sense because it insures that you will not run the system out of memory, and yet there should be plenty of virtual address space to map in files of this size.</li> </ol> Conclusion</h2> mmap() and sparse file support provide a very convenient solution for dealing with large, file-backed data structures. One example of this type of data structure is any type of large matrix such as maps used in mapping applications. Writing a “from scratch” solution to solve this problem would be a fairly large and difficult task. Processing large amounts of data efficiently is becoming more and more important in many embedded systems. This example provides another compelling reason why implementing modern, data-centric embedded systems using Linux makes a lot of sense.</p> MontaVista Linux 6 is based on OpenEmbedded Technologies 2009-05-13T00:00:00+00:00 Very interesting news from MontaVista yesterday as they announced MontaVista Linux 6</a>. It turns out they are using bitbake, which is the core of the OpenEmbedded</a> build system. Along with the adoption of OpenEmbedded in many leading Embedded Linux efforts (Beagleboard, Gumstix, Bug Labs, etc), this is a resounding affirmation that the OpenEmbedded way of building distributions is worthy of consideration. It is a well known fact that OpenEmbedded has some deficiencies: a steep learning curve, the testing/quality efforts could be improved, and the organisational aspects of the project are somewhat lacking. That said, OpenEmbedded is still the most effective tool for building Embedded Linux distributions that I’ve found yet, and is a very viable solution if you are willing to spend some time to learn it, or hire a consultant to get your project set up and work through the rough spots. MontaVista is attempting to address these difficulties with their MVL6 commercial offering.</p> Below are few a notes from watching a few videos on the MV web site:</p> Discussion with Jim Ready</a></p> around 2006 things changed</li> Semiconductor vendors now porting Linux to all their CPUs, so MV has moved up the stack</li> MVL6 first embedded linux distro produced in post-victory phase (Linux has won)</li> very friendly with other open source pieces (I assume OpenEmbedded)</li> enable open source</li> The decision the use Linux is the right decision technically. What is the right business decision to accompany the technical decision?</li> </ul> </li> MVL6 Demonstration</a></p> toolchains pre-packaged</li> sources are all downloaded from MV servers</li> </ul> </li> Interview with Joe Green</a></p> manager of developer tools team</p> </li> MLV6 is new approach</p> </li> previous editions were consistent for all targets</p> </li> MLV6 more flexible model. Market specific distros depending on market.</p> </li> source driven product. Previous products based on binary RPMs. MLV6 is a lot easier to build the whole environment. Very customized/fine-tuned distro.</p> </li> key problems MLV6 solves</p> Complete starting point for target hardware</li> complete build system, so you have total control vs pre-canned distribution that sets all the rules</li> </ul> </li> Integration platform based on Bitbake. Working closely with community. Enhanced in a number of ways. Customers have the option of bringing in software developed by the OpenEmbedded community and bringing it into their project.</p> </li> Gives you a system that you can configure at almost any level.</p> </li> Enhancements to make it easy to make changes, as well as provide a pre-built starting point.</p> </li> Features to control sources, and environment so the host environment does not contaminate your build.</p> </li> Essentially hardening, and commercializing the bitbake and openembedded projects.</p> </li> Why MVL6 vs roll-your-own.</p> Qualified starting point</li> Support. Community or Semi-conductor vendor support is hit or miss.</li> Consistent environment accross multiple systems.</li> Includes devrocket graphical tools</li> </ul> </li> </ul> </li> </ul> Overall, this seems like a very good approach. I’m glad they are not re-inventing the wheel, but instead have chosen to build on top of a proven open source solution, and leverage the work that is being done by a very active community of OpenEmbedded developers. The OpenEmbedded project currently has 61 developers (not all are active), and weekly contribution rate is very good. This type of effort would be very difficult to match by any but the largest commercial organizations. This should be a win-win situation for all involved. As a common base technology, the Bitbake and OpenEmbedded projects should improve due to MV’s involvement. Through the MVL6 product, the OpenEmbedded project and way of doing things will get a lot more exposure and there will be a lot more developers who understand how it works. Developers who are familiar with the OpenEmbedded build system will be able to apply their knowlege to a greater number of projects. And developers building products will have more options for getting the support and techology they need.</p> How to set up a NFS root filesystem for embedded Linux development 2009-05-11T00:00:00+00:00 Although ssh and friends work really well for embedded systems, occasionally you want to set up a NFS root for development. One of the scenarios where nfsroot is useful is if you are making a lot of rootfs changes, and you don’t want to spend the time to reprogram the flash on the target system. Fortunately, the Linux kernel includes complete support for NFS root, and does not require any userspace changes. This setup assumes the following:</p> the target system has Ethernet support built into the kernel</li> the target system is on the same network as a Linux workstation</li> said network includes a DHCP server</li> </ol> This configuration emphasises “simple” and does not require you to spend 4 hours trying to get a bootp server configured on a test network.</p> Kernel Support</h2> Make sure you have the following options turned on in the kernel:</p> CONFIG_IP_PNP_DHCP=y</li> CONFIG_ROOT_NFS=y</li> </ul> Then, add the following to your kernel CMDLINE:</p> ip=dhcp root=/dev/nfs nfsroot=<nfs server IP>:/path/to/nfsroot </code></pre> </li> </ul> Workstation Setup</h2> The following setup is for Ubuntu.</p> apt-get install nfs-user-server</p> (edit /etc/exports to contain something like the following)</p> /path/to/nfsroot 192.168.1.0/255.255.255.0(no_root_squash,sync,rw)</p> /etc/init.d/nfs-user-server restart</p> If you are using OpenEmbedded, then instruct OE to generate a rootfs tar image, and extract to your nfsroot directory:</p> cd /path/to/nfsroot sudo tar -xvf <path to OE images dir>/Angstrom-image...rootfs.tar </code></pre> And the result</h2> eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1 Sending DHCP requests ., OK IP-Config: Got DHCP answer from 192.168.1.1, my address is 192.168.1.112 IP-Config: Complete: device=eth0, addr=192.168.1.112, mask=255.255.255.0, gw=192.168.1.1, host=192.168.1.112, domain=hq.bec-systems.com, nis-domain=(none), bootserver=192.168.1.1, rootserver=192.168.1.11, rootpath= Looking up port of RPC 100003/2 on 192.168.1.11 Looking up port of RPC 100005/1 on 192.168.1.11 VFS: Mounted root (nfs filesystem) readonly. Freeing unused kernel memory: 220k freed Write protecting the kernel text: 2792k Write protecting the kernel read-only data: 788k INIT: version 2.86 booting </code></pre> Compulab cm-x270 kernel updated to 2.6.29 in OE 2009-04-24T00:00:00+00:00 The cm-x270 kernel support in OpenEmbedded has just been updated</a> to version 2.6.29.</p> Wi2Wi W2CBW003 Wifi/Bluetooth module review 2009-03-19T00:00:00+00:00 The Wi2Wi W2CBW003</a> is a highly integrated module that provides both Wifi and Bluetooth radios for embedded designs. This module is ideal for embedded designs, as it provides a lot of functionality in a small package and includes standard interfaces like SPI, SDIO and serial that connect with most embedded CPUs. With the availability of modules like the W2CBW003 and standard drivers in the Linux kernel, including radio functionality in an embedded device is very doable, even for low volume products. Wi2Wi provides an evaluation board for the W2CBW003 with a SDIO connector, UART connector, and BT Audio Connectors. For this review, the eval board was connected to a Marvel PXA270 ARM processor, and evaluated with current Linux and associated software.</p> W2CBW003 Overview</h2> The W2CBW003 module integrates both WiFi and Bluetooth functionality in a 12mm x 12mm x 1.6mm package. The WiFi portion is based on the Marvell 88W8686, and the Bluetooth on the CSR BC04. Both of these components are well supported by Open Source software. Some other features include:</p> Separate Antennas for WiFi and BT.</li> ROHS compliant</li> Single Supply at 3.3V</li> 802.11g support (54Mbps)</li> Both SPI and SDIO interfaces for WiFi</li> UART interface for BT</li> PCM audio interface for BT</li> </ul> Pictures of the module and the demo board are shown below.</p> </p> </p> Evaluation System Configuration</h2> The test software included with the Wi2Wi eval board is for a Windows PC, is provided in binary format only, and was not used in this review. For this review, I plugged the W2CBW003 demo board into a Compulab cm-x270 board (PXA270 cpu) running a 2.6.29-rc7 Linux kernel and a recent OpenEmbedded Angstrom distribution. With the exception of the Marvell Wifi Firmware, all software in this setup is Open Source and is available in the Linux kernel, and as packages in the OpenEmbedded Project.</p> When booting the kernel, you will see the following messages:</p> mmc0: new SDIO card at address 0001 libertas_sdio mmc0:0001:1: firmware: requesting sd8686_helper.bin libertas_sdio mmc0:0001:1: firmware: requesting sd8686.bin libertas: 00:19:88:06:0b:2e, fw 9.70.3p25, cap 0x00000303 eth2 (libertas_sdio): not using net_device_ops yet libertas: PREP_CMD: command 0x00a3 failed: 2 libertas: PREP_CMD: command 0x00a3 failed: 2 libertas: eth2: Marvell WLAN 802.11 adapter </code></pre> The “command 0x00a3 failed” messages are harmless, and have to do with features that are not supported. After the system boots, you will now see a new ethX network device:</p> root@cm-x270:~# ifconfig -a ... eth2 Link encap:Ethernet HWaddr 00:19:88:06:0B:2E BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:65902 errors:0 dropped:0 overruns:0 frame:0 TX packets:1758 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:13550002 (12.9 MiB) TX bytes:251627 (245.7 KiB) </code></pre> The “iwlist eth2 scanning</em>” command will list available access points.</p> Connecting to Open WiFi Networks</h2> A connection to an open wifi network can be accomplished by placing the following in /etc/network/interfaces:</p> /etc/network/interfaces: iface eth1 inet dhcp wireless_mode managed wireless_essid any </code></pre> And now execute “ifup eth2”:</p> root@cm-x270:~# ifup eth2 udhcpc (v1.13.2) started run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 Sending discover... Sending select for 192.168.1.115... Lease of 192.168.1.115 obtained, lease time 86400 run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 adding dns 208.67.222.222 adding dns 208.67.220.220 root@cm-x270:~# iwlist eth2 iwlist: unknown command `eth2' (check 'iwlist --help'). root@cm-x270:~# iwconfig eth2 eth2 IEEE 802.11b/g ESSID:"bec3" Mode:Managed Frequency:2.437 GHz Access Point: 00:18:39:C1:AD:4A Bit Rate:1 Mb/s Tx-Power=13 dBm Retry short limit:8 RTS thr=2347 B Fragment thr=2346 B Encryption key:off Power Management:off Link Quality=84/100 Signal level=-37 dBm Noise level=-87 dBm Rx invalid nwid:0 Rx invalid crypt:14707457 Rx invalid frag:0 Tx excessive retries:58 Invalid misc:3 Missed beacon:0 </code></pre> WPA Secured WiFi Networks</h2> The OpenEmbedded console image includes the WPA Supplicant packages which is used to manage wireless connections to secured networks. To set up the system for WPA encryption, modify the following files:</p> /etc/network/interfaces: iface eth2 inet dhcp wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf wpa-driver wext </code></pre> /etc/wpa_supplicant/wpa_supplicant.conf: ctrl_interface=/var/run/wpa_supplicant ctrl_interface_group=0 eapol_version=1 ap_scan=1 fast_reauth=1 network={ ssid="bec" proto=WPA2 key_mgmt=WPA-PSK pairwise=CCMP TKIP group=CCMP TKIP scan_ssid=1 psk="ascii passphrase" priority=10 } </code></pre> Then, execute “ifup eth2”, and you should see something like:</p> root@cm-x270:~# ifup eth2 WPA: Configuring Interface udhcpc (v1.13.2) started run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 Sending discover... Sending select for 192.168.1.115... Lease of 192.168.1.115 obtained, lease time 86400 run-parts: /etc/udhcpc.d/00avahi-autoipd exited with code 1 adding dns 208.67.222.222 adding dns 208.67.220.220 </code></pre> root@cm-x270:~# iwconfig eth2 eth2 IEEE 802.11b/g ESSID:"bec" Mode:Managed Frequency:2.412 GHz Access Point: 00:40:10:10:00:03 Bit Rate:1 Mb/s Tx-Power=13 dBm Retry short limit:8 RTS thr=2347 B Fragment thr=2346 B Encryption key:<too big> Security mode:open Power Management:off Link Quality=64/100 Signal level=-68 dBm Noise level=-89 dBm Rx invalid nwid:0 Rx invalid crypt:-1463809279 Rx invalid frag:0 Tx excessive retries:22524 Invalid misc:3 Missed beacon:0 </code></pre> Other Observations</h2> With the above networks, the bec</em> access point was much further away than the bec3</em>AP, so you will notice the difference in link quality. “iwlist eth2 rate</em>” can be used to list the current connection rate. When the network is idle, it sits at 1Mb/s. When downloading a large file, it will climb to 36 or 54Mb/s, depending on link quality.</p> Production Issues</h2> The review demonstrates that it is fairly simple to set up a demo quality Embedded Linux system with WiFi. Some of the issues that would likely need addressed for a production system include link management, test software for certification, and power management.</p> There are several ways to programatically control WPA Supplicant including linking to the wpa supplicant control interface, or using DBus. There are several WiFi management applications available including Gnome NetworkManager</a> (used in desktop systems), and connman</a> which seems a little better suited for embedded systems.</p> Unless you use a completely pre-certified module + antenna solution, you will likely need to do some level of agency certification. As many products use a custom antenna, this is an important issue to consider and plan for. While Marvell provides test software and firmware, it will likely require some work, as their software is designed to be used with their in-house drivers rather than the libertas driver which is available with modern kernels.</p> Also, if you want to minimize the power usage, some work will be required to figure out the power modes supported, and how to implement/control them. With this module, the Wifi and BT portions run off the same crystal, so if you only want the BT active, you will need to actively power manage the Wifi portion to a low power state instead of completely disabling it.</p> Conclusion</h2> TheW2CBW003 module is an attractive solution for products that need WiFi functionality. With the availability of modules like this, and mainstream open source software, the technology is available to about anyone, including low volume manufacturers. Standard interfaces such as SDIO make it possible to interface this module with about any modern ARM processor that can run Linux. Software support in the Linux kernel, wpa supplicant, and the Linux wireless tools provide the needed software support to implement a very complex system with relatively little effort.</p> Memory Performance on various Embedded Systems 2009-03-04T00:00:00+00:00 Marcin just published an interesting article</a> about memory performance on various embedded systems using the hdparm -T as a simple benchmarq. This test gives a pretty good indicator of memory performance in the system. From the hdparm man page:</p> Perform timings of cache reads for benchmark and comparison purposes. For meaningful results, this operation should be repeated 2-3 times on an otherwise inactive system (no other active processes) with at least a couple of megabytes of free memory. This displays the speed of reading directly from the Linux buffer cache without disk access. This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.</em></p> A few results I find interesting:</p> modern desktop systems have an order of magnitude more memory bandwidth than ARM systems.</li> the i.MX31 is the highest performing ARM device tested</li> the i.MX31 performs better than the OMAP3 in this test – why is this? As the ratio is 2, I’m guessing the bus is twice as wide?</li> </ul> Who is using OpenEmbedded? 2009-02-20T00:00:00+00:00 Having used OpenEmbedded for a number of Embedded Linux projects over the past 5 years, it is interesting watching new users come on board. The most recent announcement came from Koan</a> that the KaeilOS</a> distribution will be joining the OpenEmbedded project. In the announcement, they summed up my thoughts well:</p> We at KOAN wanted to avoid waste of energy converging KOAN efforts into a stable and solid Open Source project like OE is.</em></p> “Avoid waste of energy” – isn’t this what OSS is all about? Life is way too short to re-invent every wheel. Considering OpenEmbedded has dozens of active contributors, it would be very difficult for any one company to replicate the functionality of OpenEmbedded. Although there are many ways to build embedded Linux distributions, and some tools do some things better than OE, there still does not seem to be any other embedded Linux build tool that fulfills the following requirements as well as OE:</p> build from scratch on any x86 Linux system</li> large number of contributors</li> large number of packages</li> support a large number of architectures</li> Can modify images and integrate custom applications in a consistent way as part of the build system, and not a post processing hack.</li> Open Source</li> </ol> Some other companies using OpenEmbedded include:</p> TI</li> AMD</li> Intel (was OpenedHand)</li> Gumstix</li> Bug Labs</li> OpenMoko</li> Atmel</li> Techsol</li> Compulab</li> Toradex</li> 4G-Systems</li> Dream Multimedia TV</li> M&N Solutions</li> emQbit</li> Siemens</li> Embedded Alley</li> and the list goes on</a> …. (If you are using OpenEmbedded, please add your name to this list</a>.)</li> </ul> I’m looking forward to seeing the KaeilOS improvements such as mdev, and boot time improvements.</p> Sprint 598U USB Broadband Modem in Embedded Systems 2009-02-07T00:00:00+00:00 One thing that works really well in Linux is support for USB Broadband modems. While it usually takes a little fiddling with PPP scripts, it is generally not too difficult to get working. One of the reasons for this is most modems implement a USB serial interface, and then the modem is controlled with standard AT commands. A customer who currently supports a Verizon USB modem</a> in their product wanted to add support for a Sprint modem. After a little research, we settled on the 598U from Sierra Wireless</a>. Below is a picture of the 598U plugged into an industrial terminal.</p> </p> Why are USB Broadband modems useful in Embedded Systems?</h2> As systems become more complex, and time to market is shrinking (thus less testing), it is becoming increasingly important to be able to remotely diagnose problems. With a broadband modem like the Sprint 598U, developers can remotely log into a device using SSH, and debug problems. This is a huge benefit. The cost of the modem and a 1 year data plan becomes fairly insignificant when you consider the cost to fly a developer on-site to debug issues. While there are other ways to remotely access an embedded system, such as reverse ssh</a>, the simplicity of the broadband modem is attractive as it bypasses the firewall issues. Even with reverse ssh, I’ve run into paranoid sys admins who block the outgoing port 22 on their networks, which makes reverse ssh difficult. Once you have the IP address of the broadband connection, you can connect to it directly with ssh from anywhere in the world. I’ve accessed systems that were being field tested in New Zealand from the United States, and it worked great.</p> Increasingly, the value of complex embedded systems is shifting from control to data. While the control aspect is as important as it ever was, access to data is what is giving many products and businesses the competitive edge. Being able to collect data and visualize how the system is performing is becoming increasingly important to maximize performance. And sometimes this data must be collected in real-time. Scenarios where USB broadband modems might make sense:</p> remote sites that do not have internet connectivity</li> mobile applications such as vehicle terminals</li> temporary installations where the system is only located in one place for a limited time</li> situations where it is very difficult to get administrators of the local network to cooperate in adding the embedded system to their network</li> </ul> Finding PPP Config information</h2> I usually develop as much as I can on x86 systems (vs. an embedded target) as it is a lot more convenient, so I plugged the modem into a Ubuntu 8.10 laptop. While Ubuntu 8.10 recognizes and configures a UM150 Verizon modem, it does not automatically configure the 598U. Typically, the next step is to google around for information on various support forums on how to set up the PPP config files, as most manufactures don’t provide support for Linux. As I could not find any existing information for the 598U, I decided to see if Sierra Wireless published any information. Was I ever suprised when I found this page</a>. Sierra Wireless provides fairly comprehensive information on using the modem under Linux, and their PPP scripts actually worked – great job! But in the end, when I plugged the 598U into the embedded device that already supported the Verizon modem, it just worked with my existing PPP scripts. This is another indication of the value of using common code and interfaces in Linux.</p> The Linux Advantage</h2> When you run Linux in an embedded system, using a USB broadband modem becomes fairly simple. It is actually a little bit deceiving as to what all has to happen. Different modems use different chipsets, and thus different drivers. While they all eventually expose a USB serial interface, there are enough differences to require separate drivers. As an example, the 598U uses the “sierra” USB-serial driver. This driver is 782 lines, plus it utilizes a common USB serial framework that is many times this size. On the application side, the pppd framework is used to manage the modem and the network connection. Starting the connection is as simple as running “pppd call cdma” from your application. The pppd application manages getting an IP address, setting up nameservers in resolve.conf</em>, etc. Because all broadband modems are supported by a common application interface and very similiar drivers, supporting multiple modem vendors is actually quite easy. Contrast this to the Windows driver scenario where every vendor provides their “own” driver, and their “own” application user interface, and their “own” way to initiate an internet connection, and it is not obvious to me how you could easily control all this in a consistent, automated fashion in an embedded device. The way we set it up is you plug the modem in, and a few seconds later the system is automatically connected to the internet. There is nothing for the user to configure, initiate, etc. Printing from Linux in embedded applications</a> is similiar.</p> This is another example of how it is possible to leverage the work of many to add advanced functionality and features to a device with very little development cost. Getting more done with less should be an attractive (and perhaps critical) way to operate for any company seeking to remain competitive.</p> Socketcan utils and test apps added to OpenEmbedded 2009-01-28T00:00:00+00:00 The Socketcan test applications and utilities are now available in OpenEmbedded</a>. The socketcan kernel modules already exist in OE</a>. To build and deploy:</p> get the latest version of OE metadata</li> bitbake socketcan-utils-test</li> scp /build/angstrom-2008.1/tmp/deploy/glibc/ipk//socketcan-utils-test_0.0+svnr917-r0_armv5te.ipk root@:</li> </ul> And then on the target system:</p> opkg install socketcan-utils-test_0.0+svnr917-r0_armv5te.ipk</li> opkg files socketcan-utils-test</li> </ul> /usr/bin/tst_bcm_rx_sendto /usr/bin/tst_bcm_tx_sendto /usr/bin/canlogserver /usr/bin/tst_raw_filter /usr/bin/tst_raw /usr/bin/isotpsniffer /usr/bin/isotpsend /usr/bin/tst_bcm_throttle /usr/bin/tst_packet /usr/bin/vcan /usr/bin/isotprecv /usr/bin/cansniffer /usr/bin/tst_bcm_tx_read /usr/bin/tst_err /usr/bin/log2asc /usr/bin/tst_bcm_single /usr/bin/tst_filter_master /usr/bin/canplayer /usr/bin/cangen /usr/bin/slcanpty /usr/bin/log2long /usr/bin/tst_bcm_rtr /usr/bin/asc2log /usr/bin/slcan_attach /usr/bin/tst_bcm_filter /usr/bin/cansend /usr/bin/canecho /usr/bin/tst_filter_server /usr/bin/tst_bcm_cycle /usr/bin/tst_bcm_dump /usr/bin/isotpdump /usr/bin/candump /usr/bin/tst_raw_sendto /usr/bin/isotptun /usr/bin/tst_bcm_server /usr/bin/tst_proc /usr/bin/canbusload </code></pre> An impressive array of utilities!</p> Socketcan CAN-bus drivers added to OpenEmbedded 2009-01-26T00:00:00+00:00 I just added a recipe</a> to OpenEmbedded to build the Socketcan</a> kernel modules from the socketcan SVN. So if you are using the latest OpenEmbedded metadata, you can:</p> bitbake socketcan-modules</li> scp /build/angstrom-2008.1/tmp/deploy/glibc/ipk//socketcan-modules_0.0+svnr917-r0_cm-x270.ipk root@:</li> </ul> and then on the target system:</p> opkg install socketcan-modules_0.0+svnr917-r0_cm-x270.ipk</li> opkg files socketcan-modules</li> </ul> Package socketcan-modules (0.0+svnr917-r0) is installed on root and has the following files: /lib/modules/2.6.27/extra/net/can/can-bcm.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/ems_pci.ko /lib/modules/2.6.27/extra/net/can/can-raw.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/sja1000.ko /lib/modules/2.6.27/extra/drivers/net/can/softing/softing.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/ems_pcmcia.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/sja1000_platform.ko /lib/modules/2.6.27/extra/drivers/net/can/can-dev.ko /lib/modules/2.6.27/extra/drivers/net/can/softing/softing_cs.ko /lib/modules/2.6.27/extra/drivers/net/can/sja1000/pipcan.ko /lib/modules/2.6.27/extra/drivers/net/can/mcp251x.ko /lib/modules/2.6.27/extra/net/can/can.ko /lib/modules/2.6.27/extra/net/can/can-isotp.ko /lib/modules/2.6.27/extra/drivers/net/can/vcan.ko </code></pre> Then to use the modules:</p> depmod</li> modprobe mcp251x</li> modprobe can-bcm</li> </ul> What is CAN? Can stands for Controller-area network and is popular in industrial and automotive applications. A convenient way to add CAN to your Embedded Linux system is with the Microchip MCP2515</a>. This device connects to a SPI bus which means it can be interfaced with a number of popular SOC’s such as the PXA270, OMAP3, 91SAM9xxx, etc.</p> More on Socketcan later …</p> BTW, this recipe really illustrates how easy it is to compile kernel modules outside a kernel in OpenEmbedded:</p> DESCRIPTION = "Socketcan kernel modules" HOMEPAGE = "http://developer.berlios.de/projects/socketcan/" SECTION = "kernel/modules" LICENSE = "GPL" DEPENDS = "virtual/kernel" PV = "0.0+svnr${SRCREV}" SRC_URI = "svn://svn.berlios.de/socketcan/trunk;module=kernel;proto=svn" S = "${WORKDIR}/kernel/2.6" inherit module </code></pre> This recipe tells OE to download the source, cross-compile it against your target kernel build dir, and then package it for easy install on the device. This is the way things should work – no messing around figuring out make options, kernel source paths, compiler env variables, etc.</p> Intel Atom vs TI OMAP3 2009-01-22T00:00:00+00:00 As we look at new projects, both the Intel Atom</a> and the TI OMAP3</a> processors generate considerable interest. As we have already shown</a>, the OMAP3 does offer a considerable performance improvement over earlier generations of ARM CPUs. The following video</a> I found on YouTube shows a similar comparison of a OMAP3 and Atom systems rendering web pages:</p> As one would expect, the Atom does perform better (about 14%), but considering the power differences, the OMAP does surprisingly well. It is also unknown in this demo if the screen size would make a significant difference in the results. Like most things, the choice depends on the application, and no two applications are the same, and each solution has advantages. Some things to think about:</p> Power: OMAP3 platform consumes on the order of <1-2W while the Atom is more in the range of 2-5W.</p> </li> High Speed I/O Interfaces: Atom supports PCI and PCI expansion interfaces where OMAP3 is limited to more special purpose user interfaces such as SD, Camera, Asynchronous bus, etc. Both Atom and OMAP support High Speed USB.</p> </li> Packaging: OMAP3 packaging is very aggressive with the stacked Package-on-Package. To get an idea how much space an OMAP3 solution takes, check out the module from Gumstix</a>. There are basically only two chips in the system: the OMAP3+stacked RAM/Flash and a power management+I/O chip. This is very high integration!</p> </li> Module availability: for many embedded systems with volumes in the 1000’s of units per year, a module solution is very attractive compared to a full custom design. This drastically reduces the engineering effort and time to market. A sampling of the modules available include:</p> OMAP3:</p> Gumstix Overo</a> ($117 for 3000 units)</li> LogicPD OMAP35x SOM-LV</a> (Cost unknown)</li> </ul> </li> Atom:</p> Eurotech Catalyst</a> (Cost unknown)</li> </ul> </li> </ul> </li> Software support: TI and the open source community have done a remarkable job of supporting the OMAP3 with the BeagleBoard</a> effort. Gumstix maintains open source software for their devices, and has a very active development community. Intel also has invested significantly in software with their Moblin</a> project. Other factors to consider is the boot software (bootloader vs BIOS, is it open?), are there 3D graphics libraries available, etc.</p> </li> Multimedia processing: The OMAP3 is available with an on-chip DSP. Intel has traditionally offered extensions for multimedia processing such as SIMD.</p> </li> </ol> Tradeoffs!</p> GTK performance on PXA270 vs. OMAP3 2009-01-21T00:00:00+00:00 Several of my customers have built applications using the GTK+</a> tookit. While GTK+ works fairly well for what we have done, I have been wondering how the performance compares on the new Omap3</a> processors from TI. As we are evaluating the OMAP3 for several projects, I did a simple comparison with an existing application. Below is a video</a> that shows a fairly complex application running on both a PXA270, and a OMAP3530. While the PXA270 gets the job done, the result on the OMAP3 is much more pleasing. With that advent of a OMAP3 module available for $117 in volume</a>, it seems like the OMAP3 will be a popular solution for upcoming Embedded Linux projects.</p> Fix for PXA270 MMC/SD Controller Write Corruption 2009-01-17T00:00:00+00:00 During system verification for a customer, we noticed occasional failures when writing to a SD card using the Marvell PXA270 MMC/SD controller. The failure is a 4KB block of data is shifted on byte, where the first byte is duplicated and the last byte is dropped. The test app that found this problem is available here</a>. This test app simulates an application that writes data to a SD card at a certain rate. We found that typically in about 2000 - 10000 hours of simulated testing, we would see a failure. Its very interesting that the field failure rates matched this rate as well.</p> This failure had all the signs of a hardware (likely DMA) problem, and sure enough there is a Marvell Eratta for this issue. A patch has been developed and tested by several people and is available in this mail thread</a>.</p> Installing Ubuntu on a Core i7 DX58SO motherboard 2009-01-15T00:00:00+00:00 UPDATE: 2009-03-17 – It appears that hard drive issues I encountered are likely due to the HW RAID formatting on the drive I tried. I just tried a brand new hard drive, and Ubuntu 8.10 installed flawlessly, and everything works.</strong></p> I’m not sure why computer upgrades are always such an epic struggle for me, but here I go again …</p> The goal is to upgrade my aging AMD X2 system to a new Intel Core i7 system. So after researching it a bit, I decided on the Intel DX58SO</a> motherboard. I liked the architecture, and have had good luck with Intel motherboards. And since I don’t upgrade all that often, I try to get something good when I do. As I learned later, I should have researched Linux compatibility a little more …</p> The plan was to simply replace the CPU/MB/RAM in my existing box as I already have a raid controller, decent case, etc. However, the first challenge was the existing PS does not supply a EPS/ATX12V8 pin connector – it only had the 4-pin version. So, I got to upgrade my PS to a Corsair CMPSU-650TX</a> which seems very nice, and quiet.</p> After I put everything together, the raid array (3ware 9500S-4LP PCI) was detected, but the system would not boot. So I booted from a Ubuntu install disk and selected boot from 1st hard disk, and then it seemed to boot fine. However, the Ethernet controller and CDROM don’t work under Ubuntu 8.04 (64-bit) which was currently installed on the raid array. However, everything else works pretty well, and it is quite fast. Because the core-i7 brought back hyperthreading, Linux now shows 8 cores.</p> Several days later in an attempt to get the system booting from the hard drive, I upgraded the BIOS to the latest version. However, this broke booting from the raid array; even with a Ubuntu install disk.</p> At this point, I decided to just rebuild the system by re-configuring the 4 hard drives currently in a hardware raid-5 array to two raid-1 software raid arrays. As the raid card is only PCI, I think this would provide better performance and be simpler to maintain in the future. So, I copy all my data off to a USB HD, and then try to install Ubuntu to a HD attached to the motherboard. To my suprise, I can’t get MB attached hard drives to work. Lots of messages like:</p> [ 361.943329] ata1.01: status: { DRDY ERR } [ 361.943332] ata1.01: error: { ABRT } [ 362.043760] ata1.01: configured for UDMA/133 [ 362.043771] ata1: EH complete [ 362.218194] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 [ 362.218200] ata1.01: BMDMA stat 0x45 [ 362.218207] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in [ 362.218209] res 51/04:08:00:00:00/04:00:1d:00:00/f0 Emask 0x1 (device error) </code></pre> I tried booting a 32-bit Ubuntu 8.10 live CD and I get the same thing when trying to fdisk a hard drive attached to the MB. So for now, the only way to get this running is to go back to the old RAID controller. Fortunately, Intel keeps old BIOS versions on their web site and after two tries, I was able to find one that works again with the HW RAID controller.</p> The Ethernet and CD ROM did work with the Ubuntu 8.10 32-bit live CD, so once I upgrade to 8.10, hopefully that will all work. So for now, I’ll limp along with my raid controller and a USB->Ethernet adapter for networking.</p> Anyone else have success with Linux on the DX58SO?</p> I’ve filed a bug on launchpad</a>.</p> Switching to Wordpress 2009-01-13T00:00:00+00:00 After using Joomla for several years, I’m switching my web site to wordpress. As I mostly use my site for writing technical articles, it makes sense to use something that works very well for just that. So far, I’m finding Wordpress very polished, and it has all the functionality I need built-in, so I don’t have to rely on 3rd party modules for very basic operations like tagging, RSS feeds, comments, etc. Now I have no excuse for now writing more as Wordpress works so well.</p> Mono 2.2 RC1 added to Openembedded 2009-01-10T00:00:00+00:00 I recently addedMono 2.2 RC1 to OpenEmbedded</a>. The last usable version of Mono in Openembedded was 1.2.6, so this is a big step up. Lots of interesting things have been happening since then (like Full Static Compilation</a>) that continue to make Mono interesting for embedded. There is still plenty to do yet such as clean up the packaging, and update the other related recipes such as mono-xsp, and gtk-sharp.</p> Using the Vala Programming Language in Embedded Systems 2009-01-10T00:00:00+00:00 Recently I’ve been following the Vala programming language</a> and using it some in embedded systems. Vala is a new programming language that aims to bring modern programming features to GNOME developers without imposing additional runtime requirements and without using a different ABI compared to applications and libraries written in C. A few notes and observations about Vala:</p> language syntax that resembles C#, so you can write code faster with less mistakes</li> Vala compiles to C, so it starts fast and runs fast like native applications</li> binaries are not platform independent (like Mono or Java)</li> lots of bindings already exist, as it is very easy to write bindings to existing C libraries</li> easy to write programs that are mixed C and Vala</li> documentation is still a bit sparse, so you end up reading the binding files to figure out how to use the libraries</li> </ul> Much of the core Vala functionality is built on top of Glib. Having programmed extensively with Glib, I can say programming in Vala is a lot more fun, and a lot less tedious. Vala also provides dynamic D-Bus bindings which makes if very nice for writing system daemons or other bits of software that need to implement a D-Bus server.</p> As far as real world experience, in a recent application, we ported a system monitoring application from C# to Vala. There were no real serious problems with the original application, but it started slow (7 seconds or so), and the customer wanted to reduce the boot time of the system. Once you get past the core language syntax, the libraries for Vala are all different than C#, so most of the library function calls needed tweaked a little. Most of the functionality was implemented with Vala, but there were a few minor functions that were implemented in C. It now starts fast, which is what we needed.</p> We are also using the fsod</a> application from the OpenMoko project in a customer project. This is an excellent example of a well written, advanced Vala project that uses features such as plugins and D-Bus.</p> Watching the releases from the Vala project, it is amazing the progress this language is making. I’m sure we’ll be hearing more about Vala in the future.</p> How to implement an interrupt driven GPIO input in Linux 2009-01-10T00:00:00+00:00 With Linux, some of the things that seem like they should be easy are not – at least at first glance. For example, how do you read an interrupt driven GPIO input in a Linux application? With simpler microcontroller systems, this is straightforward, but with a system like Linux, you have to navigate through several layers of software (and for very good reasons). You can’t handle interrupts directly in a Linux application, so this means you need a kernel component involved. This operation of reading a GPIO resembles a key press, so the Linux input subsystem might be a good place to start looking. Once we take that route, we discover the gpio_keys</a> driver.</p> Configuring the gpio_keys driver</h2> The gpio_keys driver is configured with a few lines of code in your Linux board configuration file. An example is below:</p> static struct gpio_keys_button svs_button_table[] = { { .code = KEY_RECORD, .gpio = PP_GPIO_MIC_EN, .active_low = 1, .desc = "MIC_EN", .type = EV_KEY, .wakeup = 0, }, }; </code></pre> In this application, we are reading button presses on a cell phone style headset. Once the gpio_keys driver is configured, a new entry will show up in /dev/input/eventX. An application can then do a blocking read on this device.</p> Reading the GPIO in an application</h2> To read the GPIO, we simply do a blocking read on the new /dev/input/eventX device. The read will block until there is a change in GPIO state. An example is show below:</p> #define MIC_INPUT_DEV "/dev/input/event0" static gboolean mic_button_callback(GIOChannel source, GIOCondition condition, gpointer data) { struct input_event ev; int bytes_read; g_io_channel_read_chars(source, (gchar )&ev, sizeof(ev), &bytes_read, NULL); if (bytes_read > 0) { if (bytes_read != sizeof(ev)) { s_debug(1, "warning, only read %i bytes from mic input"); return TRUE; } } else { return TRUE; } if (ev.type != EV_SYN && ev.value == 1) { /* button pressed, do something ... / } return TRUE; } void mic_button_init() { GIOChannel micbutton = g_io_channel_new_file(MIC_INPUT_DEV, "r", NULL); if (micbutton == NULL) { s_debug(TRUE, "Error initializing mic button"); return; } g_io_channel_set_encoding(micbutton, NULL, NULL); guint id = g_io_add_watch(micbutton, G_IO_IN, mic_button_callback, NULL); } </code></pre> The above example also shows how to incorporate the GPIO read into a GLib mainloop so that you don’t need to create a separate thread. ( As a side, GLib mainloop programming is worth learning!) Using this method, reading a GPIO interrupt is easy and requires very few lines of code. This is typical of complex systems like Linux – if you know how to do something, it is relatively easy, but getting started down the right path is sometimes the challenge.</p> Tips for reading a serial data stream in Python 2009-01-09T00:00:00+00:00 Interfacing with a RS232 serial device is a common task when using Python in embedded applications. The easiest way to get python talking to serial ports is use the pyserial project found at http://pyserial.sourceforge.net/</a>. This module works on most platforms and is straightforward to use (see examples on project web site). However, getting the read function in this module to operate in an optimal way takes a little study and thought. This article investigates how the pyserial module works, possible issues you might encounter, and how to optimize serial reads.</p> We start out with several goals as to how we want the application to behave in relation to the serial port:</p> application must block while waiting for data.</li> for performance reasons, we want to read decent size chunks of data at a time if possible. Python function calls are expensive, so performance will be best if we can read more than one byte at a time.</li> We want any data received returned in a timely fashion.</li> </ul> A key parameter in the pyserial Serial</em> class is the timeout</em> parameter. This parameter is defined as:</p> timeout=None, #set a timeout value, None for waiting forever </code></pre> The Serial</em> class read</em> function also accepts a size</em> parameter that indicates how many characters should be read. Below is the source for the read function on Posix systems (Linux, etc):</p> def read(self, size=1): """Read size bytes from the serial port. If a timeout is set it may return less characters as requested. With no timeout it will block until the requested number of bytes is read.""" if not self.fd: raise portNotOpenError read = '' inp = None if size > 0: while len(read) < size: #print "\tread(): size",size, "have", len(read) #debug ready,_,_ = select.select([self.fd],[],[], self.timeout) if not ready: break #timeout buf = os.read(self.fd, size-len(read)) read = read + buf if self.timeout >= 0 and not buf: break #early abort on timeout return read </code></pre> The easy way to use this module is to simply set the timeout to None, and read size to 1. This will return any data received immediately. But, this setup is very inefficient when transferring large amounts of data due to the Python processing overhead.</p> To meet our goal of reading multi-byte blocks of data at a time, we need to pass the read function a size greater than 1. However, if timeout is set to None, the read will block until size bytes have been read, which does not meet the goal of returning any data read in a timely fashion. The solution then is to:</p> set the read size high enough to get good performance</li> set the timeout low enough so that any data received is returned in a reasonable timeframe, but yet the application spends most of its time blocked if there is no data.</li> </ul> As an example, a size of 1000 and a timeout of 1 second seems to perform well. When used this way, the pyserial module performs well and returns all data read quickly.</p> How to Optimize Python reads for general data 2009-01-09T00:00:00+00:00 The Python file object read() function acts a little differently than the standard read() found in C. This article describes some of these differences and how to optimize reads for general continuous data streams such as reading data from a collection device through a pipe.</p> Python read()</h2> The Python read function seems to be optimized for reading files and text oriented streams. By default, a read() call will block until a EOF is encountered. This is very handy for reading files from a disk. You can just slurp them up with one read() statement. If you pass a size parameter to read(), it will block until size bytes have been received. This is less than ideal for reading a continuous data stream where some of thye data may be stuck until the size threshhold is reached.</p> Non-blocking read()</h2> The way you get read to return with whatever data is available even if it is less than the read size parameter is to set the file object up in non-blocking mode. This can be done using the fcntl module:</p> flags = fcntl.fcntl(fp, fcntl.F_GETFL) fcntl.fcntl(fp, fcntl.F_SETFL, flags \| os.O_NONBLOCK) </code></pre> However, now the application does not block while waiting for data and spins using CPU resources.</p> select()</h2> Enter the select call. The Python select module does much the same thing as the C select() function. In this case, it can be used to block waiting for data from a non-blocking file object with the added benefit of a timeout. So the resulting code might look like:</p> fp = os.popen(<application that returns data to stdout>, 'r') flags = fcntl.fcntl(fp, fcntl.F_GETFL) fcntl.fcntl(fp, fcntl.F_SETFL, flags \| os.O_NONBLOCK) while 1: [i, o, e] = select.select([fp], [], [], 5) if i: s_ = fp.read(1000) else: s_ = '' if s_: logging.debug("received %i bytes of data, total = %i" % (len(s_), total)) <do something with s_> </code></pre> The above select statement blocks until data is available from the fp object or times out after 5 seconds. Reading continuous data streams in Python is very possible, but usually requires the file object to be set in non-blocking mode and a select used to block while waiting for data.</p> Implementing Asterisk 2009-01-09T00:00:00+00:00 Over the past month, we have been working on implementing Asterisk in our office. Asterisk (http://www.asterisk.org/</a> ) is a complete software PBX (Private Branch Exchange) that runs on top of Linux. Asterisk supports a number of features commonly found in a PBX including voicemail, hosted conferencing, call queuing, and many other features. Having a real PBX provides many benefits for a business and has the potential to improve our service and lower our costs. Over the next couple months we will be writing about our experiences implementing Asterisk, tips for setting it up etc.</p> One of the most amazing things about Asterisk is its flexibility. You can mix and match IP phones, traditional analog phones, soft phones, traditional PSTN lines, and VOIP services in about any way you want. Our setup is fairly simple – IP phones and connections to the outside world using our PSTN land line. We are also evaluating several VOIP providers and trying to determine if VOIP is viable. Stay tuned …</p> How to Implement a Web Application Framework in an Embedded Linux System 2009-01-09T00:00:00+00:00 As devices are increasingly more networked, an embedded web server is becoming a standard way for users to interact with and configure an embedded device using a standard web browser. As an example, most pieces of networking equipment (such as routers and wireless access points) are configured this way. There are many ways to implement a web server. One way is to just write a monolithic program that handles the requests and outputs HTML using print statements. Without a lot of planning, you usually end up with something where logic is intermixed with presentation. The disadvantage of such an approach is that it quickly becomes very difficult to maintain your web application as it grows and changes. The current best practice with web application frameworks (such as Ruby on Rails) splits a web application into 3 distinct components (Model, View, Controller – MVC) so that changes to one component can be made with minimal impact to others. This article details a solution used in a recent project to implement such an architecture using the following open source components: SQLite, Clearsilver, and Python. The challenge was to find a solution that performed acceptably on a resource constrained 130MHz ARM-Linux system.</p> Components Used</h2> The requirements for the web application framework for this system are:</p> Footprint must be fairly small – less than 10MiB.</li> Must enable us to implement a clean MVC type architecture.</li> Must support a high level language like python for rapid development.</li> Includes a database.</li> Reasonable performance – less than 1 second to render a typical page.</li> </ul> After researching and testing several options, the following components were chosen to implement the framework:</p> Web Server: Cherokee (1MiB)</li> Model: SQLite (290KiB)</li> View: Clearsilver (170KiB)</li> Controller: Python (2-3MiB)</li> </ul> </p> Total size for the above components is about 4MiB.</p> The following sequence occurs during a typical web transaction:</p> An HTTP request is received by Cherokee</li> If the request matches the URI for the web application, Cherokee forwards the request to the web application framework via CGI.</li> Clearsilver parses the HTTP headers (including CGI GET and POST parameters) and provides the data to python in a easy to access HDF format.</li> Python looks at the URI and dispatches the request to the appropriate function. Data is extracted from the SQLite database and an HDF datastructure is created.</li> Clearsilver is called to render the appropriate template. Data from the HDF datastructure is used in the template to provide the dynamic content in the page.</li> The rendered HTML is passed back to Cherokee, and is then returned to the user’s web browser.</li> </ul> Each component is discussed in more detail in the following sections.</p> Performance</h2> One of the issues with modern web frameworks is that most require a lot of processing power. Many frameworks are written in an interpreted language which tend to be not very efficient on embedded systems such as the 130MHz ARM-Linux system used in this project. As most of the application development for this project is done in Python, I tried several other Python based solutions with the following results:</p> Django</strong>: takes 10</strong> seconds to render a page</li> webpy</strong>: takes 5</strong> seconds to render a page</li> Clearsilver/Python</strong>: takes about 1</strong> second to render a page</li> </ul> Anything over 1 second is very slow for a web interface. With fastcgi, webpy could probably have been sped up to be acceptably fast, but I don’t think there is much hope for frameworks like django on this type of system. I have read of similar experiences trying to run Ruby-on-Rails on ARM-Linux systems. As I’m currently just using a basic CGI interface, the Clearsilver solution could also be sped up significantly as well, if the Python portion could be kept running between requests with fastcgi or a similar mechanism.</p> Cherokee Web Server</h2> A web application typically consists of a web server (like Apache) and a program that provides the dynamic web content. Some of the functions of a web server are:</p> receives HTTP requests</li> handles the request by serving static files/images or routes the request to a program that outputs dynamic content</li> authentication</li> encryption (SSL, TLS)</li> </ul> There are many web servers that can be used in embedded devices. Some of them are:</p> Lighttpd (http://www.lighttpd.net</a> )</li> thttpd (http://www.acme.com/software/thttpd/</a> )</li> Appweb (http://www.appwebserver.org/</a> )</li> Boa (http://www.boa.org</a> )</li> Cherokee (http://www.cherokee-project.com</a> )</li> SHTTPD (http://shttpd.sourceforge.net/</a> )</li> </ul> Cherokee was chosen for this application because it provides a good balance between the size and functionality I am looking for. It is also included in the Openembedded (http://openembedded.org</a> ) build system I am using. Other systems will have different requirements – use the one that fits your application best.</p> Clearsilver</h2> Clearsilver (http://www.clearsilver.net/</a> ) is the real gem that was discovered during this exercise. Clearsilver is a language-neutral HTML template system written in C. It is used as the templating system for many high volume sites such as Google Groups 2. Clearsilver also provides CGI handling functions and bindings to several languages including Python. The fact that it is written in C and is fast for high volume sites also makes it fast enough</em> on slower embedded systems. Because Clearsilver is written in C, the templating system is already fast. Parts or all of the application can also be moved to C as needed to get the required performance. Having a performance upgrade path is nice. Clearsilver also forces a strict separation of application logic and presentation templates, which keeps things in line with the MVC architecture.</p> SQLite</h2> SQLite (http://www.sqlite.org/</a> ) is a small C library that implements a self-contained, embeddable, zero-configuration SQL database engine. SQLite works as expected and also has bindings to a number of languages including Python. A SQLite database is just a single file that requires no configuration, making it very easy to use. The database engine runs in the same process as the application using it, which increases performance because there is no context switch when running database operations. Because SQLite does not have its own process, it does not handle concurrency as well as other databases such as MySQL, but in an embedded system you seldom have more than one process using the database. SQLite provides some support for concurrency with reader/writer locks for the entire database. If your application is not real busy, this is often adequate if you need two processes to share some data.</p> Python</h2> Python is the glue that ties things together in this web application framework. It sits between Clearsilver and the database. The advantages of using a language like Python are rapid development and an extensive library. Python is considerably slower than C, but there is always the option to move parts of the application into C as needed.</p> Openembedded Build System</h2> How does one put together a system with all these components? The answer is the Openembedded Build System (OE) (http://openembedded.org</a>). OE includes support for all of the components discussed in this article.</p> Summary</h2> The combination of Python, Clearsilver, and SQLite provides a compelling solution for a web application framework in an Embedded Linux System. The solution is reasonably small, performs well, and provides a path to improve performance if needed. We also get a clean architecture to create clean, maintainable web applications.</p> About the author</strong> – Cliff Brake owns BEC Systems, a consulting operation that helps customers utilize modern computer technologies in their products. BEC offers a range of services to make your embedded project a success including technology selection, development, troubleshooting, and training. Please visit our website (http://bec-systems.com</a> ) for more information and other free resources.</p> Do you have an Exit Strategy? 2009-01-09T00:00:00+00:00 Some time ago, eWeek published in interesting article by Scott McNealy named “Technology’s Barriers to Exit</a>”</em>. In this article, Scott makes an interesting point about considering the cost of technology. There are three costs to consider:</p> cost of acquisition</li> ongoing cost</li> cost to exit</li> </ul> Scott states that most people analyze the first two costs fairly well. What is not thought about as much is the the cost to exit – how do I move on to the next thing. I think these basic questions apply very much to embedded technology decisions.</p> Most of the time we carefully consider our requirements, think about ongoing manufacturing costs, etc. There may be some thought given to upgrading (the exit), but we encounter several difficulties when planning for the future with complex embedded systems:</p> we often can’t predict what technologies will be available in the future</li> if may be difficult to predict our product requirements in the future</li> </ul> As an example, we may have a product with an ethernet networking interface, and two years after product launch we have an opportunity that requires a wireless network interface. How much is it going to cost to add wireless support to our product? If the product is based on technologies that do not fundamentally support wireless networking, we are looking at very costly exit strategy. Another example is application-specific, off-the-shelf solutions. While these are great for many problems and provide a quick way to get something working, what happens when we want to upgrade? Do we have to re-implement the entire system?</p> How does one hedge against the unknowns in the future? One way is to try to anticipate every possible requirement and design it in. I think this is bad strategy because you usually end up spending a lot of time working on things you don’t really need and the quality of the product suffers. A better approach is make sure your product does what it needs to do well and is based on a flexible platform where significant parts of your design and software can be re-used in future versions. This approach has many parallels to recent trends in software development and manufacturing such as Agile Software Development, and Lean Manufacturing.</p> Follow Scott’s advice and make technology selection decisions with the following questions in mind:</p> Is the technology based on open standards?</li> Are programming interfaces and protocols open, published, and been adopted by industry?</li> Is there a community process behind the technology (not just one company)?</li> Can the same technology be sourced from multiple companies?</li> </ul> The use of Linux in embedded systems is an excellent example of a technology that meets the above criteria. You can develop your prototype on a x86 PC. Move it to an embedded ARM single board computer (SBC) for the first production run. Switch to a different SBC or design your own board when you run into new product requirements. Sure, it will cost you more to develop the product initially due to the complexity of a system like Linux, but you now have options to expand and change your product while directly re-using the work you have already done.</p> But wait, as long as I write my application in C, it is portable – right? Most embedded systems support development in C so I will just port my application to a new platform when needed. C is a great language and the language</em> is more or less portable between systems. But as systems get more and more complex, embedded engineers find themselves spending less time writing original code and spending more time putting together pieces that already exist. A modern embedded application is often little more than some glue between various libraries and system functions. The fact that you can develop in C for a particular system does not really mean a lot. What about the threading model, driver model, library APIs, etc. I just spent a significant amount of time recently porting a large C application from a proprietary platform to new hardware platform as the original system was going obsolete. It was a huge amount of work. The driver model was completely different, APIs were different. Changing platforms can be very expensive.</p> It rarely pays to be short sighted when choosing technologies for complex embedded systems. The best strategy is to stay flexible and choose a platform with a future.</p> Gumstix Overo Connector Spreadsheet 2008-12-15T00:00:00+00:00 One of the first things I do when designing a system based on a processor module is create a spreadsheet listing all the connector I/O. This is the easiest way I’ve found to make sure nothing gets missed. My first pass at a I/O spreadsheet for the Gumstix Overo is available at:</p> http://redmine.bec-systems.com/bec/documents/1</a></p> How to disable SSH host key checking 2008-12-12T00:00:00+00:00 This article</a> presents a very nice overview of options to disable ssh host key checking. When working with embedded systems where you are constantly reloading the rootfs, it is really annoying to have to edit your ~/.ssh/known_hosts</em>every time you update the rootfs on the target device. The solution is to simply disable host key checking for your local subnet by adding something like the following to the beginning of /etc/ssh/ssh_config</em>:</p> Host 192.168.1.* StrictHostKeyChecking no UserKnownHostsFile=/dev/null </code></pre> Anything not on the 192.168.1.* subnet is still checked.</p> Update 2009-09-26:</strong></p> Another place to put these config settings is: ~/.ssh/config</em></p> Printing from Embedded Systems 2008-12-02T00:00:00+00:00 How does one implement support for printing in embedded systems? I recently had the opportunity to add printing support to an embedded Linux system. The device is an industrial touch screen powered by a Compulab cm-x270 module (PXA270 CPU), and runs a GTK+ application. The customer is implementing a device calibration system where customers bring their equipment in to get calibrated, and the system prints out a report on a local printer. This article describes how components of Hewlett Packard’s HPLIP solution along with Cairo can be used to implement printer support in a non-desktop Linux system.</p> Requirements</h3> The requirements for this project were fairly basic – we needed to print a single page report that contained text and some elementary graphics. We initially wanted to support several low-end Inkjet and Laser USB printers. As the system was powered by a fairly slow ARM processor (slow compared to modern desktop systems), the solution needed to be efficient, and not require excessive amounts of memory.</p> Desktop Linux Print Flow</h3> With desktop Linux systems, the standard printing flow looks something like this:</p> Application -> PS output -> Ghostscript -> Rasterized output -> Printer Driver (filter) -> I/O Backend -> Printer</p> CUPS is typically used to manage this flow, provide queuing, etc.</p> Print Architecture</h3> As the application is written in GTK+, we decided to generate the report using Cairo, which is a 2D graphics library. Cairo is easy to use, and is well suited for this application. With a Cairo generated report, we already had a raster image of the report, so it seemed that the PS output/Ghostscript steps were not really needed and only added more processing to the data flow. Also the PXA270 CPU does not have a FPU, and PostScript processing can be floating point intensive. So the need was now to figure out how to get a Cairo generated raster image to a USB printer.</p> Hewlett Packard offers lots of interesting software for their printers. Their APDK is a OS independent library in source code form. However, implementing this would have required quite a bit of integration, and writing the I/O layer. It seemed like there should be something available that would work with a little less effort. The HPLIP project</a> is a comprehensive set of software for Linux printing, and is used by desktop Linux systems. It is not obvious at a glance how all the components of HPLIP fit together, but after spending some time digging through source code, and asking questions on several maillists/forums, we were able to figure out that the basic flow is:</p> Ghostscript -> HPIJS (driver) -> HP backend -> Printer</p> In this application, we simply launched the HPIJS driver directly from our application instead of Ghostscript.</p> HPLIP</h3> There were several challenges using HPLIP in our ARM system. The HPLIP is not cross-compilation friendly out of the box, so we had to fix a few issues, and ended up disabling all the Python pieces as we only needed the HPIJS and backend components. We also had to figure out the data flow from the application to HPIJS, and then to HP backend. In a nutshell, the process is:</p> start the hp backend process by forking, and obtain a file handle to STDIN for the hp backend process</li> start the HPIJS process, send various parameters to it including the file handle for the backend STDIN</li> send HPIJS the print raster and tell it to print</li> </ul> There are some additional details such as handling margins, error conditions, cancel support, etc. Some of the details can be gleaned from the Ghostscript source code, and the IJS reference implementation</a> provides some very useful library code for implementing the IJS client functionality in the application that is doing the printing. When finished, our printing module was 403 source lines of code (SLOC) – not bad considering the functionality.</p> How well does it work?</h3> Overall, we are very pleased with the result. With a simple industrial terminal, we can now support just about every USB printer made by HP with the exception of some of their very low end LaserJet printers that require a binary plugin. The HP backend can be used to detect what printer is attached at run time, so everything is plug-n-play with no</em> user configuration. Let me repeat as this is significant – on a simple Industrial terminal, we can support about every HP USB printer available with no user configuration. This is easier for users than their desktop as they simply need to buy a printer and plug it in! Kudos to HP for their excellent open source software. Because the source code is available, we were able to customize it for our application with very little support from HP. Performance wise, the printing process is quite fast; the printer starts almost immediatly after the user initiates the print. Now if I could just get HP to build a ARM version of their binary plugin for the few low end lasers we can’t support … but this is not a critical issue for this vertical application.</p> Future Direction</h3> Hopefully, this type of solution can evolve into a standard printing solution for Embedded Linux systems. Epson also supports the IJS driver model, so adding support for their printers should be possible. It may eventually make sense to integrate portions of CUPS for queing, and other management tasks. Some of the tasks I hope to implement in the future:</p> Clean up the HPLIP build for inclusion in OpenEmbedded.</li> Clean up the IJS reference code library build and packaging in OpenEmbedded.</li> Figure out portions of CUPS that may make sense.</li> Keep conversations going with the Linux printing group so we can someday have a “standard” printing solution for embedded Linux systems.</li> </ul> Thanks to Matt Gessner for helping implement this solution, and for providing feedback on this article.</p> Things that work! 2008-11-19T00:00:00+00:00 Running a small business has a special set of challenges, and if you are not careful, the overhead can quickly consume you. This article is a list of things that I use in my business that work well for me, and help me to be efficient when there is so much to do …</p> Entire article at: /web/content/view/101/9/</a></p> Asterisk</h3> While I can’t say the same for VOIP proiders, the Asterisk PXB is a very solid product. I run it on an ancient 1GHz celeron system, and use it for my office PBX. In combination with a VOIP service, I can:</p> handle multiple calls at once</li> easily conference multiple callers</li> forward calls to my cell phone or other numbers</li> provide extensions for associates who are working with me</li> </ul> and all at a very low cost. To mitigate some of the reliability problems I’ve had with VOIP providers, I set the VOIP provider to use my cell phone as a backup if the VOIP link is down. I also use multiple VOIP providers for outgoing calls so that if one is not working, I can usually use the backup. Overall, I really like the features, and for my business the occasional hassle is worth it.</p> SQL-Ledger</h3> SQL-Ledger is a web based accounting program that is open source. It provides all the features I need to run my business, and offers the following advantages over more traditional programs such as Quickbooks:</p> web based so I can access it anywhere, even remotely.</li> subcontractors and associates can log time remotely directly in my accounting system.</li> granular permissions for different people who need to use it.</li> upgrades just work.</li> consistent, reliable operation that I expect from an accounting system.</li> </ul> OpenOCD</h3> We finally have a low cost, open source JTAG solution that works reasonably well. While there are still issues, it gets the job done for most of my projects, and offers my customers a low cost way to program their systems using JTAG. Often customers are not doing extensive development, and it is nice to be able to provide a solution for the occasional need. One customer fought with a very expensive commercial unit for weeks. I finally shipped them a $50 Olimex unit that did everything they needed. There can’t be enough said for everyone on a team using the same tools, and they only way this can happen easily is if the tools are fairly low cost.</p> Tracks</h3> Tracks is a nice web-based task managment tool to help implement the GTD methodology. The user interface is supurb and better than most native applications I’ve used. I use the “starred” items for my “next action” on each project. Very nice!</p> Printable CEO Emergent Time Tracker</h3> I use David Seah’s handy form</a> for tracking my time during the day. It forces me to use a pen and paper every day which is a pretty rare thing, and is a nice compromise between simplicity and automation.</p> OpenEmbedded</h3> I’ve written a lot about the benefits</a> of OpenEmbedded. It makes the huge task of creating a build system for Embedded Linux systems very manageable and repeatable. When build systems are fully automated, things are so much easier. As an example, I’ve been working on one project for over two years, and have built 313 releases. We have had almost no image failures due to forgetting to put something in the image, or some other manual step you tend to find in build systems. This is the power of a fully automated build system with _no_ manual steps.</p> Ubuntu</h3> Once I become convinced that Ubuntu could do distro upgrades, I finally quit using Gentoo. Between servers, workstations, and home computers, I now maintain about 10 Ubuntu systems. Over the past several years, I’ve had very few problems using the distro upgrade mechanism. Also, the fact that Ubuntu includes fairly recent Linux technology is a big plus for experimenting with components I use in embedded Linux systems.</p> DHL/UPS Pickup</h3> Working from a small office where I do not regularly make shipments, it is really convenient to be able to schedule a pickup with shipping companies like DHL and UPS. This can be done over the web in 15 minutes or so, and a shipping label then prints out on your printer. The DHL website will also provide commercial invoices for overseas shipping. This sure beats driving to the local UPS shipping place, filling out paper work with a pen, waiting in line, etc. Just another thing that makes it easier for a small business to run efficiently.</p> Intel Bare Bones Server</h3> Last year, I was looking for a cost effective 1U rack server. I selected a Intel SR1530 Bare Bones server. I installed it over a year ago and have not touched it since. Last time I checked, the uptime was 254 days.</p> 3-ware Raid Controllers</h3> (I guess it is AMCC now …) They just work with Linux – currently running in two systems.</p> Sandusky Cabinets</h3> Doing embedded system development means I get a lot of equipment from customers. As some of the development stretches over several years with occasional ongoing support, its nice to be able to store this equipment and then find it efficiently. I’ve found the “Sandusky Standard Storage Cabinets” from Staples to work very well. They are fairly cheap, yet durable enough for what I need, and keeps everything enclosed which looks neat, and keeps the dust problem under control. And, Staples offers free delivery.</p> Bob Pease organization system</h3> Bob Pease had an article</a> in Electronic Design some time ago about organizing things in his refrigerator. He used small boxes to group things in his refrigerator. I use the same method in my Sandusky cabinets and it makes it very easy to find something when I need it.</p> </p> GTD Labeling System</h3> I’ve slowing been adopting some of David Allen’s organization ideas from “Getting Things Done”. One of his best ideas is to get a labeller. I used to use a Brother handheld labeller, but lately, I’ve been using a Dymo LabelWriter 400 left over from a project. The LabelWriter connects to your computer via USB, works well with Linux, and allows me to use my standard keyboard, and OpenOffice to create labels. Note, the labels on the boxes in the above picture. The high contrast black on white makes it much easier to find things versus hand writing notes on cardboard boxes, file folders, etc.</p> GTD Filing System</h3> I used to use hanging file folders, but no more. David Allen’s suggestion of simply using plain manila file folders with nice labels works very well. Once again the large, high contrast labels makes it easy to find things.</p> </p> Snaketray Cable Management</h3> I’ve already written about my cable management system</a>. A year later, I still like it and no longer dread digging through cables.</p> Steelcase Leap Chair</h3> I’ve sat in various chairs over the years – mostly the cheap office store variety that most companies buy. I must be getting old enough now that they no longer cut it. I tried a Herman Miller Aeron for a week as that is what everyone recommends, but I found it too hard. I then tried a Steelcase Leap chair and it works great for me. I no longer even think about the chair I’m sitting in, even after a long day. Expensive, but well worth it if you spend as much time sitting as I do.</p> Tomboy Notes + gnome-do</h3> Tomboy notes is a Linux desktop note-taking application that is essentially a personal wiki. It automatically tracks links to other pages, links to file locations, etc and organization is done mostly through links instead of hierarchy. It is simply the best note taking system I have ever used. Coupled with gnome-do, I can instantly bring up any note with only a few keystrokes. In this day and age where information abounds, being able to take and recall notes efficiently is critical to being effective.</p> Konsole</h3> Being able to easily name the tabs in the KDE console application is very handy as I’ll typically have 5-10 named consoles going at one time.</p> </p> 8-port USB serial expander</h3> As I work with a lot of different devices, it is very handy to have lots of serial ports available. The SerialGear device</a> available from http://www.usbgear.com works out of the box with modern Linux distributions.</p> </p> Git</h3> I’m starting to really appreciate the benefits of distributed source control systems. For commercial embedded development, it allows me to easily track upstream projects, yet have a local source control system for tracking project changes. Having everything in one version control system makes it much easier to extract patches to push back upstream, etc.</p> Redmine</h3> I’ve been a happy trac</a> user for years, but have recently switched to using Redmine</a> for project management. Redmine offers some addition features such as multiple project support, and the user interface is a little more polished. As all of my projects are accomplished with distributed teams, it is very important that there be clear visibility as to what is going on in the project. It also helps to give customers the “Warm Fuzzy Feeling</a> TM” as they have clear visibility into the project, and they can easily verify there is progress being made on the project.</p> Benefits of OpenEmbedded switching to Git 2008-10-22T00:00:00+00:00 Recently the OpenEmbedded project has switched to the Git version control system. This is good news for many reasons. The obvious reasons are Git is faster than Monotone, handles branching better, has lots of nice features, larger user base, etc. Monotone has served us well, but as new tools become available, it is time to change. For those of us helping customers use OE for embedded projects, Git will make using OE much easier. Companies want to be able to track OE with minimal effort and typically maintain a few minor tweaks to OE. As most Embedded Linux projects now implement Git infrastructure for kernel development, the OE source tree can be maintained using the same infrastructure. Branching is very natural in Git, so it is fairly easy to create a OE “topic branch” and periodically merge with upstream changes as needed. The ability for easy repo hosting and branching allows OE users to lock down a version of OE for periods of project development, and still maintain a connection to the upstream OE repository for easily updating to new versions, or cherrypicking changes. So a big thank you to all who helped make this change happen.</p> more fun with git branches 2008-08-26T00:00:00+00:00 One thing you learn after using git for awhile is you get used to trying random things, and it will often just work. For example, if I want to know the differences between Linus’s kernel tree, and the Wolfson dev branch for a particular directory:</p> cbrake@happy:/build/linux-2.6$ git diff --stat origin/master..wolfson/dev sound/soc/codecs sound/soc/codecs/Kconfig \| 125 +++ sound/soc/codecs/Makefile \| 34 + sound/soc/codecs/ad1939.c \| 690 +++++++++++++++ sound/soc/codecs/ad1939.h \| 70 ++ sound/soc/codecs/ad1980.c \| 309 +++++++ sound/soc/codecs/ad1980.h \| 23 + sound/soc/codecs/cs4251x.c \| 771 +++++++++++++++++ </code></pre> Another way to get similar information is:</p> git log origin/master..wolfson/dev sound/soc/codecs </code></pre> If I’m working on a topic branch, and I want to see a summary of all the changes in my Topic branch:</p> git log --stat origin/master..origin/my_topic_branch > my_topic_branch_changelog.txt </code></pre> One of the neat things about git branches is you don’t have to be on a branch to interact with it. You can diff, log, and checkout from branches other than the one you are currently in, and it is all very fast. For instance, if I want to grab the latest copy of a file in the Wolfson dev branch, I can do something like:</p> git log checkout wolfson/dev sound/soc/codecs/wm9713.c </code></pre> Not that this is a very good idea, but just to illustrate a point. Git branches are amazing and make development work so much less tedious.</p> Linux Input Testing and Debugging 2008-08-22T00:00:00+00:00 The Linux input layer has made a lot of progress in recent years. When writing a new input driver (such as keyboard, trackball, etc), it is useful to be able to monitor input events using a test application. This article describes two ways to accomplish this using kernel input debugging, and the evtest utility.</p> kernel input debugging</h3> The Linux kernel includes some support for printing input events as debug messages. To use, set the following in the kernel config:</p> CONFIG_INPUT_EVBUG=y </code></pre> Because the print messages are at the KERN_DEBUG level (7), we typically need to change the debug level:</p> echo 8 > /proc/sys/kernel/printk </code></pre> At this point, you will see messages like the following on the console:</p> evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 108, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 106, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: gpio-keys/input0, Type: 1, Code: 103, Value: 1 evbug.c: Event. Dev: gpio-keys/input0, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 1 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 1 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 1, Code: 6, Value: 0 evbug.c: Event. Dev: <NULL>, Type: 0, Code: 0, Value: 0 </code></pre> evtest</h3> The linux-input project (http://linuxconsole.sourceforge.net/input/input.html</a>) includes a neat little tool named evtest that is very handy for testing input devices. If you are using OpenEmbedded, simply:</p> bitbake linux-input </code></pre> If you are not using OpenEmbedded, you can glean enough details about how to get and build the this package from the OE recipe:</p> http://gitweb.openembedded.net/?p=org.openembedded.dev.git;a=blob;f=packages/linux-input/linux-input_cvs.bb</a></p> Install the resulting package on your device. To use, run evtest with a /dev/input/eventx argument:</p> root@machine1:~$ evtest /dev/input/event0 Input driver version is 1.0.0 Input device ID: bus 0x19 vendor 0x1 product 0x1 version 0x100 Input device name: "gpio-keys" Supported events: Event type 0 (Sync) Event type 1 (Key) Event code 103 (Up) Event code 105 (Left) Event code 106 (Right) Event code 108 (Down) Event code 116 (Power) Testing ... (interrupt to exit) Event: time 1009.433605, type 1 (Key), code 108 (Down), value 0 Event: time 1009.433737, -------------- Report Sync ------------ Event: time 1010.735605, type 1 (Key), code 108 (Down), value 1 Event: time 1010.735740, -------------- Report Sync ------------ Event: time 1011.075586, type 1 (Key), code 106 (Right), value 1 Event: time 1011.075720, -------------- Report Sync ------------ Event: time 1011.103379, type 1 (Key), code 108 (Down), value 0 Event: time 1011.103550, -------------- Report Sync ------------ Event: time 1011.333924, type 1 (Key), code 108 (Down), value 1 Event: time 1011.334058, -------------- Report Sync ------------ Event: time 1011.339880, type 1 (Key), code 106 (Right), value 0 </code></pre> Notice that evtest lists the keymap when it is initially run which is very handy, and then lists key events as they occur. The key descriptions are also included with events so you don’t have to manually decode the keycodes.</p> Using a Verizon USB720 modem in an Embedded Linux system 2008-06-02T00:00:00+00:00 </p> The Verizon USB720 Cellular modem is a quick and easy way to add wireless internet connectivity to your embedded system. Why might you want to add cellular connectivity to your embedded device? Remote access and diagnostics is a big reason. One customer I have is planning to supply all their distributors with a USB720 so they can easily upload log files from devices in the field. The alternative is copying the log files to a flash drive, finding a PC connected to the internet, and then uploading the files – too much hassle, therefore it does not get done. Also, think about the cost of sending a field support person on-site to troubleshoot problems versus shipping them a USB720 and accessing the device remotely.</p> The USB720 is a fairly small device that plugs into the ubiquitous USB port. This articles covers some of the details on how to configure an embedded Linux system to work with this device.</p> Why is the USB720 a good fit for wireless connectivity in Embedded Systems?</h3> Some reasons the USB720 (or other USB based cellular modems) is an attractive option:</p> fairly fast: provides typical download speeds of 600-1400kbps, and upload speeds of 500-800kbps</li> convenience: it plugs into a USB port. No taking apart the box and installing an embedded module, etc.</li> affordable: with plans around $60/mo, its a lot cheaper than sending field support people on-site to debug every problem.</li> uses standard PPP protocols for connection. That means it works with Linux.</li> supported by the mainstream Linux kernel</li> </ul> </p> Kernel Configuration</h3> The USB720 looks like a USB-serial device to the host system. If you plug it into your Ubuntu desktop Linux system, all the needed kernel pieces are there and it will just work. With your embedded system, you need to make sure a number of kernel options are enabled including:</p> CONFIG_PPP=m CONFIG_PPP_ASYNC=m CONFIG_PPP_DEFLATE=m CONFIG_SLHC=m CONFIG_USB_SERIAL=m CONFIG_USB_SERIAL_OPTION=m </code></pre> The CONFIG_USB_SERIAL_OPTION selection enables a driver that is specifically optimized to work with the USB chip found in many cellular modems. Once you enable the above modules, and if you are running udev, simply plugging in the device will cause the appropriate modules to load and a /dev/ttyUSB0 device will now appear.</p> PPP Scripts</h3> The most difficult part of this exercise is figuring out how to configure the PPP scripts. Verizon of course does not tell you how to configure their device and only supplies software for Windows and MAC. There are a number of howtos available on the internet, but most of them are tailored to using kppp or wvdial. Neither are very practical for embedded systems. Fortunately, the standard pppd daemon is fairly easy to use, is easy to build for embedded systems (it is already included in OpenEmbedded), and seems to work very well. The following scripts are needed:</p> /etc/ppp/peers/verizon</p> noauth connect "/usr/sbin/chat -v -f /etc/ppp/peers/verizon_chat" defaultroute usepeerdns ttyUSB0 921600 local usepeerdns debug -detach </code></pre> /etc/ppp/peers/verizon_chat</p> '' 'ATZ' 'OK' 'ATDT#777' 'CONNECT' '' </code></pre> /etc/ppp/chap-secrets /etc/ppp/pap-secrets</p> "<phone number>@vzw3g.com" * "vzw" </code></pre> At this point, you can run pppd call verizon</em> and you should see something like:</p> root@cm-x270:~$ pppd call verizon Serial connection established. using channel 9 Using interface ppp0 Connect: ppp0 <--> /dev/ttyUSB0 Warning - secret file /etc/ppp/pap-secrets has world and/or group access sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x539657b7> <pcomp> <accomp>] rcvd [LCP ConfReq id=0x0 <mru 1500> <asyncmap 0x0> <magic 0xb9b652e5> <pcomp> <accomp>] sent [LCP ConfAck id=0x0 <mru 1500> <asyncmap 0x0> <magic 0xb9b652e5> <pcomp> <accomp>] rcvd [LCP ConfAck id=0x1 <asyncmap 0x0> <magic 0x539657b7> <pcomp> <accomp>] sent [CCP ConfReq id=0x1 <deflate 15> <deflate(old#) 15>] sent [IPCP ConfReq id=0x1 <compress VJ 0f 01> <addr 192.168.1.119> <ms-dns1 0.0.0.0> <ms-dns3 0.0.0.0>] rcvd [LCP DiscReq id=0x1 magic=0xb9b652e5] rcvd [LCP ProtRej id=0x2 80 fd 01 01 00 0c 1a 04 78 00 18 04 78 00] rcvd [IPCP ConfReq id=0x0 <addr 66.174.61.4>] sent [IPCP ConfAck id=0x0 <addr 66.174.61.4>] rcvd [IPCP ConfRej id=0x1 <compress VJ 0f 01>] sent [IPCP ConfReq id=0x2 <addr 192.168.1.119> <ms-dns1 0.0.0.0> <ms-dns3 0.0.0.0>] rcvd [IPCP ConfNak id=0x2 <addr 70.210.82.191> <ms-dns1 66.174.95.44> <ms-dns3 66.174.92.14>] sent [IPCP ConfReq id=0x3 <addr 70.210.82.191> <ms-dns1 66.174.95.44> <ms-dns3 66.174.92.14>] rcvd [IPCP ConfAck id=0x3 <addr 70.210.82.191> <ms-dns1 66.174.95.44> <ms-dns3 66.174.92.14>] not replacing default route to eth0 [192.168.1.1] local IP address 70.210.82.191 remote IP address 66.174.61.4 primary DNS address 66.174.95.44 secondary DNS address 66.174.92.14 Script /etc/ppp/ip-up started (pid 2303) Script /etc/ppp/ip-up finished (pid 2303), status = 0x0 </code></pre> root@cm-x270:~$ ifconfig ppp0 Link encap:Point-to-Point Protocol inet addr:70.210.82.191 P-t-P:66.174.61.4 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:4 errors:0 dropped:0 overruns:0 frame:0 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:3 RX bytes:64 (64.0 B) TX bytes:94 (94.0 B) </code></pre> You are connected :-)</p> Using the OE ppp package</h3> Add ppp support to your image using OE is fairly straightfoward:</p> add the kernel modules to your image</li> addppp</em> to your image</li> add a custom package that includes the configuration scripts listed above</li> </ul> One thing that may trip you up is the ppp scripts are expecting resolv.conf to be in the following location: /var/run/resolv.conf. The reason for this is we want resolv.conf in a RAM file system to prevent unneeded flash wear. Therefore, /etc/resolv.conf should be configured in your rootfs as a soft link to /var/run/resolv.conf.</p> Summary</h3> The Linux kernel support for USB720 and the pppd package offer a quick, robust way to add cellular network connectivity to about any embedded Linux system that has a USB host port. As systems become more complex, it will be increasingly important to be able to remotely access systems in the field to troubleshoot problems, help users with configuration, etc. The USB720 offers a quick way to provide this type of connection without going through the expense of designing cellular connectivity into your product. One thing I usually encourage all my customers to do when designing new products is expose a USB host port, even though they may not see any immediate use for it. This is one of those reasons.</p> Below is a picture of a USB720 connected to a Compulab cm-x270 development system.</p> </p> Autotools quick reference 2008-05-03T00:00:00+00:00 At some point, most Linux developers need to master Autotools. For most of us, this is a fairly painful process, but like any good tool, Autotools is extremely useful and well worth learning. For example, if your program is set up correctly using Autotools, it will cross compile with almost zero effort in OpenEmbedded, or with slight effort manually. Compare this to the tedious effort of manually referencing toolchain locations, cross toolchain library and header file locations, etc. Despite first impressions, Autotools will make your life easier. This article contains a collection of tips on how to do standard operations using Autotools.</p> Minimal Autotools files required for a simple C application</h2> configure.ac</strong></p> AC_INIT([fooprog], [1.0], [email@company.com]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_PROG_CC AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([Makefile]) AC_OUTPUT </code></pre> Makefile.am</strong></p> bin_PROGRAMS = foo foo_SOURCES = foo.c </code></pre> The above files will create a binary name foo (compiled from foo.c) when you run the following steps:</p> touch NEWS AUTHORS ChangeLog</li> autoreconf -i</li> ./configure</li> make</li> </ul> How to add libraries to the link step</h2> For example, if you want to link in the pthread library to your application, add the following line to Makefile.am:</p> foo_LDADD = -lpthread </code></pre> How to add compiler options</h2> You can easily add compiler options using the foo_CFLAGS or foo_CPPFLAGS variables.</p> foo_CFLAGS = -DMY_DEFINE</p> How to use pkg-config in Autotools</h2> pkg-config is a helper tool that gives you library and include file information for various libraries installed. For example, to link to alsa and glib in your project:</p> configure.ac</strong></p> PKG_CHECK_MODULES(PACKAGE, alsa glib-2.0) AC_SUBST(PACKAGE_CFLAGS) AC_SUBST(PACKAG_LIBS) </code></pre> Makefile.am</strong></p> foo_CFLAGS = @PACKAGE_CFLAGS@ foo_LDADD = @PACKAGE_LIBS@ </code></pre> How to add a command line option to the configure script</h2> AC_ARG_ENABLE(option-foo, [ --enable-option-foo help for option foo], [CFLAGS="$CFLAGS -DOPTION_FOO_DEFINE"], []) </code></pre> The above example sets a define on the compiler command line if configure is run with the --enable-option-foo</em> command line option.</p> Disable shared libraries in your project</h2> Add the following to configure.ac:</p> AC_DISABLE_SHARED </code></pre> Pass a variable from configure.ac to Makefile.am</h2> Often there is a need to pass a variable in configure.ac to a Makefile. This is done with the AC_SUBST macro:</p> configure.ac</strong></p> AC_SUBST(VAR_FOO) </code></pre> Makefile.am</strong></p> foo2 = @VAR_FOO@</p> Installing data files</h2> By default, Autotools will install executables built when you run “make install”. To install other non-executable files, you need to tell autotools explicitly about those files. The standard place to install data files for an application is in $(datadir)/appname.</p> Makefile.am</strong></p> fooextradir = $(datadir)/@PACKAGE@ </code></pre> fooextra_DATA = \ foo_data1.txt \ foo_data2.txt </code></pre> The above will install the foo_data1.txt</em> and foo_data2.txt</em> files in the $(datadir)/fooprog</em> directory. $(datadir) is typically /usr/share/ or /usr/local/share, but it can be anything.</p> How does my program know where $(datadir) is located</h2> Because $(datadir) can be anywhere in the system, we need some way to tell the program where it was specified to be at time of compilation so that it can locate foo_data1.txt and foo_data2.txt. This can be done by passing a define at compile time:</p> Makefile.am</strong></p> AM_CFLAGS = -DDATADIR=\"$(datadir)\" </code></pre> How to install data files during development without cluttering up my system directories</h2> If you use the above method to install data files needed by your project, you may run into a case during development where you want to run the program from the build directory, but the program needs the data files at run time. Since, the program is looking for them in the location specified by the DATADIR define, they need to be installed. You could just run “make install”, but this would install them into /usr/local/share, require you to run as root, and clutter up your system directories. A much more elegant solution is to specify the install prefix at configure time to be a temporary “install” directory in your source tree. The --prefix option for configure requires an absolute directory, so the `pwd` trick is used to figure out what the current directory is.</p> ./configure --prefix=`pwd`/install make install </code></pre> Your project will now be installed in a directory in your source tree named ./install. This is also a nice way to test that the install for the project is working correctly as you can easily see everything that gets installed. Now, the program has been compiled to reference extra files in ./install and will run just fine from the build directory.</p> The Linux kernel container_of macro 2008-04-02T00:00:00+00:00 The Linux kernel contains a very useful macro named “container_of” that is used extensively in back-casting a data structure to its containing data structure. This article includes a simple program that illustrates how this macro is used, and explains why it is so useful.</p> If you do a lot of C programming, this program is worth figuring out :-).</p> The program source can also be downloaded from the following location: /linux/linux_container_of_example.c</a></p> /* test code to illustrate use of Linux kernel container_of macro * * Copyright (c) 2008 Cliff Brake, BEC Systems LLC * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. * / #include <stdio.h> #include <string.h> #include <stdlib.h> / This program illustrates how the container_of macro works. * The container of macro is very useful in multi layered * software systems where you have progressivly more detailed * software layers. Below is an example of a bus layer, * and then a device layer where a number of different * devices might register with the bus. * The device registers itself with the bus subsystem, and * then the bus subsystem makes a callback into the device. * Normally if there are multiple devices registered, the * bus subsystem must store and pass a device structure * when making callbacks. With the container_of macro, this is * no longer necessary, and the bus subsystem only has to * know about one generic device structure, and does not need visibility * into lots of different device structures, or do tricks * by casting void pointers, etc. With the container_of macro * we can backcast from the generic data structure, to the containing * datastructure. This forces good separation of code in that * that bus layer cannot modifiy data structures that are specific * to the device layer. * / /* * (from Linux kernel source) * container_of - cast a member of a structure out to the containing structure * @ptr: the pointer to the member. * @type: the type of the container struct this is embedded in. * @member: the name of the member within the struct. * / #define container_of(ptr, type, member) ({ \ const typeof( ((type )0)->member ) __mptr = (ptr); \ (type )( (char )__mptr - offsetof(type,member) );}) #define offsetof(TYPE, MEMBER) ((size_t) &((TYPE )0)->MEMBER) /========================================================== BUS layer code ==========================================================/ /* generic bus device structure / struct bus_device { int general_device_param_a; int general_device_param_b; void (device_callback)(struct bus_device * bd); }; /* the following is a global list of * devices that have registered with the * bus subsystem. Normally this would * be something like a dynamic linked list. / struct bus_device bd_list[5]; /* function to register a device with the bus / void register_with_bus(struct bus_device bd) { /* since this example only deals with one * device, will put it in slot 0 / bd_list[0] = bd; } void start_bus() { int i; struct bus_device bd; /* make callbacks to all devices on bus / for (i=0;i<sizeof(bd_list)/sizeof(bd_list[0]);i++) { bd = bd_list[i]; if (!bd) continue; / call device callback with generic * bus device structure / bd->device_callback(bd); } } /========================================================== * device X specific code * this would normally be in a different module ==========================================================/ /* structure that holds device X specific stuff, as well as * generic bus_device structure / struct device_x { int device_x_specific_param_a; int device_x_specific_param_b; struct bus_device bd; }; void device_x_callback(struct bus_device bd) { /* if we know the structure type that contains the bus_device structure, * we can extract a pointer to the containing structure using the container_of * macro / / ptr type member / struct device_x devx = container_of(bd, struct device_x, bd); /* the above statement expands to * struct device_x * devx = ( * { * const typeof( ((struct device_x )0)->bd ) __mptr = (bd); * (struct device_x )( (char )__mptr - ((size_t) &((struct device_x )0)->bd) ); } * ); / printf("device_x_callback called!, device_x_specific_param_a = %i\n", devx->device_x_specific_param_a); } void device_x_init() { / dynamically allocate structures / struct device_x devx = malloc(sizeof(devx)); memset(devx, 0, sizeof(devx)); /* set a parameter in the device_x structure so * we can test for this in the callback / devx->device_x_specific_param_a = 1001; / set up callback function / devx->bd.device_callback = device_x_callback; / we register the generic bus device structure * as the bus layer does not need to know * about the device_x stucture. Note, the * devx structure is not stored anywhere, yet * its location is being preserved without * specifically passing it to the bus * layer. / register_with_bus(&devx->bd); } int main() { / test the above system / / first, initialize device_x / device_x_init(); / now, start the bus. This should make * a callback into the device_x / start_bus(); } / when run, this program returns: * device_x_callback called!, device_x_specific_param_a = 1001 / </code></pre> Should you be using monotonic timers? 2008-03-19T00:00:00+00:00 In a previous article</a>, I covered some of the basics of Linux timers. Any time you are doing any type of fixed time delay in a program, you should really be using monotonic times, so the delay will not be affected by system time changes. In an effort to save cost, some embedded systems today do not have a battery backed up RTC, and instead get the time via GPS, NTP servers, or other clever means. What this means is your applications had better be able to handle the system time changing as the system time may not be set until well after the unit boots. This article describes how you can quickly test your system for timer problems.</p> There are two cases where delays may fail if you are using non-monotonic timers. The first is if the time advances forward by a large amount. Delays will expire immediately in this case. The other case is if the time advances backward by a large amount. Delays will never expire in this case. To test for these situations, write a simple test application (sample included below) that rapidly changes the time and then test your application while the time is changing.</p> #include <stdio.h> #include <time.h> int main() { time_t system_time; static int count = 0; printf("Starting test application...\n"); while(1) { // Sleep for 0.01 seconds usleep(10 1000); system_time = time(NULL); system_time += 30; stime(&system_time); printf("Cycle %d - system date/time set to %s", ++count, ctime(&system_time)); } } </code></pre> Compulab cm-x270 kernel update to 2.6.24 2008-02-29T00:00:00+00:00 The kernel support</a> in OpenEmbedded for the Compulab cm-x270 has been updated to version 2.6.24. The 2.6.24 kernel opens up several possibilities including better real time and high resolution timer</a> functionality, an improved SD Card driver, and a full SDIO stack with support for SDIO Wifi devices</a>. Also included is a patch to set the default framebuffer depth to 16 for several commonly used displays. Xfbev really does not like when you give it a 8-bit framebuffer when it is expecting 16, and typically returns a very unhelpful error message like: “error: Invalid argument”.</p> GESBC-9302E kernel update to 2.6.24, and reboot fix 2008-02-18T00:00:00+00:00 I recently updated the GESBC-9302 machine support in OpenEmbedded to include the 2.6.24 released kernel. Also, Glomation has kindly provided me with a patch for the software reboot problem which I have integrated into the OpenEmbedded build for the GESBC. The patch files are part of the OpenEmbedded dev branch or can be obtained here:</p> http://cgit.openembedded.net/cgit.cgi?url=openembedded/tree/recipes/linux/linux-2.6.24/gesbc-9302</a></p> Other related OpenEmbedded bits:</p> http://cgit.openembedded.net/cgit.cgi?url=openembedded/tree/recipes/linux/linux_2.6.24.bb</a></li> http://cgit.openembedded.net/cgit.cgi?url=openembedded/tree/conf/machine/gesbc-9302.conf</a></li> </ul> Future features to be integrated include RTC driver configuration, and support for loading Initramfs images from NOR flash.</p> Glomation GESBC-9302E Review 2008-01-29T00:00:00+00:00 Recently, I’ve had the opportunity to work with a Glomation GESBC-9302E single board computer. This SBC contains a Cirrus Logic EP9302 ARM processor and options for plenty of RAM and Flash memory which means you have many options for writing your applications including high level languages like C# and Python. In this review, I’ll cover some of basic features of this board and where this board might be useful.</p> Why the GESBC-9302?</h3> One of the most attractive features of the GESBC-9302E is cost. The base model with 32MB of RAM, and no NAND flash costs $95 at quantities of 1 unit. For a model with 64MB of RAM, and 128MB of flash, the cost is $120. Looking at the board, its simplicity is very evident. There is basically a CPU, flash, ram, a few linear power supplies, and a very limited amount of support circuitry. There really is not much to it.</p> </p> We are helping a customer use the GESBC-9302E in a project where it is functioning as a gateway between a local Zigbee wireless network, and a server located on the internet. The customer had originally considered using a Linksys NSLU2 for the gateway device during the alpha testing, but upon consideration it seemed like the GESBC-9302E is a better choice in this application for the following reasons:</p> the GESBC does not cost a lot more than a NSLU2</li> you can buy the GESBC with plenty of NAND flash.</li> the GESBC provides two serial ports</li> the GESBC provides many signals on headers than can be used with cables or a daughter card.</li> </ul> Having raw NAND flash on the board is a significant advantage in my opinion. I’ve run into considerable hassles with consumer grade flash devices in the past (SD, USB Flash, etc). Although there are industrial grade flash devices, they tend to be more expensive. My preference at this point for reliable storage in an embedded Linux system is JFFS2 on a 64-128MB raw NAND flash device. With 128MB of NAND flash, there is plenty of space</a> for a full featured Linux distribution.</p> With 2 USB ports, plenty of general pupose I/O pins (GPIO), and lots of memory, there is not a lot you can’t do with the GESBC in your typical headless control or gateway type application. Being able to interface to a number of USB peripherals, and support for high level languages makes this SBC+Linux ideal for implementing a device that may need to talk web services or other complex network protocols.</p> Software Support</h3> Most of the GESBC functionality is fairly well supported in the mainstream Linux kernel. Simply configure the kernel for the EDB9302 machine and most things will just work. There are several pieces missing from the mainstream kernel for the GESBC that are useful: NAND flash support, and a change needed to make the Ethernet work. For the NAND flash, I extracted a MTD driver from the Glomation kernel source code. The Ethernet change is a small tweak to the platform files. These changes are available for the 2.6.24 kernel in a public git repository at: http://dev.bec-systems.com/linux-2.6-gesbc-9302.git.</p> The GESBC uses Redboot for the bootloader. No complaints with the bootloader so far – it does everything I need including loading images from a TFTP server. The bootloader does not support writing images to NAND flash, so to load an OS image into NAND flash you typically have to boot a Linux OS from NOR flash, USB flash, or NFS root, and then use the userspace mtdutils to write an image into NAND flash.</p> Glomation provides a sample Linux image, but I prefer to use images generated by OpenEmbedded so that I have support for things like C#</a> , Python, and thousands of other packages. Support for the GESBC has been added to OpenEmbedded and I’m in the process of updating the kernel support to 2.6.24.</p> Summary</h3> Overall I’m very pleased with this SBC. Glomation has been very responsive to requests and has been a good company to work with. I still need to apply a fix for a known soft reboot issue with the 2.6 kernel and enable a driver for the RTC on the SBC. Future articles will detail how to load Linux images built with OpenEmbedded on this device.</p> Marvell Embedded SDIO Wifi Success 2008-01-09T00:00:00+00:00 As detailed in the article I wrote back in September of 2007 (/web/content/view/75/9/</a>), getting embedded wifi modules functioning is not a simple task. However, due to recent advances in the Linux kernel, it looks like a viable solution for low-mid volume products is emerging. This article provides a few details on how to get a Zcomax (Zcom) XG-180MU module working with a PXA270 processor.</p> System Setup</h3> The setup used includes:</p> Zcomax XG-180MU SDIO Wifi module</li> Compulab cm-x270 + sb-x270 system (includes the PXA270 processor)</li> Linux kernel 2.6.24-rc5</li> OpenEmbedded</li> </ul> Support for the Marvell 8385 and 8686 based Wifi chipsets is now included in the mainstream Linux kernel 2.6.24 release candidate source code. To use this driver, enable the CONFIG_LIBERTAS_SDIO option. I also had to enable the CONFIG_WIRELESS_EXT option to prevent a compile error, but this is probably required anyway to build a functioning wireless system.</p> Firmware</h3> The firmware for the Marvell SDIO wifi solutions is downloaded at runtime, so the next challenge is to find firmware files. I am working on a project that has access to a driver from Marvell, so I was able to extract this firmware from their source code header files. I’m sure there are other places where this firmware is available. The firmware files must be placed in the following location:</p> root@cm-x270:~$ ls /lib/firmware/sd* /lib/firmware/sd8385.bin /lib/firmware/sd8385_helper.bin </code></pre> The version of udev used in OpenEmbedded determines the location of the firmware files. The Libertas Linux driver requests these firmware files, and a userspace component of udev provides these files to the kernel.</p> Success</h3> The following kernel modules are required:</p> root@cm-x270:~$ lsmod Module Size Used by pxamci 7008 0 libertas_sdio 8488 0 mmc_core 46868 2 pxamci,libertas_sdio libertas 85288 1 libertas_sdio ieee80211 29956 1 libertas ieee80211_crypt 4768 1 ieee80211 root@cm-x270:~$ </code></pre> And the result:</p> root@cm-x270:~$ modprobe pxamci root@cm-x270:~$ mmc0: new SDIO card at address 0001 libertas: eth1: Marvell WLAN 802.11 adapter </code></pre> ifconfig -a </code></pre> ... </code></pre> eth1 Link encap:Ethernet HWaddr 00:60:B3:34:77:66 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) </code></pre> Very nice!</p> So it looks like there is finally an embedded Wifi solution for products that sell less than 500,000 units per year. I also briefly tried this on a AT91SAM9260 system, but it did not work. So there is still some work to do on the Linux SDHC driver for the AT91SAM9260.</p> C# on embedded ARM Linux systems is now practical 2007-12-15T00:00:00+00:00 Update</strong>: Michael Dominic K. has kindly provided feedback on this article and has some additional points</a> about Mono and embedded systems.</p> Mono has included support for ARM systems for some time, but with the release of Mono 1.2.6, and its inclusion in OpenEmbedded, it is now practical and easy to build and run C# applications on your everyday ARM Linux system. Several ARM issues in 1.2.5 have been fixed, and thus far it seems the average program runs quite well. In this article, we’ll explore what advantages C# offers over more traditional languages such as C or Python, and how OpenEmbedded simplifies adding C# support to your embedded system.</p> Why C#</h3> Why is C# an attractive language for embedded Linux systems? There are many reasons including memory protection and exception handling, native types, good threading support, extensive library, and developer productivity. In the past, I’ve typically used C or Python for implementing large applications on embedded Linux systems. This section will compare C# to my experiences with both languages.</p> A typical experience when implementing a large application in an embedded system is to deploy the system for field testing (before it is ready) and start getting reports that the application is crashing. Somewhere in the mass of C code your team just wrote there is a stray pointer clobbering something it should not. Of course this does not happen in the lab where you have access to the machine, but in some piece of industrial equipment many miles away. And, you cannot reproduce the problem in the lab because with embedded systems, simulating a real world situation is challenging because it is difficult to simulate real I/O. This is less of a problem with desktop applications where you are less dependent on external I/O. Sound familiar? Even if you can reproduce the problem on a machine you have access to (perhaps even remotely</a>), the problem can be caused by a stray pointer corrupting data in a completely unrelated piece of code. So while the debugger may tell you what got clobbered, you still don’t know exactly who did it. If you have lots of time and a team to do lots of testing, this is less of an issue, but available resources and time to market sometimes don’t give you all you need to do the job right before it gets to the field. One solution to this problem is use a language that has exception handling (I’ve used Python a lot in the past</a>), and log all exceptions. If something bad happens in the field, I ask for the log. With the stack trace in the log, and some focused testing I can usually find the problem relatively quickly. Add in the memory protection and things get a lot easier. For small applications this is not so much of an issue, but lets face it, now that we have access to very capable hardware at a low cost</a>, applications in embedded systems tend to get large and complex. C# gives you exception handling and memory protection that makes problems much easier to find and debug; especially during field testing.</p> While its hard to beat Python or Ruby for developer productively and ease of use, there are performance issues like threading support, and support for static types that must be weighed in the balance. With Python, everything is an object or a reference. The checking is done at run time to make sure a variable is of the correct type. With C#, this is done at compile time (much like C). While it is generally more work to write code in C# than Python, the compiler tends to catch more errors for you. There is probably also a performance advantage to being able to implement simple variables statically. Threading support is another consideration and is often critical in embedded systems because you are usually tending to I/O, processing data, running a user interface, etc. This type of problem is well suited to a threaded application. Threading in Python is implemented using a global interpreter lock mechanism that is inefficient in some situations. C# threading support is more similar to C. While it may require you to do more manual locking than Python, it gives you more control.</p> When it comes to developer productivity, in my experience C# is more work to write than Python or Ruby, but less work than C. So it seems to fall somewhere in the middle of the spectrum.</p> So, C# seems to fall nicely into the middle ground between C and very high level languages such as Python and Ruby. It offers many of the advantages at both ends of the spectrum and seems to be a nice balance for programming embedded systems.</p> Mono</h3> What is Mono? From the Mono web site</a> :</p> Mono provides the necessary software to develop and run .NET client and server applications on Linux, Solaris, Mac OS X, Windows, and Unix.</em></p> Mono includes all the necessary components to build and run C# applications on Linux systems including embedded ARM Linux systems. The Mono source tree is fairly clean and is nicely segmented between native and managed code. This is critical when it comes to cross compiling. The Mono team has also stuck with traditional build tools such as autotools which makes the project a lot easier to cross compile for an embedded target.</p> Mono includes a very extensive C# library that is comparable to Microsoft’s implementation. Most of what you might ever need is built right into the Mono class library; much like Python or Java. With C or C++, you typically need to add a number of additional libraries to the system.</p> Mono Support in OpenEmbedded</h3> OpenEmbedded includes comprehensive support for Mono including crosscompiling support, granular packaging of the Mono class library, and automatic run-time dependency generation for Mono applications. In this section, we’ll see how OpenEmbedded can make your life a lot easier.</p> Mono is typically cross compiled in two steps</a> :</p> The managed portions are generated by compiling Mono for the host machine architecture. A very nice feature of the Mono build process is that it can bootstrap itself.</li> The native only portion of Mono is then cross compiled over top of the above output and the native host binaries will then get replaced with the cross compiled target binaries.</li> </ul> To implement this, there are two recipes implemented in OpenEmbedded. The mono-mcs-intermediate</em> is used to generate all the managed binaries, while the mono</em> recipe builds only the native portion of Mono. The first step when building the mono</em> cross recipe is to extract the results of the mono-mcs-intermediate</em> in the install directory and then proceed to overlay the native binaries produced by the mono</em> recipe in that same directory. As mentioned before, the use of standard tools like autotools makes all this work pretty well. All of this is automated in OpenEmbedded and happens when you run “bitbake mono”.</p> One issue with embedded systems is you typically have limited flash/disk space. So you only want to include that pieces of Mono in your target image that your application needs. The OE build of Mono makes this very easy by packaging up the Mono class library into a number of different packages. The dependencies between packages is automatically tracked so that if you add a Mono package to an OE build, its dependencies will automatically be included with no extra work on your part. This mechanism uses the monodis</em> tool to figure out what each Mono library provides and the dependencies of each library. This saves a huge amount of very tedious manual work to have all this automated.</p> Building and Running your Mono application</h3> Now that we have Mono running on our target system, there are a number of ways to build Mono applications to run on the target. The quick-n-dirty method is to simply build the mono binary on any system, copy it to your embedded system and run it. Building and running your applications is easy as Mono binaries are platform independent and will run on any system that supports Mono. So, you don’t need to worry about cross compiling, toolchains, etc.</p> The quick-n-dirty method works well for getting started, experimenting, etc, but there are several advantages to integrating your Mono application into an OpenEmbedded build, even though you technically don’t need to worry about cross compiling. The first reason is packaging. A very simple OpenEmbedded recipe provides an easy way to automatically fetch your project from a separate version control system, build it, and package it properly into the OS image for your target. It also gives you the ability to deploy updates to target systems using the ipkg package manager. The second reason is automatic run-time dependencies. If you inherit the mono</em> OpenEmbedded class, all of the Mono components your application needs are automatically added to the image. An example OpenEmbedded recipe for a Mono application is shown below:</p> DESCRIPTION = "My Application" DEPENDS = "mono" SRCREV = "${AUTOREV}" PV = "1.0+svnr${SRCREV}" PR = "r1" SRC_URI = "svn://svn.mycompany.com/svn/myproject/;module=myapplication;proto=https" S = ${WORKDIR}/myapplication </code></pre> inherit autotools mono </code></pre> The above recipe assume you have set up your application to be built with autotools, which is a good idea. This page</a> provides lots of good information on building Mono applications using autotools. You can also look at a number of open source applications such as Tomboy</em> for additional clues about how to use autotools with Mono. For a simple application that I built, the following run time dependencies where automatically determined by OpenEmbedded: mono, libmono0, libmono2.0-cil, libmono-corlib2.0-cil.Each of these dependencies may have additional dependencies, and in this case a dozen or so mono packages ended up being installed.</p> Summary</h3> As embedded systems become larger and more capable, we can expect that higher level languages such as C# and Java will start to become more common. In the past, the focus on embedded systems programming was often efficiency, minimizing resource utilization, etc. Now that very capable hardware is becoming cheaper</a>, the focus in some systems is shifting to features and time to market. Operating systems like Linux and languages like C# help us reach those goals and develop competitive products quickly with small teams. C# provides the features needed to rapidly develop, deploy, and debug complex applications that still perform reasonably well. OpenEmbedded provides a way to manage builds, so that you can focus on real work instead of tedious build issues.</p> Mono 1.2.6 added to OpenEmbedded 2007-12-14T00:00:00+00:00 I recently updated the Mono recipes in OpenEmbedded to include the Mono 1.2.6 version. Several critical bugs that were present in 1.2.5 have been fixed, and its looking very good. More on this later …</p> Embedded Linux development on Windows??? 2007-11-30T00:00:00+00:00 From http://www.embedded.com/design/opensource/204300381</a> :</p> Host platform</strong> Although everything I will describe can be also done on a Windows host, I recommend using Linux. It’s more convenient, and more tools and utilities are available. And if you rely on a few Windows applications such as Word and Outlook, you can still run them on Linux in emulation using VirtualBox, Wine, or other commercial package. If you’re new to Linux, using a Linux host will also force you to learn the new platform faster.</em></p> </blockquote> How true. Those who think they can do embedded Linux development without learning the basics of using a Linux desktop computer are in for a long hard road, and they are missing out on a valuable opportunity to learn a lot about embedded Linux development. This is the primary advantage of embedded Linux – there are many things in common with desktop and server systems. There is a tremendous amount of reuse. Things can be easily prototyped on your desktop Linux system before deploying to an embedded system. Instead of viewing the suggested that a Linux workstation be used for embedded Linux development as stumbling block, it should be viewed as a huge advantage.</p> Compulab EM-X270 Review 2007-11-07T00:00:00+00:00 The EM-X270 is a full featured computer board from Compulab (http://www.compulab.co.il/x270em/html/x270-em-datasheet.htm</a>) designed for handheld/mobile applications. The board includes options for about anything you might need in a portable system including a PXA270 processor, GSM radio, GPS, Wifi, Bluetooth, Audio, SD/MMC, USB, battery circuitry, and the list goes on. The EM-X270 is designed to give you a big head start in designing a handheld computing device, and can significantly reduce time to market and design costs. This article provides a high level overview of this product, how it might be used, and lots of photos.</p> EM-X270 Overview</h3> The EM-X270 is a computer board about the size of a display you might find in a typical PDA. While the size is probably a little large to be used in consumer devices like phones and PDAs, it seems like a nice size for slightly larger products used in vertical industries.</p> </p> As already mentioned, the EM-X270 includes about all the base features you might need in a mobile computing device. An expansion connector is also included that allows you to easily add custom circuitry to the system. The stack-up between the EM-X270 and an expansion board is shown below:</p> </p> The EM-X270 differers from most other embedded computer modules in that it is a little more application specified, and quite a bit more circuitry is included in the design such as power supplies, battery charging circuitry, radios, etc.</p> Why EM-X270?</h3> Why are we interested in the EM-X270? If all you need is a generic handheld computer, there are plenty of options from PDAs to off-the-shelf industrial handhelds. However, if you need a handheld that provides some additional features not typically found in existing devices, then you can use the EM-X270 plus a custom baseboard and packaging to add your high value features. Depending what you need to add, the expansion board can be a simple 4 layer board which is very easy to design and manufacture. All of the complex PCB design and assembly is already done for you on the EM-X270 module. This allows low to moderate volume manufactures design products where they would not otherwise have the time and budget for designing a full custom processor board. Some example applications might be handheld products for the medical, industrial, and test and measurement industry.</p> Compulab’s pricing model is also very interesting. Compulab will custom build EM-X270 modules with just the options you need. The base price without any extra features (such as radios) is $122 for 1000 pieces.</p> The EB-X270 Evaluation Kit</h3> Compulab provides a EB-X270 Evaluation kit for getting started with EM-X270 development. Like their other Evaluation klits, is is very full featured and very well done. An extender board is provided that adds Ethernet, JTAG, and several other useful connectors for development.</p> </p> The eval kit also includes all necessary cables, a display, battery, etc.</p> </p> Software for the EM-X270</h3> Compulab provides software support for both Windows CE and Linux. This includes very up-to-date support for kernel version 2.6.23. EM-X270 machine support has also been included in the OpenEmbedded project for building full featured Linux images. This combination will drastically reduce development costs compared to solutions provided by many other vendors. All too often, single board computer vendors provide an ancient kernel, a hacked up root file system and call it a “Linux BSP”. Actually, any time the term “Linux BSP” is used, you should probably be a bit wary. A much more attractive solution for those of us building products using Linux is for processor and module vendors to say “our products are supported in the mainstream kernel and OpenEmbedded sources”. Compulab is doing it right.</p> OpenEmbedded provides many interesting options for devices like the EM-X270. One of the most interesting might be the ability to run the OpenMoko software on industrial devices.</p> Compulab provides a Linux demo image built with OpenEmbedded that runs the GPE Palmtop Environment. The method for loading this image is interesting, and similar to the method I have been using for some time with systems based on the cm-x270 module (/web/content/view/82/9/</a>) in that it used mtd-utils from a running Linux system to erase and write to the NAND flash. The bootloader in the EM-X270 is able to boot a kernel and rootfs directly from a USB Flash disk. This “LiveDisk” image, as Compulab calls it, contains a small utility that programs an image from the USB flash disk directly into NAND flash.</p> As with any modern Linux based embedded system, there is a lot going on in a very small package. Managing this complexity effectively requires some amount of experience. Stay tuned for future articles that discuss using OpenEmbedded to build software for the EM-X270.</p> Additional Photos</h3> </p> </p> </p> Linux 2.6.23 for the Compulab cm-x270 2007-10-22T00:00:00+00:00 Updated 2.6.23 kernel patches for the Compulab cm-x270 are now available in OpenEmbedded:</p> http://www.openembedded.org/filebrowser/org.openembedded.dev/packages/linux/linux-2.6.23/cm-x270</a></p> The cm-x270 support in OpenEmbedded has undergone a lot of clean-up in recent weeks. Now is a good time to give OpenEmbedded a try if you need a full featured Linux distribution for the cm-x270.</p> Compulab cm-x270 NAND flash eraseblock sizes 2007-10-08T00:00:00+00:00 If you are having trouble mounting a JFFS2 file system on a Compulab cm-x270 module, you may have a device that has a NAND flash with 16KiB eraseblocks. All of the devices I have personally used to date have had 128KiB eraseblocks, but I just helped a cm-x270 user through some flash issues and after much pain discovered the eraseblock size was 16KiB on his module. A variable has been added to the cm-x270.conf file in OE to set the eraseblock size that can be overridden in your local.conf file if you are using OpenEmbedded to generate jffs2 images:</p> http://git.openembedded.net/?p=openembedded.git;a=blob;f=conf/machine/cm-x270.conf;h=3c21c546b1320914c9b5d2c44e7332782fc58748</a></p> Howto load Openembedded on the Compulab cm-x270 computer module 2007-10-08T00:00:00+00:00 There are quite a few people using (or would like to use) OpenEmbedded on the Compulab cm-x270. The cm-x270 is a high performance, low cost computer module that can be used with a custom baseboard in embedded systems (see /web/content/view/62/9/</a> for a review). I get a lot of questions on how to load OpenEmbedded on this system, so this howto attempts to detail a procedure that can be used to load OpenEmbedded.</p> The NAND Flash wrinkle</h3> The Linux solution provided by compulab includes a proprietary NAND flash driver. Most developers are interested in using standard Linux MTD drivers with the JFFS2 filesystem, but its not readily obvious how to flash a JFFS2 NAND image using the cm-x270 bootloader. Fortunately, you can boot a small Linux image stored in NOR flash, and then use standard MTD utilities to then flash the NAND jffs2 image.</p> Creating the needed images</h3> For this method, you need to generate two images: a small image to run from NOR flash, and a larger full featured image that will run from NAND flash. Because you only have about 2.3MB of NOR flash, you want to use uclibc in the NOR image as it is much smaller. A base Linux image that will boot into a shell using uclibc and contains a few utilities is about 1MB in size, which leaves plenty of space for custom apps should you need them in this partition. The local.conf settings to build this image are:</p> MACHINE = "cm-x270" DISTRO = "angstrom-2008.1" ANGSTROM_MODE = "uclibc" IMAGE_FSTYPES = "jffs2 tar cpio.gz" </code></pre> At this point, “bitbake angstrom-minimal-image-with-mtd-utils”, and OE will generate a “minimalist-image-mtdutils-cm-x270.cpio.gz” image that can be programmed into the NOR ramdisk partition using the cm-x270 bootloader, and tftp download. A kernel will also be built that should be programmed into the kernel partition.</p> To build the NAND image, simply remove the ANGSTROM_MODE line, and “bitbake angstrom-console-image”. This will generate a “console-image-cm-x270.rootfs-summary.jffs2” that can be programmed into NAND flash.</p> Programming Images</h3> Loading the software onto the cm-x270 is a multi-step process. The first step is to the load the kernel image and the minimalist-image into NOR flash using the cm-x270 bootloader and tftp. After this is accomplished, run the following commands to boot into the NOR flash rootfs:</p> ramdisk on</li> bootos</li> </ul> If successful, you will be presented with a prompt – log in as user root, and no password.</p> You now need to download the jffs2 NAND image into RAM on the cm-x270. One way to do this is to use the tftp download feature in busybox:</p> tftp -g -r console-image-cm-x270.rootfs-summary.jffs2 <tftp server IP> </code></pre> At this point you can program the jffs2 image into NAND flash:</p> flash_eraseall -j /dev/mtd3</li> nandwrite /dev/mtd3 console-image-cm-x270.rootfs-summary.jffs2</li> </ul> After programming completes, reboot the system and enter the following in the cm-x270 bootloader:</p> ramdisk off (this will disable the initramfs in NOR flash)</li> bootos</li> </ul> At this point, your system should boot into the main NAND rootfs. With a little imagination, it should be obvious that you can do a lot of neat things with this type of setup by extending the functionality in the NOR rootfs including field upgrades and system recovery.</p> Mono support in OpenEmbedded and Openmoko packages 2007-10-04T00:00:00+00:00 Original article published</em>2007-10-04</p> As of today, the Mono build in OpenEmbedded is working pretty well. Mono support for ARM and other embedded architectures should add some interesting capabilities for embedded Linux devices. Having written a number of large embedded Linux applications, I can really appreciate the advantages of high level languages for exception handling. It is also interesting to note that Java support in OpenEmbedded is making progress as well (http://jalimo.org/wiki/doku.php</a>). Windows CE developers have already taken to the .net compact framework in a big way, so its nice to see similar capabilities for embedded Linux systems.</p> A feed has been set up for installing mono packages on your Openmoko phone. Add the following line to a /etc/ipkg/-feed.conf file:</p> src/gz mono-armv4t http://dev.bec-systems.com/feed/openmoko/mono </code></pre> And then:</p> ipkg update </code></pre> ipkg install mono </code></pre> The packaging is fairly granular, so look at the listing in the above URL to see a complete list of what is available.</p> How to implement realtime periodic tasks in Linux applications 2007-09-25T00:00:00+00:00 (see also should you be using monotonic timers?</a>)</p> Have you ever wondered what is the best way to implement periodic tasks in Linux applications – something better than usleep()? This article covers a number of issues related to this subject including real-time tasks, the different timers available, timer resolution, and how to implement periodic tasks accurately so that error is not accumulated. The recent inclusion of the high-resolution timers in the mainstream kernel makes this a very interesting subject.</p> High Resolution Timers</h3> Historically Linux has done all timing off the OS timer tick which is usually between 1ms and 10ms. While this is adequate for many tasks, it is really nice to have a high resolution timer for timing tasks. With the integration of the high resolution timers into the mainstream Linux source tree, this is now possible. From a user space perspective, there are no API changes. The only difference you will notice is that now you can sleep for less than OS timer tick period. clock_getres() can be used to check the timer resolution and will tell you instantly if you have high resolution timer support. If the clock resolution is 1ns, you have high res timer support. Realistically, you can’t delay for 1ns in a Linux application, but delays in the range of 100us should be possible, and depending on the configuration, much better performance is possible. The kernel config entry for high resolution timers is CONFIG_HIGH_RES_TIMERS.</p> The difference between CLOCK_REALTIME and CLOCK_MONOTONIC</h3> Some of the Linux timer functions take a clockid_t</em> argument that can be specified as CLOCK_REALTIME, or CLOCK_MONOTONIC. The big difference between these two is that changing the system time will have an affect on CLOCK_REALTIME, thus affecting your timers. Changing the system time will have no affect on CLOCK_MONOTONIC – it will always count upward. For periodic tasks, CLOCK_MONOTONIC may be more applicable. The best way to get burned using CLOCK_REALTIME is when your application takes a timestamp, does something, takes another time stamp and then compares them. If you are using CLOCK_REALTIME, and the system time gets changed between the two timestamps, your comparison will not be valid. For most timeouts and relative timekeeping in Linux, use CLOCK_MONOTONIC. This issue becomes more important as many systems now have a process that periodically sets the time automatically from network time servers, and you have no idea when this might happen.</p> Preemption</h3> Kernel preemption (/web/content/view/69/9/</a> ) makes a huge difference in the performance of real time applications by allowing the kernel to be preempted by higher priority application processes. Historically, any kernel code that was runnable ran before the kernel returned control to applications. This all changes with kernel preemption. Kernel preemption has been available in mainline kernels for some time now (I think since 2.6.16). The worst offender I’ve found for locking the kernel for long periods of time have been flash drivers – especially proprietary ones. But, even jffs2 can cause problems in realtime applications without kernel preemption.</p> The PREEMPT_RT patch</h3> Much of the realtime work being done for Linux is maintained in the PREEMPT_RT patch. Bits of this patch have already been merged into the mainline kernel, but there is still a lot of very useful functionality in the patch and it should be considered if you are doing any type of realtime work. More details will be presented in a future article.</p> Absolute vs Relative timekeeping</h3> When implementing a periodic task, you really want to base your timekeeping off an absolute time, versus relative delays such as usleep(). It is not possible to achieve precise periodic activation with a relative sleep such as usleep(). The reason for this is you must first get the current time, make a calculation to determine how long to sleep, and then call the relative sleep function. If your process gets preempted between the time you acquire the timestamp, and the sleep, your relative sleep time will probably be wrong. This problem is solved by using the clock_nanosleep() function. clock_nanosleep() can be called with the TIMER_ABSTIME value in the flags argument. If the TIMER_ABSTIME flag is set, then clock_nanosleep() will sleep until the absolute timer value is reached. It does not matter if you get preempted between the time you take a timestamp, and the sleep function.</p> Summary</h3> In summary, if you want to implement accurate, realtime, periodic tasks in a Linux application, consider the following:</p> Use high resolution timers. Although Linux is still limited in its response time, at least the scheduling resolution is now quite high.</li> Enable kernel preemption. This makes sure long kernel processes don’t get in the way of your real-time application process.</li> Use the CLOCK_MONOTONIC for relative timekeeping. You don’t want your application to lock up due to the system time changing.</li> use the clock_nanosleep() function instead of relative delays like usleep(). This is the only way to accurately schedule periodic tasks.</li> if needed, apply the PREEMPT_RT patch.</li> </ul> The clock_nanosleep man page also includes a lot of useful information about timer functions.</p> The Embedded WiFi module Quest 2007-09-19T00:00:00+00:00 original article written in Sept, 2007</em></p> How does one implement WiFi functionality in vertical, low volume portable products? This is a good question, and one I’ve been struggling with for the past 3 months. I have a customer who is designing a portable data acquisition system based on a AT91SAM9260 processor, and needs WiFi functionality. The fundamental problem is that no one has time to deal with low volume customers, and the task of implementing WiFi functionality is obviously complex. At volumes of 5000 units a year or less, it makes a lot of sense to go with a WiFi module rather than trying to integrate the WiFi chipset on the board. This article covers some of the options we have looked at and some of the possible solutions.</p> How is WiFi done on x86 systems?</h3> With desktop Windows, most devices are well supported by device manufacturers. With x86 Linux some of the devices are well supported by OSS drivers, but these tend to be PCI or USB devices. Some of the devices that are not supported can still be used by running the Windows NDIS drivers inside of ndiswrapper, which allows you to run the Windows driver on a x86 Linux system, but this obviously does not work very well on non-x86 processors such as ARM.</p> WiFi solutions for portable, embedded systems</h3> While there are many off-the-shelf solutions for PCs, Notebooks, etc, the current solutions are generally in the USB formfactor. The ideal solution for portable systems running non-x86 processors are the small modules that connect to the host processor using the SDIO or SPI bus – like the modules you would find in newer cell phones and PDAs. Older modules tend to use the Compact Flash interface, but these modules tend to be larger and are not packaged as nicely for deeply embedded applications. So Ideally we want:</p> a modules with a SDIO or SPI interface</li> packaging options such as solder down for robust packaging</li> </ul> The scope of this article is limited to modules that implement a SPI or SDIO interface. Why are we interested in modules? Putting a WiFi chip down on a custom PCB is a lot of work. Because the WiFi silicon includes a processor, if things do not work, it may be very difficult to debug. There is also the issue of factory calibration which usually requires proprietary PC based tools that can communicate directly with the WiFi silicon. If the WiFi silicon is buried in your product, getting all this to work can be a challenge. The test equipment required for calibration is expensive. Add to this the difficulty in getting support from anyone who makes WiFi silicon …</p> Complexity</h3> It should be noted that WiFi modules are fairly complex devices. They are typically based on a highly integrated IC that includes an ARM processor, Flash memory, and radio circuitry. The fact that these modules include a significant amount of firmware on the module contributes significantly to the complexity of getting WiFi solutions to work properly. Hopefully no bugs are encountered in the module firmware, because I can only imagine how difficult it would be to get these fixed for a small customer. The fact that the module firmware is involved in many of the WiFi functions like authentication further complicates the problem with the abundance of authentication and encryption options available for WiFi. The worst possible problem is the case where you must design an embedded WiFi system that must operate at the enterprise level in every environment. It is one thing to design a system that works most of the time in most environments where WiFi is more of a convenience (like a Cell phone or PDA). It is another thing to design an industrial grade system where it must work all the time and in all environments.</p> The Fundamental Problems with low volume embedded WiFi</h3> I have never worked with a technology with so many dead ends as embedded WiFi modules. Many of the companies I email or call never even bother to return emails and phone calls. This is certainly true of the chipset manufactures. From what I can gather, the fundamental problems are:</p> WiFi implementation is difficult; therefore, a manufacturer’s scarce resources are dedicated to high volume customers. Low volume customers are a significant distraction.</li> WiFi companies are very busy right now – they are not hungry for business from small customers.</li> WiFi solutions are very competitive and highly proprietary. Many companies are very secretive and will not release driver source code. This is certainly true at the module firmware level.</li> </ul> There are many WiFi module manufacturers and resellers out there, but very few of them offer any type of driver solution for Windows CE or Linux other than binary modules that are supposed to work on perhaps one reference platform. Sorry, this is not going to cut it. So, with plenty of hardware available, the gating item is the availability of software drivers and knowledge of how to use the software. It would seem to me that this is the perfect opportunity for chipset vendors to open source some drivers and develop vibrant support communities so that their modules can be used without a lot of hand-holding by the chipset vendors.</p> So What are the options?</h3> Atheros AR6001</h4> The Atheros AR6001 seems like a nice solution. The driver situation for the AR6001 is progressing. For Windows CE, there is an opensource WinCE 6.0 driver available at: http://www.codeplex.com/CEWifiDriverAR6000/</a>. Unfortunately, this project appears to be a snapshot of code and does not have any significant amount of community activity or development. Perhaps once more AR6001 modules are available, this will change.</p> For Linux, there are several options:</p> Atheros has released a driver that is available: http://sourceforge.net/project/showfiles.php?group_id=186068</a>. There are several issues with this driver in that it is written for a SDIO stack that will likely never be part of the mainstream kernel.</li> A driver is being developed as part of the OpenMoko project.</li> </ul> Embedded Works supplies Atheros based modules and development boards: http://www.embeddedworks.net/wlan/oem_sdio_80211g.html</a>. AR6001 development boards are also available from Cardaccess: http://www.cardaccess-inc.com/products/index.php?a=wlan_sdio</a>.</p> Marvell</h4> The Marvell 88W8385 seems to be a very popular IC in WiFi modules available from a number of different companies. OSS Linux drivers are in progress and are reportedly somewhat functional at this point, so it is probably just a matter of time before these devices are well supported in Linux. This chipset/driver is often referred to as “libertas”. The SDIO stack being developed for the Linux mainline (http://lwn.net/Articles/242747/</a>) is where support for this device is being developed.</p> For Windows CE, it seems the only option is to get the drivers from Marvell, which is a very difficult and time consuming process – at least for low volume customers. First you have to find a module reseller that can get you the source code from Marvell, and then the process takes about 3 months.</p> Other Options</h4> There are a number of other WiFi Silicon manufacturers and module vendors, but the driver options from them seem to be very limited in the form of binary only drivers that will only work with certain processors/operating systems. As noted before there is almost no hope of getting anyone to even talk with you if your volumes are low. So your only hope is finding a module vendor that can do any driver work for a NRE fee, obtain the source from the silicon manufacturer, or use OSS drivers.</p> Summary</h3> The Embedded WiFi situation is changing fast, so by the time you read this, it is probably already out of date. We are currently at the phase where it is very difficult to deeply embedded WiFi in low volume products. I expect this will change during the next year. Eventually, WiFi modules re-sellers will figure out that supplying hardware is just one side of the equation. As things get more and more complex, the software availability and support is becoming the gating item.</p> Many thanks to James Nahra, Dave Anders, and Erik Strack for sharing information and their WiFi experiences with me over the past few months.</p> The correct way to add packages to an OpenEmbedded Image 2007-08-30T00:00:00+00:00 Update 2007-10-14: use IMAGE_INSTALL in image recipe</strong></p> Update 2007-10-22: a few corrections, added full path for include and comments</strong></p> Update 2010-06-07: use recipes instead of packages directory</strong></p> As more and more OpenMoko developers are coming on-line, it is becoming obvious that my previous post (/web/content/view/59/9/</a>) about adding packages to OpenEmbedded could use some improvements. This post suggested using the DISTRO_EXTRA_RDEPENDS variable, which was intended to only be used in distro.conf files. A better approach is to create a custom image recipe that includes the packages you want. This article covers how to create a custom image recipe.</p> What are the problems with the DISTRO_EXTRA_RDEPENDS approach?</h3> As detailed by Marcin Juszkiewicz in his blog post “Why using of DISTRO/MACHINE variables in local.conf is wrong</a>”, and from discussions with OE developers, the following issues surfaced:</p> You have to remember to rebuild task-base every time you update the DISTRO_EXTRA_RDEPENDS variable.</li> The image is no longer angstrom-console-image or whatever you have extended, so when asking for support, other developers do not realize exactly what you are building. If it is a custom image, then that is obvious by the image name.</li> Changing DISTRO_EXTRA_RDEPENDS in local.conf requires that you bitbake reparse the entire recipe tree (which takes minutes). Changing a custom image recipe only requires one file to be reparsed.</li> </ul> Creating a custom image file</h3> Creating a custom image file is quite easy, and works much better than modifying DISTRO_EXTRA_RDEPENDS in your local.conf file. A custom image is created by simply copying or including an image file closest to what we want to use, and then adding additional packages to IMAGE_INSTALL variable. In the following example, we require angstrom-console-image.bb and extend it with a few lines:</p> require recipes/images/console-image.bb IMAGE_INSTALL += "\ xset xserver-kdrive-fbdev xinit \ kernel \ kernel-module-mmc-block \ kernel-module-mmc-core \ kernel-module-pxamci \ kernel-module-ac97-bus \ " export IMAGE_BASENAME = "my-custom-image" </code></pre> Note, this method requires your custom image file to be in the same location as the original image recipe. If you use a bbcollections overlay, then you may want to copy the original recipe.</p> Note the use of a full path specified in the require statement. This allows you to have your custom image recipe in a bbcollections overlay, and bitbake will still find the required recipe in the main OE tree.</p> Thats it! With the many advantages to this method, there is no reason not to create your own image recipes – it works better.</p> Howto use a Git topic branch to maintain a patchset, and isolate problems 2007-08-16T00:00:00+00:00 The more I use Git, the more I am really impressed with some of its capabilities. What is Git? Git is a very advanced distributed source code control system that can do some very neat things. For the average developer using open source software, we are mainly concerned about maintaining patch sets (often called a topic branch). Git includes a rebase capability that is very useful for a number of different operations related to maintaining a branch of code including moving a branch forward, moving a branch around on an upstream branch to look for breakage, and merging changesets to create patch files. This article provides a brief overview of some of these operations and several tips.</p> When in doubt, make a branch</h3> The most important concept to understand when working with Git is to get used to creating and destroying branches. This is different than most other SCMs where all branches are stored on a central repository, and you don’t want to clutter the branch namespace. With Git, get used to branching without even thinking about it. For example, the following are the branches in a Git repository I use to maintain patch sets for the Compulab cm-x270:</p> git branch cmx270 cmx270_2.6.20 cmx270_2.6.21 cmx270_2.6.22 master svs test v2.6.20_test </code></pre> Most of the names are self explanatory. The cmx270 branch is where I track the latest kernel version. The svs branch is for a customer platform based off the Compulab cm-x270. To view the changes on any particular branch:</p> git-log cmx270_2.6.22 </code></pre> an Git will start showing you a list of changes on that branch. This works even if you are not working on that branch, so it is very handy for seeing what changes are in other branches. With Git is also very easy to move changes between branches using git-cherry-pick</em>. For example, if we are working on the cmx270 branch, and we want to pull the 3rd changeset from the HEAD of the cmx270_2.6.22 branch into the cmx270 branch, we would do something like:</p> git-cherry-pick cmx270_2.6.22~2 </code></pre> Git is really good at maintaining patch sets</h3> Many developers use quilt to maintain patch sets. Once you understand how Git works, Git is even better at maintaining patch sets. Although there are some add-on tools available for maintaining patch files in a git repository like Patchy GIT and Stacked GIT, I’ve found that core Git works well enough. Actually the Patchy GIT maintainer has quit working on Patchy GIT because core GIT now meets his needs. The fundamentals are you simply export each Git commit as a patch using the git-format-patch command as shown below:</p> git-format-patch -o tmp/ master </code></pre> Produces the following files (all of the commits since the branch has left the master): </code></pre> tmp/0001-cm-x270-base2.patch tmp/0002-cm-x270-match-type.patch tmp/0003-cm-x270-ide.patch tmp/0004-cm-x270-it8152.patch tmp/0005-cm-x270-pcmcia.patch tmp/0006-ramdisk_load.patch tmp/0007-mmcsd_large_cards-r0.patch tmp/0008-cm-x270-nand-simplify-name.patch </code></pre> Pretty slick! HEAD~8 means to back up 8 revisions from the current HEAD and then start outputting changes as patch files. The name of the patch file is the commit comment.</p> Moving your patch set forward in time</h3> If you are maintaining a patch set for the Linux kernel, you might want to move your patchset up to the latest version for testing, and also maintain patchsets for each stable version of the kernel. Lets say we already have a patch set for the 2.6.20 version. The first thing we need to do is create a branch. In this example, I am maintaining a patch set for the Compulab cm-x270 computer module.</p> git-branch cmx270_2.6.20 v2.6.20 (branch name, tag name in repository) </code></pre> git-checkout cmx270_2.6.20 </code></pre> At this point, I would manually apply each patch, and then commit each patch as its own changeset. Now, I want to move the changeset forward to kernel version 2.6.21. The easiest way to do this is create a new branch, and simply rebase your patchset to the new kernel version. Make sure you create a new branch first, or your existing branch will be moved!</p> git-checkout -b cmx270_2.6.21 cmx270_2.6.20 </code></pre> git-rebase --onto v2.6.21 master (tag point to relocate branch, upstream branch) </code></pre> The upstream branch is required to be specified so that Git knows where our branch changes start – specifically where the cmx270_2.6.21 branch diverges from the master branch.</p> There may be a few things to clean up along the way if changes don’t apply cleanly. The most critical thing to know is that if you want to skip a patch, you must run “git-reset --hard</em>” before running “git-rebase --skip”</em>. This is often handy if you have an especially troublesome patch that you don’t want to apply now. You can skip it and then cherry-pick it later. Otherwise the process is fairly straightforward. After you are finished, you can now export a set of patch files that can be applied to the 2.6.21 kernel.</p> Bisecting the master branch to determine when things broke</h3> You may run into a situation where something broke on the main branch, and you don’t know where. Most of the time, you would just run git-bisect</em> to do a binary search to find where things broke. This does not work so well if the platform you are testing is not supported in the main branch – the changes in your topic branch are required to make the system run. Once again, git-rebase</em> to the rescue. The first thing we must do is find the mid-point between the known good and bad points.</p> git-rev-list --bisect v2.6.20 v2.6.21 13f7e5acc8b329080672c13f05f252ace5b79825 </code></pre> Now, rebase out changes to the above rev and test.</p> git rebase --onto 13f7e5acc8b329080672c13f05f252ace5b79825 master </code></pre> The build failed, so now we have a new “bad” point and can continue our binary search.</p> git-rev-list --bisect v2.6.20 13f7e5acc8b329080672c13f05f252ace5b79825 </code></pre> If you can build working kernels from the main tree, then you can directly use git-bisect – another good reason to get your changes into the mainstream source, but this is not always possible.</p> Another option for doing this type of binary search is to use git-bisect, and then cherry pick your changes at each bisect point. It may also be possible to use git-merge, but I have not figured out how yet.</p> Summary</h3> That is it for now. Git is very useful for maintaining multiple patch sets for different versions of an open source project, and is just another thing that makes working with open source projects a real pleasure. Future articles will cover how to further manipulate patches such as how to combine two patch files.</p> Linux 2.6.22 for the Compulab cm-x270 2007-08-13T00:00:00+00:00 I just completed porting the cm-x270 Linux kernel patches to the 2.6.22 kernel. Kernel build is available in OpenEmbedded. Patches are available in the OE tree:</p> http://www.openembedded.org/filebrowser/org.openembedded.dev/packages/linux/compulab-pxa270-2.6.22</a></p> Shortly I will be making 2.6.22 the default kernel for the cm-x270 in OpenEmbedded and posting instructions on how to use JFFS2 with the NAND flash on the cm-x270.</p> Design tips for your embedded Linux product 2007-07-20T00:00:00+00:00 When designing a system that runs embedded Linux, there are number of things you can do to make your life a lot easier and provide for future expansion to cover cases you did not anticipate. Following these simple guidelines can drastically improve the efficiency of developing the system, and troubleshooting problems during field testing, and provide the flexibility needed in todays products.</p> #1 Have a dedicated serial console</h3> This is the single most important interface to provide. A serial console is your gateway into the system, and allows you at any time to poke around at the internals of the system, debug problems, run applications, change configuration, etc. A serial console does not have to be a big DB9 connector accessible outside the system. It can can be a simple 3 pin header and you make a special cable to connect to a PC. Also, do put the RS232 level translators on the board. This saves so much hassle during the development and testing cycle. You can always not populate them later, but you will probably find that the debug console is so useful, you will leave it in – even in production.</p> Another consideration is to make sure you use a serial port provided by the processor if possible. The advantage of this is very little hardware has to be working for the serial console to work. If you use a serial port off an external UART, then the bus interface between the processor and the UART, the UART and the processor need to be working. If something is wrong with the system, you are less likely to get debug messages out the serial console which will help you debug the system.</p> #2 Implement a USB Host port</h3> A USB host port is so useful it should be considered for about any system. Some of the things USB can be used for include:</p> keyboard and mouse if you are designing a display based system</li> USB networking</li> Wifi networking</li> extra serial ports</li> WAN modems</li> </ul> Many embedded processors that can run Linux include a USB Host interface, so assuming you have one of these, it is very cheap to provide this interface as it only requires two wires, and a 5V power supply.</p> Once again, you don’t have to put a full size USB connector in your device – a simple header wired to a connector might work in most situations.</p> I can’t begin to describe how useful a USB keyboard was on a recent touchscreen based product. The product is used in agricultural vehicles, so anytime in the field we can connect a USB keyboard and look at logs, run commands, and other operations that are very difficult to do though a touch interface.</p> #3 Implement a Network Interface</h3> Working on any Linux system is very painful without a networking interface. Even if you don’t plan to use Ethernet in the final product, make sure a networking interface is available for development. Some reasons why a network interface is critical:</p> can easily log into the system remotely for development and debugging</li> easily copy files to and from the device quickly</li> install software updates easily during the development process</li> </ul> Trust me on this one. The neat thing about Linux, is if you have a USB Host interface, and a reasonably modern kernel, you can use off-the-shelf USB->Network devices for Ethernet.</p> #4 Provide a fast way for the bootloader to download SW updates</h3> This is important. Don’t depend on JTAG or a serial port for programming large OS images. Some options for BL image download include:</p> network (preferred)</li> MMC/SD</li> USB Client</li> USB Host (using a USB Mass storage device)</li> PCMCIA/CF</li> </ul> The good news is if you did steps #2 & #3, this might be taken care of – just make sure you check what the bootloader actually supports. One project I worked on recently only had a serial interface and a USB client interface that the BL did not support. We ended up creating a mini rootfs that included just enough smarts to download the main rootfs over the USB Client interface. So, there are ways to work around some of these issues, but its a lot easier if you do a little planning up front.</p> Summary</h3> There is a lot to be said for using industry standard interfaces and components like Linux, USB, and Ethernet. When you design a system using standard building blocks, your product will have the flexibility to implement features you never thought of when the product was originally designed. And, your development and testing will go so much smoother.</p> Open JTAG debugging tool for the PXA270 2007-07-10T00:00:00+00:00 Looking for a low cost, high functionality JTAG solution for the PXA270? As of several weeks ago, the openocd project now works with the Intel/Marvell PXA270 processor. There are a number of USB based JTAG debuggers available for under $100, so the cost is minimal. This article provides a brief overview of JTAG, suggestions for your hardware design, and how to use OpenOCD (Open On-chip-debugger) with the PXA270.</p> OpenOCD</h3> Over the past 8 years, I’ve used a number of JTAG solutions for programming and debugging embedded processor systems. At the low end, there are JTAG solutions for flash programming only that just wiggle pins on a PC parallel port which in turn wiggle pins on the scan chain of the processor. This way of doing things works, but is very slow because you have to fabricate the bus cycles in software and clock the entire jtag chain just to change the state of one pin. The jtag tools from the openwince project on Sourceforge fall into this category.</p> More advanced JTAG tools use the debugging capability of a processor to speed up flash programming and add debugging features. This typically involves downloading a tiny bit of code into the cache or internal SRAM in the CPU. This bit of code then can receive blocks of data or instructions and run bus cycles directly through the memory controller in the device. This makes flash programming very fast as most of the work is being done by the CPU, instead of manually controlling CPU pins using a jtag chain.</p> In past years, there have been various attempts at creating an open source JTAG solution, but none of them have gotten very far. When I had a project that needed to get done quickly, I simply bought or upgraded a BDI2000 debugger – which is an excellent tool. The OpenOCD project now provides much of the same functionality as a BDI2000, but as in the case of the PXA270, it sometimes takes some work to get it all working.</p> OpenOCD was created by Dominic Rath as part of a diploma thesis (http://openocd.berlios.de/web/</a>). It provides support for a wide range of target processors and JTAG debuggers. OpenOCD includes functionality for On-Chip Debugging, In-System Programming, and Boundary-Scan Testing.</p> Hardware design considerations</h3> Although openocd has worked with other Xscale parts for some time, functionality for the PXA270 has not worked until recently. The primary reason for this is the way the power supplies are sequenced in most PXA270 designs. There is one supply to the PXA270 that is always on (VCC_BATT). All of the other supplies are often kept off until reset to the processor is de-asserted and the SYS_EN and PWR_EN signals go active. Because of this, some special considerations need to be taken during the sequencing of the reset signals, and the setting of the hold_rst bit in the PXA270 DCSR register.</p> Make sure you power the JTAG connection (Pin #1 of your JTAG connector) with VCC_BATT or some power that is always active. If you power the JTAG connector with a power supply that is controlled by the SYS_EN or PWR_EN signal, then you will loose JTAG power during the connection process and things obviously will not work.</p> JTAG Debugger Hardware</h3> Most of the JTAG debugger debuggers used with OpenOCD are based on the FTDI FT2232 USB chip. The debugger I used is the arm-usb-ocd device from Olimex. At a cost of $80, you can’t afford not to have one.</p> </p> OpenOCD Installation</h3> OpenOCD can use two different libraries that provide access to the FTDI chip: an open source library named libftdi</em>, and a closed source library from FTDI named libftd2xx</em>. I could not get the opensource libftdi library to work, so I suggest you use the FTDI library for now. Otherwise, installation is pretty straightforward. Find yourself a reasonably modern Linux computer and follow instructions on the following page: http://openfacts.berlios.de/index-en.phtml?title=Building_OpenOCD</a> . Make sure you use the --enable-ft2232_ftd2xx</em> option when running configure</em>. On Ubuntu 7.04:</p> Install the FTDI library:</p> wget http://www.ftdichip.com/Drivers/D2XX/Linux/libftd2xx0.4.13.tar.gz</li> mkdir libftd2xx; cd libftd2xx; tar -xzf ../libftd2xx0.4.13.tar.gz</li> sudo cp .h /usr/local/include/</li> sudo cp libftd2xx.so.0.4.13 /usr/local/lib</li> cd /usr/local/lib/; ln -s libftd2xx.so.0.4.13 libftd2xx.so; sudo ln -s libftd2xx.so.0.4.13 libftd2xx.so.0</li> cd /usr/lib; ln -s /usr/local/lib/libftd2xx.so.0.4.13 libftd2xx.so.0</li> </ul> Install OpenOCD:</p> sudo apt-get install build-essential</li> sudo apt-get install automake</li> sudo apt-get install autoconf</li> svn checkout svn://svn.berlios.de/openocd/trunk openocd</li> cd openocd</li> ./configure --enable-ft2232_ftd2xx</em></li> make</li> sudo make install</li> </ul> OpenOCD Configuration</h3> OpenOCD is configured using a configuration file. An example is:</p> # config for Intel PXA270 # not, as of 2007-06-22, openocd only works with the # libftd2xx library from ftdi. libftdi does not work. telnet_port 3333 gdb_port 4444 interface ft2232 ft2232_layout olimex-jtag ft2232_vid_pid 0x15BA 0x0003 ft2232_device_desc "Olimex OpenOCD JTAG" jtag_speed 0 # set jtag_nsrst_delay to the delay introduced by your reset circuit # the rest of the needed delays are built into the openocd program jtag_nsrst_delay 260 # set the jtag_ntrst_delay to the delay introduced by a reset circuit # the rest of the needed delays are built into the openocd program jtag_ntrst_delay 0 #use combined on interfaces or targets that can't set TRST/SRST separately reset_config trst_and_srst separate #jtag scan chain #format L IRC IRCM IDCODE (Length, IR Capture, IR Capture Mask, IDCODE) jtag_device 7 0x1 0x7f 0x7e #target configuration daemon_startup reset target xscale little reset_halt 0 pxa27x # maps to PXA internal RAM. If you are using a PXA255 # you must initialize SDRAM or leave this option off working_area 0 0x5c000000 0x10000 nobackup run_and_halt_time 0 30 #flash bank <driver> <base> <size> <chip_width> <bus_width> # works for P30 flash flash bank cfi 0x00000000 0x1000000 2 4 0 </code></pre> Programming Flash</h3> After OpenOCD is installed, copy the above config file contents to a file name openocd.cfg, and run openocd in the same directory as your configuration file.</p> Currently, there is an issue in that the openocd is expecting a debug handler file at a specific relative path from the openocd binary. The easiest way to get around this for now is to to run openocd from the src directory in the openocd source code.</p> If you get the following message: Error: ft2232.c:1338 ft2232_init_ftd2xx(): unable to open ftdi device: 2, there is a permissions problem with the USB device. To get around this, run openocd using sudo, or add the appropriate rules to udev to allow the usb device to be used by any user.</p> OpenOCD runs as a daemon waiting for telnet or GDB connections. After openocd starts, you can connect to it by running the following command: telnet localhost 3333</p> At this point you should be presented with a OpenOCD prompt. The following is an example of how to program a bootloader into flash memory:</p> > flash probe 0 flash 'cfi' found at 0x00000000 > flash protect 0 0 5 off cleared protection for sectors 0 through 5 on flash bank 0 > flash erase 0 0 5 off erased sectors 0 through 5 on flash bank 0 in 4s 615249us > flash write 0 /build/tmp/u-boot-holon.bin 0 wrote 121832 byte from file /build/tmp/u-boot-holon.bin to flash bank 0 at offset 0x00000000 in 6s 138592us (19.381735 kb/s) > </code></pre> Debugging</h3> In this example, OpenOCD waits for GDB connections on port 4444. To debug code running on the target, you would use gdb in the same way you would use a BDI2000, or most any other hardware debugger with GDB.</p> Summary</h3> OpenOCD is a robust, full-featured JTAG debug solution that supports many processors and low cost JTAG debuggers. There are still a few rough edges to clean up with PXA270 support, but OpenOCD is already a very useful tool for embedded software development.</p> How to control the cable nightmare 2007-06-20T00:00:00+00:00 Do you ever wonder if there is a better way to manage cables in your lab? I typically have have 5-10 embedded systems on the bench powered up at any give time. When you start thinking of all the cables required to connect these systems (every one requires at least power, serial console, and network), it starts to add up. I recently ran across a product called snaketray (http://snaketray.com/products/wall.html</a> ) and decided it might be the solution to my rats-nest of cables. This article includes a few pictures of my implementation, and some ideas. I’m quite pleased with the result and have high hopes that I’ll keep my sanity for a while longer.</p> I’m sure there are many solutions to cable management. The IT folks managing racks of computers no doubt have lots of good solutions. There are lots of different cable ties available, etc. But, the needs of a developer are a little different in that you need a solution that can easily be changed. You need something that is not permanent where you can quickly add, change, and re-route cables as needed. Most cable management solutions are more permanent solutions not designed for flexibility. Then I read the following article: http://www.joelonsoftware.com/articles/BionicOffice.html</a>. It occurred to me that snaketray might be the perfect solution for an embedded systems development lab. The idea of snaketray is very simple as shown in the picture below:</p> </p> I installed two runs about 1.5 feet above my lab work space. There are power strips along the back of each lab surface. All cables now run in the snaketray. The idea is cables go straight into the snaketray and exit exactly where the are needed. If the cable is too long, I loop it back in the snaketray. The result is pretty amazing when you compare it to what it was before. My wife is not real impressed with how it looks – apparently she has not spend enough time under tables untangling cables :-)</p> Some of the benefits I’ve discovered so far:</p> even though it still looks pretty messy, I’m finding it very easy to pull cables out of the tray and re-route them – even network cables which are notorious for getting hung up due to the little plastic clip on the connector. I think the reason is the cables are always running orthogonal – in the tray and the come out at a 90 degree angle where they are needed. So if cables are not crossing each other they do not get tangled.</li> bench space is not used for cables, so I have lots of space for more systems</li> I’ll be more likely to try builds routinely on more targets. One of my goals is to routinely do OpenEmbedded builds for a variety of embedded systems to keep my finger on the OE pulse, and help with QA. Obviously crawling around on the floor untangling cables to connect various systems is a negative motivator for such activities.</li> I have a place to put cables when I’m not using them – just put them in the tray. Before, they were left on my lab bench taking space, or falling on the floor. Now I have a few extra network and serial cables waiting in the tray for when I need them.</li> </ul> The bottom line is this makes me more efficient at what I do, so that I can better serve my customers and the projects I work on. I imagine snaketray would also be very useful in factory setups and other situation where you routinely need to change cabling.</p> What you you found that works in your lab?</p> </p> </p> </p> Atmel AT32AP7000 development board review 2007-06-09T00:00:00+00:00 I recently purchased an Atmel NGW100 development board which includes an Atmel AT32AP7000 CPU. This CPU is based on the new Atmel AVR32 architecture, and is capable of running full Linux. This review covers the basics of this development board, the AVR32 architecture, options for building Linux for this system, as well as opinions on how a new architecture can succeed in a field of well established architectures like ARM and MIPS.</p> What is AVR32?</h3> The AVR32 architecture defined by Atmel is a 32-bit RISC load/store architecture. There are several different versions of the core. Atmel currently supplies two classes of AVR32 cpus:</p> UC3 Flash MCUs - low end devices with integrated RAM and flash, no MMU, and look similar in capability to ARM7 microcontroller type devices.</li> AP7 Application Processors - faster devices, include MMU, seem similar in capability and speed to ARM9 application processors such as Atmel’s AT92SAM devices.</li> </ul> As I am most interested in CPUs that can run Linux, this review will focus on the AP7 (AVR32 AP) class devices.</p> It seems that AVR32 is Atmel’s answer to ARM Cortex. Some features of the AP7 devices:</p> good code density</li> DSP instructions that execute in a single cycle</li> Full MMU</li> Instruction and Data caches</li> SIMD extensions</li> dynamic branch prediction</li> </ul> The MMU in the AP7 devices is interesting in that page tables are handled in software similar to a MIPS device. In a ARM CPU, the page tables are located in physical RAM and are strictly defined by the ARM architecture and page faults are handled in hardware. With MIPS (at least some devices) and AVR32, the MMU basically consists of a TLB (Translation Lookaside Buffer) that maps virtual addresses to physical address. As the TLB is limited in size, any misses are handled in software.</p> NGW100 Development board</h3> The NGW100 development board is available from a number of distributors (including Digikey) for about $70. The board includes the following major components:</p> Atmel AT32AP7000 CPU</li> 32MB of SDRAM (MT48LC16M16A2)</li> 8MB Parallel NOR Flash (AT49BV642D-70TU)</li> 8MB Serial NOR Flash (AT45DB642D)</li> 2 Ethernet ports</li> SD/MMC slot</li> RS232 port</li> USB Client port</li> Microcontroller for board control (ATtiny24-20SSU)</li> </ul> </p> Linux Support</h3> Atmel is directly supporting the development effort for the AP7 processors. They seem to be doing it right in that they are pushing support directly into the mainstream kernel. Currently, Atmel provide patches for GCC and uClibc. There is already efforts underway to provide AVR32 support in Openembedded, which will enable developers to quickly build full featured Linux distributions for AVR32 based products.</p> Summary</h3> It will be interesting to see how the AVR32 based products work out. On one hand, one could argue that a custom architecture like AVR32 can’t succeed as it does not have an extensive ecosystem of tool vendors in place like ARM and MIPS. But, the availability of open source software such as GCC and Linux is rapidly lowering the barrier to entry for creating tools for new architectures. No longer do you need to wait for commercial tool vendors to provide software support. The model for software development is changing.</p> In comparison to Atmel’s AT91SAM parts, its difficult initially to see what the AVR32 parts offer over the SAM parts. For one, they are lacking a USB Host port which is becoming increasingly important to have in embedded Linux systems. Perhaps they offer other advantages such as lower cost and power. As we gain experience with these parts, I’m sure we’ll find applications where they are an excellent fit.</p> The Embedded Industry Transformation 2007-06-09T00:00:00+00:00 I just ran across an interesting article written by Doug Gaff titled “Is the embedded industry dead?</a>”. The article points out the transformation that is happening in the “embedded” industry. For many systems, we are no longer constrained by minimal resources (/web/content/view/67/9/). Embedded Systems are no longer isolated systems, but parts of larger systems.</p> A few of my own observations:</p> There is more to modern embedded systems than control and logic. We now have to worry about data. In the past the task might be to get a microcontroller to run a state machine, control motors, etc. Today, we need to present a calendar to the user through a web interface, collect information about the location of the Sun, upload statistics to a remote server, support a rich GUI application on a VGA color display. The control aspect is suddenly the small part of the problem, and managing data is often the primary concern.</p> Embedded system development now encompasses many disciplines including OS development, web services, advanced application development in high level languages, and the traditional hardware debugging and low level coding tasks. It is becoming less about writing code and more about integration of the right pieces. As Embedded Systems become more complex, access to specialized knowledge is becoming more important. There is just too much for one person (or even one company) to know. Companies developing competitive products in the future will be the ones who have access to experts. Is your company taking advantage of this transformation?</p> Can you afford not to use Linux in your product? 2007-05-29T00:00:00+00:00 How much does hardware capable of running Linux cost? Recently, I priced components for a fairly minimal system.</p> CPU: Atmel AT91SAM9260 (< $10)</li> 32MBytes of SDRAM ($6)</li> 256MBytes of NAND flash ($6)</li> </ul> These are prices for relatively low volumes on the order of 1000 pieces. So, you can get a processor, RAM, and flash that is more than capable of running Linux for about $20. Think about the possibilities. The AT91SAM9260 is a rather impressive part and includes interfaces for about anything including USB, SD/MMC, Ethernet, and the list goes on. It also includes that capability to boot from NAND flash which eliminates a dedicated NOR boot flash and reduces cost. If you don’t want to do a full processor design, processor modules can be obtained for less than $100</a> .</p> So, at this price you have to ask: can I afford not to embed Linux in my product? Can I afford not to take advantage of standard interfaces like Ethernet and USB? Can I afford not to take advantage of 1000’s of high quality applications and drivers? Can I afford to not implement a system with virtually unlimited storage</a> and capability for expansion? Think about this carefully because your competitors are. Have questions or need help getting started? Contact us</a> – we can help.</p> Is Linux ready for real-time applications? 2007-05-12T00:00:00+00:00 Have you ever wondered about the real-time response of Linux and how well it performs? I recently developed a system where I had the requirement to send a packet of data out a serial port every 30ms. This article describes how stock Linux can be used to accomplish this and solutions to several problems I encountered. There are several concepts that need to be understood such as how to set a Linux process to real-time, and kernel preemption.</p> All of the work referenced in this article was done on a Compulab cm-x270</a> system which is based on the PXA270 ARM processor. Before utilizing any real-time optimizations, the packets sent out the serial port where anything but consistent. There was a significant amount of jitter in the packet timing and occasionally we would see delays of up to several seconds. Obviously this was not good enough.</p> Real Time Process Priority</h3> The first step was to partition the application into several threads, and give the serial communication thread a priority of real time. Linux supports several scheduling policies:</p> SCHED_OTHER: default universal time-sharing scheduler policy used by most processes.</li> SCHED_BATCH: intended for “batch” style execution</li> SCHED_FIFO: First In-First Out scheduling. Can only be used with static priorities higher than 0. A SCHED_FIFO process will always run before a SCHED_OTHER or SCHED_BATCH process. A SCHED_FIFO process runs until it is blocked by an I/O request, or is preempted by a higher priority process.</li> SCHED_RR: Round Robin scheduling. SCHED_RR is similar to SCHED_FIFO, except the process will only run for a maximum time quantum if there is a process of equal priority waiting to be run.</li> </ul> Practically, in a simple system there is really not much difference between SCHED_FIFO and SCHED_RR as you will typically only have at most a couple real time processes and you will probably assign them different priorities.</p> The sched_setscheduler() system call can be used to set a process or thread priority – see the Linux man pages for more details.</p> After setting the communication thread to real time, there was a noticeable improvement in the packet scheduling, but there was still way too much jitter and the occasional several seconds of silence with no packets.</p> Preemptible Kernel</h3> After some experimentation, and reading, it was suspected that the NAND flash driver in the system was locking the kernel for long periods of time. With older Linux kernels, the kernel was not preemptible. Any process running in the kernel had to finish before something else could run. Recent 2.6 Linux kernel include an option to enable Kernel Preemption (CONFIG_PREEMPT). This allows a real-time process to preempt the system even if it is running in kernel space. After enabling the CONFIG_PREEMPT option in the kernel, the scheduling of packets was very consistent most of the time with almost no perceptible jitter.</p> Be careful what you put in your real-time thread</h3> Now the system was running fairly well, but we still noticed at system startup that there were still some long delays between packets. It turns out the application was writing log messages to a file on NAND flash in the real-time thread using the standard glib logging functions. At system startup, the NAND driver blocks for long periods of time (up to several seconds). After the system has been running for a short time, this went away. The solution in this case was to send any logging messages from the communication thread to a separate normal priority logging thread. This fixed any remaining issues and the packet scheduling is now rock solid all the time. The leads to an fundamental real-time concept which seems very obvious in hindsight:</p> Do not access any I/O (such as storage or network interfaces) that may block for long periods of time**in a real time process.</strong></p> Although we had designed all the primary I/O to be handled this way, we had not given any consideration to log file writes, which are an important part of any program.</p> The glib library provides several very nice mechanisms for sending data between threads.</p> Linux Real-time</h3> With improvements in the 2.6 Linux kernel, Linux works very well for some real-time applications. While I would still be very careful in using Linux for critical, hard-real time requirements due to its overall complexity, Linux works very well for multi-media, communication, and data acquisition systems.</p> NAND Flash is almost "free" and what it means for Embedded Systems 2007-02-26T00:00:00+00:00 original article p</em>ublished in March, 2007</p> EETimes recently ran an article titled “Plummeting prices make NAND appear almost free” (http://www.eetimes.com/news/latest/showArticle.jhtml?articleID=197002923</a>). 1GBytes parts cost around $55 in Nov 2005. As of 2007-03-28, a 1GBytes part costs around $8. This is almost an order of magnitude reduction in a little over a year. This article examines the impact this price drop will have on embedded systems and the changes we will likely see including: storage is now at post-scarcity, will “embedded” distributions become obsolete, will uclibc become obsolete, and challenges with large NAND flash parts.</p> 3 major components in an embedded computer</h3> The 3 major components in an embedded computer are:</p> processor</li> RAM</li> Flash Storage</li> </ul> All of these components have been gradually decreasing in price over the years, and there has been a corresponding gradual increase in the use of complex embedded systems. However, the drop in NAND flash in the last year represents more of a step change, and will likely result in some more drastic changes.</p> We are to the point of post-scarcity for Flash storage</h3> Up to this point, most full featured embedded Linux systems contained 16-64MB of NOR flash. A full featured graphical Linux terminal requires about 20-30MB of flash disk space just for applications and libraries. So, in the past we were still concerned about application footprint. The was a lot of incentive to not cross the 32MB to 64MB boundary, etc. With 1GiB of flash now costing less than $10, we essentially have unlimited flash storage in the context of an embedded Linux distribution. We will probably see the following:</p> faster time to market as we need to spend less time optimizing</li> increased use of GTK+ as the standard GUI toolkit. We no longer care how much space GTK+ takes.</li> more debugging features can be left in the production system. Why not leave VIM, Minicom, and SSH in the build if we have plenty of space. These non-critical features will lower the cost of maintaining a product.</li> </ul> Will “embedded” distributions become obsolete?</h3> There has always been a great deal of interest in running Debian and other “standard” distributions in embedded systems. With 1GiB of storage, this is going to be a lot more practical. However, with a little experience, you soon realize that distribution size is only one of the many factors. Another equally important factor is the ability to easily automate the build process for the entire project, including custom applications and image generation for programming. With complex systems like Linux, it is easy to get into the hack-n-slash mode where it takes a lot of manual steps to build the system. While this may seem OK when you building the system, it ends up causing a lot of pain and errors once you get in a hurry, of down the road when you are trying to maintain the system. Ability to easily customize the distribution is another requirement. OpenEmbedded (http://openembedded.org), and other build systems provide these features. I think we will see a big increase in Debian and other standard Linux distributions in Embedded systems, but OpenEmbedded and other embedded build systems will continue to grow in use as well because there are many factors in the equation. This is the nice thing about open source – use what works best for you.</p> Building and maintaining an embedded system is a lot different than setting up and maintaining a desktop or server system. Other articles related to this topic include: /web/content/view/63/9/</a> and /web/content/view/35/37/</a>.</p> Will uclibc become obsolete?</h3> Because flash is now cheap, why do we need to use uclibc (small footprint libc) – can’t we just use glibc. For full featured embedded Linux systems, this argument makes sense. However, there is push toward higher integration at the low end. Many companies are offering 32-bit “microcontrollers” with integrated flash and RAM. The processors in these devices are more than capable of running Linux. The limitation to date has been the RAM and Flash size. The available memory in 32-bit microcontrollers will increase over time enabling them to run a low end uclibc based system. The idea of running Linux on a system that is composed of a single chip is very interesting.</p> Challenges with large NAND flash parts</h3> What challenges will using large NAND parts present? In general, dealing with NAND is a lot more complex than with NOR flash. For reading, NOR parts present a simple memory mapped interface. A NAND interface is not memory mapped and looks more like a FIFO or a Hard Drive interface. This makes booting from NAND flash much more difficult. Some parts present a memory mapped interface for an initial loader. Some processors build NAND flash bootloading into the processor. Other systems boot from a small NOR flash and use NAND after the system has booted.</p> Filesystems are also a challenge with NAND flash. The standard method is to use JFFS2 which was designed for smaller NOR parts. JFFS2 works well, but is a log based filesystem. This means it has to scan the entire file system on mount and store the entire inode chain in memory. This takes lots of time and uses lots of RAM. For small devices (64MB), this is tolerable, but for large devices the mount time can take 10’s of seconds. YAFFS2 is another log based flash filesystem designed specifically for NAND flash which includes a feature called checkpointing that bypasses the normal mount scanning. Another development is LogFS which looks promising, but will probably not be ready until mid/late 2007.</p> Programming NAND flash can also be a challenge. Because NAND flash can contain bad blocks, programming is a lot more complex. Traditionally, flash programming in a NOR system was done by the bootloader. For a system that boots from NOR flash, programming the NAND flash can be done from a small Linux system (see /web/content/view/66/9/</a>). If you are booting from NAND flash, programming may be done with a JTAG programmer that supports NAND flash, or by downloading a small bit of bootstrap code into the processors internal memory using JTAG, and this bootstrap code then has enough functionality to program the NAND flash.</p> Summary</h3> The lower NAND prices will bring about change in the way embedded systems are built. Please comment if you have addtional ideas or observations.</p> Do you need "software update" functionality in your Embedded Linux system? 2007-02-21T00:00:00+00:00 In this day and age, most embedded systems include a way for users to easily update software once the device has been deployed. This article discusses the requirements for a field update mechanism along with pointers for how to implement.</p> Update Mechanism Requirements</h3> Requirements for a field update mechanism might be:</p> easy for users to perform updates.</li> little chance of “bricking” a unit if reset occurs during update.</li> ability to update all of the software in the unit</li> in this case, update from files on a USB Storage device</li> ability to program NOR and NAND flash</li> </ul> Other systems may have requirements to update from a network, Compact Flash card, etc.</p> The USB wrinkle</h3> System update is typically done by the bootloader. In this case, we are using a Compulab cm-x270 module (/77/compulab-cm-x270-pxa270-module-review/</a>) that has a proprietary boot loader and does not support updates from a USB storage device. Therefore, we need to run the updates from the context of the Linux operating system which has drivers for USB and the NAND flash. We decided that the best way in this case is to have a small Linux OS image that contains enough functionality to update the main filesystem. This also provides us with the capability to recover if the main filesystem ever gets corrupted. The flash layout for the system is:</p> Partition</strong></td> Flash Device</strong></td> Size</strong></td> File System</strong></td> Description</strong></td> </tr> Kernel</td> NOR</td> 1536KiB</td> NA</td> Linux kernel</td> </tr> Update rootfs</td> NOR</td> 2304KiB</td> Initramfs (RO)</td> Small rootfs that contains just enough functionality to update the system</td> </tr> Main rootfs</td> NAND</td> 512MiB</td> JFFS2 (RW)</td> Main rootfs including application</td> </tr> </tbody> </table> There are other schemes that could work as well. One way might be to have two identical application partitions and only update one at a time. A reset while updating the kernel or the update rootfs does have the potential of “bricking” the system, but most of the updates will be for the Main rootfs, so we think this risk is low enough to be acceptable in this system. The update rootfs is loaded into RAM before it is used, so there is little chance of it being corrupted during normal operation. Running the update rootfs from ram is also convenient in that we can easily re-program the update rootfs in flash as we are not directly mounting the filesystem in flash.</p> Building the Update rootfs</h3> The base update rootfs is built with OpenEmbedded as described in /77/compulab-cm-x270-pxa270-module-review/</a>. Once we had the system booting into the update rootfs, we had to add the following functionality:</p> Boot the main rootfs from NAND flash</li> look for update files on a USB Storage device and flash the kernel, update rootfs, and main rootfs as needed.</li> </ul> Booting the NAND flash image is accomplished using the busybox switch_root</em> application. This utility allows you to easily switch to another rootfs from an initramfs. There are a few gotchas to be aware of when working with switch_root:</p> switch_root must be “exec’d” from the init process with PID=“1”.</li> switch_root requires a “/init” file to be present</li> </ul> The above functionality was accomplished by writing a shell script called /init. If an error occurs, the /init shell script then launches the standard init process in /sbin/init which provides a terminal and allows for easy debugging. The mtd-utils package provides a handy utility called nandwrite</em> which can be used to write jffs2 images to NAND flash (handles bad blocks, etc). At some point, we may remove all the extra functionality from the update rootfs like the standard sysvinit, terminal login, etc, but for now it is handy to have for debugging. To give you an idea how easy it is to write to flash from a shell script, consider the following snippets:</p> # write Image into update rootfs parition </code></pre> cat /usb/nor_rootfs.img > /dev/mtd2 </code></pre> # write Image to NAND rootfs partition </code></pre> flash_eraseall /dev/mtd3 </code></pre> nandwrite /dev/mtd3 /usb/nand_jffs2.img </code></pre> # and to launch rootfs in nand flash </code></pre> exec switch_root -c /dev/console /jffs2 /sbin/init 5 </code></pre> Summary</h3> Having typically used bootloaders for system update in the past, I’m very pleased with how this mechanism worked out. Thanks to the flexibility of Linux and OpenEmbedded, this only took several days to implement and debug. Writing update programs with shell scripts at the application level gives you a lot of flexibility and allows you to use standard Linux drivers for USB and Flash access which are very robust. At some point if there is interest, we may look at cleaning up our code and contributing functionality to OpenEmbedded to generate a more generic update initramfs. Please contact us or leave a comment if you are interested in collaborating on something like this.</p> Building a tiny "safe" boot image using OpenEmbedded 2007-02-16T00:00:00+00:00 We are currently working a project that contains cm-x270 module (/web/content/view/62/9/</a> ) which contains 4MB of NOR flash and 128MB of NAND flash. The Linux kernel and a small root file system (rootfs) will be stored in NOR flash, and the main rootfs in the NAND flash. The rootfs in the NOR flash will be used to update the system from a USB Flash drive in the field. This article describes how to build a small boot rootfs using OpenEmbedded. We also look at various ways to store the image in flash.</p> Requirments</h3> In many embedded systems, field upgrades is a requirement. There are many ways to organize this, but one of the most reliable is to have two rootfs partitions. The first partition is a “safe” rootfs that is only used to update the second partition. The second partition contains the main application and can be updated in the field. If a programming error, or power loss occurs while updating the 2nd partition, you can always boot into the first rootfs partition and restart the process.</p> The kernel and the bootloader in this system take up about 1.5MB (out of 4MB total) of the NOR flash. This leaves us with about 2.5MB for the update rootfs. Because we are building the rest of the system using OpenEmbedded (OE), it may make sense for us to also build this update rootfs using OE to keep everything in one build environment.</p> uclibc, glibc, or klibc?</h3> There are several libc libraries that are commonly used to build Embedded Linux systems. glibc is the standard that is used in desktop and sever systems and is what is being used in the main NAND rootfs partition. uclibc is smaller than glibc is often used in space constrained systems. klibc is a very minimal libc subset that supports operations typically used in an initramfs. Because we are already using glibc, we decided to try this first as we could then use the same toolchain and build tree and just generate two images – one for NAND, and one for NOR.</p> OpenEmbedded support for small images</h3> OE includes a task-base-minimal</strong> meta target that can be used as a basis for a small images, or you can create your own based on task-base. You will then need to create a custom image recipe that will use this meta task. An example is shown below:</p> PR = "r0" export IMAGE_BASENAME = "svs_nor" export PACKAGE_INSTALL = "task-min" # keep extra language files from being installed export IMAGE_LINGUAS = "" RDEPENDS = "task-min" IMAGE_FSTYPES = "tar jffs2 ext2 cramfs squashfs squashfs-lzma cpio.gz" IMAGE_ROOTFS_SIZE_ext2="10240" #EXTRA_IMAGECMD_jffs2="--pad=0x1000000 --eraseblock=0x40000" EXTRA_IMAGECMD_jffs2="" inherit image </code></pre> In this case, I had defined a custom recipe for task-min based on task-base, which installs the following packages:</p> root@compulab-pxa270:~$ ipkg list_installed base-files - 3.0.14-r58 - base-passwd - 3.5.9-r2 - busybox - 1.2.1-r11 - initscripts - 1.0-r86 - ipkg - 0.99.163-r1 - ipkg-collateral - 1.0-r5 - libc6 - 2.5-r4 - libgcc1 - 4.1.1-r10 - libipkg0 - 0.99.163-r1 - makedevs - 1.0.0-r2 - sysvinit - 2.86-r32 - sysvinit-inittab - 2.86-r32 - sysvinit-pidof - 2.86-r32 - task-min - 1.0-r1 - tinylogin - 1.4-r3 - update-rc.d - 0.7-r0 - </code></pre> Storing the NOR filesystem in flash</h3> OE can generate images for many different flash filesystems and initrd mechanisms. These are specified in the IMAGE_FSTYPES</strong> variable in the above recipe. In this case, I chose to generate many different filesystems so I could see evaluate how much compression I would get with each filesystem type. The results:</p> Rootfs Type</strong></td> Size (bytes)</strong></td> </tr> squashfs-lzma</td> 1785856</td> </tr> tar.bz2</td> 1918243</td> </tr> cpio.gz</td> 2024558</td> </tr> squashfs</td> 2109440</td> </tr> cramfs</td> 2265088</td> </tr> jffs2</td> 2455396</td> </tr> ext2 (uncompressed)</td> 5481000</td> </tr> </tbody></table> I ended up choosing the cpio.gz format as it can be loaded directly into an initramfs filesystem by the kernel. There are several advantages to using an initramfs for this task:</p> You don’t have to mess around setting up tmpfs filesystems for directories that need to be writable.</li> You can update the NOR flash partition while you are running out of the initramfs filesystem.</li> It is fairly small with only squashfs-lzma being smaller.</li> </ul> Future Optimizations</h3> For now, the glibc based solution is good enough</em> because it fits in the flash space we have and it is easy to build with our existing build environment. In the future, we may move to a uclibc or klibc based solution and try to reduce our flash size (OpenEmbedded supports both klibc and uclibc). The above image sizes can also be reduced by removing ipkg and other components that are not needed.</p> This exercise illustrates many of the advantages of Linux and OpenEmbedded. With very little work, I can produce exactly what I need. The ability of the 2.6 Linux kernel to load a cpio archive into an initramfs is a very elegant solution for small boot images.</p> Compulab cm-x270 PXA270 module review 2007-01-30T00:00:00+00:00 We are in the process of supporting a customer who is designing a product that uses the Compulab cm-x270 “Computer-on-module”. The cm-x270 is a small computer module 66x44x7mm that contains an Intel PXA270 ARM processor similar to those found in many PDAs or smart phones. This review provides an overview of the cm-x270, why it was selected, and how we are using it.</p> Project Needs</h3> For this project, we needed a low cost embedded computer that could drive a VGA (640x480) color display. The graphical user interface for the device was fairly advanced, so we were also looking at a system that could run a modern OS with an advanced graphical toolkit. After discussing the display requirements with the customer, we concluded that we needed a 32-bit processor with a LCD display controller. As there was some custom circuitry needed, a configuration where the customer could design a custom baseboard and use an off-the-shelf computer module seemed optimal. The schedule was also very aggressive and we did not have the time or budget to develop a full custom processor board.</p> CM-X270</h3> After researching a number of computer modules, we chose the cm-x270 from Compulab. The cm-x270 is a module that includes the following components:</p> Intel PXA270 ARM processor (up to 520Mhz)</li> 802.11b Interface</li> up to 128MB SDRAM</li> 128-512MB NAND flash</li> up to 4MB NOR flash</li> PCMCIA</li> PCI</li> Serial Ports</li> Touch panel controller (UCB1400)</li> Host and Slave USB</li> and other features …</li> </ul> For more details visit the compulab web site (http://http://www.compulab.co.il</a> ). The cm-x270 provides a very impressive number of features in a very small space.</p> Below is a picture of the cm-x270 mounted on the Compulab SBC-X270 baseboard with a VGA display. The system is running the GPE Palmtop Environment built with OpenEmbedded.</p> </p> Low Cost</h3> Perhaps the most impressive feature of the cm-x270 is the cost. When ordering in volume, Compulab will build the module with only the components you need. For prototyping, you can order a standard offering with all components populated. Starting a price of $50, it is the lowest cost module we found. The configuration we will be using costs around $98 in 2K quantities (64MB DRAM, 4MB NOR Flash, 512MHz + 128MB NAND Flash + Audio/Touchscreen + RTC). Eventually, we may be able to reduce some of the features and get the price down around $70 per module.</p> At this price, the cm-x270 provides a very cost effective way to develop a low volume product (a few thousand per year) with advanced features provided by the PXA270 processor.</p> Documentation</h3> The bootloader and hardware design of the cm-x270 is proprietary. Compulab does not supply schematics for the design or source code to the bootloader. The documentation provided is adequate and provided all the information we needed to complete the design. Compulab also provides schematics for their development baseboard, which is helpful.</p> Bootloader</h3> The cm-x270 bootloader provides typical features found in an embedded processor bootloader, including: a serial console, NAND and NOR flash programming, download images from a tftp server, downloading images over a USB connection to a PC, and a number of other configuration commands. The tftp download works very well and is fast. The USB download is much slower, but is convenient for systems that do not have a network connection. The USB download requires a windows PC with a specific version of Microsoft ActiveSync installed, which is unfortunate. A more general solution that uses something like libusb (http://libusb.sourceforge.net/</a>) instead of ActiveSync would be preferred as it could then run on both Windows and Linux computers.</p> Linux Support</h3> Compulab provides a fairly complete port of the 2.6.16 Linux kernel for the cm-x270 in the form of a kernel patch. Most of the peripherals we tested seem to work with the following exceptions:</p> The SD card driver is not preempt safe. In this application, we need kernel preemption turned on to meet certain real-time requirements.</li> The Touch driver contains a small bug that does not allow applications using it to exit properly</li> </ul> Overall, support is pretty good and enabled us to get started with development quickly.</p> NAND Flash Driver</h3> As the Compulab module only provides 4MB of NOR flash, using the NAND flash is a requirement when implementing a full featured Linux system with an advanced graphical user interface. Compulab provides a binary module that is linked into a kernel driver to support the NAND flash. This driver provides a block interface on which standard filesystems like ext2 can be used. The NAND driver has worked fairly well, although we have seen several cases where we get I/O errors if we remove power to the device without shutting down the system properly. The way to recover is to do a low level format the NAND flash. This should not be an issue with this product as we will have a backup battery that will be used to power the system during shutdown if power is removed. We may eventually move to an open source JFFS2 flash driver as time permits.</p> The proprietary NAND driver presents an additional wrinkle in that the driver cannot be linked into the kernel due to GPL licensing requirements. One solution is to boot a small root file system from NOR flash that includes the NAND driver module, and then proceed to boot from NAND flash.</p> GNU/Linux Root File System</h3> Compulab provides an example root file system (I think based on Debian) and suggests using scratchbox to build applications. We looked at it briefly, but then decided to build our own rootfs using OpenEmbedded for the following reasons:</p> We can easily add additional packages from the extensive OpenEmbedded catalog as needed.</li> Allows us to have an rootfs independent of the module – we can more easily move to other processor modules or architectures in the future.</li> OpenEmbedded provides us with mechanisms for package management and building complete images that include our applications and tweaks to the OS.</li> </ul> For more information on OpenEmbedded, read our “Introduction to OpenEmbedded</a> “. With OpenEmbedded, we were able to quickly build a toolchain and a root file system that included the kdrive version of Xwindows and GTK+ libraries. This rootfs fits in less than 32MB of flash disk space, which is much smaller than Compulab’s OS image.</p> We have also added machine configuration files to the OpenEmbedded project to support the cm-x270: http://www.openembedded.org/repo/org.openembedded.dev/conf/machine/compulab-pxa270.conf</a></p> Design Support</h3> Compulab seems mainly interested in customers who are interested in purchasing modules in volume. They do not sell low cost development boards such as LogicPD’s $500 board. Compulab requires you to purchase a fairly high cost development system ($1900) in order to get started. Part of this cost is to pay for Compulab’s support costs. For companies who do not have experience designing systems with PXA270 class processors, we highly recommend getting some up-front help from someone who has experience with these systems. Even though the module takes care of a lot of the design complexity, it helps to have a detailed understanding of the PXA270 and associated components, as you still need to interact with them. Consulting companies like BEC Systems can provide assistance in the form of hardware design reviews and Linux OS support to accelerate projects and help avoid costly mistakes and delays. As Compulab designs only the module, there is only so much they can do when you run into problems – as shown in the next section.</p> Issues Encountered</h3> Overall, the project using the cm-x270 has gone well. The only real challenge we have faced is a latch-up issue with the ITE PCI controller on the cm-x270 module. It appears that this part is very sensitive to the characteristics of the 3.3V supply to the module. After a considerable amount of debugging and testing, we were able to solve this problem with some filtering and slowing down the rise time of the 3.3V supply. This is a good example of where a project can get difficult very quickly when things don’t work as expected.</p> Summary</h3> So far, the cm-x270 has worked very well in this project. With its low cost, it is a very attractive option for companies wanting to implement advanced functionality in their products, but don’t have the time, sales volumes, or experience to justify a full processor board design. Feel free to post comments about your experiences using embedded computer modules.</p> Introduction to OpenEmbedded part 1 2007-01-12T00:00:00+00:00 This is the first in a multi-part series on the OpenEmbedded development environment. This and subsequent articles will provide an introduction to OpenEmbedded, why to use it, how it works, and will provide examples of how BEC has used OE in various projects.</p> Part 1 – Why do we need OpenEmbedded? This article discusses some of the problems developers face when building embedded Linux systems, such as collecting the pieces needed, setting up a repeatable build environment, cross-compiling, customizing a system, and licensing issues. Future articles will cover how OE addresses some of these issues.</p> Embedded Linux = Complex System</h3> The fundamental problem with building Embedded Linux systems is the sheer complexity. Lets face it, there is a lot going on in a typical Embedded Linux system. Millions of lines of code are being compiled. Typically there are tens to hundreds of applications and libraries that need to be built. Linux supports many different processor architectures. The size of Linux leads to the conclusion that a good deal of automation is required to manage all this. In my paper “Tips for Planning your Embedded Linux Project” (/web/content/view/35/37/</a> ), the analogy of the Iceberg is presented. The application specific software in the project is the part of the Iceberg that is above water – the part you see. Most of the system (Linux and system applications) is below water – you don’t see it. The part under water is the big unknown and has the potential to add a lot of variance to a project. Note: this problem is not specific to Linux – any system of this complexity that is targeted to so many different applications faces the same problem. As a side, one might ask why use Linux if it is so complex? The answer is simple – features. Linux has the features and device support required for many Embedded products to be competitive.</p> Collecting and building all the pieces</h3> There are a lot of pieces of software that need to come together to build an Embedded Linux system. This may include a toolchain, kernel, system libraries, and many system applications and configuration scripts required to make a system work. The is not something you want to do manually, or building an embedded Linux system quickly becomes impractical for most developers.</p> Repeatable Builds</h3> With any software project, it is highly desirable to have a build system that is easily repeatable – which means it can be reproduced at any time on a wide variety of development workstations. Some of the of the situations I have run into that are less than desirable are:</p> Being dependent on a vendor to provide a pre-compiled root file system image for a system. What happens when you need to add something two years down the road (such as foreign language support) and you don’t have a build system where you can easily reproduce the current build?</li> Being dependent on a golden build machine that takes a lot of work to set up and is very difficult to reproduce. If you have a build environment where all the developers use one machine to build software, you may want to investigate if you are in this situation.</li> </ul> What is really needed is a build system where you can check the build system out of a source control system onto about any Linux workstation at any point in time, type make, and a few hours later you have an image.</p> GPL Compliance</h3> If you distribute binaries in your product compiled from GPL software, you must offer to provide the source code for these binaries. Do you have easy access to the source for everything that is in your product? Do you even know what all is in your product? It is not good enough to simply point to someone else’s distribution or web site and say “I use that”. As the founder of the MEPIS distribution discovered</a>, the GPL license explicitly states the distributor of the GPL code is obligated to provide source code for up to 3 years. So if you build a distribution by copying binaries from a pre-build distribution, you may have a problem. In one case I am familiar with, a company was in the last stages of a multi-year product development cycle. They then discovered that they did not have the source code used to build the GPL binaries in the system. The company that built the distribution for them was out of business and the open source project the binaries where originally obtained from no longer maintained the sources. Although it may be possible to find all of the old sources, it is a huge amount of work and would probably be easier to just start over and build new distribution.</p> Customizing the system</h3> For most embedded systems, the distribution needs to be customized. There are often requirements such as custom software update schemes. Ideally, a good build system should easily allow you to integrate changes into the build system in a automated way, so that you don’t have any manual steps in the process of creating an image for the embedded system.</p> Cross Compiling</h3> The above “repeatable build” requirement that you be able to build an image for your embedded target on about any Linux workstation leads to a requirement for cross-compiling. Cross-compiling is the process of creating executable code for a platform different than the one on which the cross compiler is run (http://en.wikipedia.org/wiki/Cross_compiler</a>). Cross compiling many standard open source applications and libraries can be difficult on a good day, and near impossible on a bad day, as many of them were not written with cross compiling in mind. Therefore, a good build environment should have tools that automate the process of cross-compiling as well as provide the ability to easily cross compile the many common programs and libraries that may be needed in an embedded Linux system.</p> Summary</h3> As we examine some of the challenges, it should be obvious that putting together a production embedded Linux system is not something to be done in an ad-hoc fashion. There are several ways to approach this problem. Stay tuned for future articles where we explore how the OpenEmbedded build system solves these problems (subscribing to our blog or newsletter is a good way to do this).</p> Also, please post comments about your experiences – we like to hear and learn from you. We are also interested in what problems you face when building Embedded systems. This input will be used when writing future articles. Thanks!</p> Remote access for embedded systems behind a firewall 2006-12-19T00:00:00+00:00 When deploying embedded systems, wouldn’t it be nice if you could easily set up remote access for troubleshooting and updates? With Embedded Linux systems, this is quite easy to do. This article will explore how to use SSH to set up remote access to an embedded system behind a firewall without using VPN or special network setup and presents a few tips for making this setup work well.</p> The Problem</h3> With today’s complex embedded systems, testing and deployment can be a challenge. During the initial testing and roll-out, a new system may need to be carefully monitored to assure it is operating correctly. Embedded Systems programming is different than application programming in that often you are interfacing with external I/O. The behavior of things external to the embedded system are often very difficult to model during development; therefore time must be spent in the field testing the system in real-world situations. Being able to remotely access the system during testing is very convenient.</p> As with most networked computer systems, networked embedded systems are typically deployed on a network behind a firewall. It is very easy to log in and administer any Linux system you have direct network access to – the challenge is getting to a system behind a firewall.</p> Options for remote Access</h3> There are several general options for remote access to networks:</p> VPN</li> Port forwarding at the firewall</li> PC remote access technologies. These are typically single session type connections that are initiated by the PC user.</li> </ul> The disadvantage of the above technologies is they require involvement from the IT department or users of the remote network. BEC is currently involved in the deployment of a system that uses embedded computers in six different buildings. Each building has its own firewalled network. Getting remote access to all of these networks is a big task and everyone is busy. An ideal solution would be a way to communicate directly with each embedded computer from a central location, which brings us to SSH …</p> SSH</h3> SSH (Secure Shell) is a set of standards and an associated network protocol that is used to establish a secure channel between a local and remote computer. SSH encrypts all traffic (including passwords) and is generally considered to be very secure if used properly. Some uses for SSH are:</p> remote terminal access.</li> transfer files between computers.</li> tunnelling and port forwarding between computers</li> </ul> The fact that SSH can be used to forward ports over an encrypted tunnel is what makes remote access possible using what is called a Reverse SSH</strong> tunnel. Because the embedded system is behind a firewall with no direct access from the Internet, the connection must be initiated from the embedded system. A reverse SSH tunnel is set up as follows:</p> the embedded systems make a SSH connection to an external server that is internet accessible.</li> The SSH connection is set up so that a port on the external server (let’s pick 2900) is forwarded back over the SSH tunnel to port 22 (port used for SSH) on the embedded system.</li> The developer connects to 2900 on the external server using SSH. Because port 2900 on the external server gets forwarded over the SSH tunnel to port 22 of the embedded system, the developer is actually logging into the embedded system.</li> </ul> This connection is illustrated in the figure below:</p> </p> In the above figure, the external server is typically a system located in the developer’s office with only port 22 exposed to the Internet. Port 2900 is only accessible to developers on the same private network as the server.</p> Security</h3> When talking about any type of remote access technology involving the internet, it is important to consider the security implications. The reverse SSH mechanism is fairly secure, as everything is encrypted. SSH is widely used for remote access to servers; therefore it receives a lot of testing and scrutiny. With the reverse SSH remote access setup, the most likely attack is probably to compromise the external server and then try to access the remote network the embedded system is located on through the embedded system. If the external server is properly secured, then this mechanism should be fairly safe. One of the advantages of this scheme is that the embedded system is not directly exposed to the Internet, therefore securing the embedded system itself is not all that important.</p> It is obviously very important that the people administering the remote network understand what you are doing – that is where this article can come in handy. If there are concerns about external developers accessing a network, one mode of operation may be for the the SSH tunnel to be manually started and stopped so that administrators of the remote network know exactly when remote access is enabled.</p> TIPS</h3> A few ideas for making this work well:</p> It may be useful to put the reverse SSH connection in a cron job on the embedded system so that the connection is automatically set up. If you do this, I advise killing the connection and restarting every so often. Reverse SSH connections are somewhat fragile and occasionally need restarted if there is a network glitch.</li> Use the SSH TCPKeepAlive option. Some firewalls/routers will drop a network connection if there is no activity.</li> </ul> Summary</h3> Using SSH for remote access is another example of the “Big Win” with Embedded Linux – how industrial grade technologies developed for use in server and desktop systems can be readily used in embedded systems (/web/content/view/35/37/</a> ). Remote access during the deployment and testing of embedded systems can drastically reduce the amount of time and effort required to roll out an embedded system.</p> How to build GNU/Linux for an embedded x86 computer 2006-11-29T00:00:00+00:00 Recently, I had a customer who needed an embedded Linux distribution running on an Advantech PCM-9371 single board computer (SBC). The PCM-9371 contains a low voltage Celeron or Pentium III processor. This article describes why the Openembedded build system was chosen and a few tips for running Openembedded on a x86 system.</p> The hard way!</h2> There are many ways to put Linux on a x86 embedded system. There are hundreds of distributions out there and the temptation is always to do something “quick” and just try to trim down one of the standard distributions. Other shortcuts are often taken, such as compiling your application on a workstation that is running a different distribution and different versions of the libraries used in the embedded systems. This approach can work, but there are many possible pitfalls:</p> Several years down the road, it will be difficult to compile applications for your target system as the libraries on the target system are very dated compared to tools and libraries used in standard desktop distributions. What this means is you will need to keep a “golden” compile machine around for the life of your project – not fun! I went through an experience like this on a previous version of this same project where it took us a week to set up a build machine to compile applications using a very old version of Debian.</li> It is difficult to gather GPL sources used to build the distribution you are using. You are required to supply GPL source code upon request to customers who purchase your product.</li> To build applications correctly, you really need a build machine that is running the same distribution as the target, unless you set up a cross-compilation or scratchbox environment.</li> Most distributions are quite large and have difficulty fitting on CF card.</li> Most distributions are difficult to customize in a clean, systematic way.</li> </ul> OpenEmbedded</h2> Considering the disadvantages of using an standard x86 distribution for this project, I chose to use OpenEmbedded (OE). The OpenEmbedded project (http://openembedded.org</a> ) is a full featured build system for building embedded Linux distributions. The development is done on a Linux workstation of your choice and the entire distribution is cross-compiled to the architecture of your choice. The entire build process is automated, including building the toolchain and an image that can be installed on the target embedded system. Some of the features of OpenEmbedded are:</p> Over 3000 packages that can be built. About anything you would ever need for an embedded system.</li> Support for a wide variety of target architectures. ARM and x86 processors are well supported.</li> Build system is very self contained. This means you can do the builds on a variety of different host machines. Many of the tools OE needs are built by OE, so there is very little dependence on host tools besides a few basics such as Python, make, and other standard tools. This also means it is much easier to pull the build system out of the archives in several years, load it on a current machine, and do a build.</li> All source code used in the build is downloaded, so GPL compliance is easy.</li> and the list goes on …</li> </ul> Build and Installation</h2> Setting up an OE build is beyond the scope of this article, but it basically involves selecting a machine and distribution. In this case, I chose the x86</em> machine and the generic</em> OE distribution. Several hours later, I had an image in the form of a tar.gz file that I decompressed to a Compact Flash (CF) card. After installing grub on the CF card, I had a booting Linux distribution that included all the basics (like ssh, ipkg package manager, shell, standard file utilities, etc) in less than 4MB of disk space.</p> Adding GTK+ and Xwindows</h2> This system required support for GTK+ and Xwindows. To add this involved running the following commands: bitbake xserver-kdrive</em> and bitbake gtk+</em>. The OE build system then automatically downloaded and compiled the need packages. The image size is now around 20MB – still fits very easily on a CF card.</p> Getting X running</h2> There are several options for running X on a x86 system. In this case, I chose the quickest way so that my customer could get started with application integration as quickly as possible. Linux and Xwindows support the standard VESA Framebuffer mode. Then can be enabled by selecting the CONFIG_FB_VESA kernel config option. The kdrive version of X can then be started by running Xvesa. In the future, we may move to an accelerated X driver if needed.</p> Summary</h2> We now have Linux running off of a CF card with full X/GTK+ support and the customer can proceed with integration, testing and application development. Getting to this point took about 1 day of development as we did not encounter any major problems – much less time than it took to resurrect an old build environment used in previous versions of the project. Future work will involve integrating applications into the OE build system, and customizing the distribution as needed. Openembedded has proved to be a very valuable tool for getting embedded Linux systems running quickly and maintaining them in a controlled way.</p> Adding software to a Moxa UC7408 using OpenEmbedded 2006-10-30T00:00:00+00:00 One of the big advantages of using Linux in embedded systems is the thousands of applications and drivers available for it (/web/content/view/35/37/</a> ). One of the challenges of Embedded Linux is building these applications. Building applications for an embedded system often involves cross-compiling which is frustrating on a good day. This article details how to use OpenEmbedded (OE) with the existing toolchain Moxa provides to build a number of applications for the Moxa UC7408.</p> Web Application Support</h2> My needs for this project were to build a web application that ran on the UC7408 (See /web/content/view/55/9/</a> for a review of the Moxa UC7408). The Moxa firmware is fairly full featured and provides php for web application development. This is probably fine for many applications, but I prefer to build web applications as detailed in this article: /web/content/view/46/9/</a> . Because the rest of the application is written in Python, there are advantages to also implementing the web application in Python. I also wanted to use the sqlite database. A quick look at the php supplied with the Moxa firmware suggested it was built without sqlite support. In summary, I needed the following components:</p> Python</li> Clearsilver</li> sqlite</li> minicom</li> vim</li> strace</li> subversion</li> wget</li> </ul> The above is a fairly small list, but what you don’t see are all the dependencies that also need to be built and installed. This ends up being many more components. Some of the items are for development only, but this is Linux – why not have nice tools running on the target system?</p> OE to the rescue</h2> Ideally, the Moxa system would have a complete root file system built with OE running on it. But I did not have the time or budget to implement this, so I did the next best thing – use the rootfs and toolchain supplied by Moxa along with OE to compile just the applications I needed. Typically, OE builds the toolchain and Glibc, but fortunately OE provides a very slick way to use an external toolchain and glibc with OE. How to do this is detailed in the OE manual: http://www.openembedded.org/user-manual&dpage=commonuse_prebuilt_toolchain</a>. Below is the setup I used:</p> Environment variables:</p> TOOLCHAIN=/usr/local/mxscaleb/bin/ export PRE_BUILT=/usr/local/mxscaleb/armv5b-linux export CCACHE_DIR="${TOPDIR}/ccache" export PYTHONPATH="${TOPDIR}/bitbake/lib" export BBPATH="${TOPDIR}:${TOPDIR}/openembedded:${TOPDIR}/bitbake" export PATH="${TOPDIR}/bitbake/bin:${TOOLCHAIN}:${PATH}" export LD_LIBRARY_PATH= export LANG=C </code></pre> Local.conf file:</p> DL_DIR = "/build/sva_oe/downloads" BBFILES := "/build/sva_oe/openembedded/packages//.bb" BBMASK = "" PREFERRED_PROVIDERS = "virtual/qte:qte virtual/libqpe:libqpe-opie" PREFERRED_PROVIDERS += " virtual/libsdl:libsdl-qpe" PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}gcc-initial:gcc-cross-initial" PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}gcc:gcc-cross" PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}g++:gcc-cross" PREFERRED_PROVIDER_virtual/libx11 = "diet-x11" ASSUME_PROVIDED += " virtual/${TARGET_PREFIX}gcc " ASSUME_PROVIDED += " virtual/libc " MACHINE = "moxa" IPKG_ARCHS = "all arm armv4 armv5te xscale ${MACHINE}" TARGET_CC_ARCH = "-mcpu=xscale" TARGET_ARCH = "arm" PACKAGE_ARCH="xscale" TARGET_OS = "linux" TARGET_FPU = "soft" DISTRO = "moxa-disro" DISTRO_NAME = "moxa-distro" DISTRO_VERSION = "0.0.1" DISTRO_TYPE = "release" INHERIT += " package_ipk package_tar debian" IMAGE_FSTYPES = "jffs2 tar" PARALLEL_MAKE = "-j 4" BBINCLUDELOGS = "yes" CVS_TARBALL_STASH = "http://www.oesources.org/source/current/" export CC = "mxscaleb-gcc-3.3.2 ${HOST_CC_ARCH}" export CXX = "mxscaleb-g++ ${HOST_CC_ARCH}" export CPP = "mxscaleb-gcc-3.3.2 -E" export LD = "mxscaleb-ld" export AR = "mxscaleb-ar" export AS = "mxscaleb-as" export RANLIB = "mxscaleb-ranlib" export STRIP = "mxscaleb-strip" TARGET_CPPFLAGS_append = " -I${PRE_BUILT}/include " TARGET_LDFLAGS_prepend = " -L${PRE_BUILT}/lib -Wl,-rpath-link, ... </code></pre> ${PRE_BUILT}/lib -Wl,-rpath-link,${PRE_BUILT}/qt2/lib " </code></pre> With the above setup, I was able to build the applications I needed with very little effort. There were a few little problems I ran into, but I was able to quickly work around them. The result is a number of packages that can be installed on the target system:</p> bigreqsproto-dev-X11R7.0-1.0.2-r0.tar.gz python-crypt-2.4.3-ml0.tar.gz busybox-1.2.1-r1.3.tar.gz python-curses-2.4.3-ml1.tar.gz busybox-udhcpd-1.2.1-r1.3.tar.gz python-datetime-2.4.3-ml0.tar.gz clearsilver-0.10.3-r0.tar.gz python-db-2.4.3-ml0.tar.gz clearsilver-dbg-0.10.3-r0.tar.gz python-devel-2.4.3-ml0.tar.gz clearsilver-dev-0.10.3-r0.tar.gz python-distutils-2.4.3-ml0.tar.gz clearsilver-doc-0.10.3-r0.tar.gz python-email-2.4.3-ml0.tar.gz inputproto-dev-X11R7.0-1.3.2-r0.tar.gz python-fcntl-2.4.3-ml0.tar.gz ipkg-0.99.163-r1.tar.gz python-gdbm-2.4.3-ml0.tar.gz ipkg-dbg-0.99.163-r1.tar.gz python-hotshot-2.4.3-ml0.tar.gz ipkg-dev-0.99.163-r1.tar.gz python-html-2.4.3-ml0.tar.gz kbproto-dev-X11R7.1-1.0.3-r0.tar.gz python-idle-2.4.3-ml0.tar.gz libapr-0-0-0.9.12-r0.tar.gz python-image-2.4.3-ml0.tar.gz libapr-0-bin-0.9.12-r0.tar.gz python-io-2.4.3-ml0.tar.gz libapr-0-dev-0.9.12-r0.tar.gz python-lang-2.4.3-ml0.tar.gz libaprutil-0-0-0.9.12-r0.tar.gz python-lib-old-and-deprecated-2.4.3-ml0.tar.gz libaprutil-0-bin-0.9.12-r0.tar.gz python-logging-2.4.3-ml0.tar.gz libaprutil-0-dev-0.9.12-r0.tar.gz python-mailbox-2.4.3-ml0.tar.gz libcrypto0.9.7-0.9.7g-r1.tar.gz python-math-2.4.3-ml0.tar.gz libexpat-bin-2.0.0-r1.tar.gz python-mime-2.4.3-ml0.tar.gz libexpat-dev-2.0.0-r1.tar.gz python-mmap-2.4.3-ml0.tar.gz libexpat-doc-2.0.0-r1.tar.gz python-netclient-2.4.3-ml1.tar.gz libexpat1-2.0.0-r1.tar.gz python-netserver-2.4.3-ml0.tar.gz libgcrypt-dbg-1.2.2-r0.tar.gz python-pickle-2.4.3-ml0.tar.gz libgcrypt-dev-1.2.2-r0.tar.gz python-pprint-2.4.3-ml0.tar.gz libgcrypt-doc-1.2.2-r0.tar.gz python-profile-2.4.3-ml0.tar.gz libgcrypt11-1.2.2-r0.tar.gz python-pydoc-2.4.3-ml0.tar.gz libgdbm-dbg-1.8.3-r2.tar.gz python-pyserial-2.2-r1.tar.gz libgdbm-dev-1.8.3-r2.tar.gz python-pysqlite2-2.2.2-ml1.tar.gz libgdbm-doc-1.8.3-r2.tar.gz python-pyxml-0.8.4-ml0.tar.gz libgdbm3-1.8.3-r2.tar.gz python-re-2.4.3-ml0.tar.gz libgnutls-bin-1.4.0-r1.tar.gz python-readline-2.4.3-ml0.tar.gz libgnutls-dbg-1.4.0-r1.tar.gz python-resource-2.4.3-ml0.tar.gz libgnutls-dev-1.4.0-r1.tar.gz python-shell-2.4.3-ml0.tar.gz libgnutls-doc-1.4.0-r1.tar.gz python-stringold-2.4.3-ml0.tar.gz libgnutls-extra13-1.4.0-r1.tar.gz python-subprocess-2.4.3-ml0.tar.gz libgnutls-locale-en+boldquot-1.4.0-r1.tar.gz python-syslog-2.4.3-ml0.tar.gz libgnutls-locale-en+quot-1.4.0-r1.tar.gz python-terminal-2.4.3-ml0.tar.gz libgnutls-locale-pl-1.4.0-r1.tar.gz python-tests-2.4.3-ml0.tar.gz libgnutls-openssl13-1.4.0-r1.tar.gz python-textutils-2.4.3-ml0.tar.gz libgnutls13-1.4.0-r1.tar.gz python-threading-2.4.3-ml0.tar.gz libgpg-error-dbg-1.3-r1.tar.gz python-tkinter-2.4.3-ml0.tar.gz libgpg-error-dev-1.3-r1.tar.gz python-unittest-2.4.3-ml0.tar.gz libgpg-error-locale-de-1.3-r1.tar.gz python-unixadmin-2.4.3-ml0.tar.gz libgpg-error-locale-fr-1.3-r1.tar.gz python-xml-2.4.3-ml0.tar.gz libgpg-error-locale-pl-1.3-r1.tar.gz python-xmlrpc-2.4.3-ml0.tar.gz libgpg-error-locale-ro-1.3-r1.tar.gz python-zlib-2.4.3-ml1.tar.gz libgpg-error-locale-vi-1.3-r1.tar.gz sqlite3-3.3.7-r2.tar.gz libgpg-error0-1.3-r1.tar.gz sqlite3-dbg-3.3.7-r2.tar.gz libice-dbg-X11R7.1-1.0.1-r0.tar.gz strace-4.5.14-r3.tar.gz libice-dev-X11R7.1-1.0.1-r0.tar.gz strace-dbg-4.5.14-r3.tar.gz libice6-X11R7.1-1.0.1-r0.tar.gz strace-doc-4.5.14-r3.tar.gz libipkg-dev-0.99.163-r1.tar.gz subversion-1.3.1-r1.tar.gz libipkg0-0.99.163-r1.tar.gz subversion-dbg-1.3.1-r1.tar.gz liblzo-dbg-1.08-r14.tar.gz subversion-dev-1.3.1-r1.tar.gz liblzo-dev-1.08-r14.tar.gz subversion-doc-1.3.1-r1.tar.gz liblzo1-1.08-r14.tar.gz subversion-locale-de-1.3.1-r1.tar.gz libneon-bin-0.25.5-r0.tar.gz subversion-locale-es-1.3.1-r1.tar.gz libneon-dev-0.25.5-r0.tar.gz subversion-locale-fr-1.3.1-r1.tar.gz libneon-doc-0.25.5-r0.tar.gz subversion-locale-it-1.3.1-r1.tar.gz libneon25-0.25.5-r0.tar.gz subversion-locale-ja-1.3.1-r1.tar.gz libpython2.4-1.0-2.4.3-ml5.tar.gz subversion-locale-ko-1.3.1-r1.tar.gz libreadline-dbg-4.3-r3.tar.gz subversion-locale-nb-1.3.1-r1.tar.gz libreadline-dev-4.3-r3.tar.gz subversion-locale-pl-1.3.1-r1.tar.gz libreadline-doc-4.3-r3.tar.gz subversion-locale-pt-br-1.3.1-r1.tar.gz libreadline4-4.3-r3.tar.gz subversion-locale-sv-1.3.1-r1.tar.gz libsm-dbg-X11R7.1-1.0.1-r0.tar.gz subversion-locale-zh-cn-1.3.1-r1.tar.gz libsm-dev-X11R7.1-1.0.1-r0.tar.gz subversion-locale-zh-tw-1.3.1-r1.tar.gz libsm6-X11R7.1-1.0.1-r0.tar.gz tcl-8.4.11-r3.tar.gz libsqlite-bin-2.8.17-r2.tar.gz tcl-dbg-8.4.11-r3.tar.gz libsqlite-dbg-2.8.17-r2.tar.gz tcl-dev-8.4.11-r3.tar.gz libsqlite-dev-2.8.17-r2.tar.gz tcl-doc-8.4.11-r3.tar.gz libsqlite0-2.8.17-r2.tar.gz time-1.7-r0.tar.gz libsqlite3-0-3.3.7-r2.tar.gz time-dbg-1.7-r0.tar.gz libsqlite3-dev-3.3.7-r2.tar.gz time-doc-1.7-r0.tar.gz libssl0.9.7-0.9.7g-r1.tar.gz tmp libx11-6-X11R7.1-1.0.1-r1.tar.gz update-rc.d-0.7-r0.tar.gz libx11-dbg-X11R7.1-1.0.1-r1.tar.gz util-macros-dev-X11R7.1-1.0.2-r0.tar.gz libx11-dev-X11R7.1-1.0.1-r1.tar.gz vim-7.0-r0.tar.gz libx11-doc-X11R7.1-1.0.1-r1.tar.gz vim-doc-7.0-r0.tar.gz libx11-locale-X11R7.1-1.0.1-r1.tar.gz vim-help-7.0-r0.tar.gz libxau-dbg-X11R7.1-1.0.1-r0.tar.gz vim-syntax-7.0-r0.tar.gz libxau-dev-X11R7.1-1.0.1-r0.tar.gz vim-tutor-7.0-r0.tar.gz libxau-doc-X11R7.1-1.0.1-r0.tar.gz wget-1.9.1-r6.tar.gz libxau6-X11R7.1-1.0.1-r0.tar.gz wget-dbg-1.9.1-r6.tar.gz libxdmcp-dbg-X11R7.1-1.0.1-r0.tar.gz wget-doc-1.9.1-r6.tar.gz libxdmcp-dev-X11R7.1-1.0.1-r0.tar.gz wget-locale-bg-1.9.1-r6.tar.gz libxdmcp6-X11R7.1-1.0.1-r0.tar.gz wget-locale-ca-1.9.1-r6.tar.gz libxml2-2.6.22-r3.tar.gz wget-locale-cs-1.9.1-r6.tar.gz libxml2-dbg-2.6.22-r3.tar.gz wget-locale-da-1.9.1-r6.tar.gz libxml2-dev-2.6.22-r3.tar.gz wget-locale-de-1.9.1-r6.tar.gz libxml2-doc-2.6.22-r3.tar.gz wget-locale-el-1.9.1-r6.tar.gz libxml2-utils-2.6.22-r3.tar.gz wget-locale-es-1.9.1-r6.tar.gz libxt-dbg-X11R7.1-1.0.2-r0.tar.gz wget-locale-et-1.9.1-r6.tar.gz libxt-dev-X11R7.1-1.0.2-r0.tar.gz wget-locale-fr-1.9.1-r6.tar.gz libxt-doc-X11R7.1-1.0.2-r0.tar.gz wget-locale-gl-1.9.1-r6.tar.gz libxt6-X11R7.1-1.0.2-r0.tar.gz wget-locale-he-1.9.1-r6.tar.gz libz-dbg-1.2.3-r1.tar.gz wget-locale-hr-1.9.1-r6.tar.gz libz-dev-1.2.3-r1.tar.gz wget-locale-hu-1.9.1-r6.tar.gz libz1-1.2.3-r1.tar.gz wget-locale-it-1.9.1-r6.tar.gz minicom-2.1-r0.tar.gz wget-locale-ja-1.9.1-r6.tar.gz minicom-dbg-2.1-r0.tar.gz wget-locale-nl-1.9.1-r6.tar.gz minicom-doc-2.1-r0.tar.gz wget-locale-no-1.9.1-r6.tar.gz ncurses-5.4-r8.tar.gz wget-locale-pl-1.9.1-r6.tar.gz ncurses-dbg-5.4-r8.tar.gz wget-locale-pt-br-1.9.1-r6.tar.gz ncurses-dev-5.4-r8.tar.gz wget-locale-ro-1.9.1-r6.tar.gz ncurses-doc-5.4-r8.tar.gz wget-locale-ru-1.9.1-r6.tar.gz ncurses-terminfo-5.4-r8.tar.gz wget-locale-sk-1.9.1-r6.tar.gz ncurses-tools-5.4-r8.tar.gz wget-locale-sl-1.9.1-r6.tar.gz openssl-0.9.7g-r1.tar.gz wget-locale-sv-1.9.1-r6.tar.gz openssl-dbg-0.9.7g-r1.tar.gz wget-locale-tr-1.9.1-r6.tar.gz openssl-dev-0.9.7g-r1.tar.gz wget-locale-uk-1.9.1-r6.tar.gz openssl-doc-0.9.7g-r1.tar.gz wget-locale-zh-cn-1.9.1-r6.tar.gz python-audio-2.4.3-ml0.tar.gz wget-locale-zh-tw-1.9.1-r6.tar.gz python-bsddb-2.4.3-ml0.tar.gz xcmiscproto-dev-X11R7.0-1.1.2-r0.tar.gz python-codecs-2.4.3-ml0.tar.gz xextproto-dev-X11R7.0-7.0.2-r0.tar.gz python-compile-2.4.3-ml0.tar.gz xf86bigfontproto-dev-X11R7.0-1.1.2-r0.tar.gz python-compiler-2.4.3-ml0.tar.gz xproto-dev-X11R7.1-7.0.5-r0.tar.gz python-compression-2.4.3-ml0.tar.gz xtrans-dev-X11R7.0-1.0.0-r1.tar.gz python-core-2.4.3-ml1.tar.gz </code></pre> Installing the needed packages is just a matter of uncompressing a subset of the above files on the target system.</p> Summary</h2> This exercise demonstrates how a developer can quickly (took me less than 4 hours) add a large number of applications (and dependencies) to about any Embedded Linux system. Leveraging embedded Linux is being able to use the components you</strong></em> need. Please contact us if you would like additional details or assistance.</p> Moxa UC7408 Review 2006-09-29T00:00:00+00:00 I am currently evaluating a Moxa UC7408 for one of my customers. The UC7408 is a small fanless industrial computer that runs Linux or Windows CE. This article provides a basic overview of this unit and a review of some of the UC7408 features and the Linux distribution Moxa provides.</p> </p> The basic specifications for the UC7408 are:</p> relatively small size and fanless</li> Intel Xscale IXP-422 266MHz Processor</li> 8 RS232/422/485 serial ports</li> 8 digital Input and 8 digital output ports</li> dual 10/100 Ethernet</li> PCMCIA, CompactFlash, Wireless LAN Expansion (supports 802.11b/802.11g)</li> Runs Linux or WinCE</li> Console serial port</li> </ul> Documentation</h3> The documentation Moxa provides is adequate and fairly detailed. The moxa documents include a quick install guide, hardware user’s manual, and a 114 page user’s manual. As good as this documentation is, a general knowledge of Linux is always helpful when working with embedded Linux systems. Basics like using SSH, FTP, Telnet, and basic Linux system administration are essential when working with embedded Linux.</p> Linux Distribution</h3> The Linux Distribution provided by Moxa is fairly full featured. A few highlights with the v1.8 Linux firmware:</p> Provides a 26MB user JFFS2 flash partition. This partition is read/write and is about 45% full leaving about 14.5MB for user files. It is probably also possible to delete some of the files that are not required, freeing up more space.</li> Apache & PHP support</li> SSH</li> a number of other utilities are provided such as telnet, ftp, iptables, etc.</li> VI editor</li> Many command line utilities found in most Linux systems.</li> 2.4.18 Kernel</li> </ul> The Linux distribution seems to be based on MontaVista Linux and seems fairly solid. I ran through some of the basics like mounting a CF card and everything seems to work.</p> Firmware Updates and Recovery Mode</h3> Moxa provides a mechanism to update the firmware in the system. This update is a global flash update that programs the entire flash and will erase all user changes. The update works by enabling a RAM file system and copying the new flash image to the ram file system. Running an update command that copies the firmware update file into flash.</p> The 7408 contains a stripped down version of Linux in a separate flash partition that can be booted in a recovery mode if the User flash partition becomes corrupted or unusable. This mode can be enabled by pressing the “Reset to Default” button on the unit and powering it on. In this mode, there is no ssh support and you must ftp an image from an ftp server to the device and then reflash it. This requires you to set up an ftp server that the Moxa system can access. While this is an entirely workable solution, it may be a challenge for inexperienced Linux users.</p> Tools</h3> Moxa provides a Linux and Windows toolchain to use with the device. I have built a number of packages with the Linux toolchain and it seems to work well. It is based on GCC 3.3.2. Cross compiling applications for embedded Linux is always a challenge and in a future article I will present a way to use OpenEmbedded to compile a number of packages using the Moxa toolchain.</p> Possible Improvements</h3> There would be some benefits if the system was a little more open in the following areas:</p> access to the bootloader console and documentation. This would allow developers to more easily flash their own software.</li> readily provide source for GPL components. Moxa provides a form that you can fill out to receive GPL source code on a CE. The cost is $100.</li> </ul> One of the big advantages of using Linux in an embedded system is the ability to use the many open source components available. The easier an embedded solution provider makes this for developers, the more value their system provides. See the white paper Tips for Planning Your Embedded Linux Project</a> (/web/content/view/35/37/) for more ideas on this subject.</p> Summary</h3> Overall, the Moxa system seems very usable and the hardware is quite nice. Stay tuned for future articles about how to get the most out of this system.</p> </p> </p> </p> How to evaluate VoIP providers with Asterisk 2006-09-20T00:00:00+00:00 There are many options when selecting a VoIP provider to terminate your Asterisk PBX. One of the most important factors in this selection is the quality of the internet connection between your PBX and the VoIP provider. The route between you and your VoIP provider is one of the most important considerations when selecting a provider. This article details a simple method for evaluating this connection using Asterisk.</p> Asterisk includes a feature that can be used to monitor the latency between your system and a peer</em> or friend</em>. This feature is enabled by setting the qualify</em> setting in the iax.conf</em> configuration file. Valid options are yes</em>, no</em>, or a time in milliseconds. If qualify is enabled, NOTIFY messages are sent periodically to the peer and the latency between replies is measured. The peer is determined unreachable if the number of milliseconds is greater than the qualify value or 2,000 if qualify is set to yes</em>. If a peer is unreachable, events are logged in /var/log/asterisk/messages as shown below:</p> Sep 19 21:48:19 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now UNREACHABLE! Time: 49 Sep 19 21:50:09 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now REACHABLE! Time: 49 Sep 20 07:16:38 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now TOO LAGGED (769 ms)! Sep 20 07:16:48 NOTICE[752] chan_iax2.c: Peer 'NuFone2' is now REACHABLE! Time: 50 Sep 20 12:09:31 NOTICE[752] chan_iax2.c: Peer 'teliax' is now TOO LAGGED (2923 ms)! Sep 20 12:17:27 NOTICE[752] chan_iax2.c: Peer 'teliax' is now REACHABLE! Time: 83 </code></pre> I am currently using two different VoIP providers and I get qualify notice messages about once every day or two with the qualify set to 400ms. This is a very useful information as it allows me to determine which provider has the most reliable route from my office. If you get very few qualify NOTICE messages, you can probably assume your VoIP connection will be fairly reliable.</p> Have connections with two different providers also tells me when the congestion is with my Internet connection or if it is something beyond my connection. If both providers go out at the same time, it is most likely the internet connection at my office or problems with my ISP. I rarely if ever see this scenario, so I can conclude that my internet connection is fairly reliable and the congestion problems are most likely between me and the VoIP provider or problems at the VoIP provider.</p> Overall, things are now working fairly well. As we continue to resolve problems, VoIP is now starting to become a very usable option for us.</p> Implenting QoS for VOIP in a Linksys router 2006-08-28T00:00:00+00:00 In our quest to implement an Asterisk PBX in our office, we are evaluating several VOIP (Voice Over IP) providers. VOIP provides several benefits over traditional phone lines including:</p> low cost (typically around $0.02 per minute)</li> available in pre-paid plans</li> can support concurrent concurrent calls on the same number</li> </ul> However, implementing VOIP successfully is not trivial. You must</strong> have a reliable internet connection and we have found you must implement some type of Quality of Service (QoS) in the router. The first problem I ran into was calls sounded great to me, but people I called mentioned the calls sometimes sounded garbled to them. This article discusses the diagnosis of this problem and how to implement QoS in a Linksys router to fix this issue.</p> Our connection is a standard DSL connection – 5Mb/sec down and 640Kb/sec up. There is plenty of bandwidth for incoming voice, but bandwidth for outgoing voice is much more limited. A few tests confirmed this – when a ftp upload or some other bulk transfer was occurring, the outgoing voice sounded really bad.</p> The solution to this problem is to implement some type of Quality of Service (QoS) mechanism that prioritizes the VOIP traffic over all other types of traffic. Vonage and other “boxed” solutions no doubt do this as they recommend placing their box between the modem and the router. It turns out that QoS can be implemented in a standard Linksys router/firewall. I spent some time trying to get QoS working using a Linksys WRT54G v5 router with no success, I ended up loading the dd-wrt firmware (http://www.dd-wrt.com</a>) on an older Linksys router I had laying around. The dd-wrt firmware is quite nice and is worthy of consideration if you have a v4 or earlier Linksys router or the WRT54GL version. QoS may have worked on the WRT54G v5 with Linksys firmware with some more effort, but after trying dd-wrt, I’m not going back.</p> Setting up QOS</h2> The QoS settings can be found under Applications & Gaming -> QoS.</p> There are many ways to assign priority to your VOIP system. I just used the IP address as shown below. 192.168.1.10 is the LAN private IP of the Asterisk system.</p> </p> With the priority of the Asterisk box set to high, we should be all set. It turns out that unless you cap the bandwidth, you will still have QoS problems. Enter around 85% of your up and down bandwidth in the Uplink and Downlink boxes. Capping the bandwidth is required to keep the latency low. Otherwise, your bandwidth is filled up with large packets and the priority settings are not effective for VOIP traffic. See http://www.dd-wrt.com/wiki/index.php/Quality_of_Service</a> for more information.</p> </p> With the above settings, VOIP calls are now clear even with other concurrent activities.</p> UPDATE:</strong></p> Several people have reported being able to install dd-wrt on v6 WRT54 routers and the QoS configuration is the same.</p> Embedded is a Service Industry 2006-06-02T00:00:00+00:00 Linuxdevices.com has recently published an article titled: Let’s call Embedded what it is – a Service Industry</em>(http://linuxdevices.com/articles/AT6989225710.html</a>). In this article, Curt Schacker discusses how little growth there has been in the Embedded Systems industry. He claims that most embedded companies are trying to establish themselves as product companies, when the market is really a services market. Having worked for two Embedded companies in the past that went out of business, I think he is right. I have often heard the argument that service companies are too much work, they don’t scale, etc. But, to me services has always been a more natural fit for myself and an embedded systems company – one of the reasons BEC Systems exists.</p> TRAC and Distributed Development 2006-05-29T00:00:00+00:00 I have long been a big fan of source control tools and issue tracking systems. The “concurrent” model made popular by CVS is a tremendous benefit any time there are multiple people working on the same project. For most of the projects BEC participates in, we end up setting up a source control system (usually Subversion these days) and an issue tracking system (typically Bugzilla). On one of my recent projects, I decided to try the trac integrated SCM & Project Management tool. Trac brings together several functions into a clean, easy to use web based tool for managing projects. This article relates some of my experiences and the benefits of using trac.</p> What is trac?</h2> Trac provides the following functions (from the trac website - http://www.edgewall.com/trac/</a> ):</p> An integrated system for managing software projects</li> An enhanced wiki</li> A flexible web-based issue tracker</li> An interface to the Subversion revision control system</li> </ul> Why is trac so interesting?</h2> Although all of these functions are available as separate tools, having them in one tool has several benefits:</p> The tool is more likely to be used if everything is in one place</li> The ease of cross referencing wiki pages, issues, and Subversion changesets compounds the usefulness of each tool.</li> </ul> I have been using and implementing collaboration tools for years and one thing you eventually learn is that getting people to use collaboration tools is a big challenge (see http://www.linux-watch.com/news/NS9450464356.html</a> for some more thoughts on this subject). Having mutiple tools in one place greatly increases the likelihood they will be used. A typical session might go something like this:</p> A developer might want to see what has changed recently in the code base, so he logs into trac to view the timeline – after all, most developers are mainly concerned with code.</li> As the trac timeline lists all changes (wiki and issues as well as source code changes), he notices that a wiki page with notes about driver X has changed – that looks interesting, so he view the page. While viewing the page, the developer notes there is some missing information, so he adds the information by simply editing the page.</li> Once this habit is established, it can greatly improve the productivity of the team as the project becomes more transparent to all team members.</li> </ul> Any collaboration tool must be easy to use and provide an obvious benefit to those using the tool.</p> The wiki markup used in trac is used for both wiki pages as well as issue/bug reports. Referencing an issue in a wiki page or another bug report is simple as typing #. It is also very easy to reference code changsets in the wiki markup by simply typing r (see image below).</p> </p> Clicking on a issue or changeset number will take you directly to the information. The trac interface is very clean and very easy to use.</p> Distributed teams</h2> Tools like trac allow for very effective distributed teams. The success of Open Source software has proven that development can occur efficiently in a distributed manner. BEC has integrated this model and we regularly work with people all over the country and in Europe. When there is a process for all people involved in the project to keep up with its status and progress, distributed development becomes very possible. Often, the customer is part of the development team.</p> Project Transparency</h2> Another benefit of trac for BEC, is that we can easily provide project transparency to our customers. It is BEC’s practice to provide customers with access to the project as it develops, so we give our customers trac logins and they can follow the progress by watching the timeline and milestone pages. This gives the customer an assurance that progress is being made on the project and some insight into some of the issues encountered. Having a comprehensive view into the project also allows customers to provide early feedback and reduces BEC’s project management and reporting overhead.</p> Summary</h2> Tools like trac improve our customer relationships, increase efficiency and reduce our costs. If you are interested in implementing tools like trac, BEC can assist you in selecting and implementing the right tools to make your projects run smoothly.</p> Opencores SPI Controller Review 2006-05-08T00:00:00+00:00 Recently, I had the opportunity to use a SPI controller from Opencores in a customer design based on an Intel PXA255 processor and a Xilinx CPLD. The customer did not want to change the processor or the CPLD family, but needed an extra SPI interface in the device. I had my doubts, but mentioned I would look into a SPI controller from Opencores. To my surprise, the core fit in a Xilinx CPLD and worked out very well. We have since upgraded to a Altera MAXII CPLD. This review details this effort.</p> SPI Controller Overview</h3> The SPI controller is based on the interface for a standard Motorolla SPI controller and is available at http://www.opencores.org/projects.cgi/web/spi/overview</a> . The core includes the following features:</p> Full duplex synchronous serial data transfer</li> Variable length of transfer word up to 128 bits</li> MSB or LSB first data transfer</li> Rx and Tx on both rising or falling edge of serial clock independently</li> 8 slave select lines</li> Fully static synchronous design with one clock domain</li> Technology independent Verilog</li> Fully synthesizable</li> </ul> A datasheet type document is provided with the core that provides the necessary information for developers writing software for this core.</p> The Wishbone bus</h3> Most of the cores from Opencores uses the Wishbone bus standard. The Wishbone bus is a synchronous bus designed to be used inside an IC. The CPLD was connected to the PXA255 processors asynchrous bus, so most of the effort required was to create an PXA255 <-> Wishbone interface module. At some point, I hope to release this module on Opencores.</p> Simulation</h3> The simulation was done using the Icarus Verilog simulator under Linux. Below is a screen shot of the simulation displayed in GTKWave:</p> </p> Conclusion</h3> The Opencores SPI core worked out very well in our design and I was very impressed with quality of this core. The documentation is more than adequate. Standards like Verilog make it very easy to move designs between different vendors devices and allows designers to use tools of their choice such as the Icarus Verilog simulator. As FPGAs and CPLDs become more larger and more standand, Opencores is valuable resource for adding standard interfaces to these devices.</p> A Really Nice Hex Calculator 2006-05-02T00:00:00+00:00 When working with embedded software, I often have a need for a hex calculator. Over the years I’ve used a number of different tools. Through college and for several years after, I used my HP48 calculator. It is a great tool. Around 1998, I discovered the Python programming language. As Python supports an interactive session, it works well as a desktop calculator – I don’t think I have used my HP48 since I discovered Python. Recently, I’ve been using Ruby more in place of Python and am finding it is even nicer. This article presents several tips and snippets of code for using Ruby as a very effective hex calculator.</p> For those of you not familiar with Ruby, Ruby is an interpreted scripting language for quick and easy object-oriented programming – similar in concept to Python or Perl.</p> Ruby provides an interactive Ruby shell called “irb”. After you install Ruby, you can run irb from a command prompt. You can immediately do useful calculations like the following:</p> irb(main):001:0> a = 10 => 10 irb(main):002:0> b = 23 => 23 irb(main):003:0> a+b => 33 </code></pre> Hexadecimal numbers are prefixed with “0x” as in many other languages:</p> irb(main):005:0> c = 0x1000 => 4096 </code></pre> It is easy to convert from hex to decimal – just type the hex number, hit return and Ruby will give you the decimal equivalent as shown above. To convert from decimal to hex, I often just use the printf</em> statement. The syntax is similar to C, so it is very natural for embedded programmers.</p> irb(main):006:0> printf "0x%x", (310241024) 0x300000=> nil </code></pre> Ruby provides standard shift operators similar to those found in C. For example, if you want to know the hex value when bits 15 and 23 are set high:</p> irb(main):019:0> printf "0x%x", (1<<15)\|(1<<23) 0x808000=> nil </code></pre> It is often useful to know what bits are set in a particular number when determining if a bit in a status register is high, what address bits are used, etc. Ruby provides a bit reference method in the form.</p> fix[n] </code></pre> n is the nth bit in a binary number fix</em> where fix[0] is the least significant bit. Some examples:</p> irb(main):010:0> 2[0] => 0 irb(main):011:0> 2[1] => 1 irb(main):017:0> a = 310241024 => 3145728 irb(main):018:0> 31.downto(0) {\|n\| print a[n]} 00000000001100000000000000000000=> 31 </code></pre> For convenience, I wrote a simple module called bits.rb</em> that extends the basic Integer number class and provides a nice display of the bits in a number. To use this, create a file called bits.rb with the following contents and drop it in your Ruby site_ruby</em> directory.</p> M = 10241024 K = 1024 class Integer def bits print "3322222222221111111111n" print "10987654321098765432109876543210n" print "---\|---\|---\|---\|---\|---\|---\|---\|n" 31.downto(0) { \|x\| print self[x] } end end </code></pre> And to use this extension to see what address bits are used to decode 8MB of address space:</p> irb(main):020:0> require 'bits' => true irb(main):021:0> (8*M).bits 3322222222221111111111 10987654321098765432109876543210 ---\|---\|---\|---\|---\|---\|---\|---\| 00000000100000000000000000000000=> 31 </code></pre> From this example, we can see that address bits 0 through 23 are required to decode 8MB of address space.</p> ===========================================</p> Update 2009-07-30</strong></p> The following was posted as a comment to my old web site:</p> An even easier way of converting between bases in Ruby:</p> irb(main):001:0> puts 300.to_s(16) 12c => nil irb(main):002:0> puts "12c".to_i(16) 300 => nil irb(main):003:0> puts 255.to_s(2) 11111111 => nil </code></pre> Very nice!</p> Specialization or not 2005-11-15T00:00:00+00:00 I have often wondered about specialization. We can often find ourselves in ruts if we specialize too much. But on the other hand, specialization can lead efficiencies in some cases. In my work as a consultant, I often find myself in new situations where I have to learn fast. As one of my associates states – problem solving is the key skill. Kevin Barnes article Three theories on how to use developers efficiently</a> is a good read about the subject.</p> Total Phase USB Protocol Analyzer review 2005-11-03T00:00:00+00:00 I recently purchased a USB protocol analyzer from Total Phase</a>. The device will analyze full speed USB traffic and works well with Linux. On Linux, the software makes use of libusb, so there is very little setup required to get the device working – very nice. At a price of $400, there is really no reason not to have one if you are doing any type of USB development.</p> </p> Economics and Finding Outsourcing Talent 2005-11-03T00:00:00+00:00 Economics In One Lesson by Henry Hazlitt presents an interesting and logical view of economics. The book presents many ideas that seem right (like saving) and provides very logical arguments to back them up. Even though the book was originally published in 1946, the topics covered seem very applicable to today (The Blessings of Destruction, Public Works Mean Taxes, Taxes Discourage Production, Credit Diverts Production, The Curse of Machinery, Spread-the-Work Schemes, Disbanding Troops and Bureaucrats, The Fetish of Full Employment, Who’s “Protected” by Tariffs?, The Drive for Exports, “Parity” Prices, Saving the X Industry, How the Price System Works, “Stabilizing” Commodities, Government Price-Fixing, What Rent Control Does, Minimum Wage Laws, Do Unions Really Raise Wages?, “Enough to Buy Back the Product”, The Function of Profits, The Mirage of Inflation, The Assault on Saving)</p> The book can be found:</p> online: http://jim.com/econ/contents.html</a></li> download: http://www.fee.org/pdf/books/Economics_in_one_lesson.pdf</a></li> amazon.com</li> </ul> The following article presents some of the realities of finding talent in places like India. It is encouraging that the basic economic laws of supply and demand still work.</p> Finding coders on the subcontinent</a></h3>}

BEC Systems

🧱 The hardest things

🧹 zephyr-snmp v0.1.0 - a huge cleanup!

🚣 Rowing the race you need at the end

🛰️ Simple IoT now processes GPS data

🧊 Why small teams have to be good at borrowing

⚖️ Exercise: Build vs Borrow

📈 Do schedule estimates get more or less confident over time?

🚧 Gatekeepers are ineffective

✉️ Drafting emails in Markdown

🛣️ The two ditches

🚀 Accelerating or Decelerating?

🧩 From PCB change to mechanical model in one script

📥 Email is not a workflow

🔌 GitPLM: An easy way to manage your KiCad library using CSV files in Git

🧊 Web application architecture

🔩 Generating Mechanical Models with Claude Code

🪑 The missing leg of reliability

💻 Why the terminal still matters

📧 How to solve the email problem?

📝 You can't improve an email

🛠️ Tools or Processes?

🛩️ Why did the Wright Brothers succeed?

🔧 What has changed around edge build systems?

🔧 Why Go Works Well for Building CLIs

🔌 API + CLI + Skills = the Future of AI Agents

❓ Better questions get better answers

🏞️ Communicating Status Without Demanding Attention

🤝 How to manage up

👥 The release is a language everyone understands

📦 What is a release used for?

⚡ The essence of shipping faster

🔀 What is the difference between MVP and Release First?

✅ Each release is 100% done

💡 A Simple Example

🏷️ The simple approach to versioning

📦 The bare minimum to call something a release

❓ Release Questions

🤔 Two types of unknowns

🧊 The hardest thing to manage

🎉 The BIG 1.0 Release

❓ What is the difference between "releasing" and "shipping"?

🚨 Problems with not Releasing First

🌅 Why optimizing the user release experience early is so important

🔵 The Big Blue Button

🧠 The Human Factors

☝️ What do we mean by "First"?

🟢 What is in the Big Green Button?

🟢 The Big Green Button

🔄 The release circle

🧩 Why things fall apart at the end

🪞 The illusion of project meta-data

🚢 What is the #1 problem in product development?

📦 More definitions of a release

🛠️ Do things that don't scale

🔮 Can we predict product development schedules?

🚀 How to make releases painless

📖 When should you write the user manual?

🎬 Is a Demo the same as a Release?

⏱️ How do you find time for higher value tasks?

⚖️ Reliability tradeoffs

⚖️ Reliability, Cost, and Value

🔗 What defines reliability?

🚀 Shipping faster - what are the stakes?

🔍 It is often more work to review something than to do it

🚀 How far do you take releases?

🎯 Standards for internal vs external releases

📦 What do we release?

🤝 A release is what unifies Engineering and Business

📦 What is a release?

🚀 Shipping faster leads to predictability

🔗 Platform thinking does not mean we don't use other platforms

🔍 AI coding requires better inputs and robust verification

✨ Impressive is not always effective

🥄 The beauty of human creativity

📖 The definition of the word 'Platform'

🔄 Platform Thinking is a habit

🎯 Who is Platform Thinking for?

⏳ The long-term compression timeline

🗜️ Long-term compression