Go for IIoT Systems

As developers, we have many options for programming languages. On one hand, it is great to have choices. On the other hand, it can be a little overwhelming. Many times in life, we need to make decisions before we have the experience to know the best answer. There are many opinions as to what is best, and this article is just another one. There is no one-size-fits-all — a lot depends on your team and the task. In this article, we will examine the challenges we face in developing industrial IoT (IIoT) systems, review desirable attributes in a programming language, and discuss how Go meets these needs.

Peter Bourgon defines the industrial programming context as:

in a startup or corporate environment;
within a team where engineers come and go;
on code that outlives any single engineer; and
serving highly mutable business requirements.

Additionally, an industrial IoT system can be thought of as a system that gives users access and control of remote systems. There are many variations of this, but let’s focus on a simple case that is typically composed of the following pieces: Edge device (remote equipment, sensors, etc), Cloud, and Browser.

All of these pieces must be dealt with at some level and will require some programming. The fundamental requirement of this system is networking — data must be moved between the above three instances. IIoT systems are inherently distributed systems. Distributed systems are hard, thus the distributed problem needs to be solved well if you want to have a scalable, reliable IIoT system. It is natural to focus on the end application at the edge and add IIoT as an afterthought. However, this rarely works very well.

The context of this article is that of an average programmer working in small teams where engineers do a lot of different things. Many of us are not full-time programmers doing deep theoretical work, but find ourselves building these complex systems. Some of the challenges/requirements we face include:

resources are constrained at the edge (CPU, Disk space, etc)
networks can be unreliable
config/state needs to exist in multiple places (edge & cloud)
software updates need to be deployed to remote places (edge & cloud)
end-to-end (edge <-> browser) real-time response is often desired
many parallel operations are happening (collecting data, running rules, managing the system, etc)
physical access to edge systems so debugging problems can be difficult
edge systems typically use Embedded Linux, which can be a challenge to develop and maintain
IoT systems are inherently concurrent and distributed
many of us are doing this part-time, are not computer science gurus, and need pragmatic solutions.

Attributes of Go

Go has several attributes that make it a good fit for this problem:

excellent reliability. I’ve written about this before. (our systems are not crashing in the field)
it is a compiled language (reasonably fast and small)
runtime is small, embedded, and efficient (this means there is very little overhead to distributing the entire runtime with each binary. This makes deployment easy.)
statically linked (this makes it much easier to deploy updates as everything you need is embedded in one binary file)
easy to embed assets (again, this makes it easy to deploy in one file without containers)
is garbage collected, so we don’t have to worry about managing memory (we don’t have memory leaks, invalid pointers, and other problems that can cause field failures)
typed language (compiler catches more problems early instead of having crashes in the field)
popular and well-supported (will continue to improve and be around for a long time)
excellent concurrency support (a lot is going on in these systems, so this helps)
popular in cloud and distributed systems (there are plenty of stdlib features and 3rd party packages — metrics, monitoring, logging, communication, etc to help us build an IoT system)
language is simple (easy to learn, read, and maintain)
tooling is excellent, compile is blazing fast, no Makefiles, etc, built-in support for profiling, and easy to cross-compile for other systems (we spend less time messing around and more time deploying value to our customers)

Go is not the best solution in every case, but if your situation matches what is described above, it might be a good fit.

Go removes a lot of complexity

Cloud and edge systems typically run Linux. You may outsource parts of the cloud portion and perhaps even parts of the edge device, but you are generally responsible for the edge device. It must run 24/7. You likely need to deploy updates, maintain security, and add functionality over the product lifecycle to meet customer needs. You will run into issues and need to be able to debug them. You can’t walk over to an edge or cloud system and press the reset button or hook up your JTAG debugger. Most embedded software in the past has been developed for MCUs (microcontrollers) and written in C/C++. This is a fine solution for disconnected control-focused products, but a Linux MPU (microprocessor) solution provides you with more flexibility and functionality for data-focused connected devices that process/store a lot of data and have complex user interfaces. Linux provides unparalleled support for a large number of devices, USB and networking functionality is reliable and mature, and it scales to processing and storing large amounts of data. In an MCU platform, you may write most of the software in the system. In some cases, you may use an RTOS (Real-time operating system), but even these are relatively small as resources are limited. In a Linux system, you are running millions of lines of code you did not write. We can use the iceberg analogy — the part underwater includes Linux and system libraries which you don’t see most of the time. The part you see above water is your application, which is a tiny portion of the total software in the system.

Part of the value of modern systems is the huge number of software components you can reuse in your system as there are software libraries/packages for many tasks. Building modern systems is partly the creative combination of existing/reusable technology and the creation of new technology (the code you write).

So the question arises: How am I going to get up to speed on all this? One solution is to minimize the amount you need to interact with the stuff underwater. If you build applications with C++, the compiler does not give you a lot of help in linking to libraries and cross-compiling to edge system architectures. (Cross compiling is compiling native code on one architecture for another — typically developing on an x86 workstation and then running on an ARM Linux target system). Thus, you need to rely on complex build systems (CMake, Yocto, Buildroot, etc). Every time you add/update a library/package, you are faced with the following questions:

How do I integrate this library into my app build?
How do I cross-compile to my edge system?

It’s not easy, thus projects such as Yocto and Buildroot exist. All developers need to have a cross tool-chain (compiler and libraries) installed on their development computer that matches the libraries in the edge system. Deploying an application update often means deploying additional libraries, or library updates. So we add container technology to try to manage all this, which is another layer of complexity.

Some solutions (like Raspbian for the Raspberry PI) advocate developing directly on the target device to avoid the complexity of cross-compiling. This works fine for small one-off maker projects, but for the development of large industrial programs, it is a poor solution as it is like developing on a PC from the 90s. Sure, cross-compiling is easier, but everything else is much harder.

Python or Node.js may look attractive in that they are easier to program than C++ and the program is interpreted on the target device, but many of the underlying packages for these languages depend on C++ code, as interpreted languages are slow, as a result the build problem is worse because it is even more difficult to cross-compile this code in the Python and Node.js build systems. Additionally, a separate run-time is required on the edge system which is relatively large and must be kept in sync with the application.

Go sidesteps most of this because it is a mostly pure ecosystem. There are no host library dependencies in a pure Go application — everything it needs is embedded in a single binary file (typically 2-10MB compressed). The Go standard library is rich and has many needed functions. Additionally, there is a rich selection of pure Go 3rd party packages that cover most other needs. Adding packages to a Go application is as simple as go get some/package/name. There are no development tool requirements other than a standard Go installation and an editor. No complex build tools are required to add 3rd party packages or to cross-compile to different target systems (macOS, Windows, Linux). It has built-in support for profiling. If it crashes, you always get a stack trace. It’s so easy, and the learning curve is small.

Many software tools today add layer upon layer to solve the problems with the underlying technology. While this helps in some cases, it often adds additional complexity. You can’t gloss over complexity — when things go wrong, you still need to understand the entire stack anyway. Abstraction has its place but can only take you so far. If there are problems in one technology layer, adding additional layers sometimes just makes it worse. Go threw everything out and started over. Simplicity is the only sustainable approach to technology.

Go is a relatively new language (released in 2009), so many people have not heard of it. Learning a new language does take some time. Go has a small learning curve, but that is nothing compared to the learning curve of building, deploying, and maintaining C++ code in edge systems. Go allows you to quickly become productive and spend very little time fighting tools over the product lifecycle. Moving your Go application forward over time often does not require any changes to the base Linux system because they are decoupled.

Go is familiar and simple

Google originally developed Go as a solution to the pain they were experiencing with C++, and they are still investing in it. Go syntax is similar to C and was developed by some of the same people who were involved in the development of the C language and Unix back in the 1970s. Thus, it feels very familiar to those of us who have spent time developing in C, Java, C#, etc. Go eschews the complexity that has overtaken C++. Language features are added slowly and carefully. Compile errors are generally easy to understand and debug. Backward compatibility has been excellent. Since 2014, I’ve implemented several systems in Go and have not had any significant issues after updating the Go or 3rd party package versions. Go developers take API compatibility seriously, so this helps.

Go is a simple language but does not force you to do things correctly (as a language like Rust does). You can still make a concurrency mess with channels. Go has nil values, so you can still crash your program if you try to reference a nil. If you don’t check returned error values, you’ll probably run into problems. But, if you follow a few simple best practices and idioms, it works well and is a nice experience. Simple programming languages are much easier to read and maintain.

Go Concurrency Support

IIoT systems have a lot going on, so you need good support for concurrency. Interpreted languages like Python and Node.js are weak in these areas. Go has excellent support for concurrency by providing channels and the go keyword as language features. This allows for running concurrent tasks and communication between them with very little overhead. Go channels are easy to misuse, so it requires a little effort and practice to learn how to use them correctly, but once understood, it is a very simple and reliable way to manage concurrency in a system.

Go is an Internet language

While older languages like C/C++ and Python have been excellent tools for solving programming problems in the past (and are still the best in some domains like MCU development, machine learning, etc), the problems we are solving in IIoT systems are different today in several ways:

we are dependent on large ecosystems of 3rd party packages to implement all the functionality we need
we need better tools for concurrency and distributed systems
we need simpler models for network programming
we need software that is easier to build and deploy

Communication between systems is the fundamental problem IoT systems solve and Go is good at this. Go is the best-in-class for a new breed of software: an Internet programming language. As evidence of this, most cloud and distributed software infrastructure is written in Go these days (Kubernetes, Docker, NATS, etc). Cloud companies that scale to thousands of systems are very sensitive to cost, thus they need efficient platforms. This efficiency in Go also benefits us at the edge where resources are constrained. An added benefit is that you can write your edge and cloud applications in the same programming language.

But Go has no GUI support …

Although there have been many excellent efforts to develop a native GUI library for Go, none of them have emerged as the standard. There is a good reason for this … the browser is the standard UI platform these days. Nothing else can come close to touching its capabilities. It is cross-platform, reasonably performant, and very well-supported. The browser has so many options for programming, packages, etc. The browser has built-in support for video and other forms of media and is highly optimized. Developing applications in the browser is more difficult to get started than in a C++ GUI toolkit. But there is a much richer ecosystem of packages and functionality available in the browser platform. The browser platform is stable and you are ensured of backward compatibility — it will be around and well-supported for a long time. The web platform is built on solid engineering principles that have proven to scale. Many UI technologies designed to replace the browser programming model have come and gone: Java, Macromedia Flash, and Microsoft .net — yet the standard browser programming model of HTML5 + Javascript has displaced all of these and is growing at an astounding rate. Google Flutter is the latest attempt to displace the web platform, but it remains to be seen yet if it will be successful. Most C++ GUI toolkits are highly dependent on a single company that may be here today and gone tomorrow (look at the history of Qt for an example), or licensing requirements may change at any time — there are a lot of unknowns. Plus, with the web platform, you can view the UI remotely just as well as locally. You can also use the same UI building blocks in the cloud as in the edge device. If designed correctly, web applications also function very well on mobile devices, eliminating the need to develop separate mobile apps. With the web platform you can address all of the following:

on-device LCD UI (runs a browser full screen)
remote access to the device using a browser
cloud application UI
mobile device access (browser, PWA, Cordova, etc)

The experience of a native phone app might be slightly better than a web application on mobile devices, but how many of us can afford to develop 4 different UI applications for a product (native, web, Android, iOS) when one will do in most cases? The question is not what is possible or technically best, but what is practical with the limited resources we have.

Languages like Elm simplify front-end (web) development in many of the same ways that Go has simplified back-end development.

But Go is not the best at …

Very true — Go is not as easy as Python, as widespread as C++, as popular as Javascript, as safe as Rust, or as good at concurrency as Elixir. But, Go is almost as good as the best at everything and its tooling is much better than any other language ecosystem — especially for distributed applications that need to be deployed to remote and different architectures. It does not have any of the significant drawbacks other languages have. And especially for networked IoT systems, it is an excellent choice. The below decision matrix likely reflects my biases, but illustrates this point (green is best, red is worst):

But what if I need to use C++ …

There are certainly cases where you may need to use C++ or some other language. You may choose to implement a native UI using Qt or LVGL. Machine learning code is often written in C++. There is no reason your system needs to be written in one language — Go can be used in combination with other languages. There are several ways this can be implemented:

call the C++ code directly from Go (CGO)
run the C++ and Go code as separate applications and communicate between them using a message bus

Method #2 is preferred in many cases as it keeps your Go code clean, and the C++ code can be built using standard C++ build systems instead of trying to force the Go build system to build C++. Simple IoT is one framework that is designed for exactly this scenario:

This architecture keeps your Go code easy to build/develop/deploy. Your core logic, which manages config/state and communicates with other systems, can be written in Go. This core logic must be reliable and easy to maintain. Specialty code like machine learning models can be dedicated C++ applications. NATS is a modern message bus written in Go that is easy to embed in any Go program and is a great way to connect different applications.

Optimize for where you will spend most of your time

Linux-based IoT edge systems have plenty of memory, storage, and processing power. Thus, features are continually added over the product lifecycle, which can be 5-10 years in an industrial system. The time spent maintaining and improving a product often swamps the initial development. Therefore, it makes sense to optimize for the long term — adding features and deploying updates over the next ten years. Go makes it easy to build and deploy updates with the latest Go and package versions.

Friction

Developer productivity is difficult to fully understand. We often think of the features we need in our languages/tools. Commercial software vendors focus on features. Development methodologies are designed to increase productivity by better organizing what needs to be done. But another aspect that is easy to overlook is the amount of time we spend messing around working on stuff that does not directly add value to our product. I call this “friction.” This friction can manifest itself in unforeseen problems that are more common in complex systems: learning curves, build issues, issues with libraries/packages, lack of flexibility, security problems, deployment problems, bugs, race conditions, stability problems, technical debt, lack of control over parts of the system, etc. Go does an excellent job of reducing friction in the development process. You focus on what needs to be done, and the rest is mostly invisible.

Summary

In this article, we explored many of the attributes of Go that make it ideal for implementing IoT systems. Go provides a modern programming ecosystem where the build tools are first class, the language is simple, and there is little friction to get things done. Go also provides features such as garbage collection and concurrency support in the language which are very helpful in IoT systems where data is distributed and a lot is going on. Go is an Internet language that is increasingly used to develop distributed systems, which includes IoT systems. It is a stable (almost boring in many respects) technology and is a great option for developers who are familiar with C/C++ and want to expand into Embedded Linux/IoT — the skills you have are very applicable in Go.

What is the main constraint in your projects?

Application performance?
Getting things done and shipped?

If #1, then perhaps C++ or Rust is a good choice. If #2, then take a look at Go.

Go is not the best solution for every team/project, but for the scenario described in this article, it has worked incredibly well. With Go, we can develop and deploy remote systems with confidence.