The Ability to Improve

(while this article contains many general ideas, it is written from the perspective of product development.)

If you don’t continually improve, you soon lose the ability to do so …

This thought came after an associate described a company that is struggling to manufacture one of their products:

They have been making a product since way back in the last century. EOL (end-of-life) is catching up with them. The circuit is really hairy analog stuff with ridiculously high precision. Pots everywhere to get it dialed in. But their voltage regulator is drifting in value significantly, whereas the old parts are rock steady.

How do we get into situations like this, and how can they be avoided?

A few more examples

Below are a few more examples of situations that are very difficult to deal with in the lifecycle of a product:

1. A part becomes obsolete in a printed circuit design, and no one knows how to use the CAD system it was designed in. Rather than updating to an open/modern CAD system years ago at a relaxed pace, exorbitant efforts are now spent trying to track down a few more parts or find a replacement. When this fails, it is now a frantic crash-and-burn project to revise the design with no time for testing and verification.
2. A serious security problem is found in a product, but the software build system has fallen into disrepair and no one knows how to set it up, and it will not run on current operating systems.
3. A new Linux kernel driver is needed to support a new USB peripheral, but the kernel we are shipping is 7 years old, and the SOC/SOM vendor has not provided any updates. Updating to the latest kernel would take months. Backporting the new driver is nearly impossible due to the many changes in the kernel and its subsystems.
4. Manufacturing was outsourced to another country years ago. Then COVID happened and the manufacturer shut down for an extended time. No one knows where the design files are or how to build the product to move it to another manufacturer.
5. The manufacturing process for a complex product has been the same for years. No one knows why things are done the way they are, but are afraid to change anything for fear that problems may crop up.
6. A code base has not been refactored as it grew and no tests were written. Adding new functionality is very difficult and usually causes bugs in unrelated areas, requiring an exorbitant amount of manual testing with each release.
7. An application does not have a cohesive architecture. Global variables and unrestrained function calls between various subsystems are common. As time goes on, the application becomes less and less stable. Fixing one bug often causes another. All developer resources are consumed in a circle of support, continuous bug fixing, and damage control.
8. A new customer is interested in purchasing a product, but it is missing a critical feature. The original development was contracted to an outside development firm, and there is no knowledge in-house on how to modify the software. The managers at the company are scrambling to find someone who can figure out how to make the changes needed.

Technical debt is another term to describe systems that are not adequately maintained — eventually, it takes so much effort just to keep things going that no time is left for improvements.

Maintaining Systems

How do you maintain the ability to react to changes, security problems, part shortages, and feature requests? How can you add value to your products over time? Maintaining systems (both hardware and software) is what keeps them malleable and flexible. If you don’t maintain stuff, then it soon becomes brittle and harder and harder to change. Modern systems are not ship-it-and-you-‘re-done, but rather more similar to a living organism. If it is not maintained, it is dying.

The only way to make sure you can maintain something is to maintain it. Make it better. Replace a part with a cheaper/better one. Improve something. Add a feature. Make it faster. Streamline and simplify manufacturing. If we don’t continually improve, then it becomes difficult to do anything without starting over. It is much easier to absorb a series of small costs over time than a large expense all at once. Small changes over time allow us to improve our processes. If we are not under pressure and something goes wrong, then we simply revert to the previous version or write tests to keep it from happening again.

This does not have to be hard, but it does require an investment in methodology and workflow. It requires a commitment to open tools, automation, and testing. It requires process — not the type that results in stifling bureaucracy but rather improves efficiency through automation and transparency.

There are several concerns related to change including return on investment (ROI), quality, and efficiency.

The ROI Objection

There are two financial mindsets:

1. View R&D as a one-time capital expense where you design something, lock it down, and then try to extract as much value as possible — the “minimizing costs” mindset.
2. View R&D as an ongoing operating expense, improve things over time, and continually increase value — the “maximizing value” mindset.

The first is easier to measure and forecast. Financial people like and understand this model. The second requires intuition and vision. It involves some risk. As your processes become more efficient, the value you get from an improvement begins to outweigh the costs of making the change. Over time, these improvements have a compounding effect.

It depends if you are playing the short or long-term game …

The Quality Objection

Stability and Quality is always a concern with any change in a system. With changes, mistakes will be made — especially early on. When this happens, there is an opportunity to learn and proactively improve the testing and verification process for next time, or reactively place blame, lock things down even harder, and remove responsibility. When someone introduces a bug, the question should be: “Did you create a test so it does not happen again?” If no mistakes are being made, not much is likely being done. The upside of growth and improvement should far exceed the costs associated with the occasional mistakes made in the course of changes. Done right, quality should follow the same value curve as the value shown in the graph above. The key is to not get discouraged by an initial drop in quality and focus on improving the processes, not minimizing or restricting change and responsibility.

The Efficiency Objection

Efficiency is always a concern, and again there are two schools of thought:

1. It is more efficient if we just do things the way we’ve always done them, not risk change, and get what we can out of our investment. Change brings risk.
2. Efficiency can always improve, bringing lower costs and more value.

Again, the key is to be patient with the initial declines and persevere until you get to the point where efficiency is increasing and the cost and value lines cross.

Case Studies

Tesla is an interesting case study on continuous improvement. From a design and manufacturing perspective, their accomplishments are notable. There is always a better way to design or manufacture something.

Google’s blameless postmortem practice enables proactive improvements for the future instead of reactively placing blame for the past.

Amazon’s “You build it, you run it” philosophy allows people who know the most about the system to improve how it is run instead of going through endless rounds with gatekeepers who don’t understand the system.

Toyota’s implementation of “kaizen” has produced excellent results. Kaizen is a concept referring to business activities that continuously improve all functions and involve all employees from the CEO to the assembly line workers.¹

These practices are used at large successful companies, but in concept, they are simple and can be applied at any scale.

The Irony of Kaizen

We can develop products with the thought — we are going to get this perfect and then go into production. Logically it would seem that if we expect to make improvements after it is in production, then we must be starting with an inferior product with flaws. However, Toyota has proved the opposite. Even when a model is first introduced, their quality is likely better than the average in the automobile industry. Kaizen does not mean we slop something together of inferior quality, but rather we do the best we can but have the humility to realize it can always be better. Kaizen is not limited to the assembly line, but all aspects including design, prototyping, quality control, testing, etc. If all of these processes are continually getting better, then the first item produced will also be better.

Requirements for Improvement

Every person and every organization is different, so there is no one-size-fits-all recipe that will always work. But there are several ingredients that are generally beneficial including:

1. Metrics
2. Testing
3. Openness
4. Simplicity
5. Leveraging outside improvements
6. Iterate faster
7. Owning your platform
8. The right people

Metrics

Not all change is an improvement — change for the sake of change is just churn and generally negative. You have to be able to measure something to know if things are improving or not. Improvement is very difficult without some metrics, because we are then just guessing and going off our intuition, and human intuition is often distorted by emotion, social pressures, and a host of other distractions.

Testing

Testing is trendy today and many talk about it, but fewer individuals and teams have the discipline to implement testing consistently. If you are not confident enough to deploy new software to production after your tests pass without doing some manual checking, then you possibly need better tests. Automated end-to-end and performance testing is critical. Consider the following statement about SQLite:

The reliability and robustness of SQLite is achieved in part by thorough and careful testing.

As of version 3.42.0 (2023-05-16), the SQLite library consists of approximately 155.8 KSLOC of C code. (KSLOC means thousands of “Source Lines Of Code” or, in other words, lines of code excluding blank lines and comments.) By comparison, the project has 590 times as much test code and test scripts – 92053.1 KSLOC.

https://www.sqlite.org/testing.html

Most of SQLite’s tests are not open-source. This is the real IP of SQLite.

Openness

Openness is another necessary ingredient for improvement. This takes many forms:

• Open access to tools, design files, and source code. Open-source software and tools are preferred when available.
• Good documentation is available to all involved.
• Work being done is in the open and can be reviewed by anyone at any time. Pull requests are created when work starts to provide visibility into work being done rather than when the work is finished. Transparency is a powerful ingredient for quality.
• Metrics, including financial information and customer feedback, are accessible to the team.
• Anyone on the team to allowed and encouraged to fix or improve anything. There are no “turf battles” where only certain people are allowed to do certain things. Within practical limits, anyone can do any work, but the process (transparency and testing) ensures that substandard work will not be automatically used.

The above does not mean everything is open-source to the world, but rather those involved with the product — a team, organization, company, customers, partners, etc. We can learn a lot from the success of open-source projects and their associated culture and methodologies.

Simplicity

As the complexity of systems continues to increase, we need to also be aware that humans interacting with these systems are finite — we can only handle so much complexity. Thus, efforts need to be made to simplify where possible. We often hear the phrase: “keep it simple!” But, from whose perspective? Design, Manufacturing, Users, Service, Ops, Support, Sales, Financial, CEO, etc. If we look at how much time these various groups interact with the product, we can arrange them in an inverted pyramid:

The CEO is only one person and hopefully, there are many Users, so if we can simplify things for the Users, that will probably provide more benefits to the product than simplifying things for the CEO. Likewise, if Design can spend a little more time making Manufacturing and Support simpler, that is probably a good investment.

Simplifying a system’s architecture often provides the biggest wins. This can take the form of common data structures, how data is structured, how data is communicated and synchronized, lifecycle management, etc. A simplification at this level has the potential to improve the entire system.

Faster Iterations

The most efficient way for humans to design complex systems is through iteration. Some call this failing fast. The faster you can design/develop, test, get feedback, and repeat, the more likely you will be successful.

There is certainly a place for some up-front planning, but you don’t want to get stuck there. Build a minimal slice of your product as soon as possible, even if it only has a few features. This can even be done with hardware — put down only the bare minimum parts on the PCB and build something. The rest can be added later. If it takes too long to build a PCB prototype, find ways to speed it up, or switch to another vendor. Automation and process is very important for fast iterations. CI (continuous integration) tells you quickly if your code fails tests. CD (continuous delivery) continually builds and deploys your work so that it can be tested in the complete system and evaluated by others. CAD and PLM automation allows you to quickly package, track, and build new versions of PCBs.

A focus on iteration compensates for our many human weaknesses. It reveals our blind spots. It improves teamwork and project transparency. It helps avoid the scenario where the lone genius works for three months behind closed doors while the rest of the team is largely stalled. It ensures we are building what the customer actually needs. It encourages automation to avoid the tedium of manual testing and releases. It accelerates learning. Above all, it improves the efficiency of the entire process. It ensures we can quickly make changes and release them when needed, with confidence.

Owning Your Platform

Owning your platform means you “own” and take responsibility for key parts of whatever it takes to build and maintain your products. It might be a printed circuit design, a build system, an application architecture, automation, infrastructure, process, etc. Owning your platform likewise is an investment for the future. It gives you more control and the ability to improve things. You are not locked into what another company provides at one snapshot in time. Certainly, there are limits to this and we cannot create an entire system from scratch so we all must use software and components that other companies create. However, we can build a system that has flexibility so that we can easily swap out one component for another. We can understand it well enough to test it. We can manage the build systems so that if something needs to be fixed or updated, we can do that. We build our device software when possible instead of relying on binary images that we can’t control. We are not afraid of developing our own automation and process software instead of relying entirely on outside vendors. Perhaps the most important thing is to own the integration points. This might include:

• Top-level PCB (printed circuit board) designs (we can change electronic components and submodules as needed)
• Software build systems (we can add, remove, and update software components)
• PLM (project lifecycle management) (we write our own or use a system that can be easily extended through programming to meet our needs)
• Core application architecture (can be adapted to specifically meet our needs)
• Hosting (we can run our own servers or move applications between hosting providers as needs change)
• CI/CD (can be adapted to any task/technology)

Integration is the most difficult (but also the most valuable) part of product development, so it is tempting to outsource integration. But this often locks you into one vendor and does not give you the flexibility to scale to higher volumes or additional products, as this running commentary illustrates.

Ideally, you can reuse parts of your platform across many products, thus compounding your efforts over time. Even if your design files and source code are open-source, owning your platform (which includes so much more than design files) is what gives you the ability to efficiently design, test, maintain, and produce something. This is the hard part. If you are moving fast, a static snapshot of your product design at any point in time is of marginal value to competitors.

Leveraging Outside Improvements

Leveraging outside improvements is another important activity. This can include upgrading to a new open-source software component, designing in a new integrated circuit, etc. It is easy to get stuck in the mindset that the “old” was “better.” In some aspects of life, this may be true, but with technology, things are generally getting better. Open-source software is generally more stable as time goes on due to better testing. Newer integrated circuits are more capable and lower cost. Many products today are largely an integration of many outside technologies, many of which are now open-source. To improve, we must continually track and integrate these improvements. Developing using the mainline (upstream) branch of open-source projects is a good strategy that companies are increasingly adopting. Knowing when to adopt new technology is a key skill.

If you use an outside development firm to develop parts of your product, it is critical that you receive a deliverable that is maintainable. Several suggestions for ensuring this include:

1. run the project like an open-source project. All activity is transparent and is committed to Git often.
2. own the integration points and testing.
3. use small iterations when possible. This ensures you are engaged and what is being built is what you need.
4. use open tools/technology that everyone has easy access to where possible. Ensure that you can build or generate usable outputs from the design/source files. Even if you don’t have the expertise to use the tools, there is a better chance you can find someone else who can.

The Right People

There is a saying often used by hikers: “Leave it better than you found it.” If we are working on a section of code that does not have tests, do we write them? If a manufacturing problem is found, do we improve the process to try to prevent the problem in the future, or just write it off? Do we give more responsibility and opportunity to everyone involved to improve things, or do we let a few mistakes restrict responsibility to only a few people?

For this to all work, you have to have people with integrity — who are genuinely interested in improvement and not just gaming the metrics. You have to have people with the discipline to write tests, which goes above and beyond just getting the job done and the ticket closed.

This all applies personally too … do you want to improve how you use your time? Then measure how you spend your time and share these measurements with someone. Use one of the many tools available to help track your time. Create your own tooling and process, and then improve it. Look for better ways to do all this.

Fundamentally, people don’t usually change much, so if someone is not interested in improving themselves, they likely will not be interested in improving the organizations they are a part of and be a drag on the process. So perhaps the most fundamental thing to learn in an interview is not only past accomplishments, but also if the candidate still has that burning desire to learn and improve themselves. It is a frustrating situation when a person who is wired for improvement is part of an organization that is not, or vice-versa. Both types of people and organizations exist.

Small Changes vs Large Changes

We can make small or large changes. This might be the difference between an improvement vs. a pivot, slow and steady vs drama. The drama captures attention in the news — the new charismatic CEO brought in from the outside to shake things up. You typically don’t see great companies in the news, but you do see them in investment portfolios. In his book “Good to Great,” Jim Collins studied several companies that produced exceptional results and concluded the following:

We keep looking for change in the wrong places, asking the wrong questions, and making the wrong assumptions. There’s even a tendency to blame Wall Street for the “instant results” approach to change. But the companies that made the jump from good to great did so using Wall Street’s own tough metric of success: a sustained leap in their stock-market performance. Wall Street turns out to be just another myth—an excuse for not doing what really works. The data doesn’t lie.

Now picture a huge, heavy flywheel. It’s a massive, metal disk mounted horizontally on an axle. It’s about 100 feet in diameter, 10 feet thick, and it weighs about 25 tons. That flywheel is your company. Your job is to get that flywheel to move as fast as possible, because momentum—mass times velocity—is what will generate superior economic results over time.

Right now, the flywheel is at a standstill. To get it moving, you make a tremendous effort. You push with all your might, and finally you get the flywheel to inch forward. After two or three days of sustained effort, you get the flywheel to complete one entire turn. You keep pushing, and the flywheel begins to move a bit faster. It takes a lot of work, but at last the flywheel makes a second rotation. You keep pushing steadily. It makes three turns, four turns, five, six. With each turn, it moves faster, and then—at some point, you can’’t say exactly when—you break through. The momentum of the heavy wheel kicks in your favor. It spins faster and faster, with its own weight propelling it. You aren’t pushing any harder, but the flywheel is accelerating, its momentum building, its speed increasing.  

This is the Flywheel Effect. It’s what it feels like when you’re inside a company that makes the transition from good to great. Take Kroger, for example. How do you get a company with more than 50,000 people to embrace a new strategy that will eventually change every aspect of every grocery store? You don’t. At least not with one big change program.  

Instead, you put your shoulder to the flywheel. That’s what Jim Herring, the leader who initiated the transformation of Kroger, told us. He stayed away from change programs and motivational stunts. He and his team began turning the flywheel gradually, consistently—building tangible evidence that their plans made sense and would deliver results.  

“We presented what we were doing in such a way that people saw our accomplishments,”Herring says. “We tried to bring our plans to successful conclusions step by step, so that the mass of people would gain confidence from the successes, not just the words.”  

Think about it for one minute. Why do most overhyped change programs ultimately fail? Because they lack accountability, they fail to achieve credibility, and they have no authenticity. It’s the opposite of the Flywheel Effect; it’s the Doom Loop.  

Companies that fall into the Doom Loop genuinely want to effect change—but they lack the quiet discipline that produces the Flywheel Effect. Instead, they launch change programs with huge fanfare, hoping to “enlist the troops.” They start down one path, only to change direction. After years of lurching back and forth, these companies discover that they’ve failed to build any sustained momentum. Instead of turning the flywheel, they’ve fallen into a Doom Loop: Disappointing results lead to reaction without understanding, which leads to a new direction—a new leader, a new program—which leads to no momentum, which leads to disappointing results. It’s a steady, downward spiral. Those who have experienced a Doom Loop know how it drains the spirit right out of a company.  

https://www.jimcollins.com/article_topics/articles/good-to-great.html

Change is easiest to implement and absorb if it is incremental and not a drastic overhaul. Incremental change allows you to measure, adjust, and backtrack if needed. Incremental improvements can be implemented by anyone without permission and can then be evaluated and improved by others.

It is also important to understand the difference between foundational truths/values (things you should not quickly change) and inefficiencies that and ripe for improvement. The foundational principles of your business provide stability and need to be understood and valued.

Summary

Do you want to be able to respond quickly to unexpected opportunities or issues that come up? The solution is simple — practice doing so now. If you regularly practice on the little things that by themselves may not really be worth the time to do, several things will happen:

1. you will be ready and practiced to make critical changes that will make a big difference such as fixing a critical security problem, responding to supply chain shortages, or making a change necessary to land a big sale.
2. the compounding effect of small improvements will eventually provide a sizable return in the future.
3. each small improvement is a small push on the flywheel of momentum. This adds positive energy to your organization and this momentum helps carry you through difficult times.

A culture of improvement is a long-term game. There will be some up-front pain as you develop your processes, but this is much easier than the long-term pain of accumulated technical debt which cripples your ability to do anything. You can compound the value from improvements or the cost of technical debt — take your choice.

1. https://en.wikipedia.org/wiki/Kaizen ↩︎