Recently, we brought up a new PCB (printed circuit board) a customer had designed based on the Microchip ATSAMA5D27C-D1G processor. We encountered several interesting challenges.
The first issue was that the design only had serial ports (and possibly a USB Host port that could be used in device mode) for bootstrapping the system. Microchip provides a tool named SAM-BA to do this. There are two implementations of SAM-BA. The older implementation does not support this CPU. The newer implementation does support this CPU, but the documentation and examples are sparse (at least when we were using it). The standard way to use SAM-BA is to directly program the bootstrap and bootloader into SPI flash. However, this design only has eMMC flash. So we came up with a way to bootstrap at91bootstrap and u-boot over the serial port, and then load a Linux kernel with an embedded initramfs from a USB flash disk. This initramfs contains an updater that is used to program the eMMC flash. After the updater runs the first time, the system will then boot from eMMC. We developed and verified this process on a development board, and it is documented in a Github issue (The SAM-BA team has since disabled Github issues on this repository), and as part of the Yoe Distro sama5d27 support. Microchip has also since provided us with a procedure to program eMMC using SAM-BA, but the procedure is complex and we never had a need for it once we figured out the above method.
After receiving 10 prototypes of the new board, we learned pretty quickly that VCCBU must have voltage for the CPU to start. After that, the first board loaded u-boot and everything looked good, but things went south from there. U-boot would only load perhaps one out of twenty times. Basic checks of power supply voltage levels all looked good. The custom board used a 12MHz crystal for the main clock, where Microchip’s SOM1 module uses a 24MHz crystal. We decided to try a 24MHz crystal, and after that change, u-boot would reliably load over SAM-BA. We’re not sure why (12MHz is slower than 24MHz, so things should run better), but initializing a clock tree is complex and there is likely something that was not configured correctly in software. Lesson #1: Unless you want to spend a lot of time writing chip initialization software and debugging strange problems, follow a reference design as closely as possible. Working at the level of chip initialization often requires intimate knowledge of the MPU and likely the support of those who designed the chip. This is much more effort than most teams have time for and is best left to the MPU vendor.
Now that u-boot was loading reliably, we could turn our attention to the kernel. At this point, a couple boards would start the kernel, but most failed with no kernel messages after the “Starting kernel …” message on the serial console. The boards that would start the kernel would crash partway through the boot process. We started looking at power supplies and observed a 100mV sawtooth on the core power supply.
The SAMA5D27 datasheet specifies a maximum of 15mV of ripple for the core supply.
The ADP5023 power supply used for the core voltage had a mode pin that can be used to force it into PWM-only mode, instead of PWS/PWM mode. After forcing the supply into PWM-only mode, some boards would start Linux. However, there were still stability problems later in the boot cycle.
After trying many other things, we finally raised the voltage of the core supply slightly by changing one of the feedback resistors. Presto! Lesson #2: If a CPU has stability problems and everything else looks OK, trying raising the core voltage. The system was now stable. After communicating with Microchip, we learned that the CPU actually required a higher voltage than the documentation specified. Microchip provides a power supply that is designed to be used with the SAMA5D27 — if this supply would have been used, then we would have gotten the right voltage.
This bringup experience was harder and longer than most — especially for a board of this complexity. But, it illustrates an important point — full custom MPU design is hard — even with a part like the SAMA5D27-D1G which integrates SDRAM in the CPU package. It can require a significant amount of time. This design was done by a competent designer and was extensively reviewed, but it still contained serious problems that took weeks to resolve. In the end, we were still able to make the first PCB revision work with minor modifications. However, it is very possible that the first revision of a PCB like this might need to be scrapped and chalked up to a learning experience leading to months of project delays.
Lesson #3: Use a SOM (System on Module) unless you have a good reason to do a full custom design, and you have the time and resources to pull it off. You are likely not Google or Apple. Yes, you will pay slightly more for a SOM than the raw MPU system components, but the SOM cost may be offset by a lower cost baseboard. A full custom MPU design may require 8 layers and very small routing features where a baseboard for a SOM may only require 4 layers and use larger features, resulting in a less costly PCB. Additionally, a baseboard for a SOM may have a lot less parts on it, and requires less effort to set up for assembly — simpler ones can even be hand built to bypass weeks of assembly lead-time.
There are dozens of SOM vendors offering products today that fit about any price/quality/certification price point. Some SOMs contain pre-certified WiFi and Bluetooth radios. Others meet various mil-spec requirements. This situation today is much different than 10 years ago when there were only a few expensive SOM models available. SOM modules allow us to bring advanced technology to a problem, and implement a solution with a small team in a reasonable amount of time. Implementing a SOM based MPU design still requires a fair amount of knowledge, but the unknowns are drastically reduced, and chances of success are far better. Much of the design complexity is already handled, freeing you up to work on the value a product provides. Any time we can simplify our development effort, our chances of success are much greater.
Working with Microchip on this project was a good experience. Without their support, we would not likely have resolved the issues we faced in a timely fashion. This brings us to Lesson #4: When developing complex system, you are likely going to need some level of support. This may come from IC vendors, open source communities, consultants, etc. In this case, the MPU documentation was not up to date, and the chip designers were able to point out the problem. We could be critical of Microchip for this oversight, but with a 3000+ pages datasheet, this is a lot of detail, and there will be mistakes and oversights. The fact that most of the datasheet is available and accurate is a commendable accomplishment. Getting this level of documentation from some vendors is like pulling teeth. Before any of us cast the first stone, we must ask the question — is everything we do completely and accurately documented? Overall, the SAMA5D27 documentation is pretty good.
The support provided on the SAM-BA Github issues was also essential for us to figure out a way to program the device. The procedure we found for programming this system using only a serial port is relatively simple and easy to implement. Though it may violate some SDRAM refresh timing specs, in practice it works very well — and that is what matters — getting things done. It is unfortunate that Microchip has since closed the Github issues for the SAM-BA project, indicating they are not really interested in building a community around the project, but rather Github is only a way to distribute the tool that is developed in-house. As systems get more complex, Open Source communities are proving to be the only viable mechanism to develop and support all the various needs users might have. Large companies readily acknowledge this as many core technologies are now Open Source, and much effort is put into building communities around them. But, lack of Open Source vision aside, Microchip is a reliable partner — especially for smaller companies wanting to use advanced technology and needing support.