During system verification for a customer, we noticed occasional failures when writing to a SD card using the Marvell PXA270 MMC/SD controller. The failure is a 4KB block of data is shifted on byte, where the first byte is duplicated and the last byte is dropped. The test app that found this problem is available here. This test app simulates an application that writes data to a SD card at a certain rate. We found that typically in about 2000 – 10000 hours of simulated testing, we would see a failure. Its very interesting that the field failure rates matched this rate as well.
This failure had all the signs of a hardware (likely DMA) problem, and sure enough there is a Marvell Eratta for this issue. A patch has been developed and tested by several people and is available in this mail thread.