When the Plan Laughs Back

Part 1: The Lesson

At some point in our careers we learn the same painful truth: a component loved is a component that will eventually betray us. Burns said it plainly—the best laid schemes of mice and men go sideways no matter how carefully we plan.

The lesson is simple: never architect a system around a single point of failure, especially one you are emotionally attached to. When a piece of silicon becomes the make-or-break of the entire design, that is exactly when you need contingencies the most.

Risk does not disappear because we are enamored with the solution. And in this business, a lost bet does not ship product.

My first painful experience came during a design contest between two divisions of our parent company. I was Chief Engineer of the smaller group, and the goal was to create a unified hardware and low-level firmware framework. That is when I made my first mistake: I fell in love with a new microcontroller.

I became entranced by the shine. I ate, breathed, and dreamed that datasheet. Fast execution, plenty of memory, lightning-quick context switching, generous I/O—it looked like the Holy Grail. I could even cut component cost because of the peripheral set. I was all in. And because I was all in, I was blind.

The flaw was obvious in hindsight: it was single-sourced silicon with no alternative. If it failed, the design failed. If the design failed, so did I. Warnings did not matter—I was too enamored to hear them.

The part failed. Worse, it did not just fail—it did not exist. The early production silicon the sales representatives swore was real turned out to be marketing hype and pre-release literature. There was nothing to ship, nothing to test, nothing to salvage.

I lost the contest, a chunk of my ego, and more of my reputation than I would like to admit. But I gained something far more valuable: the understanding that a balanced design process matters.

Part 2: The Lesson Redux

A recent experience involved the development of a high-speed pattern generator with a brutally tight schedule. We had roughly eight weeks to ship working prototypes, and if that was not enough pressure, we were racing the Chinese New Year shutdown at the board house.

The schedule alone eliminated the possibility of an FPGA design. We needed a microcontroller with as much flash as we could get, 32-bit GPIO ports, and the speed to sustain a 20 ns flash-to-I/O cycle.

On paper, the DMA engine was supposed to be the hero—the backbone of the architecture. In reality, it could not transfer blocks larger than 64 kB without chaining descriptors, and the stalls between chains created long dropouts in the I/O stream. We tried everything to close that gap. Nothing worked.

We dropped to timed, cycle-counted assembly routines to deliver the deterministic I/O the design demanded. It was not elegant. It was not pretty. But it shipped.

That is having a plan B. That is how you keep the plan from laughing back.

Originally published on LinkedIn as “When the Plan Laughs Back”.