jcyamo.dev

The Planning Delusion: Why Your "Next Time" is Doomed to Fail

We’ve all been there. The project is finally done. The last feature is shipped, the bugs are squashed, and we’re in the retro looking back at the winding road we took to get here. And someone, with the best of intentions, says, “Wow. Okay. Next time, we’ll plan better.”

It feels right. The path to the solution now seems so obvious. All the dead ends, the pivots, the late-night realizations—if only we’d known then what we know now, we could have drawn a straight line.

This is the planning delusion. It’s a well-intentioned lie we tell ourselves, and it’s the reason so many teams are doomed to repeat the same cycle of over-planning and under-delivering.

The Shifting Foundation: R&D’s Hierarchy of Needs

I learned this the hard way. My team was tasked with building a new project management system. We had a solid understanding of the solution we wanted to build, but only a vague grasp of the actual customer need. We anticipated needing complex systems and started building them to “get the hard stuff out of the way.” The estimate was six months.

We were building a beautiful, elaborate house on a foundation we hadn’t even checked.

That experience led me to a model—a riff on Maslow’s hierarchy, but for engineering. I call it “R&D’s Hierarchy of Needs.” Just like Maslow’s pyramid, the most critical layer is at the bottom. It’s the foundation. It’s Validation. The simple, terrifying question: Are we building the right thing?

On top of that foundation, you stack the things engineers love to talk about:

These are good things. We strive for them. But they are a false summit. We pour our time and energy into perfecting the top of the pyramid, convinced that this is what makes a project successful. But here’s the thing: None of that matters if the foundation of Validation is cracked. Without the base layer, everything you build on top of it, no matter how beautiful or robust, is destined for irrelevance.

The Diagnosis: Cynefin and the Curse of Knowledge

The “Next time, we’ll plan better” delusion is a promise to get better at the top of the pyramid. It’s a vow to build a more beautiful, more robust, more confident system. But it completely ignores the fact that the real failure was a lack of Validation at the foundation.

Why do we keep making this mistake? It comes down to a cognitive blind spot the Cynefin framework calls the “curse of knowledge.” Cynefin helps us categorize problems. A Complicated problem is like building a bridge. The solution is discoverable through expertise and planning. You can map it out ahead of time. A Complex problem is like raising a child or launching a new product. The variables are too numerous and interconnected to predict. The only way forward is to experiment, see what happens, and adapt.

Building new software is almost always a Complex problem. But here’s the trap: once you’ve navigated that complexity and arrived at a solution, your brain rewires the past. The emergent solution now feels obvious. You fall victim to retrospective coherence, and it becomes almost impossible to imagine not knowing the answer. That’s the curse of knowledge. You mistake the messy, emergent path of a Complex problem for the straightforward blueprint of a Complicated one. And so you vow, “Next time, we’ll plan better.”

The Antidote: Gall’s Law as a “Needs” Filter

So how do we break the cycle? How do we focus on Validation instead of just polishing the upper levels of the pyramid? We follow Gall’s Law: A complex system that works is invariably found to have evolved from a simple system that worked.

This isn’t just a nice idea; it’s the operational antidote to the planning delusion. Gall’s Law is the only practical method for building a solid foundation of Validation.

This is the principle that saved our project. My team was already deep into building the complex system we’d envisioned. They were doing good work, but we were all operating on a faulty premise. I started asking some uncomfortable questions, prodding at the actual need and challenging if the path we were on was the right one.

The team, to their credit, was open to the challenge. We decided to pivot together. We scrapped a decent amount of work, simplified the code, and decoupled it from the existing system. The six-month estimate shrank to a few weeks. We built the simplest possible version of the thing to get real-world feedback.

Think of this as the “Minimum Demoable Product.” It’s not a scaled-down version of the final vision; it’s a probe. It’s the first, critical step in laying that foundation of Validation, and because it’s minimal, it gives you the freedom to pivot. That simple, working system was our test of the pyramid’s base. It answered the “Are we building the right thing?” question faster and more definitively than any architectural diagram ever could. The result was the company’s most successful beta release at the time. The complexity—the beauty, the agility, the robustness—could emerge later, built securely on top of that proven foundation.

Reclaiming “Next Time”

The planning delusion is a promise to perfect the past. The wise approach is to reframe the promise. Instead of saying, “Next time, we’ll plan better,” we should be saying, “Next time, we’ll start simpler to validate our foundation first.”

This shift in language is critical. It moves the focus from the illusion of foresight to the reality of discovery. It puts Validation, not Robustness, at the base of our priorities from day one. And it’s the only way to ensure you’re not just building something perfectly, but that you’re building the right thing.