Two initiatives landed on our roadmap at the same time this summer: bring in a new large data set, and migrate existing infrastructure onto a new platform. The team used AI to plan both, something we’ve been rapidly adopting into our process for complex initiatives and big projects. The AI agent proposed something clever. There was a specific kind of live connection we could use between the old system and the new one - we could route the new data through the new platform while still serving everything out of the old system during the transition time. Instead of two projects spanning months, we could tackle both in a fraction of the time.
The AI agent didn’t just propose two smart plans, but a shortcut. It explained the shortcut so simply and matter of factly, that nobody stopped to ask if that shortcut would actually work. It was now a perfect, combined, speedy plan.
Three senior contributors spent three weeks building against the new plan. There was real deadline pressure as two of the three were about to roll off the project, and the first phase needed to land before they left. In the last few days of the critical first phase, they found out the live connection between platforms didn’t behave the way the plan assumed.
In fact, the new “shortcut” plan wouldn’t work at all. The mechanism the whole plan was built on just didn’t hold up. The two projects were split apart again, because they were never actually one project that could be successful together. That illusion came from the AI agent’s plan, not from anything true about the systems.
One of the initiatives, which was business critical, was three weeks in and had to restart close to zero.
Although painful to our team, this failure of ours seemed especially curious. None of the team skipped testing because they were lazy or too busy. They skipped it because the plan was good enough that testing felt like it would just confirm what everyone already believed. AI is extremely good at producing a plan that reads like every complex problem has a set of simple solutions. The confidence in the writing and the truth of a claim come from two completely different places.
They skipped it because the plan was good enough that testing felt like it would just confirm what everyone already believed.
A mediocre human plan gets questioned because it reads like a rough draft. A fluent AI plan doesn’t get questioned, because it reads like it’s already been thought through. The elegance is the trick, not reassurance. When a plan makes two hard problems dissolve into one easy one, that’s usually the exact spot where an assumption or set of assumptions is doing work that testing & verification should be doing instead.
Researchers have been studying automation bias since the late nineties, well before any of this was generative: people favor automated output even when the evidence in front of them disagrees with it. The original studies were about pilots trusting an instrument over what they could see out the windshield. Nothing about that changes when the automated thing writes in full sentences and sounds like a colleague: a bad instrument reading looks like a bad instrument reading, and a bad plan reads like a plan.
The fix isn’t a trust problem, it’s a location problem…
AI doesn’t just get accepted uncritically, some of the newer work on sycophancy in these models shows it tends to agree with the frame you hand it, which makes a plan built to merge two projects even more convincing once you’ve already decided that’s what you wanted. The errors don’t stay small either. A plane one degree off course ends up a mile off for every sixty it flies. At the speed AI lets you move, a wrong assumption at the start doesn’t cost you a correction later, it costs you the three weeks you spent building on top of it.
The fix isn’t a trust problem, it’s a location problem: we need to know exactly where in a plan the unverified claim is hiding before we move forward to implement it.
Protect Planning with First Principles
First, find the claim the whole plan rests on. This is not the “risks section” that the AI Agent also generated, but the thing or things so central the plan never flagged it because it’s treated as settled. Ask: what does this plan assume that, if false, the whole thing falls apart? What’s the part that made two hard problems dissolve into one easy one, that’s usually where it’s hiding. In our case that was “this live connection works under our conditions.” That’s always in there somewhere, and it’s usually load-bearing for the part of the plan that feels like a win.
Second, run a pre-mortem before you run the plan. Gary Klein’s old technique still works here: assume the project already failed, and work backward to why. It’s a thirty-minute exercise that would have surfaced the actual question, does this mechanism hold up under production conditions, instead of letting everyone nod along to how clean the merged plan looked.
Third, test the load-bearing claim directly, cheaply, before it gets people and resources. You cannot simply ask “does this sound right.” You must understand the conditions of the plan and try to simulate what could fail under those conditions. Does the connection actually hold up under real load, does high latency kill it as an option? Does the vendor actually do what the plan says they do? Ask it the way you’d ask a first-year engineer to show their work, not the way you’d nod along to a good presentation.
Fourth, keep a hard line between what the AI verified and what it assumed. Those read identically in the output. They are not the same thing, and the plan will not tell you which is which. AI agents don’t lie, but they do take what’s in their context as real facts. For example, here’s what we found our AI had verified versus assumed in the plan from this piece: it verified the connection type existed and the syntax was valid, that’s checkable against public documentation. It assumed, without flagging it, that the same connection would hold up under our actual production conditions. Those two claims read identically, but only one of them is true.
The plan will always sound finished, that’s what a plan is for. Whether it’s actually something that can be delivered is a separate question, and it’s still ours, as the humans, to go answer.