AI Projects That Never Leave the Pilot: Where Do We Get Stuck? 🚧
In my last article I said that when choosing a model you should look at the test set built from your own data, not at the parameter count. Let's say you did exactly that: you picked the right model, built the pilot, ran the demo in the meeting and everyone was impressed.
Then what happens?
What I see in the field is this: most AI projects die not because they failed technically, but because they got stuck at the pilot stage. And nobody ever says “this project is cancelled.” The project simply stops being mentioned; the deck stays in a folder and the process keeps running by hand, exactly as before.
So let's go through the points where we get stuck moving a pilot into production, one by one. They all share one property: none of them are about the model.
⚠️ First, Let's Kill One Illusion: “But It Worked in the Demo”
85% accuracy is an impressive number in a pilot. In production, that same 85% means “30 documents to fix by hand every day” for a team processing 200 documents a day.
The gap between demo and production is not a gap in percentages; it is a gap in volume and accountability. In a demo an error is an anecdote. In production an error is a cost. And that is precisely why pilots look easy: we always run them on the cleanest data, the best-case scenario and the most enthusiastic user.
🚧 The 6 Places We Get Stuck
1. Starting with the wrong question. Most projects begin with “what could we do with AI?” That question puts the technology at the center and ends up producing an elegant demo nobody needed. The right question is: “which task ate most of my team's time this month?” The proforma → bank instruction automation I described in my first article was born exactly that way: not out of technology, but out of a recurring pain.
2. Ignoring the exceptions. A pilot solves the happy path; production asks about the exceptions. Think of my customs document example: in the pilot you always have properly scanned, single-language, complete PDFs. In the field you get an upside-down page, a document with a handwritten note on it, a file missing two pages, two different declarations inside the same file. If you automate 80% of a process and leave the remaining 20% undefined, the team has to stop every single time and ask “which bucket does this one fall into?” — which cancels out the speed you expected from automation. You don't have to solve every exception; but the system must be able to recognize one and say “route this to a human.”
3. The last-20-meters problem. The model produces the output and it looks great on screen. But how does it get from there into the ERP? If the answer is “the user copies and pastes it,” that process isn't automated — it has merely moved. Most pilots stall right here, because the part everyone assumed was hard (the model) is done, and what's left is the tedious part: permissions, record formats, rollback on failure, logging. What carries a project into production is not the model's intelligence, but the output's ability to reach its destination on its own.
4. A project with no owner. A pilot is usually driven by one curious person. When that person goes on holiday, the project goes with them. Going live means the process has an owner, a backup, and a clear answer to “who do we call when it breaks?” Without those, the team won't take on the risk and will quietly fall back to the old method. The old method is slow, but it is familiar — and faced with uncertainty, people choose the familiar every time.
5. You cannot defend what you didn't measure. “It really worked out great” is not a sentence that wins budget. Before starting the pilot you have to measure the current state: how many minutes does this task take today, how often is it done per month, how many errors occur, what does an error cost? Without a baseline, all you have when the pilot ends is an impression. With one, you have a defensible number like “40 hours a month and this many errors.” The go-live decision is usually not a technical decision at all — it hinges on whether that number exists.
6. Never building the trust and accountability layer. This brings me back to the question from my article on vision: who is at the wheel? If the system misreads the IBAN on a bank instruction, who is responsible? Without a clear answer, nobody presses that “send” button, no matter how well the system performs. The solution isn't to make the AI flawless; it is to identify the critical fields and route them through human approval. Amount, IBAN, tax number, quantity: shown side by side on screen and confirmed. Everything else flows. Trust comes from the design of the process, not from the accuracy of the model.
✅ 5 Rules That Move a Pilot Into Production
1️⃣ Start the pilot with your dirtiest data, not your cleanest. Collect the 20 worst examples you have and run the system on those. Clear that bar and the rest follows.
2️⃣ Measure the baseline on day one. Duration, volume, error rate. You cannot go back and measure these later.
3️⃣ Define the scope to include the last 20 meters. Wherever the output has to land, the pilot must reach it. Every step left as copy-paste is a reason the project won't go live.
4️⃣ Write the critical-field list up front. Decide in advance what requires human approval. Debating it after an error will not bring the trust back.
5️⃣ Go narrow, but end to end. Solving one process completely is incomparably more valuable than solving half of five. Five half-finished processes just give the team five new things to chase.
🎯 Final Word
A pilot shows that something is possible. Production shows that it is sustainable. The distance between the two is almost never about model quality; it is about process, ownership, integration and trust.
That is why the first question I ask myself when starting an AI project is no longer “can we build this?” It is: “if this works, who uses it tomorrow morning, and who fixes it when it breaks?”