Your AI agent works in the demo and dies in production

The problem

The agent works perfectly when you show it to the client. Two weeks later it is quietly making things up, and nobody notices until a customer does.

I see a lot of companies stuck in exactly this loop right now. They build something impressive, demo it, get approval, deploy it, and then watch it degrade. MIT's number on this gets quoted constantly because it is brutal: around 95% of AI pilots never make it to production. The ones that do often get switched off within a quarter.

After 4 years building and maintaining these systems for 30+ businesses, here is what I think is actually going on.

The demo is a rigged test

A demo is three or four inputs that you chose. You picked them because you already knew the agent handled them well. You ran it at 2pm on a Tuesday with a clean context window and no concurrent load.

Production is none of those things.

Production is the customer who writes in three languages in one message. The record with a null where every other record has a value. The API that changed its response shape last Thursday without telling anyone. The conversation that is now 40 turns long.

The agent did not get worse. You just stopped hand picking its inputs.

It is almost never the model

This is the part that costs companies the most money. Something breaks, and the instinct is to reach for a better model, or to rewrite the prompt for the ninth time.

Shopify published a number on this that I think about a lot. They took an AI tool from 71% to 93% accuracy by changing what information the model received. Rewriting the prompt alone got them to 74%.

That gap is the whole story. Twenty two points from fixing the information, three points from fixing the wording.

When I audit a broken agent, I am looking at what went into the window, not how politely the instructions were phrased. Nine times out of ten the agent is reasoning correctly over bad, stale, or bloated input. It is not confused. It is misinformed.

The three failure modes I actually see

Stale context

The agent is reading a cached copy of something that changed. Pricing, inventory, a lead status, a policy doc from 2024. It answers confidently and it is confidently wrong.

Fix: make freshness explicit. Every piece of retrieved data gets a timestamp in the window, and the agent gets told what to do when data is older than a threshold. Ask, do not guess.

Overloaded context

Someone decided that more context is safer, so they dump the entire CRM record, the last 50 messages, and four knowledge base articles into every call.

This backfires measurably. Research from Google DeepMind found that information placed in the middle of a long window gets recalled 20 to 30% less accurately than content at the start or end. You did not give the agent more to work with. You buried the one line that mattered.

No failure path

The agent has no way to say "I do not know." So it does not. Every single query gets an answer, because that is what you built it to produce.

Fix: give it an exit. An explicit escalation path, a confidence threshold, a handoff to a human queue. An agent that escalates 10% of cases is worth more than one that guesses on 100%.

What I build instead

The systems I ship that survive past month three have four things in common.

A defined context budget. I decide what goes into the window and in what order, and I can tell you why each piece is there. Anything I cannot justify gets cut.

Real inputs from day one. Before I write a line of logic I pull 50 to 100 real records, including the ugly ones. If the agent cannot handle your actual data, it does not matter how it handles clean data.

Logging on every call. Input, output, which context was retrieved, how long it took. When something drifts in week six, I need to be able to see when it started.

An owner. Somebody is responsible for this system after launch. Usually that is me, because upstream APIs change and models get deprecated and nobody at the client is watching for that.

The test I run before anything ships

I take the 20 weirdest records in the client's database and run them through. Not the clean ones. The duplicates, the ones with missing fields, the one where someone typed their whole life story into the notes field.

If the agent handles those, it will handle Tuesday. If it only handles the demo set, all you have built is a very expensive demo.

What this means if you are buying

If someone is showing you an AI agent, ask them two questions.

What happens when it does not know? And who is watching this in month four?

If the answer to the first is a shrug, it will hallucinate on your customers. If the answer to the second is "you are", you are buying a prototype and paying production prices.

I build these systems end to end and I stay on them after launch, because that is the part that decides whether it still works next year. If you have an agent that demoed well and is now quietly failing, that is usually a two week fix, and it usually starts with the context, not the model.