AI proof of concept: why pilots stall before production

An AI proof of concept is a short, bounded build that answers one question: can this AI idea do a specific job, on your own data, to a standard you set in advance. Most never reach production. One reason, and the cheapest to fix, is that nobody wrote down what "it works" means before the build started.

Without a written target, a result that impresses the room counts as success, the team moves on to the next idea, and six months later nobody can say whether the first one should have been built.

This guide covers what a proof of concept has to prove, why so many stall, how to write the targets on day one, and what the work should hand you at the end.

What an AI proof of concept has to prove

People use the term for two different pieces of work, and they are easy to confuse. One is a demo on sample data. The other is a test on your own data, which some teams call a pilot. Production is the third stage, shown here for comparison.

With today's models, the first column is rarely in doubt. A language model can summarise a call, answer a question from a document or sort a ticket. You can see that in an afternoon. The open question sits in the second column: will it do that on your documents, with your edge cases, at a cost you can live with.

That is why we call our own version a pilot and build it on the client's data from the first day. In this article, proof of concept and pilot both mean the work in that second column. If yours stops at a demo on sample data, the question that matters is still open.

Why pilots stall before production

The best-known figure is bleak. IDC research with Lenovo found that 88% of the AI proofs of concept it observed never reached wide deployment: for every 33 a company launched, four graduated to production. IDC put it down to low readiness in data, processes and infrastructure.

There is a name for the place these projects end up. McKinsey used "pilot purgatory" in a 2018 report on digital manufacturing, for companies with plenty of pilots underway and no results on the bottom line. The phrase is now used for AI pilots too. The project is too promising to cancel and too unfinished to ship, so it sits there with a sponsor and a status slide.

IDC's causes are broad. In our experience they show up as three specific gaps:

• No written target. The pilot was approved to "explore" something. Without a number, any result can be read as encouraging, and encouraging is not a reason to fund production.
• Demo data. It was built on fifty tidy examples. Real records have duplicates, missing fields and three versions of the same policy, and the accuracy seen in the demo does not survive them.
• Nobody priced running it. The build cost was approved. The monthly cost of model usage, infrastructure and someone to look after it was never estimated, so the production decision has no figure to decide on.

All three can be dealt with before any code is written, which makes them cheap to fix: agree the target, choose the test data, and put cost on the target sheet.

Set the targets on day one

A target sheet is one page. It is written and agreed by the sponsor and the team doing the build before the work starts, and it does not change during the work unless both agree in writing.

The example below is for an assistant that answers customer support questions. The figures are an illustration, not a client's.

What a good target looks like

1. One number. "Improve response quality" cannot be passed or failed. "Resolve 60% of contacts without a human" can.
2. A baseline. If you do not know what the process costs or achieves today, measure that first. It is usually quick, and it sometimes changes which idea is worth piloting.
3. A stop line. The target says when to proceed. The stop line says when to walk away. Results that land between the two mean fix and re-test, and it helps to agree that before anyone has feelings about the outcome.
4. Named test data. Which records, how many, and who scores the answers. This is what makes the number believable to a board.
5. Cost. A system that hits its accuracy target at ten times the affordable price has failed, and you want to know in week three.

A live example: the support agent we built for eSafetyFirst on Amazon Bedrock resolves 90% of customer contacts without a human, in English and French, grounded in their own course material. That is a production figure. A target sheet puts a number like it in writing before the build starts.

Run it on your own data

A pilot is only as honest as the data it runs on, so three rules apply.

Use a representative sample. Pull records from the real system, across a normal period, with the bad ones left in. If a fifth of your tickets are two-word messages with an attachment, a fifth of the test set should be too.

Hold part of it back. Split the data before the build. One part is used to build and tune. The other is locked away and only used for the final measurement. A system tested on the examples it was tuned on will always look better than it is.

Decide who scores. Some targets can be scored automatically. Others need a person who knows the work. On AWS, Amazon Bedrock evaluations supports both: you supply your own set of test questions, and the answers can be scored automatically, by a second model, or by your own staff. Whichever you choose, write it on the target sheet.

Real data brings real obligations. If the records include health information, the pilot needs the same safeguards as production, which starts with what AWS covers under its business associate agreement and what you still configure. For any sensitive data, run the pilot in its own AWS account with account-level guardrails and the baseline security controls already on. The production system needs the same account structure, so this is work done once.

What the pilot should hand you at the end

A pilot should end with a written report, and you can ask what it will contain before you sign anything. It should have five parts.

1. The measured result against each target. The same table as the target sheet, with a results column filled in.
2. What failed, and why. The wrong answers, grouped by cause. This is the most useful page of the report, because it tells you whether the fix is better data, a different design, or a narrower job.
3. A production plan. What has to be built that the pilot skipped: access control, monitoring, a fallback for when the system is unsure, integration with the tools people already use.
4. The running cost. What it costs per month at real volume.
5. A decision. Proceed, fix and re-test, or stop.

The running cost

Model usage on AWS is billed by volume, so the pilot gives you the two numbers you need: the cost per unit of work, and how many units you handle.

As an illustration, say the pilot measured $0.04 of model usage per conversation and you handle 20,000 conversations a month. That is $800 a month in model usage. Add the infrastructure around it and the time of whoever owns the system, and you have a monthly figure to set against what the process costs today.

Measure it during the pilot and not from a pricing page. Real conversations run longer than expected, the documents the system looks up are added to every request, and failed attempts that get retried are billed again. The measured figure is the only one worth planning on.

Proceed, fix or stop

If the targets are met, the production plan and its cost go to whoever signs for the launch. If the results land between the target and the stop line, the failure analysis says what to change, and the same test set measures whether it worked. If they land below the stop line, you stop.

Stopping is a result, and it is worth saying so in advance. A pilot that misses its targets has told you something true about your data for the price of a few weeks. The alternative is learning it after the production build.

Conclusion

An AI proof of concept earns its cost when it ends in a decision, and the target sheet written before the build is what makes that decision possible.

If you have an idea that worked in a demo and then stalled, that is the situation our Prove step is built for: a fixed-price pilot on your own data, measured against targets agreed on day one. See how Know, Prove, Launch works.