AI Pilot Projects: Designing a 30-90 Day Proof of Value

An AI proof of value is a time-boxed experiment, usually 30 to 90 days, designed to change one specific decision: whether to put a system into production. It differs from a proof of concept in what it has to prove. A proof of concept shows the technology can work. A proof of value shows the technology moves a number the business already tracks, under the conditions the business actually operates in, at a cost someone is willing to pay. The first is a demo. The second is evidence.

The gap between those two things is where most AI budgets quietly disappear. A pilot ends, everyone agrees it was interesting, and nothing changes. That outcome is rarely a technical failure. It is a design failure, and it is nearly always visible on day one to anyone who knows what to look for. This article is about how to design the pilot so that it produces a decision instead of a slide deck.

What is prototype theater, and how do you recognise it?

Prototype theater is a pilot that cannot fail. It runs on a hand-picked slice of data, is evaluated by the people who built it, and is judged against a standard nobody wrote down in advance. At the end it is declared a success, because there was no definition of success capable of returning any other answer.

The tell is not the demo quality. Prototype theater usually demos beautifully; that is the point of it. The tells are structural, and there are four:

• No pre-agreed threshold. If nobody wrote down the number that would trigger a production decision, the result will be interpreted after the fact by whoever has the strongest opinion.
• No baseline. The pilot reports that the model is 91% accurate. Nobody measured what the current process achieves, so 91% is unanchored. It might be a large improvement or a significant regression.
• Data that was assembled by hand. Someone in IT produced a one-off extract for the pilot. The pilot proves a model can be trained on that extract. It proves nothing about whether the data can be obtained on a Tuesday, every Tuesday, in production.
• No named owner for afterwards. If nobody is accountable for running the system in month four, the pilot is an orphan at conception, whatever its results.

All four are cheap to fix before the pilot starts and impossible to fix afterwards. That asymmetry is the entire argument for designing pilots deliberately.

What has to be written down before the pilot starts?

Three things, on one page, signed off by whoever controls the production budget. If you cannot produce this page, you are not ready to start, and the time spent producing it is not overhead. It is the cheapest part of the project.

The decision the pilot informs

State it as a conditional: if the pilot shows X, we will do Y; if it does not, we will do Z. Then check that Y and Z are genuinely different. Surprisingly often they are not. If the answer is "we would probably build it anyway" or "we would keep exploring either way", the pilot is not informing a decision and its budget belongs somewhere else.

This also settles scope. A pilot only has to reduce uncertainty enough to make that one decision safely. It does not have to handle every edge case, cover every product line, or look finished. Scope creep in pilots is almost always someone quietly substituting a different, larger decision.

The success metric and its threshold

The metric has to be something the business already measures, or could measure without a new reporting project: scrap rate, hours per case, first-pass resolution, days sales outstanding, throughput per shift. Model metrics like F1, precision, and recall are instrumentation. They tell your engineers whether the model is working. They do not tell your sponsor whether to deploy.

The threshold is the harder half, and it should be set by asking the sponsor a specific question: what result would make you deploy this without further debate? Write down that number before you see any results. A threshold agreed afterwards is not a threshold, it is a rationalisation. We worked through how to make these numbers honest, including baselines and attribution, in validating AI ROI before you build.

Measure the baseline first, with the same definition you will use at the end. This sounds obvious and is skipped constantly. If the current process has never been measured, measuring it is the first week of the pilot, not an afterthought.

The kill criteria

Kill criteria are the success threshold's mirror image, and they are the part teams resist writing. They should state what result, or what discovery, ends the project early. Useful ones are concrete: the data cannot be accessed in production without a new integration nobody will fund; the model cannot clear the threshold even with generous assumptions; the workflow change requires headcount that does not exist; the latency budget cannot be met on available infrastructure.

Name who is allowed to call it, and put a date on the check. A kill criterion nobody is empowered to invoke is decoration. The point is not pessimism, it is that a pilot stopped in week five costs a fraction of one that limps to week twelve and is then quietly shelved. Stopping early is the pilot working correctly, and it should be described that way to the team in advance so that nobody reads it as failure.

How do you scope 30, 60, or 90 days?

The duration should follow from what is uncertain, not from a calendar preference. Three shapes cover most cases.

Around 30 days

Appropriate when the data already exists in a queryable system, the task is close to a solved problem, and one decision-maker owns the outcome. Typical shape: measure the baseline, evaluate an off-the-shelf or lightly adapted model against a held-out set of real cases, and put the output in front of the people who would use it. The deliverable is a go or no-go on a narrow question. Thirty days is not enough to prove production readiness, and a 30 day pilot should not claim to.

Around 60 days

The common case. Enough to do the above and connect the system to one real workflow, with real users, for a few weeks of live operation. This is the shortest window that produces evidence about adoption, which is usually the binding constraint. A model people will not use has an effective accuracy of zero, and you cannot learn that from an offline evaluation.

Around 90 days

Warranted when data plumbing has to be built before anything can be evaluated, when the context is regulated and the oversight and documentation design is itself part of what is being tested, or when several stakeholder groups have to agree. Ninety days is also the right shape when the honest answer to "can we get this data reliably?" is "we think so". Discovering the answer is no in month one of a 90 day pilot is a good outcome; discovering it in month seven of a build is not.

Beyond 90 days you are no longer piloting. You are building without having decided to, which is a governance problem rather than a scheduling one.

What data readiness does a pilot actually require?

Less than people fear for the model, more than people expect for the pipeline. A pilot rarely needs the full historical archive. It needs enough representative examples to evaluate honestly, and it needs proof that the data can be obtained repeatedly.

Those are two different questions and pilots routinely answer only the first. The practical test is whether the pilot's data arrived through a path that could survive: an actual query against an actual system, run more than once, by someone who is not the one person who knows how. If the answer is a spreadsheet a colleague exported in March, the pilot has not tested data readiness at all. Make re-running the extract an explicit pilot task in week one, and treat a failure there as a finding, not an inconvenience. Our data infrastructure assessment goes deeper on what makes a source genuinely reachable.

Representativeness matters more than volume. A thousand cases spanning your real variation, across sites, shifts, product lines, seasons, and the awkward exceptions, will tell you more than a hundred thousand records from one clean quarter. Sample deliberately for the conditions you expect to fail in, and keep a held-out set nobody looks at until the end.

Labels are the usual bottleneck. If the task needs ground truth that does not exist yet, labelling is a real line item with real expert time, and it should appear in the plan rather than being absorbed silently by whoever is most helpful.

Which deployment constraints have to be tested during the pilot?

The ones that could stop deployment. It is worth being blunt about this: a pilot that runs in a notebook on an engineer's laptop has proven a model, not a system, and the distance between the two is where projects die. Test these inside the pilot window, even crudely:

• Where it will run, and who supports it. Cloud, on-premise, at the edge, inside an existing application. Each carries different security review, cost, and support consequences, and the review queue is often longer than the pilot.
• The latency and throughput budget. A response that takes eight seconds may be fine for a batch process and useless at a workstation. Establish the real requirement from the workflow, not from a guess.
• The integration surface. What system does the output land in? If a human has to copy a result from one screen to another, adoption will decay to zero regardless of quality.
• Security, privacy, and residency. Where the data may travel, which vendors are acceptable, what the GDPR position is. Start this conversation in week one; it is frequently the longest pole and it is entirely predictable.
• The failure path. What happens when the system is unavailable or unsure. If there is no defined fallback, the pilot has not been designed for production.

None of these needs to be built properly during a pilot. All of them need to be probed, so that the cost of doing them properly is a known number in the final recommendation rather than a surprise in month five. Several of the recurring integration mistakes are catalogued in five mistakes companies make when integrating AI.

Who needs to be aligned, and on what?

Five roles, and the pilot should not start with any of them vacant.

• The sponsor controls the production budget and owns the decision. They must personally agree the threshold. A sponsor who delegates the threshold has delegated the outcome.
• The process owner runs the workflow that changes. They know the exceptions that will break your assumptions, and they can quietly ensure the pilot never touches real work if they were not consulted.
• The data owner can grant access. Identify them in week one, because procurement and access approvals do not compress.
• The end users have to use the thing. Involve a few of them early and let them be sceptical; scepticism surfaced during a pilot is information, whereas scepticism surfaced after deployment is resistance.
• The eventual owner runs the system in month six, monitors it, and decides when it needs retraining. If this person does not exist, that is the finding, and it is worth more than any model metric.

The alignment that matters is not enthusiasm, it is agreement on the threshold and the kill criteria. Enthusiasm is abundant at kickoff and unreliable at review. A signed threshold survives the meeting where results turn out to be ambiguous, which is the meeting that decides whether the pilot mattered.

What should a proof of value deliver?

A decision, and the evidence behind it. Concretely, five artifacts:

• The measured result against the baseline, on held-out real cases, with the threshold restated and met or missed. No adjectives.
• An honest account of what broke, including data gaps, integration friction, and cases the system handled badly. This section's usefulness is proportional to its uncomfortableness.
• A cost-to-production estimate covering build, integration, and the ongoing cost of running and maintaining the system, which is the number most business cases omit. The lifetime view is covered in calculating ROI for machine learning projects.
• A recommendation with a real stop option. Proceed, proceed with a narrowed scope, revisit after a specific prerequisite, or stop. A pilot process that has never recommended stopping is not a pilot process.
• The reusable assets, meaning the evaluation harness, the labelled set, and the documented data path. These outlive the decision and make the next pilot cheaper even if this one stops.

When is a pilot the wrong instrument?

Four cases, and recognising them saves the most money of anything here.

When the answer is already known. If the approach is well established for your exact task and the real question is integration cost, you need an implementation plan, not an experiment.

When the constraint is organisational. If the process is undefined, ownership is contested, or the data is unreachable for political rather than technical reasons, a pilot will surface that expensively. An AI Readiness Audit surfaces it in a fraction of the time, which is why the audit sits before the pilot in the sequence and not after. The broader failure pattern, and the gates that catch it, is set out in how to avoid AI project failure.

When there is no baseline and no appetite to measure one. Without a before, there is no after, and the pilot will end in an argument about interpretation.

When nobody will own the result. This is worth stating plainly to the sponsor before starting, because it is the one condition that guarantees waste regardless of how well the pilot is executed.

The honest summary

A good pilot is a small, well-instrumented argument. It names the decision it serves, measures against a baseline that existed before it started, holds itself to a threshold somebody signed, tests the constraints that could actually block deployment, and is allowed to end early. Nothing in that list is technically demanding. All of it is organisationally uncomfortable, which is why it gets skipped and why skipping it is the most reliable predictor of a pilot that produces admiration and no change.

If you are deciding where to start, the sequence that works is usually the same: validate the business case first, then run one narrow proof of value against a threshold, then, once something is in production and needs sustained ownership, put experienced hands on it. That is the shape of our service packages, from a readiness assessment through a scoped proof of concept, and it is why a Fractional AI Lead is the step after a successful pilot rather than a substitute for one. The audit tells you what to build, the pilot tells you whether it works, and the ownership decides whether it lasts.

Frequently asked questions

How much data do you need to run an AI pilot?

Less than most teams assume. A pilot needs enough representative cases to evaluate honestly, typically hundreds rather than millions, plus proof that the same data can be fetched again. A thousand cases spanning your real variation beats a hundred thousand from one clean quarter.

Can a pilot run while the production system stays untouched?

Yes, and for a first pilot it usually should. Run the model alongside the existing process and compare outputs rather than replacing anything. You still learn accuracy and adoption, and you avoid the change management that makes a failed pilot expensive.

Who should run the pilot, an internal team or an external partner?

Whoever can finish it inside the window. Internal teams know the data and the exceptions; external teams bring the evaluation discipline and are not distracted by day-to-day work. The deciding question is who will own the system afterwards, because that person should be involved either way.

What if the pilot succeeds but nobody wants to deploy it?

That is a scoping failure surfacing late. It means the decision the pilot informed was never really open, or the sponsor who controls the budget did not agree the threshold. Both are fixable before the next pilot by writing the conditional decision down and having the sponsor sign it.