A useful AI proof of concept tests one risky assumption against real data within four weeks and ends with a documented go, change or stop decision. Define measurable success criteria and an evaluation set before building, secure data access in week one, and judge the result on accuracy, cost and integration feasibility rather than on a demo.
On this page
Many AI initiatives stall in what is often called pilot purgatory: a demo impressed the board, a pilot ran for months, and nobody can say whether it worked. The remedy is a proof of concept with a strict time box, a single question and success criteria agreed before anything is built. Four weeks is enough for most business use cases if preparation is done well. This guide lays out a week-by-week plan and the decisions that make the result useful.
What a proof of concept should prove
A proof of concept is not a small version of the product. It tests the assumption that would kill the project if it were false. For most AI use cases that is one of three things: whether the model can reach the required quality on your real data, whether the costs per transaction are acceptable at expected volume, or whether the necessary data and systems can be accessed at all. Pick the riskiest one and design the four weeks around it. User interface, scalability and polish are explicitly out of scope unless they are the risk.
Before week one: the preparation
Three things must exist before the clock starts. First, a written hypothesis such as "the system classifies at least 90 percent of incoming supplier emails into the correct category, with fewer than 2 percent wrongly marked as urgent". The numbers are yours to set; what matters is that they are set in advance. Second, a named business owner who can judge results and make the final decision. Third, confirmed access to real data, with data protection clearance. Waiting for data access is the most common reason proofs of concept overrun. If personal data is involved, decide early whether it must be anonymised; local tools such as Textwash Pro can clean documents on your own infrastructure before they enter any AI system, and our guide to data anonymization before AI processing explains the options.
The four-week plan
- Week 1, data and evaluation set: collect a representative sample of real cases, label the correct outcomes with the business owner and freeze an evaluation set of typically 100 to 300 examples. Set up a repository and a simple script that runs any approach against the set and reports the metrics.
- Week 2, baseline: build the simplest viable approach, often a well-designed prompt with a capable model, and measure it. Record accuracy, typical failure types, latency and cost per case. This baseline is frequently better than expected and sometimes already sufficient.
- Week 3, improvement: address the main failure types with targeted changes such as retrieval of relevant documents, better instructions with examples, a second validation step or a different model. Measure each change separately so you know what helped.
- Week 4, integration check and decision: test the critical integration path, for example reading from the real mailbox or writing to a test instance of the target system. Estimate operating costs at production volume. Write a short decision document and present it.
Measuring the right things
Accuracy alone is misleading. Separate the types of error, because a missed case and a wrong action have very different consequences. Measure how often the system says it is uncertain, since a well-designed process can route those cases to humans. Track cost per case and latency, and compare everything with the current manual process, not with perfection. A system that handles 70 percent of cases reliably and passes the rest to staff may be a strong business case, while one that handles 95 percent but occasionally makes confident, harmful errors may not be.
The decision document
End with a document of two or three pages: the hypothesis, the evaluation results, the main failure types, estimated production costs, integration findings, risks and a recommendation to go, change or stop. Attach the evaluation set and the code. A "stop" is a valid and valuable outcome; it saves the cost of a failed production project. A "go" should come with a production plan that reuses the evaluation set as a regression test. Our guide on integrating AI into existing software describes the production patterns that typically follow.
Traps to avoid
The most common trap is the moving target: changing the hypothesis midway because the original one looks unlikely to pass. Another is the curated dataset, where only clean examples are tested and production data later turns out to be messier. A third is the demo trap, where a polished interface hides the fact that nobody measured quality. Finally, beware of proofs of concept built in tools that cannot reach production for data protection or integration reasons; the result then proves little about the real system.
Who should run it
A proof of concept needs a small team: the business owner, one or two engineers who can work quickly with coding agents and model APIs, and access to someone who knows the data. Speed matters, which is why experienced AI Code Engineers are well suited; they can build evaluation scripts, baselines and integration checks within days. In venture contexts, the proof of concept is one of the stage gates described in venture building with AI. OnlineAppCreator runs four-week proofs of concept as a fixed-scope first milestone of its custom development services, with the evaluation set and code handed over regardless of the outcome.
Frequently asked questions
How long should an AI proof of concept take?
Four weeks is enough for most business use cases if data access and success criteria are settled beforehand. Longer proofs of concept tend to drift into unfocused pilots.
What is an evaluation set?
A fixed collection of real examples with known correct outcomes. Every version of the AI system is measured against it, which makes progress and regressions visible.
What if the proof of concept fails?
A documented stop decision is a valid result. It prevents a costly production project and often reveals a narrower use case that does work.