Applied AI · 4 min read
Four questions every AI feature has to answer before it ships
The demo works and everyone is impressed. Then someone asks about data, errors, cost and ownership, and the pilot stalls. Ask those questions first and it does not.
Priya Natarajan
October 28, 2026
Most AI pilots die in the same place: the week after the demo.
The demo is built to impress, and it does. Then it meets the people who would have to live with it, and they ask four questions that nobody planned for. We now ask them before we build anything, usually on paper, and it has changed what we build.
1. Where does the data come from, and who may see it?
A demo runs on a sample file. A production system runs on live data with access rules attached. Before a model touches anything we work out which systems it reads from, which records it is allowed to see for which user, and what happens to the data after the request. Half the time this question alone changes the design, because the feature everyone wanted needs data that a particular team is not allowed to see.
2. What happens when it is wrong?
It will be wrong sometimes. The question is who notices, how fast, and what it costs in the meantime. For every feature we define a confidence threshold, a review path for the uncertain cases and a way for the person reviewing to correct the output. Those corrections go into the evaluation set, which is how the system gets better rather than just older.
3. What does one task cost, and what does that look like at volume?
A demo costs nothing. A system handling ten thousand tasks a month has a bill, and the bill depends on decisions made early: which model, how much context, whether results are cached, whether a cheaper model can handle the easy cases. We estimate the cost per task before building and measure it after. A feature that saves an hour of work for two dollars is a different business case from one that saves the same hour for forty cents.
4. Who owns it on a Tuesday in March?
Model providers change things. Prompts drift. A supplier changes the format of a document and the extraction quietly degrades. Someone has to own the system after launch: watch the evaluation numbers, handle the exceptions queue, decide when to upgrade a model. If nobody is named, the answer is nobody, and the system will be switched off within a year.
Build the other way round
We answer these four first and build the model second. The result looks less exciting in week one and considerably better in month six, which is when it matters.
AI in production is mostly an engineering problem wearing an AI costume. Treat it like one and it tends to stay in production.