AI Production Systems
Why Most AI Projects Fail Before the First Prompt
Most AI projects do not fail because of a bad prompt. They fail because nobody built the engineering underneath: authentication, permissions, audit, observability, evals, token budgets, deployment and rollback. Here is what production AI actually needs.
Curated by Matt Perry
CTO
Most AI projects fail before anyone writes a prompt. They fail in the gap between a working demo and a system real people can safely use.
The demo is the easy part. You paste a clever prompt into a model, it produces something impressive, and everyone in the room gets excited. Then you try to put it in front of customers, and the questions start. Who is allowed to use this? What happens when it makes something up? How much did it cost us last month? Can we turn it off?
These are not prompt questions. They are engineering questions. And if you cannot answer them, your AI project is a prototype pretending to be a product.
The Demo Trap
A demo has to work once, for a friendly audience, on a happy path. A production system has to work every time, for strangers, on inputs you never imagined.
That gap is where the money goes. Teams spend weeks perfecting prompts, then discover the real work has not started. The model was never the hard part.
We see the same pattern repeatedly. A business builds an AI assistant that works beautifully in testing. Then someone asks it a question it should not answer, and it answers. Or the bill arrives and nobody can explain it. Or it breaks on a Friday and there is no way back to the version that worked.
The Eight Things Nobody Demos
Here is what actually decides whether your AI project survives contact with real users.
1. Authentication
Authentication means knowing who is talking to your system. It sounds obvious, but AI features are often bolted onto an existing app without it.
If your AI assistant can be reached without a login, it can be used by anyone who finds the URL. That includes people running up your model bill for fun.
2. Permissions
Permissions mean knowing what each user is allowed to see. This is where AI gets genuinely dangerous.
Traditional software checks permissions at the database. AI systems often skip that step. You connect a model to your company documents, and now it can quote a salary review to whoever asks nicely. The model has no idea some documents are private. It only knows what you gave it.
The rule is simple: the AI must only ever see what the person asking is allowed to see. Filter the data before it reaches the model, not after.
3. Audit
An audit trail is a record of what happened. Who asked what, what the system answered, and what it did as a result.
When a customer says "your AI told me the wrong price", you need to be able to look it up. Without a log, you are guessing. With a log, you have an answer in thirty seconds.
4. Observability
Observability means being able to see inside a running system. Not just "is it up", but "is it any good right now".
AI systems fail quietly. They do not crash. They just start giving worse answers, and nobody notices until customers complain. You need dashboards that track quality, latency and error rates, and alerts that fire before your users do.
5. Evals
An eval is a test for an AI system. You collect a set of real questions with known good answers, then check the model against them automatically.
This is the single biggest gap we see. Teams have unit tests for their code and nothing at all for their AI. So when they change a prompt or upgrade a model, they have no idea whether they made it better or worse. They just ship and hope.
Evals turn "it feels better" into a number. Without them, you cannot improve safely.
6. Token Budgets
Tokens are how AI usage is measured and billed. Roughly, a token is a short chunk of a word.
The cost problem is that AI spend scales with use, and use can spike without warning. One badly written loop can burn through a month of budget overnight. You need hard limits per user, per feature and per day, plus alerts when spend jumps.
7. Deployment
Deployment is how new versions reach your users. AI adds a wrinkle: prompts and models are configuration, not code, and they often bypass your normal release process.
If someone can change a live prompt by editing a field in an admin panel, you have an untested change in production. Prompts should go through the same review and release process as everything else.
8. Rollback
Rollback means going back to the last version that worked.
This is the one people skip, and it is the one that saves you. When a model update makes your assistant worse, you need to reverse it in minutes, not days. That means keeping old versions ready to run, not just old versions saved somewhere.
Demo Versus Production: What Changes
Concern | In the demo | In production |
|---|---|---|
Who can use it | You and your team | Anyone with a login, and anyone who finds the endpoint |
Data access | All of it, for convenience | Only what the individual user is permitted to see |
Failure | You try again | A customer gets a wrong answer and acts on it |
Cost | A few pounds | Scales with usage, and can spike without warning |
Quality checks | You eyeball the output | Automated evals on every change |
Going wrong | You edit the prompt | You roll back, then investigate |
What This Costs in the UK
Getting this right is not free, but it costs far less than getting it wrong. Rough guide prices for UK businesses:
Stage | Typical cost | What you get |
|---|---|---|
Proof of concept | £32000 to £4,000 | A working demo that proves the idea is viable |
Production foundation | £5,000 to £10,000 | Auth, permissions, audit, observability, evals and rollback |
Ongoing running costs | £200 to £1,000 per month | Model usage, monitoring and maintenance |
The gap between the first row and the second is exactly the gap that kills projects. Businesses budget for the demo and are then surprised by the system.
When You Do NOT Need All This
Not every AI project needs the full engineering treatment. Be honest about which one you are running.
You can safely skip most of it if all of these are true:
- Only internal staff will use it, and they already have access to the data involved
- Nothing it produces goes to a customer without a human reading it first
- Getting it wrong is embarrassing, not expensive or unsafe
- You could turn it off tomorrow and nobody would mind
A tool that helps your marketing team draft first drafts fits this description. Build it quickly, keep it simple, do not gold-plate it.
But the moment your AI touches customer data, makes decisions, or speaks to the public without a human in the loop, the eight items above stop being optional. That is not caution. It is the difference between a feature and a liability.
How to Tell Which One You Have
Ask your team these questions. If the answers are vague, you have a prototype.
- If a user asked our AI to reveal another customer's data, what stops it?
- A customer says our AI gave them the wrong answer last Tuesday. Can we find out what it said?
- We want to change the model. How do we know the new one is better?
- What is our AI spend this month, and what would happen if it went up ten times?
- The new version is worse. How long to get back to the old one?
Good answers are specific and fast. "We filter documents by user permission before they reach the model." "We have logs, here they are." "We run 200 evals on every change." Vague answers mean the work has not been done.
Why This Matters for Your Business
The uncomfortable truth is that prompt engineering is the cheapest, easiest and least durable part of an AI project. It is also the part everyone talks about.
The parts that decide whether your project succeeds are the same parts that have always decided whether software succeeds. Can you control who uses it? Can you tell what it did? Can you prove it works? Can you afford it? Can you fix it when it breaks?
These are not new questions. AI did not make them go away. It just made more people skip them.
At Original Objective, we build AI systems the way we build any production software: with the boring parts first. Our team has delivered systems for organisations including Bet365, Kohler Mira and Ticketmaster, where "it worked in the demo" was never going to be good enough.
This is the same lesson we wrote about in why most digital projects fail before a line of code is written. The failure happens early, in the things nobody thought to ask about.
If you have an AI demo that everyone loves and nobody can ship, that is a solvable problem. Book a discovery call and we will tell you honestly what stands between your prototype and production.
Because the best prompt in the world cannot save a system nobody can trust, audit, afford or switch off.
More in AI Production Systems
View all
Ready to put AI to work in your business?
Book a free 30-minute discovery call. We will discuss your goals, identify quick wins, and outline a practical plan to get started.
Book a discovery callCurated by Matt Perry
CTO
Most AI projects fail before anyone writes a prompt. They fail in the gap between a working demo and a system real people can safely use.
The demo is the easy part. You paste a clever prompt into a model, it produces something impressive, and everyone in the room gets excited. Then you try to put it in front of customers, and the questions start. Who is allowed to use this? What happens when it makes something up? How much did it cost us last month? Can we turn it off?
These are not prompt questions. They are engineering questions. And if you cannot answer them, your AI project is a prototype pretending to be a product.
The Demo Trap
A demo has to work once, for a friendly audience, on a happy path. A production system has to work every time, for strangers, on inputs you never imagined.
That gap is where the money goes. Teams spend weeks perfecting prompts, then discover the real work has not started. The model was never the hard part.
We see the same pattern repeatedly. A business builds an AI assistant that works beautifully in testing. Then someone asks it a question it should not answer, and it answers. Or the bill arrives and nobody can explain it. Or it breaks on a Friday and there is no way back to the version that worked.
The Eight Things Nobody Demos
Here is what actually decides whether your AI project survives contact with real users.
1. Authentication
Authentication means knowing who is talking to your system. It sounds obvious, but AI features are often bolted onto an existing app without it.
If your AI assistant can be reached without a login, it can be used by anyone who finds the URL. That includes people running up your model bill for fun.
2. Permissions
Permissions mean knowing what each user is allowed to see. This is where AI gets genuinely dangerous.
Traditional software checks permissions at the database. AI systems often skip that step. You connect a model to your company documents, and now it can quote a salary review to whoever asks nicely. The model has no idea some documents are private. It only knows what you gave it.
The rule is simple: the AI must only ever see what the person asking is allowed to see. Filter the data before it reaches the model, not after.
3. Audit
An audit trail is a record of what happened. Who asked what, what the system answered, and what it did as a result.
When a customer says "your AI told me the wrong price", you need to be able to look it up. Without a log, you are guessing. With a log, you have an answer in thirty seconds.
4. Observability
Observability means being able to see inside a running system. Not just "is it up", but "is it any good right now".
AI systems fail quietly. They do not crash. They just start giving worse answers, and nobody notices until customers complain. You need dashboards that track quality, latency and error rates, and alerts that fire before your users do.
5. Evals
An eval is a test for an AI system. You collect a set of real questions with known good answers, then check the model against them automatically.
This is the single biggest gap we see. Teams have unit tests for their code and nothing at all for their AI. So when they change a prompt or upgrade a model, they have no idea whether they made it better or worse. They just ship and hope.
Evals turn "it feels better" into a number. Without them, you cannot improve safely.
6. Token Budgets
Tokens are how AI usage is measured and billed. Roughly, a token is a short chunk of a word.
The cost problem is that AI spend scales with use, and use can spike without warning. One badly written loop can burn through a month of budget overnight. You need hard limits per user, per feature and per day, plus alerts when spend jumps.
7. Deployment
Deployment is how new versions reach your users. AI adds a wrinkle: prompts and models are configuration, not code, and they often bypass your normal release process.
If someone can change a live prompt by editing a field in an admin panel, you have an untested change in production. Prompts should go through the same review and release process as everything else.
8. Rollback
Rollback means going back to the last version that worked.
This is the one people skip, and it is the one that saves you. When a model update makes your assistant worse, you need to reverse it in minutes, not days. That means keeping old versions ready to run, not just old versions saved somewhere.
Demo Versus Production: What Changes
Concern | In the demo | In production |
|---|---|---|
Who can use it | You and your team | Anyone with a login, and anyone who finds the endpoint |
Data access | All of it, for convenience | Only what the individual user is permitted to see |
Failure | You try again | A customer gets a wrong answer and acts on it |
Cost | A few pounds | Scales with usage, and can spike without warning |
Quality checks | You eyeball the output | Automated evals on every change |
Going wrong | You edit the prompt | You roll back, then investigate |
What This Costs in the UK
Getting this right is not free, but it costs far less than getting it wrong. Rough guide prices for UK businesses:
Stage | Typical cost | What you get |
|---|---|---|
Proof of concept | £32000 to £4,000 | A working demo that proves the idea is viable |
Production foundation | £5,000 to £10,000 | Auth, permissions, audit, observability, evals and rollback |
Ongoing running costs | £200 to £1,000 per month | Model usage, monitoring and maintenance |
The gap between the first row and the second is exactly the gap that kills projects. Businesses budget for the demo and are then surprised by the system.
When You Do NOT Need All This
Not every AI project needs the full engineering treatment. Be honest about which one you are running.
You can safely skip most of it if all of these are true:
- Only internal staff will use it, and they already have access to the data involved
- Nothing it produces goes to a customer without a human reading it first
- Getting it wrong is embarrassing, not expensive or unsafe
- You could turn it off tomorrow and nobody would mind
A tool that helps your marketing team draft first drafts fits this description. Build it quickly, keep it simple, do not gold-plate it.
But the moment your AI touches customer data, makes decisions, or speaks to the public without a human in the loop, the eight items above stop being optional. That is not caution. It is the difference between a feature and a liability.
How to Tell Which One You Have
Ask your team these questions. If the answers are vague, you have a prototype.
- If a user asked our AI to reveal another customer's data, what stops it?
- A customer says our AI gave them the wrong answer last Tuesday. Can we find out what it said?
- We want to change the model. How do we know the new one is better?
- What is our AI spend this month, and what would happen if it went up ten times?
- The new version is worse. How long to get back to the old one?
Good answers are specific and fast. "We filter documents by user permission before they reach the model." "We have logs, here they are." "We run 200 evals on every change." Vague answers mean the work has not been done.
Why This Matters for Your Business
The uncomfortable truth is that prompt engineering is the cheapest, easiest and least durable part of an AI project. It is also the part everyone talks about.
The parts that decide whether your project succeeds are the same parts that have always decided whether software succeeds. Can you control who uses it? Can you tell what it did? Can you prove it works? Can you afford it? Can you fix it when it breaks?
These are not new questions. AI did not make them go away. It just made more people skip them.
At Original Objective, we build AI systems the way we build any production software: with the boring parts first. Our team has delivered systems for organisations including Bet365, Kohler Mira and Ticketmaster, where "it worked in the demo" was never going to be good enough.
This is the same lesson we wrote about in why most digital projects fail before a line of code is written. The failure happens early, in the things nobody thought to ask about.
If you have an AI demo that everyone loves and nobody can ship, that is a solvable problem. Book a discovery call and we will tell you honestly what stands between your prototype and production.
Because the best prompt in the world cannot save a system nobody can trust, audit, afford or switch off.
More in AI Production Systems
View allReady to put AI to work in your business?
Book a free 30-minute discovery call. We will discuss your goals, identify quick wins, and outline a practical plan to get started.
Book a discovery callFrequently Asked Questions
Why do most AI projects fail?
Most AI projects fail on engineering, not on prompts. The demo works, but the system around it is missing: authentication, permissions, audit logs, observability, evals, token budgets, deployment and rollback. Without these, an AI feature cannot safely reach real users, no matter how good the prompt is.
What is an eval, and why does my AI project need one?
An eval is an automated test for an AI system. You collect real questions with known good answers, then score the model against them. Evals matter because AI systems fail quietly: change a prompt or upgrade a model and quality can drop without any error appearing. Evals turn "it feels better" into a number you can trust.
How much does it cost to take an AI project into production in the UK?
A proof of concept typically costs £3,000 to £8,000. Building the production foundation, which covers authentication, permissions, audit, observability, evals and rollback, usually runs from £15,000 to £40,000. Ongoing running costs range from £200 to £2,000 per month depending on usage. Most budget overruns happen because businesses budget for the demo and not the system.
How do I stop an AI assistant leaking data it should not show?
Filter the data by user permission before it reaches the model, never after. A language model has no concept of which documents are private. It can only work with what you give it, so if you feed it everything, it can repeat anything. Apply your existing permission rules at the point where you retrieve data, so the AI only ever sees what that specific user is allowed to see.
Subscribe to the AI Growth Newsletter
Get weekly AI insights, tools, and success stories — straight to your inbox.
Here's what you'll get when you subscribe::
- AI for SMBs – adopt AI without big budgets or complex setup
- Future Trends – what's coming next and how to stay ahead
- How to Automate Your Processes – save time with workflows that run 24/7
- Customer Service AI – chatbots and agents that delight customers
- Voice AI Solutions – smarter calls and seamless accessibility
- AI News – how to stay ahead of the ever changing AI world
- Local Success Stories – how AI has changed business in the UK.
No spam. Just practical AI tips for growing your business.
Not sure if your app is production-ready?
Not sure if your app is production-ready?
Take the AI Readiness Quiz