Measure the return of an AI project before you build it, not after: it is the mistake we see repeated the most, and the most expensive one to fix once it has already happened. Everyone asks "is this AI worth it?", but almost nobody answers it well, because nobody defined ahead of time which number would answer it. Six months later, what is left is a feeling that "it's going well" or "it doesn't show that much", and a feeling is not a metric.
At AutoBoost we follow four steps, always in the same order, before we call the return on any AI project we build good. None of them needs a money figure: they are measured in time, error rate, or volume handled, which are numbers just as real as an amount, and in most businesses, easier to defend in front of a committee.
Step 1: pick ONE metric before writing the first line of code
The most common mistake is not measuring wrong, it is measuring late. If the project is already running by the time someone asks "how are we measuring this?", the answer is almost always a number invented after the fact to justify what was already built.
The decision rule we apply: if you cannot name the main metric in one sentence before you start, you are not ready to start. And it has to be a single one, not five at once, because five metrics at once almost always means none of them will actually get checked three months from now.
The three families we use most, ranked by how easy they are to defend:
- Time: how long a task takes today, and how long it should take. It is the easiest to explain to anyone in the company, technical or not.
- Error rate: how often the current process fails (a badly copied field, a number that does not add up, a notice that never arrives), out of how many times it runs.
- Volume handled: how much the current process can handle without someone having to review all of it by hand.
Step 2: measure the "before" with the same rigor you will demand of the "after"
This step gets skipped more often than it seems. It is tempting to launch the AI and only measure what happens from then on, but without a comparable "before", any "after" number is a figure without context.
A real, anonymized example: at an industrial distributor with over 50,000 products in its catalog, before building anything we measured how long a quote handled manually over WhatsApp took on average: checking the customer, the rate, the stock, and closing it. That average (around 30 minutes) was the starting point. Without that prior number, when the AI sales agent that today resolves the same quote in minutes went live (full case), there would be no way to say how much it had really improved, only that "it's faster now", which does not convince anyone who has to sign off on a budget.
How to measure the "before" without turning it into a project of its own:
- Pick a representative sample (a normal week, not the best one or the worst one).
- Time or count the current process exactly as it runs today, with its real interruptions.
- Also log the failures that happen in that sample, not just the time: a fast process that fails half the time is not better than a slow, reliable one.
Step 3: validate the number before you rely on it, not after you publish it
A fast AI and a correct AI are not the same thing, and the ROI only exists if both are true at once. The step that gets skipped most often under deadline pressure is this one: checking by hand, thoroughly, that the number the AI gives is the real number, before calling the whole process good.
At a data platform that unifies several offices of a group, the AI rebuilt a sales figure that the client's ERP had been calculating at almost half the real value. Before letting it act on its own recommending purchases, that reconstruction was validated against the real data down to a 0.0004% deviation (full case). That percentage, not a promise that "the AI is accurate", is what allowed it to move from "analyzes" to "decides and sends real orders to the distributor".
The decision rule here: do not validate once and trust forever. A process that gets validated at launch and never checked again quietly accumulates drift, just like data that goes missing without warning.
Step 4: measure in production, with real volume, not in the demo
A demo with ten well-chosen cases says nothing about what will happen with a real company's actual volume. The true ROI is measured once the process has been running for weeks with real data, not with the examples prepared for the pitch.
A useful case to think about volume: an accounting firm that received tax notifications from the Spanish tax agency's electronic mailbox (DEH) and needed to classify them into 15 different categories every day (full case). The number that actually matters there is not "it works in the test", it is how many real notifications, from a normal month's real volume, get classified correctly without anyone having to check the whole inbox "just in case".
The 4 steps, at a glance
| Step | Question it answers | Real example (anonymized) |
|---|---|---|
| 1. Pick ONE metric | What will we look at in 6 months to know if it worked? | Time per quote, not general "efficiency" |
| 2. Measure the before | How long did the current process take, or how often did it fail? | ~30 min per manual quote |
| 3. Validate the number | Is the AI's result correct, not just fast? | 0.0004% deviation validated to the cent |
| 4. Measure in production | Does it hold up at real volume, not just in the demo? | Classifying tax notifications into 15 categories, every day |
Checklist before calling your AI project's ROI good
- There is ONE main metric written down before building started, not invented afterward.
- There is a measurement of the "before" using the same representative sample that will be used for the "after".
- The AI's result has been validated by hand against reality, not just against itself.
- It has been measured with real production volume, not just the demo cases.
- Someone can summarize the result in one sentence with a real number in it, without adjectives ("faster", "more reliable") standing in for the data.
If after six weeks nobody in your company can complete that last sentence with a real number, your AI project does not have a measured ROI: it has a nice feeling. And a feeling does not convince anyone to expand the project, even if the result is genuinely good.
Why this is not just consultant theory
These four steps are not from a generic playbook: they are the same ones we apply on every custom software and AI project we build, because they are what let us, not just the client, know whether what we built actually works or just looks like it does. A project without a metric defined before it starts is not an AI project with uncertain ROI: it is a bet with a very good pitch deck.
If you are weighing an AI project and are not sure which of these four steps you are missing, let's talk: sometimes half an hour is enough to see whether the metric you were about to use is the one that actually matters.
