Building an AI assistant over your manuals sounds simple until you actually try it. You upload the PDFs, connect a model, ask a test question, and the answer looks perfect. The problem shows up later, when someone on your team asks something the manual doesn't cover and the assistant, instead of saying "I don't know," improvises an answer that sounds just as confident as the rest. We already covered what RAG is and when it actually works; this post is about the how: the concrete steps we follow at AutoBoost to make an assistant like this work in production, not just in the demo.
The first filter isn't technical: which documents go in
Before touching a single line of code there is a decision the business has to make, not the developer: which manuals the assistant will be allowed to read. Not everything sitting in a shared folder needs to go in.
- Process and operational documentation: usage manuals, protocols, technical sheets. This is most of what adds value and carries the least risk if it ever leaks.
- Formulas, recipes or processes that are a competitive advantage: these can go in, but with extra access controls (who can ask, what the assistant is allowed to quote back).
- Personal data about employees or clients: only goes in if there's a clear legal basis and you know exactly where it's processed, because GDPR doesn't care whether a person or a model is the one reading it.
This filter decides half the project's architecture. If your manuals are purely operational, a standard assistant is enough. If they hold trade secrets or personal data, the next decision (where the models run) stops being optional.
Where the models run, before writing anything
In a real project we built for a food industry company, the assistant answers questions about production processes and formulas using language models that run in Spain, so that knowledge never leaves toward an outside provider. That wasn't a marketing choice: it was the answer to a very specific question that any sector with product secrets or regulated data has to ask before starting. We cover the full criteria here.
If your business doesn't handle secrets or regulated data, a generic cloud model may be enough, and cheaper. If it does, decide this first: where the data lives while the model processes it, and whether it ever sits outside your control. Changing it after the system is built costs far more than deciding it upfront.
The step that actually decides if it works: teaching it to say "I don't know"
This is where almost everything is won or lost. An assistant that always answers, even when it doesn't have the data, is worse than no assistant at all: it hands out confidence where none is warranted. The rule we apply is simple to state and demanding to meet: if the answer isn't anchored in a real passage from the manual, the assistant has to say it doesn't know, rather than approximate a plausible-sounding answer.
Signs that an assistant is making things up instead of answering:
- It answers with the same level of detail for a heavily documented topic and for one that barely exists in the manuals.
- It never says "I don't have that information" or "I can't find this in the documentation."
- It doesn't point to which document the answer came from, so nobody can verify it without searching by hand.
- It gives a different answer to the same question depending on how it's phrased, instead of the same information in different words.
If your assistant does any of these four things, it isn't ready yet for people on the floor to trust it.
How to test it before handing it to your team
The easy test (two or three obvious questions you already know the manual answers well) is worthless: it only confirms what you already knew. The test that matters is the opposite one.
| Type of question | What it should answer | What it should NOT answer |
|---|---|---|
| Clear and well documented | The correct answer, citing the manual | A vague answer, or a different one each time |
| Not in any manual | That it doesn't know | A made-up but coherent-sounding answer |
| In an old manual that's since been replaced | The current version, not the old one | Mixing both versions without flagging it |
| Ambiguous, or open to two readings | Asks which case you mean | Picks one reading at random and doesn't say so |
Before rolling it out to your team, test it with questions from the last row and the second one: those are what separate an assistant ready for production from one that only works in the demo. If it fails there, the problem is almost never the model itself; it's usually how the document that should hold the answer was chunked and organized.
Maintenance: the day your manuals change
An AI assistant over documentation isn't a project you deliver and forget. The day a manual gets updated, if the assistant keeps reading the old version, it starts giving wrong answers with the exact same confidence as the correct ones, and nobody notices until someone follows an instruction that's no longer current. The fix isn't exotic: every manual update has to trigger an update of the index the assistant uses, instead of waiting on someone to remember to do it by hand.
The decision rule
If your assistant can't say "I don't know" when it should, and if you can't see which document each answer came from, it isn't ready for production, no matter how good it sounds in the demo. And if your manuals hold trade secrets or regulated data, decide where the models run before writing the first line of code, not after.
This is exactly the approach we follow at AutoBoost when we build custom software and AI: AI goes where it adds value, but only after the data is put in order and these questions are answered first. If you have manuals, protocols or technical documentation your team struggles to find, or that today only lives in two people's heads, let's talk and see if it makes sense for your case.
