Our AI watchdog's own test suite had been saying "all green" for a while, and that was only half true: it was only running 6 of the 13 tests that actually existed. The other 7 were written, sitting in the repository, and had never run once. Nobody noticed until we audited the very mechanism that decides what gets tested. Here's what happened and the rule we took away, because the same gap can be hiding today in any system, AI or not, that depends on a manually maintained list.
The problem: the green light was watching less than it claimed to
The system in question monitors the health of several billing integrations for a group of clients: it checks that documents arrive, get processed, and that the amounts add up. Like any piece of software that makes decisions on its own, it has its own automated test suite to make sure it keeps doing its job correctly every time something changes.
The process that ran that suite called each test by name, from a list typed by hand into the code itself. That worked perfectly on the day the list was written. The problem showed up afterward, every time someone added a new test (to cover a case that had just failed, for example) and either forgot to add it to that list, or assumed the system would pick it up on its own.
It didn't. The suite kept running the exact same 6 tests it had been running from day one, marking the result green, while 7 new tests sat asleep in the repository, never executed. And a green light that never changes reads as "this is still covered," not as "this hasn't actually been checked in a while."
Why "green" and "covered" aren't the same thing
This is the underlying mistake, and it applies to any business with some form of automated check, whether that's an AI, a script, or a scheduled report:
| What green seems to mean | What it actually means |
|---|---|
| "Everything that needs checking is fine" | "Everything that actually ran is fine" |
| "The system will notice if something new breaks" | "The system will notice if something it already tests breaks" |
| "If I add a rule, it's covered automatically" | "If I add a rule and don't wire it in, it doesn't exist for the test suite" |
The trap is that a test suite that fails is loud: it turns red, someone looks at it. A test suite that never gets around to running something is silent, because from its own point of view nothing unusual has happened: it ran the 6 tests on its list, and all 6 passed. The failure isn't in any single test. It's in the mechanism that decides which tests exist in the first place.
The decision rule
If you take one idea from this post, make it this one: any check that depends on a manually maintained list has, by design, a silent expiration date. It works perfectly on day one and falls a little further behind every time the real system it watches grows, until someone audits it and finds out how far behind it actually got.
The fix isn't "remember to update the list better." That was already tried, informally (nobody decided to leave 7 tests unrun; it just drifted that way over time) and it will happen again. The fix is making the mechanism itself discover what tests exist, instead of relying on a human to write it down somewhere else. In this case: we changed the process to scan the codebase and pick up every new test automatically, so a written test and an executed test are now, by construction, the same thing.
How we verified it, not just how we fixed it
Fixing the mechanism isn't the same as knowing it works. Before calling the change done, we did two things:
- We counted how many tests ran before and after the fix (6 and 13), instead of trusting that "it should be more now." A number is the one thing nobody can argue with.
- We wrote a new test on purpose and confirmed it showed up on its own in the next run, without touching any list. If the mechanism we just fixed can't pass this check, it isn't fixed, it just looks fixed.
Checklist: audit your own "green"
This isn't only an AI problem, or a technical team's problem. Any business that trusts something to "check itself" (an automated report, a dashboard, a quality checklist, an AI monitoring a process) can have the same gap. Before trusting any green light:
- Do you know how many checks exist today, or do you just know that "there are checks"? If you can't put a number on it, you can't tell if the number dropped.
- Does the system discover on its own what it needs to check, or does it depend on a list someone maintains by hand? Every hand-maintained list has an expiration date.
- The last time someone added a new rule, did they confirm it actually ran, or did they just assume it did?
- How long has the number of checks stayed flat, even as the system it watches kept growing? A frozen number while the business grows is the exact same signal we saw here.
- If an outsider (a client, an auditor) asked "what exactly are you checking?", would you have an answer with a number, or an answer that's "in general, everything"?
Why this matters more when the watchdog is an AI
When a check is done by a person, a gap like this tends to surface sooner: someone eyeballs things occasionally and asks "is this covered too?" When the check is done by an AI system, that human eye disappears by design, which is exactly what makes the AI valuable the rest of the time. The trade-off is that if the mechanism deciding what the AI watches has a gap in it, nobody is going to stumble on it by accident. It has to be audited on purpose, the way we did here.
It's the same principle behind the AI data platform for a group of pharmacies, where before letting the AI decide anything over more than a million lines of data, you need to be able to prove, with a number, how much of that data is actually being checked. And it's the same care we put into invoices that book themselves for a veterinary group: it's not enough for the process to work, you need to be able to prove it keeps working every time something changes.
The takeaway for any small business
You don't need an engineering team to apply this, just one uncomfortable question: of everything you believe your system (AI or not) is checking for you, how much of it is actually being checked today, and how much has been sitting on a list nobody has looked at in a while? The gap between those two answers is exactly where our 7 missing tests were living.
If you have automated processes or AI agents in your business and you're not entirely sure how much of what they're "supposed to" check is actually being checked, let's talk. At AutoBoost we build custom software and AI integrated into your real systems, with proof that it keeps working built in from day one. Get in touch and we'll take a look together.
