Your AI can have the urgency of every alert perfectly calculated and still let a real problem sit unresolved for days. Not because the calculation is wrong, but because the alert reaches the wrong place, or reaches the right place and nobody is responsible for looking at it. This happened to us recently with an alerting system that watches several clients at once, and the fix ended up being a design rule that applies to any business using AI to watch something and alert on what it finds.
The starting problem: one channel for everything
The system sent every alert through the same channel, and decided the medium (interrupt now or leave it for a calm review) by looking at what dominated the content of the message. The practical result: an email with several minor mismatches could carry a real incident inside it, say a dropped connection, and that urgent alert stayed buried in a message meant to be read without any rush.
The failure was not in detecting the problem. Detection worked fine. It was in how the alert got packaged and who it reached, which is exactly the part that gets the least scrutiny when designing an AI alerting system: all the attention goes to "what does it detect", and almost none to "how does it decide to interrupt someone with it".
First fix: split by urgency, not by subject
The initial solution was simple to describe and not so simple to get right: two independent channels instead of one. A real incident goes through a channel that interrupts. A mismatch that can wait for a calm review goes through one that does not. They no longer compete for the same message or the same attention.
The part that actually matters is not having two channels, it is how the system decides which one each alert uses. That routing logic was pulled out into its own function, separate from the main process, so it could be tested on its own with concrete cases ("this is a real incident, it goes here"; "this is a mismatch, it goes there") instead of trusting that the system's overall behavior would cover it by accident. This is a rule that applies to any decision logic inside an AI workflow: if it decides something important, it needs to be testable on its own, without spinning up the whole process around it.
Second fix: urgency also depends on who receives it
A few days later came the twist that actually changes the rule: an alert's urgency does not depend only on how severe the problem is, it also depends on what stage the affected client is in. A client already operating live needs to be interrupted immediately. A client still in testing can wait for a channel that gets reviewed calmly, even if the technical problem is identical.
It is a nuance that is easy to skip because it looks like it adds unnecessary complexity. In practice it is the opposite: without it, every alert from a client in testing sounds exactly as loud as one from a client in production, and that trains whoever receives it to lower their guard with everything equally, including the alerts that actually matter.
At the same time, a gap that usually goes unnoticed was closed: if the check itself stops working, it now alerts too. Before, a silent failure of the check looked exactly like "everything is fine", which is the worst place for a monitoring system to fail: in its own blind spot.
The number that put the rule on the table: 3 unowned cases
With the two channels already split, a different and less obvious problem showed up: 3 low-urgency cases had been waiting for review in their channel, with nobody assigned to them. They were not lost, or misclassified. They were exactly where they were supposed to be. There simply was no specific person checking that channel with the regularity that a "whenever there's a moment" queue needs.
That is the part almost nobody designs on purpose: a low-urgency channel without an explicit owner does not empty itself, it tends to accumulate. Not because nobody wants to handle it, but because nobody has the responsibility of clearing it assigned, and the urgent things elsewhere always win the fight for the day's attention. A channel like that, left unreviewed, ends up becoming the place where alerts go to be unseen, which is exactly the opposite of what splitting it from the urgent channel was meant to achieve.
The decision rule
Two ideas, applicable to any business using AI to watch something (an integration, several locations, several clients) and alert on what it finds:
- An alert's urgency depends on who receives it, not only on what failed. The same technical problem can be urgent for one client and expected for another, depending on their stage or real exposure.
- A low-urgency channel with no assigned owner becomes a graveyard for alerts. Splitting by urgency solves the noise, but it creates a new channel that, if nobody has it assigned as a task, accumulates without anyone noticing until someone asks about a specific case.
Checklist before you trust your AI alerting system
| Question | Why it matters |
|---|---|
| Do urgent alerts and ones that can wait go through separate channels? | If they share a channel, one hides the other. |
| Is the logic that picks the channel in a function that can be tested on its own? | If it lives buried inside the main process, nobody validates it against concrete cases. |
| Does urgency depend on who receives the alert, or only on technical severity? | The same failure can weigh differently depending on the client or their stage. |
| Does someone, by name, own clearing the low-urgency channel? | Without an owner, that channel accumulates cases nobody claims. |
| What happens if the check itself stops working? | A check that goes silent looks exactly like "everything is fine", the worst blind spot possible. |
Why this is not just "our own plumbing"
This pattern is not unique to an internal monitoring system. It shows up in any AI process that decides who to alert and how: an assistant that reads invoices and flags mismatches, an agent that watches stock across several stores, an AI that reviews tax notices and decides which ones need an immediate response. In our work with an accounting firm, we built an AI that reads the tax notifications inbox and classifies it into 15 categories: the part that actually makes the difference is not just classifying correctly, it is that each category has a clear owner and a clear urgency, or the perfect classification is useless if nobody knows it is theirs to check.
The lesson repeats in any system where an AI decides "this needs looking at now" versus "this can wait": designing who gets alerted and how weighs as much as designing what gets detected, and it usually gets a fraction of the attention.
If you are building or reviewing your own AI alerting system
Before calling an AI project that watches something (an integration, a warehouse, several locations, several clients) finished, ask the question backwards from how it is usually asked. Do not just ask "what does it detect?" Ask "who receives each type of alert, with what urgency, and who is responsible for making sure none of them go unreviewed?" If you do not have a concrete answer for all three parts of that question, your alerting system is not finished yet, even if the detection side works perfectly.
At AutoBoost we work with clients who need exactly this: not just an AI that detects what matters, but an alert that reaches the person who needs to act, with the right urgency and an assigned owner. If you want to review how yours is designed, let's talk.
