Your AI monitoring system can have perfectly sound logic and still be training your team to stop reading its emails. It happened to us recently with a system that watches the health of several integrations: as soon as it caught a problem twice in a row, it treated it as real and fired an alert. It sounded reasonable. Once we put numbers on it, it stopped sounding that way.
The problem: alerting too soon trains people to stop listening
The original rule was simple: if a check fails for two consecutive cycles, it is a real problem and it is worth alerting on. And if the next cycle comes back clean, it gets marked resolved and another email goes out. On paper, two consecutive checks seem like enough margin to rule out a one-off glitch.
In practice, they were not. Many of the problems this kind of system detects (a connection that is slow to respond, a service that restarts on its own, a queue that empties a few minutes late) fix themselves without anyone touching anything, simply because the monitored system recovers on its own. The "broken" email and the "resolved" email still went out, even though nobody had done a thing between the two.
What came out of measuring 40 real cases
Instead of tweaking the rule by feel, we went back through the last batch of "broken" / "resolved" alert pairs the system had sent: about 40 in total. The result was clear: all but 3 had closed themselves within an hour, with resolution times ranging from 10 to 70 minutes. In other words, out of every 10 alarms, less than one was a problem that actually needed someone to step in. The other nine were noise wearing the shape of urgency.
That is the real cost of alerting too soon, and it is not just the time someone loses opening an email. It is the habit it builds: if 90% of alarms turn out to be nothing, the next alarm (the real one, the one that does need action right now) gets read with the same shrug as the previous 39 that weren't. Once that happens, the alerting system has stopped doing its job, even though it is technically still "working" and still sending emails.
Before and after measuring
| Old rule (no measurement) | New rule (data-driven) | |
|---|---|---|
| When it alerts "broken" | On the second consecutive failed check | After one continuous hour of failure |
| When it alerts "resolved" | As soon as one check comes back clean | After thirty continuous minutes without failure |
| If the problem comes back before those 30 minutes | Treated as a new case, new alert | Treated as the same incident, no repeat alert |
| Real alarms out of the total measured | Not measured (it alerted on everything) | Roughly 1 in 10 |
The new rule: one hour present, thirty minutes absent
With the data from those 40 cases in hand, the rule changed from the ground up. Now a problem has to be continuously present for a full hour before it triggers a "broken" alert. And to be marked "resolved," it has to stay continuously absent for thirty minutes. If it reappears before those thirty minutes are up, it is not treated as a new incident: it is the same one as before, and it does not get announced again.
The difference from the old rule is not just the timing, it is where the number comes from. Before, "two consecutive checks" was a figure picked by feel, without checking how long real problems actually took to resolve. Now, the hour and the thirty minutes come from looking at how long the 90% of cases that turned out to be noise actually took to clear on their own, and setting the threshold above that. A problem that resolves itself in 70 minutes no longer triggers anything; one that is still there after ninety minutes does.
Why this isn't just a problem for us
Any small or midsize business running automated alerts (on an ERP, an order, a payment, a website going down) faces the same dilemma, whether they have measured it yet or not. The temptation when setting up any alerting system is to pick a threshold that "sounds reasonable" and alert as fast as possible, because that feels like the safe choice. The problem is that "as fast as possible" and "as fast as you can still trust the alarm" are not the same thing, and the only way to tell them apart is by measuring how long the problems that turned out to be nothing actually took to resolve themselves.
We ran into something similar before, when a flickering connection triggered 40 emails in a single night about the exact same failure, with nothing new to fix. The debounce window covered one kind of failure but not another. The underlying lesson is the same in both cases: an alert threshold that has never been measured against how the monitored system actually behaves is, at best, a guess wearing the shape of a setting.
Checklist before you set your own alert delay
- Pull your last 20 to 30 "broken" / "resolved" alert pairs. How long did each one take to resolve from the moment it appeared?
- Work out what share resolved on their own, with nobody stepping in between the "broken" alert and the "resolved" one.
- Set your alert threshold above the typical resolution time of the noise, not below it. If 90% clears up in under an hour, alerting after two minutes doesn't add safety, it adds emails.
- Keep the "alert" threshold and the "resolved" threshold separate. They don't have to match, and in our case they don't (one hour versus thirty minutes).
- Decide what happens if the problem comes back right after being marked resolved. Treating it as new doubles up alerts on the same incident; treating it as a continuation avoids the flicker effect.
Designing monitoring systems that alert on what actually matters, and only on that, is the same kind of work we do inside the data and AI platform that unifies information across several pharmacies and decides on its own when to recommend a purchase. If you recognize this problem, the alarms nobody looks at anymore, check out our services or get in touch and we'll take a look together.
