Your AI can have a perfectly sensible noise filter and still send you 40 emails in a single night over a bug that never actually changed. Not because the filter is badly designed, but because it was built for a different kind of failure than the one that showed up. It happened to us recently with a system that watches the connection of several pharmacies, and it's worth telling because it's an easy mistake to make in any alerting system, AI-driven or not.
The filter that worked fine, until it didn't
The system monitors the connection of several pharmacies that share the same data platform, and it has a common-sense rule to avoid crying wolf: if it detects slowness, it doesn't alert on the first read. It waits for two consecutive check cycles before escalating. And if the connection is back to normal on the next cycle, it drops the alert without further fuss.
That rule works beautifully against the exact noise it was designed to filter: a blip of a couple of seconds, the kind any network can throw at you and that means nothing. Two cycles of margin are plenty to avoid raising an alarm over that.
The problem showed up with a different kind of failure: a connection running slow in a sustained way, not fully down but degraded for minutes, and fluctuating within that degradation. It flipped between "running badly" and "just caught up for a moment" roughly every 25 minutes, faster than the two-cycle window built to absorb a few-second blip. The result: the filter would see "fixed," drop the alert, and a few minutes later see it broken again and send it out. All night long.
The number: 40 emails, one single bug
Reviewed the next morning, the count was clear: close to 40 emails in a single night, alternating between alert and resolved, over exactly the same connection issue. It wasn't 40 separate incidents. It was one, seen 40 times from angles that looked new each time because the filter compared against the previous cycle, not against whether the underlying problem was still there.
None of this affected the actual service to the pharmacies: the data platform that unifies their information, and that the AI uses to recommend purchases and validate sales down to the cent, kept working the whole time. The cost wasn't operational, it was about attention: 40 emails about the same issue train anyone to stop opening them, which is exactly what you don't want the day one of those emails is actually new.
Why a well-designed filter can fail the same way
Here's the part that generalizes to any alerting system: an "wait N repeats before escalating" rule doesn't filter noise in general, it filters the noise it was calibrated for. If the real failure changes rhythm faster than the chosen window, the window stops protecting you from anything and starts generating the exact noise it was meant to prevent.
| Type of failure | Typical duration | What the filter needs |
|---|---|---|
| One-off blip (a network spike, a stray timeout) | Seconds | Waiting 1-2 cycles is enough to ignore it |
| Sustained, stable degradation (something goes wrong and stays that way) | Minutes to hours, unchanging | The "wait N repeats" filter still works fine |
| Sustained degradation that fluctuates (goes bad, briefly recovers, goes bad again) | Minutes, changing state repeatedly | The repeat window isn't enough; you need to group by the underlying failure, not by each reading |
The middle row is the trap: a failure lasting minutes looks, at first glance, like it should be covered by "wait two cycles." And it is, as long as it doesn't fluctuate. Once it fluctuates faster than the window, the same filter that protects you from a second-long blip starts treating every oscillation as a brand-new event.
The decision rule
Two ideas, applicable to any alerting system that uses a "wait N repeats" window before escalating, whether or not an AI is deciding on top of it:
- Know the real rhythm of your failures before you set the window. Picking "two cycles" because it sounds reasonable isn't enough; you need to know whether a typical failure lasts seconds, minutes, or flips state several times within that stretch. Calibrate the window against that, not against intuition.
- A failure that oscillates isn't a new failure every time it changes state. If the system only compares against the previous cycle, it treats every oscillation as a fresh event. If it compares against whether the underlying problem is still unresolved, it groups those 40 readings into a single alert: "this has been unstable since such and such time," not 40 separate emails.
Checklist before you trust your own alerting system
If your business depends on a dashboard, a Slack channel, or an AI agent deciding when to interrupt you, before you trust the noise filter it has in place:
- Do you know how long a real blip typically lasts on your systems? Seconds, minutes, hours?
- Is your "wait N repeats" window slower or faster than that real rhythm?
- Does the system group a single fluctuating failure, or does it generate a new alert every time the state flips?
- Has anyone tested what happens when a failure lasts exactly twice as long as your filter window?
- How many duplicate emails about the same thing would it take before you stopped reading them?
That last question matters most. If the answer is "not many," that's your signal the filter needs revisiting before the night it actually matters comes around.
Designing an alerting system that tells real signal from noise that only changes shape is exactly the kind of technical work we do at AutoBoost, inside data and AI platforms already running on real businesses. If you want to review how your process monitoring is set up, or you're considering an AI that decides based on that state, check out our services or get in touch and we'll take a look together.
