How to avoid false positive alerts with multi-region checks

It's 3 a.m. and your phone buzzes: the website is "down". You open your laptop, the site loads instantly, and nobody else noticed a thing. It costs more than a broken night's sleep: each false alarm chips away at the trust you place in the next alert.

This guide covers where false positives come from, how multi-region confirmation works and which settings to review. It applies to any monitoring tool; along the way we point out what LucidStack does.

What a false positive is, and why it matters

A false positive is an alert raised while the service is working fine for its users. The probe really did record a failure, but that failure doesn't reflect reality: it came from the network path, the probe itself or a setting that's too strict.

One false alarm is harmless; a steady trickle is not. After a few pointless wake-ups, people ignore notifications and mute the channel. That's alert fatigue, and its real cost shows up on the day of a genuine outage, when the alert lands in a channel nobody reads any more. The monitoring chapter of the Google SRE book puts it well: an alert that pages a human should be urgent, actionable and real.

The usual suspects

Before changing anything, it helps to understand why a check fails when everything is actually fine.

On the probe and network side

• A transient network issue near the probe: congestion at the probe's hosting provider, packet loss on a transit link, a router restarting. Your server has nothing to do with it.
• A DNS resolver hiccup: the resolver the probe uses is slow or returns a temporary error. The request fails before it ever reaches your server.

On the monitored server side

• Rate limiting or a web application firewall (WAF): bot protection sees regular requests from the same addresses and eventually blocks them or serves a challenge (403, 429, a verification page).
• A timeout that's too tight: the page takes 4 seconds during a traffic peak, the probe gives up at 3. The site is slow, not down.
• A heavy page: a home page that aggregates ten internal calls depends on the slowest of them on every check.
• Redirects: an HTTP-to-HTTPS, language or login redirect changes the status code and sometimes the content. If the tool doesn't follow it, or you don't accept the code returned, every check fails.
• Maintenance: deployments, database upgrades and planned restarts cause real failures, but expected ones. That's not an outage anyone should be paged for.

The first two are solved by multi-region confirmation; the rest by sensible settings.

How outage confirmation works, step by step

The idea is simple: one vantage point isn't enough to declare an outage. Here's how LucidStack's multi-region monitoring does it, with probes in Paris, Amsterdam and Warsaw.

1. One region fails. For example, the check from Amsterdam times out.
2. The other regions recheck immediately. Paris and Warsaw run the same check straight away, without waiting for the next interval.
3. The majority decides. If at least two regions out of three fail, the outage is confirmed: an incident is opened and alerts go out.
4. Otherwise, the monitor is marked "degraded". If the other regions succeed, a failing majority is impossible: no outage alert is sent, but the anomaly stays visible.

In practice, a confirmation round looks like this:

03:12:04  Amsterdam  failed   timed out (30 s)
03:12:05  Paris      success  200 OK in 310 ms
03:12:05  Warsaw     success  200 OK in 420 ms
→ 1 failure out of 3: monitor "degraded", no outage alert

And if the confirmation regions don't answer? LucidStack then declares the outage only if the first region has failed several times in a row: an unreachable probe can't hide a real outage, and a single glitch can't raise an alert.

Recovery: avoiding flapping

False positives don't only happen at the start of an incident. An unstable service can alternate between failures and successes and produce a burst of "down" and "recovered" notifications: that's flapping.

The fix is to require several consecutive successes before declaring a recovery. In LucidStack, a service is only marked as recovered after several successful checks in a row, two by default. A lone success doesn't close the incident.

Tuning your monitors

Confirmation takes care of network noise. Everything else comes down to configuration.

Timeouts

Your timeout should reflect what's unacceptable for users, not your typical response time. LucidStack defaults to 30 seconds; cutting it to 5 seconds turns every load spike into an outage. Handle slowness separately.

Accepted status codes

An API returning 204, a page behind authentication answering 401, a deliberate 301 redirect: none of these are outages. Declare the codes you expect explicitly rather than settling for "any 2xx". Where possible, point the monitor at the final URL so it doesn't depend on a redirect.

A slowness threshold instead of downtime

A slow site deserves attention, but not a downtime alert. With a slowness threshold, the service switches to "degraded" above a given response time, without a downtime alert. Per-region response time monitoring, the 95th percentile and anomaly detection show you the trend without paging whoever is on call.

Keyword checks

Keyword monitoring catches pages that return 200 but display an error. Pick stable text: a footer element or a structural heading, not this week's promotion, a sentence translated according to the visitor's language, or a block under an A/B test. For text that must never appear ("Fatal error", "Database connection"), be specific so you don't trip over a blog post that happens to contain those words.

Allow-listing the probes

If a firewall, WAF or CDN filters traffic, add the probes to its allow-list. LucidStack's IP addresses are published on the LucidStackBot page, and requests carry the LucidStackBot user agent. To check whether probes are being blocked, look for them in your logs:

grep "LucidStackBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

A run of 403s or 429s points to your protection layer, not to an outage.

Maintenance windows

Schedule planned work inside a maintenance window. During the window, failing checks don't trigger alerts, and the period is excluded from uptime.

Quiet hours for what can wait

Not everything needs to arrive at 3 a.m. With LucidStack's quiet hours, non-urgent warnings (certificate reminders, degraded services) wait until the end of the quiet period, while outages and recoveries always get through. That lets you keep a demanding alerting policy without sacrificing the team's nights.

Checklist: symptom and fix

Wrapping up

False positives are rarely inevitable. Multi-region confirmation deals with network and probe issues; realistic settings, allow-listed probes, maintenance windows and quiet hours deal with the rest.

LucidStack applies these principles out of the box: three regions included in every plan, Free included, outages declared only by majority, and recoveries confirmed by several successes. If you want alerts you can actually believe, that's a good place to start.