Recheck strategies: confirming a new status before alerting
A recheck is the second opinion a monitoring system asks for before it believes a status change, repeating the failed or newly successful check from other locations so one bad vantage point cannot open an incident on its own. HostTracker checks a site on the schedule set in its monitoring configuration, and a newly detected UP or DOWN is treated as a candidate until the recheck confirms it.
Why one failed check is not an outage
A check result describes one request, from one place, at one moment. Any of those three can be wrong about the site as a whole. The network between a single checkpoint and your server can be broken while every other route is fine. A DNS record can still be propagating in one region and already correct in the next. A checkpoint's own environment can be incompatible with something your server does.
If a monitoring system acted on every first result, it would page you for all of those and would also corrupt your uptime statistics, since the outage it recorded never happened for your users. So the detected status, UP or DOWN, is held as unverified until it is confirmed, and only then does anything downstream happen: an alert going out, an episode opening, the uptime number moving.
What a recheck catches
Rechecking from other locations is what separates a real, site-wide change from a local artifact. The cases it filters out most often are:
- Short downtime during a new deployment, where the site is back before a second check runs
- DNS unavailability while a configuration change propagates, so some resolvers answer and others do not
- A local blacklisting of the site or of the checkpoint's address, visible from one network only
- A checking agent whose environment is incompatible with the technologies the server uses
- Routing or peering trouble on the path between a single checkpoint and the server
The confirming checks are drawn from the geo region configured for the monitoring task, so the second opinion comes from the same part of the world your users are in rather than from wherever happens to be free.
The three recheck strategies
HostTracker supports three ways of deciding whether a new status is real.
1. Vote of the majority. When a check returns a new status, the system allocates up to 7 new locations from the monitoring task's geo region to confirm it. The returned results are grouped by status, and the larger group wins. Say a check from Miami reports DOWN, and rechecks from New York, Boston and Chicago come back UP, DOWN, UP. The DOWN is unconfirmed and discarded, and Miami is penalized: it will not be used for checks of that site. Run the same example the other way, with Miami reporting UP after a DOWN and New York and Chicago confirming UP, and the DOWN-UP transition is registered.
2. A set number of locations confirming DOWN. Here you configure the minimum number of rechecks needed to confirm a DOWN, while an UP still requires the majority vote. With that number set to 1, in the same example, Boston alone returning DOWN is enough to confirm Miami's DOWN, even though New York and Chicago disagreed.
3. No recheck. The new status is always confirmed as soon as it is detected. This assumes complete trust in the server and in the environments of the monitoring locations. It suits a few specific monitoring needs, and it should be chosen carefully, because every transient network problem at a single checkpoint becomes a false alert and a dent in the uptime figure.
Choosing a strategy
- Start with the majority vote. It is the balanced default: it costs a few seconds of extra confirmation time and removes nearly all single-checkpoint noise.
- Lower the DOWN confirmation threshold when a fast alert is worth more than a quiet inbox. A regional outage that only two locations can see still reaches you, at the cost of more false positives.
- Keep the majority vote for UP transitions whichever way you set DOWN. A premature UP closes an incident that is still running and hides the tail of the outage from your statistics.
- Reserve no-recheck for targets you control end to end, where you would rather investigate a spurious alert than wait for confirmation.
Whichever strategy you pick, you can reproduce a disagreement yourself from two different networks and compare what each one sees:
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://example.com/A 503 from one network and a 200 from another is the same pattern the recheck is designed to resolve. If the failing status is a 5xx rather than a timeout, the server answered, which is a different problem: see 503 Service Unavailable.
What confirmation gives you
Rechecks are what let a monitoring system run frequent checks from many places without turning that sensitivity into noise. HostTracker checks from 300+ checkpoints in 158 cities, so there are always independent locations available for the second opinion, and only a confirmed transition triggers an alert by email, SMS, voice call, Slack, Telegram and more. The strategies above are not the whole possible set, and support can consider a new one for a future release if your monitoring has a demand none of them covers. See distributed availability monitoring for how the checkpoint network is used, and the rest of the monitoring concepts section for the terms behind it.