403 Forbidden: Causes, Fixes and Bot Blocks That Fool Monitors

403 versus 401

RFC 9110 draws the line clearly. A 401 Unauthorized means authentication is required or the credentials supplied were not accepted, and the response must carry a WWW-Authenticate header describing how to authenticate. A 403 Forbidden means the server refuses to authorize the request, and re-authenticating will not help.

The spec also allows a server to answer 404 instead of 403 when it does not want to reveal that a resource exists at all. That is why a URL you know is there can come back "not found" from a hardened server. If you are triaging a 403, remember that the reverse case also exists.

What produces a 403

• Filesystem permissions. The web server process cannot read the file or traverse the directory. A classic after a manual file copy or a restore that changed ownership.
• Directory listing disabled. A request for a folder with no index file returns 403 on servers configured to refuse listings. Correct behavior, confusing error.
• Application authorization rules. The user is logged in but lacks the role, plan or tenant membership required for that resource.
• IP allow and deny lists. An admin path restricted to office addresses returns 403 for everyone else, including anyone testing from home or from a mobile network.
• Web application firewall rules. A WAF blocks a request whose URL, headers or body matched a signature. Often the request looks entirely innocent to you.
• Geo-blocking. Traffic from some countries or regions is refused at the edge, so the site is fine from one continent and forbidden from another.
• Hotlink protection. A rule that rejects requests for images or media whose Referer header is missing or points at another domain. Requests with no referrer are often caught by accident.
• Bot and user agent filtering. The request is blocked because of who the client claims to be, not what it asked for. This is the case that catches monitoring.

The "403 for monitors, 200 for humans" case

This is the most common false alarm in uptime monitoring, and it is not a monitoring bug. The site is genuinely returning 403, just not to browsers.

Bot management at a CDN or WAF scores each request on its user agent, TLS fingerprint, header ordering, IP reputation and whether it solves a JavaScript or interactive challenge. An automated HTTP check is, by design, an automated client: it does not run JavaScript challenges and it identifies itself honestly. So it gets challenged or blocked while your browser sails through, and the result is a site that alerts as down while every human confirms it is up.

Two signs identify it quickly: the 403 arrives instantly rather than after a delay, which rules out overload, and it is consistent for the automated client while never reproducible in a browser on the same network.

How to allowlist a monitoring service

1. Confirm the block is identity-based. Repeat the request as a plain HTTP client and then with a browser user agent string, and compare:

   curl -sS -o /dev/null -w "%{http_code}\n" https://example.com/
   curl -sS -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://example.com/

   If the codes differ, a user agent rule is doing it.
2. Find the rule that fired. WAF and CDN dashboards log blocked requests with the matching rule id. That is faster and safer than guessing which rule to relax.
3. Allowlist by IP address where you can. Monitoring providers publish the addresses their checkpoints use. Adding them as a skip rule is precise and does not weaken protection for anyone else.
4. Or allowlist by user agent. Configure the check to send a distinctive user agent and add a rule that exempts it. This is easier to maintain than an IP list, but treat it as a convenience rather than a security boundary, since a user agent can be copied by anyone.
5. Prefer a skip rule over disabling protection. Exempt the monitoring path or client from the specific rule, and leave the rest of the ruleset in force.
6. Re-test from several regions. A rule can be applied unevenly across edge locations, so verify the fix from more than one place before closing the ticket.

Fixing a 403 that is not about bots

1. Check whether the URL should be public at all. Half of 403 investigations end with the answer that the block is correct.
2. Verify file ownership and permissions on the path and every parent directory. The server process needs execute permission on directories, not only read permission on the file.
3. Look for an index file if the URL is a directory, or enable a listing deliberately if one is wanted.
4. Read the server configuration for deny rules, IP restrictions and referrer checks on that path, then the application's authorization layer.
5. Check the edge before the origin. Request the origin directly. A 403 that disappears when you bypass the CDN is an edge rule, not an application problem.

Seeing the block from outside your own network

A 403 introduced by a WAF update or a permissions change does not look like an outage from the inside: your session is authenticated, your address is allowlisted, and the pages you check are the ones that work. A scheduled HTTP check asserts the code you expect on the URLs that matter, from outside your network, so a newly forbidden page raises an alert immediately. Geo-blocks and edge rules vary by region, so the number of vantage points matters: HostTracker checks from 300+ checkpoints in 158 cities, which turns "forbidden in some countries" into a visible difference between locations. If the refusal is temporary and tied to request volume rather than identity, the code will usually be 429 instead, covered in the 429 Too Many Requests guide, and the wider family is in the 4xx overview.