How to check robots.txt for blocked pages
Open https://yourdomain.com/robots.txt in a browser: whatever is served there is your robots.txt, and it controls what crawlers are allowed to fetch. It does not control what gets indexed, which is the single most consequential thing to understand about the file before editing it.
Where robots.txt has to live
It must sit at the root of the host, as /robots.txt, in plain UTF-8 text. It applies to exactly one scheme, host and port, so https://example.com/robots.txt governs nothing on https://shop.example.com and nothing on the http version of the same host. Every subdomain you want to control needs its own file.
The status code the file returns matters as much as its contents. A 404 is read as "no restrictions, crawl everything", which is a perfectly valid state. A repeated 5xx is read as a temporary failure and Google may pause crawling the host rather than assume it is allowed to continue. A 403 behaves like a server error for this purpose. If robots.txt is unreliable, crawling becomes unreliable with it.
The syntax, in five lines
User-agent: *
Disallow: /admin/
Allow: /admin/public-page.html
Disallow: /*?sessionid=
Sitemap: https://example.com/sitemap.xmlUser-agent opens a group. Disallow blocks a path prefix, Allow carves an exception out of a block, * matches any run of characters and $ anchors the end of a URL. Sitemap is independent of any group and takes an absolute URL. The format is standardized as RFC 9309, so the behavior described here is not vendor-specific.
Two resolution rules decide what actually happens. A crawler uses only the most specific group that names it, and ignores the * group entirely once a group names it directly. Within that group, the longest matching rule wins, and a tie goes to Allow.
The mistakes that block a whole site
Disallow: /left over from a staging deploy. One character, the entire site blocked. It ships in exactly the way a staging noindex ships: a config file promoted with a release.- Blocking CSS and JavaScript. Google renders pages, so a blocked asset directory can make a page look broken to the crawler that looks fine to you.
- Expecting robots.txt to remove a page from search. It cannot. A blocked URL can still be listed from external links, usually without a snippet, and because the crawler cannot fetch it, a noindex on that page is never seen. To remove a page, allow the crawl and serve noindex.
- Using
noindex:as a directive inside robots.txt. It was never part of the standard and Google stopped honoring it on 1 September 2019. It is silently ignored today. - Adding a group for a named bot and expecting the wildcard rules to still apply. They do not. Once
User-agent: Googlebotexists, Googlebot reads that group and nothing else, so every shared rule has to be repeated in it. - A blank line inside a group. A blank line ends the group, so the rules after it belong to nothing until the next
User-agentline. - Listing private paths. The file is public. Naming
/internal-admin/in it publishes the location of the thing you are trying to hide.
How to test a single path
Fetch the file and read the group that applies to you first:
curl -s https://example.com/robots.txtThen apply the longest-match rule by hand to the path in question. With Disallow: /blog/ and Allow: /blog/public/, the URL /blog/public/post is allowed because the matching Allow rule is longer. For a property you own, the robots.txt report in Search Console shows the fetched file, when it was last retrieved and any parse errors, which is the fastest way to confirm the crawler is reading what you think it is reading.
Confirm it is reachable from everywhere
A robots.txt that loads from your desk can still fail for a crawler. Bot-protection rules, rate limits and geographic blocks routinely return 403 to data-center address ranges while serving residential visitors normally, and the result is a host that Google slows down or stops crawling for reasons nothing in your logs calls an error.
The way to see it is to request the file from somewhere that is not your office. Point the free HTTP check tool at https://example.com/robots.txt and read the status code returned from checkpoints in different countries. A 200 everywhere is the healthy answer; a 200 at home and a 403 from three regions is the finding. Watching that same URL on a schedule, so the day a rule changes is the day you hear about it, is what a website monitoring service is for.