Skip to main content
HostTracker
HOST-TRACKER
website monitoring service
Status pages Instant check Price
Login Register

Robots.txt tester

Test whether a site's robots.txt blocks a URL for all crawlers (User-agent: *), find its sitemap and check whether the page is listed in it.

Examples: 300+ locations worldwide · results in about 10 seconds

You've used today's free checks

Create a free account to keep checking. It takes less than a minute, and new free checks are available again tomorrow.

Start 30-day free trial Already have an account? Log in
{{error.message}}

Unlock the full report

30-day free trial · cancel anytime

Loading plans…
See all plans & pricing
Recommended
{{ plan.name }}
{{ money(priceMo(plan)) }}/mo
  • {{ plan.maxTaskCount == null ? 'Unlimited' : plan.maxTaskCount }} monitored sites
  • All check locations in the report
  • TLS, headers & timing details
  • Scheduled reports
  • REST API access
Start free trial
Billed only after your trial · Compare all plans

What the robots.txt tester checks

Whether robots.txt disallows your URL for the User-agent: * group that applies to all crawlers, and whether the URL is listed in the site's sitemap.

Find out more

Blocked is not the same as removed

robots.txt stops crawlers from fetching a URL, but a blocked URL can still show up in search results when other sites link to it.

Find out more

One wrong line can hide a whole site

A Disallow: / copied from staging, or a rule that matches more paths than intended, is one of the most common reasons a site drops out of search.

Find out more

Guides

Related checkers

Catch indexing changes the day they ship See SEO monitoring

Key takeaways

  • This tool reads the robots.txt of the host you enter and tells you whether its User-agent: * group disallows the URL.
  • It also looks for the site's sitemap and reports whether the URL is listed in it, with the sitemap address and how many URLs it lists.
  • Rules for individual named crawlers are not evaluated yet; the verdict covers the group that applies to all crawlers.
  • It is free, needs no login and no credit card.
  • Robots.txt controls crawling, not indexing, so the result also shows the page's noindex and canonical tag.

Frequently asked questions.

Enter the full address of the page you care about, not just the robots.txt file. The tester fetches /robots.txt from that host, applies the rules of its User-agent: * group to your URL and reports "Allowed for all crawlers" or "Blocked for all crawlers". It also looks for the sitemap, says whether the URL is listed in it and shows the page's noindex and canonical. In Search Console, the robots.txt report, which replaced the old robots.txt Tester, shows how Google last fetched your file.
robots.txt is a plain-text file at /robots.txt in the root of a host that tells crawlers which URLs they may fetch. It is defined by RFC 9309, the Robots Exclusion Protocol, and is organized in groups: one or more User-agent lines followed by Allow and Disallow rules. A file covers only the protocol, host and port it is served from, so a subdomain needs its own. It is a request, not access control. The robots.txt guide covers the syntax and the mistakes that block a whole site.
A Disallow rule tells the crawlers in its group not to fetch URLs whose path starts with the given value. Disallow: /admin/ blocks everything under /admin/, Disallow: / blocks the whole site, and an empty Disallow: blocks nothing. Paths are case-sensitive, so a rule for /admin/ does not cover /Admin/. An Allow rule carves an exception out of a broader Disallow. Disallow controls crawling only; it does not remove a URL from search results.
No. robots.txt controls crawling, not indexing. Google says a URL blocked by robots.txt can still appear in search results, without a description, for example when other pages link to it. And because the crawler never fetches a blocked page, it never sees a noindex on it. To keep a page out of search, leave it crawlable and add noindex, or put it behind a password. The result shows the page's noindex next to the robots.txt verdict, and the noindex checker shows where it comes from.
The most specific one, which is the matching rule with the longest path. RFC 9309 measures that length in octets, and when an Allow and a Disallow rule are equally specific, Allow should win. So with Disallow: /shop/ and Allow: /shop/sale/, the URL /shop/sale/shoes stays crawlable while /shop/cart is blocked. This tester applies the same longest-match rule, with ties going to Allow.
Two characters have a special meaning: * matches any run of characters, including none, and $ marks the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*? blocks every URL with a query string. Without a $, a rule matches any URL that begins with it, so Disallow: /blog also blocks /blog-archive. This tester supports both characters and matches rules against the path and query of the URL.
Not yet. It evaluates the User-agent: * group, which applies to every crawler that has no group of its own. A crawler that does have its own group, for example User-agent: Googlebot, follows only that group and ignores the * rules: RFC 9309 uses the * group only when no other group matches, and Google says specific and global groups are not combined. If your file has named groups, read the group for the crawler you care about yourself.
A missing file (HTTP 404 or most other 4xx answers) means no restrictions: crawlers may fetch everything. A server error or network failure is different: RFC 9309 tells crawlers to assume everything is disallowed, and Google stops crawling the site for 12 hours, then falls back to its last cached copy for up to 30 days. This tester shows any answer other than HTTP 200 as "No robots.txt" and a failed connection as "robots.txt unreachable", so if you expect the file to exist, check its status with curl -I https://example.com/robots.txt.
It reads the Sitemap lines in robots.txt and, if there are none, tries /sitemap.xml on the same host. A sitemap index is followed one level down, gzipped sitemaps are read, and at most three sitemap files are fetched per check. The result shows the sitemap address, how many URLs it lists and whether your URL is one of them. Membership is checked against the first 2,000 URLs, and on a larger sitemap the result says the URL may simply be further down.
The scheduled crawl monitor walks a whole site on a schedule and cross-checks every crawled page against robots.txt and the sitemap on each run, flagging, for example, pages that robots.txt disallows while the site still links to them. It is in beta and enabled on request: write to [email protected]. SEO monitoring describes what it reports. To hear when the robots.txt file itself stops answering, point an ordinary website monitor at its URL.

Part of HostTracker's website monitoring software.