Robots.txt tester
Test whether a site's robots.txt blocks a URL for all crawlers (User-agent: *), find its sitemap and check whether the page is listed in it.
Unlock the full report
30-day free trial · cancel anytime
Loading plans…
Recommended
{{ plan.name }}
{{ money(priceMo(plan)) }}/mo
- {{ plan.maxTaskCount == null ? 'Unlimited' : plan.maxTaskCount }} monitored sites
- All check locations in the report
- TLS, headers & timing details
- Scheduled reports
- REST API access
Billed only after your trial · Compare all plans
What the robots.txt tester checks
Whether robots.txt disallows your URL for the User-agent: * group that applies to all crawlers, and whether the URL is listed in the site's sitemap.
Find out moreBlocked is not the same as removed
robots.txt stops crawlers from fetching a URL, but a blocked URL can still show up in search results when other sites link to it.
Find out moreOne wrong line can hide a whole site
A Disallow: / copied from staging, or a rule that matches more paths than intended, is one of the most common reasons a site drops out of search.
Find out moreRelated checkers
Catch indexing changes the day they ship See SEO monitoring
Key takeaways
- This tool reads the robots.txt of the host you enter and tells you whether its User-agent: * group disallows the URL.
- It also looks for the site's sitemap and reports whether the URL is listed in it, with the sitemap address and how many URLs it lists.
- Rules for individual named crawlers are not evaluated yet; the verdict covers the group that applies to all crawlers.
- It is free, needs no login and no credit card.
- Robots.txt controls crawling, not indexing, so the result also shows the page's noindex and canonical tag.
Part of HostTracker's website monitoring software.