Robots.txt tester
Test whether a site's robots.txt blocks a URL for all crawlers (User-agent: *), find its sitemap and check whether the page is listed in it.
Unlock the full report
30-day free trial · cancel anytime
- {{ plan.maxTaskCount == null ? 'Unlimited' : plan.maxTaskCount }} monitored sites
- All check locations in the report
- TLS, headers & timing details
- Scheduled reports
- REST API access
What the robots.txt tester checks
Whether robots.txt disallows your URL for the User-agent: * group that applies to all crawlers, and whether the URL is listed in the site's sitemap.
Blocked is not the same as removed
robots.txt stops crawlers from fetching a URL, but a blocked URL can still show up in search results when other sites link to it.
One wrong line can hide a whole site
A Disallow: / copied from staging, or a rule that matches more paths than intended, is one of the most common reasons a site drops out of search.
Related checkers
Catch indexing changes the day they ship
Website monitors check your key pages around the clock, and the Indexability option adds a notice whenever a page's noindex or canonical tag changes.
Keep an eye on your site around the clock
Monitor from 300+ regions with instant downtime alerts, start your 30-day free trial.
Key takeaways
- This tool reads the robots.txt of the host you enter and tells you whether its User-agent: * group disallows the URL.
- It also looks for the site's sitemap and reports whether the URL is listed in it, with the sitemap address and how many URLs it lists.
- Rules for individual named crawlers are not evaluated yet; the verdict covers the group that applies to all crawlers.
- It is free, needs no login and no credit card.
- Robots.txt controls crawling, not indexing, so the result also shows the page's noindex and canonical tag.
Part of HostTracker's website monitoring service.