Skip to main content
HostTracker
HOST-TRACKER
website monitoring service
Status pages Instant check Price
Login Register

Crawl web pages

Validate status of your site's pages to track changes and cross-references

Need one page, not the whole site? Check a single page for noindex or its canonical tag.

Examples: 300+ locations worldwide · results in about 10 seconds

You've used today's free checks

Create a free account to keep checking. It takes less than a minute, and new free checks are available again tomorrow.

Start 30-day free trial Already have an account? Log in
{{error.message}}

Unlock the full report

30-day free trial · cancel anytime

Loading plans…
See all plans & pricing
Recommended
{{ plan.name }}
{{ money(priceMo(plan)) }}/mo
  • {{ plan.maxTaskCount == null ? 'Unlimited' : plan.maxTaskCount }} monitored sites
  • All check locations in the report
  • TLS, headers & timing details
  • Scheduled reports
  • REST API access
Start free trial
Billed only after your trial · Compare all plans

Web page monitoring

Track changes to your website. The feature automatically checks all pages and reports any updates, deletions or errors.

Find out more

Importance of Load Testing Before Launch

Conducting server testing under high load reveals how your website's pages hold up when many requests arrive in a short time, so you can catch slow responses, timeouts and broken links before real visitors or a traffic spike do.

Find out more

Enhancing Website Monitoring with HTTP Load Testing

HTTP load speed testing using tools like HostTracker crawls your entire site and breaks down each page's response time into connection, TLS and data-transfer stages, making it easy to pinpoint exactly what is slowing your pages down.

Find out more

Need this checked around the clock, not just once? Set up availability monitoring

Key takeaways

  • This tool follows the links on a site and requests each page it finds, reporting the HTTP status code and the response time of every one.
  • It is how you find broken links, redirect chains and pages answering 404 or 5xx before visitors run into them.
  • The crawl runs from HostTracker's distributed location fleet, the same one the scheduled website monitors use.
  • It is free, needs no login and no credit card.
  • For continuous checks, see the distributed availability monitoring page.

Frequently asked questions.

A crawl test requests the start page you enter, reads the links in its HTML and follows them page by page, the way a search engine crawler discovers a site. This tool records the HTTP status code, the response time and the basic SEO tags of every page it reaches, so broken links, slow pages, redirects and missing titles show up in one report. It is a one-time crawl from one checkpoint in the region you pick in the form.
Up to 50 URLs without an account, up to 100 when you are signed in to a free account, and more on paid plans. Links to other sites count toward that number but are not requested. The crawl takes the pages closest to the start page first and stops when the budget is used or after 3 minutes, whichever comes first, so on a large site it covers the top levels. Without an account the results show a short preview; sign in to see every page.
A visitor only sees the links a page displays, while a crawler reads every link in the HTML source. Links hidden by CSS, tucked into collapsed menus or left behind in old templates are all followed, and the same page with a different query string, such as ?page=2 or ?sort=price, counts as a separate URL. That is how forgotten test pages and stale links surface. The reverse also happens: this crawl does not run JavaScript, so links that only appear after scripts run are not found.
For every URL the Requests tab shows the HTTP status code or the error, the time split into connection, TLS handshake, headers, body and redirects, the number of requests including redirect hops, the page size and the download speed. The Hosts tab lists each host the crawl reached with its IP address, TLS version, cipher and certificate expiry date. The SEO tab covers indexing and on-page tags, and the URL tree shows which page links to which.
On each HTML page the crawl flags a missing title or one over 60 characters, a missing meta description or one over 160 characters, a missing canonical tag or one that points to another URL, a missing H1, more than one H1 or an H1 that is not the first heading, and a missing lang attribute. It also marks pages excluded by a noindex robots meta tag or X-Robots-Tag header, and pages with exactly the same HTML as another crawled page.
A request counts as unsuccessful when the page answers with a 4xx or 5xx status, when the DNS lookup or the connection fails, or when it does not answer in time: the start page gets 20 seconds and every other page 4 seconds. A 404 almost always means a broken link, and the URL tree shows the page that links to it. A page that timed out here may still load slowly in a browser, so test it on its own with the HTTP check. The 404 guide explains the fixes.
No. Links to other domains are listed as external but never requested, and by default the same applies to other subdomains, such as a help center or a sign-in page on a subdomain of its own. The www and non-www versions of the start host count as one site. When an internal link redirects to another domain, the crawl keeps the status and timing of that request but does not read the page it landed on.
No. This crawl follows every link it finds whether or not robots.txt disallows it, so it can reach pages search engines are told to skip. To see what your robots.txt blocks, use the robots.txt tester. Each request identifies itself with the user agent Mozilla/5.0 (compatible; HostTracker/2.0; +https://www.host-tracker.com/), which you can look for in your server logs or allow in a firewall that blocks unknown bots.
For most sites, yes. The crawl sends up to 5 requests at a time, all of them GET requests with no pause between pages, and stops at the page budget or after 3 minutes, so a run adds at most a few hundred requests. It follows plain links and never submits forms. A link that performs an action through a plain GET request, such as a logout or delete link, is requested like any other page, so keep such actions behind forms.
Yes, as a beta enabled on request: write to support@host-tracker.com with the site you want crawled and it will be enabled on your account. The scheduled crawl walks up to 500 pages, once a week by default, compares each run with the previous one and reports pages added, removed or newly noindexed, along with a site health score. The SEO monitoring page describes it in full.

Part of HostTracker's website monitoring software.