What is noindex, and when to use it
A noindex directive tells search engines to keep a page out of their index, even though they are still allowed to crawl it. It is set either as a meta tag in the page's <head> or as an X-Robots-Tag HTTP response header, and it is the correct tool for pages that must exist but must not appear in search results.
The two places noindex lives
There are exactly two supported ways to say it, and they mean the same thing:
<meta name="robots" content="noindex">
X-Robots-Tag: noindexThe meta tag only works inside the <head> of an HTML document. The header works on anything the server sends, which is why it is the only option for a PDF, an image or a JSON endpoint. You can also target one crawler by name, so <meta name="googlebot" content="noindex"> applies to Google and leaves other engines alone.
The second value in content is a separate decision. noindex, follow keeps the page out of the index while letting crawlers use its links; noindex, nofollow stops both. For a paginated archive or a filtered listing, noindex, follow is usually what you want, because the links on those pages are how deeper pages get discovered.
When noindex is the right call
Use it on pages that have to be reachable but have no business ranking: internal search result pages, thank-you and order-confirmation pages, printer-friendly duplicates, user account areas, tag archives that hold one post each, and staging or demo environments as a whole.
Do not use it as a duplicate-content fix when the two pages are genuinely the same thing. That is what rel=canonical is for: a canonical consolidates the signals from both URLs into one, while noindex throws the duplicate away entirely, links and all.
The rule everyone gets wrong: crawling comes first
A crawler has to fetch a page to see the noindex on it. If you block that URL in robots.txt, the crawler never fetches it, never sees the directive, and the URL can still surface in results from external links, usually without a description.
So combining the two is self-defeating. To remove a page from search, allow crawling and serve noindex. Once the page has been recrawled and dropped, you can block it in robots.txt if you also want to save the crawl budget. Doing it in the other order leaves the page in the index indefinitely. Putting noindex inside robots.txt does not work either; Google stopped honoring that undocumented syntax on 1 September 2019.
How it leaks from staging to production
Almost every serious noindex incident starts as a deliberate, correct noindex on a staging site. Then one of these happens:
- A database copy. The staging database is restored over production, carrying the CMS flag with it. In WordPress that is the "Discourage search engines from indexing this site" checkbox under Settings, stored as an option row.
- A config promote. The staging
.envor nginx snippet that addsX-Robots-Tag: noindexto every response gets deployed with the release. - A theme or template merge. A hardcoded meta tag in a shared layout file survives a merge that nobody read line by line.
- A CDN or edge rule written to protect a preview hostname, applied to a route pattern that also matches production.
The reason this is expensive is that nothing breaks. The site returns 200, loads normally, and passes every uptime check. Traffic decays over days as pages are recrawled and dropped, and by the time someone notices the graph, the recovery also takes days.
How to spot a noindex you did not intend
- Read the header.
curl -sI https://example.com/ | grep -i x-robots-tagreturns nothing on a healthy page. Anything it prints is worth explaining. - Read the meta tag in the served HTML, not in your editor:
curl -s https://example.com/ | grep -i 'name="robots"'. Server-side logic and edge workers can inject tags that are not in your repository. - Check Search Console. URL Inspection reports "Excluded by 'noindex' tag" per URL, and the Pages report groups every affected URL under the same reason.
- Crawl the whole site, because the leak is rarely on the homepage you tested. A crawl flags noindex per URL, so a template that only affects product pages shows up as a block of flagged rows rather than a single surprise.
Site-wide noindex is the easy case. The common one is a single template or a single route, which is exactly what a spot check on the homepage misses.
Check every page, not just the one you remember
The free site crawl tool walks your site and flags the noindex pages it finds along with missing titles, missing canonicals and duplicate bodies, which is the fastest way to see whether a directive is on one page or on a whole section. If you would rather have the whole set of failures watched instead of checked by hand, that is what a website monitoring service is for.