llms.txt: what it is, who reads it, and how to write one
llms.txt is a proposed convention, not a web standard, for publishing a short Markdown file at a site's root that gives AI language models a curated list of its most useful pages. Jeremy Howard of Answer.AI introduced it in September 2024 at llmstxt.org; no browser, search engine, or major AI product is required to support it, and none treats it as authoritative today.
What is llms.txt?
llms.txt addresses a real problem: a page built for people wraps its content in navigation, ads, and JavaScript, and turning that back into clean text for a model is difficult and lossy. Instead of asking a model to crawl and parse full HTML pages, a site publishes one plain Markdown file listing, in order of importance, the pages a model should read.
It is explicitly a proposal, not a standard: no governing body, no required support from any AI tool. Whether it gets used depends entirely on whether a given tool chooses to read it.
What goes in an llms.txt file?
The format defined at llmstxt.org is specific:
- An H1 with the site or project name, the only required element.
- A blockquote under the H1 with a short summary.
- Optional plain paragraphs with more context, before the first H2.
- One or more H2 sections, each a Markdown list of links:
[name](url): optional note. - An optional H2 named "Optional" for secondary links a model can skip when short on context.
Here is a realistic example for a fictional SaaS product:
# Acme Metrics
> Acme Metrics is a hosted dashboard for product analytics events.
## Docs
- [Quickstart](https://docs.acmemetrics.com/quickstart.md): Install the SDK, send your first event
- [API reference](https://docs.acmemetrics.com/api.md): REST endpoints for events and dashboards
- [Webhooks](https://docs.acmemetrics.com/webhooks.md): Real-time event notifications
## Optional
- [Changelog](https://docs.acmemetrics.com/changelog.md): Release notes
- [Status page](https://status.acmemetrics.com): Uptime and incidents
Many sites also publish an llms-full.txt: the same map, but with the full Markdown content of every linked page pasted into one file instead of just links, so a model that will not follow links still gets everything in one fetch.
llms.txt vs robots.txt vs sitemap.xml
These three files get confused because they all sit at the site root and all relate to how machines see a site, but they do three different jobs:
| File | Purpose | Who reads it | Controls access? |
|---|---|---|---|
| robots.txt | Which paths crawlers may fetch | Crawlers that choose to honor it | Yes, advisory (a compliant bot follows it) |
| sitemap.xml | URLs you want discovered and indexed | Search engines, for discovery | No, a hint list, not a permission list |
| llms.txt | A curated Markdown map of your best pages | Only tools that choose to fetch it | No, it does not touch crawling or indexing |
robots.txt is the only one of the three that gates anything, and even that is voluntary: it works because well-behaved crawlers check it first, not because a server enforces it.
Does Google or any AI search engine use llms.txt?
Google's own documentation answers this directly. Its Search Central guide on generative AI search, under "Mythbusting generative AI search: what you don't need to do," names LLMS.txt files and states: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them" (last updated July 2026).
Google representatives have said the same in public for over a year. At Search Central Live in July 2025, Gary Illyes said Google does not support llms.txt and has no plans to, comparing it to the old meta keywords tag, a signal search engines stopped trusting once site owners controlled it. In January 2026, after Google's own docs platform briefly auto-generated an llms.txt file, John Mueller was asked on Bluesky if that was an endorsement: "I'm tempted to say something snarky since this has come up so often, but to be direct, no."
Other AI vendors have said less. What is verifiable: Anthropic, OpenAI, and Perplexity each publish their own llms.txt for their developer docs (for example platform.claude.com/llms.txt and developers.openai.com/llms.txt), which only shows they find the format useful for their own site. That differs from their crawlers reading an llms.txt on someone else's site before answering a question, and none of the three has published that. Perplexity's own crawler documentation describes PerplexityBot and Perplexity-User in detail and does not mention llms.txt. Treat a claim like "ChatGPT reads llms.txt" as unverified until the vendor says so.
Should you publish one anyway?
Probably, for modest reasons. An llms.txt is a static file with no server work beyond serving it as plain text, so the cost is close to zero. The real use case is narrower than "AI SEO": a coding agent or IDE assistant fetching pages on demand while answering a question can use a short llms.txt to find your quickstart or API reference faster than guessing from a sitemap.
It is not a substitute for a service built for agents to act on: HostTracker's own MCP server (/en/mcp-server) is a live interface an assistant calls to run a check or manage a monitor, not a file it just reads. llms.txt offers a reading list; it cannot make anything happen.
Keep it short and accurate. A stale link in a file meant to be a trustworthy map is worse than no file, and there is no evidence tying llms.txt to better rankings or more AI citations, so do not pad it with marketing copy.
How to control AI crawlers for real
If the goal is controlling what AI systems do with your content, robots.txt with named user-agent groups is the real lever, not llms.txt. Each vendor documents its own token:
- GPTBot (OpenAI): collects web content to train future models; OpenAI lists it separately from OAI-SearchBot, used for search results, so you can allow one and disallow the other.
- ClaudeBot, Claude-User, Claude-SearchBot (Anthropic): ClaudeBot collects training data, Claude-User fetches a page for a live user question, Claude-SearchBot crawls for search-style results; Anthropic's docs state all three honor robots.txt.
- PerplexityBot and Perplexity-User (Perplexity): PerplexityBot crawls to surface sites in results and, per Perplexity's docs, is not used to train models. Perplexity-User fetches a page live for a specific question and generally ignores robots.txt, since the fetch is user-initiated, not a background crawl.
- Google-Extended: a robots.txt token, not a crawler with its own user agent, controlling whether Googlebot's already-fetched content can train future Gemini models. Google states blocking it has no effect on Search ranking or indexing.
Test your rules with the free robots.txt tester: it checks whether the general User-agent: * group blocks a URL and whether the URL is in your sitemap. It does not yet evaluate a rule written for one named crawler, so a URL open under User-agent: * can still be blocked for GPTBot with its own group further down the file; check named-crawler groups by reading the raw file for now. The how to check robots.txt guide covers reading one by hand.
How to check your llms.txt is reachable
An llms.txt a model cannot fetch is worse than useless: a model that tries once and gets blocked or redirected has no way to tell you it failed. Before treating the file as live, confirm three things:
- It returns a plain 200 status, not a redirect to a login page or an error page rendered as 200.
- Its
Content-Typeistext/plainortext/markdown, nottext/html, meaning a template wrapped it instead of serving the raw file. - It is not silently dropped by a WAF or bot-mitigation rule that flags an unusual request for a root text file.
A quick command-line check:
curl -I https://example.com/llms.txtLook for HTTP/1.1 200 and a content-type starting with text/. To confirm it responds the same way from outside your own network, run the URL through the free HTTP checker, which requests it from multiple locations and reports the status and response time from each. Then use the robots.txt tester to confirm a blanket Disallow: / is not also hiding the file itself.
HostTracker's SEO monitoring watches the signals that do move search visibility: noindex meta tags, X-Robots-Tag headers, and canonical tag changes, plus a scheduled crawl that cross-checks pages against robots.txt on every run. llms.txt sits outside all of that.
Frequently asked questions
Does llms.txt replace robots.txt?
No. robots.txt controls what crawlers may fetch and predates llms.txt by three decades. A page can be listed in llms.txt and still be disallowed in robots.txt, or the other way around.
Will publishing an llms.txt improve my search rankings?
There is no evidence for that, and Google has said directly that its Search systems, including AI Overviews, do not use it. Treat it as a convenience for certain developer tools, not an SEO tactic.
What is llms-full.txt?
A companion file some sites publish alongside llms.txt that inlines the full Markdown content of every linked page in one file, for a model that will not or cannot follow individual links.
Do ChatGPT, Claude, or Perplexity automatically read my llms.txt?
No vendor has published a statement saying its crawler or chat product fetches and parses other sites' llms.txt files. All three publish their own for their documentation, which is different from reading yours.
Where should I put the file?
At the site root, as /llms.txt, served as plain text. The proposal also allows a subpath, such as /docs/llms.txt, to describe just that section.
Is llms.txt an official standard?
No. It is a community proposal published at llmstxt.org with no standards body behind it and no requirement that any tool support it.