Jeremy Howard, co-founder of Answer.AI, proposed llms.txt on 3 September 2024 as a way to hand language models a clean, hand-picked map of a site instead of making them guess from the navigation (llmstxt.org). It is a single markdown file. It is not a standard that any major AI search engine has confirmed it reads for ranking.
llms.txt is a markdown file listing your most important pages with short descriptions, meant to help an LLM find the good parts of your site fast. It lives at /llms.txt or /.well-known/llms.txt. It is a real, documented proposal with real adopters. No major AI engine has confirmed it uses the file to decide what to cite.
What goes inside the file?
The format is deliberately small. One H1 with the site or project name. A blockquote summary underneath. Then ## sections that group links, each link followed by a short description of what the reader (human or model) will find there (llmstxt.org). Howard also defined an optional companion, llms-full.txt, which inlines the actual content of those pages into one long markdown document for models that want the whole thing in one fetch.
The design goal is the context window. A model cannot hold your entire site in memory at inference time, so llms.txt is the shortlist you would hand a smart intern who has ten minutes. Documentation-heavy companies were the first movers: Anthropic, Stripe, Cursor, Mintlify, and Zapier all publish one (Answer.AI, GitHub).
Where does it actually go?
Two locations are in circulation, and this is where sites quietly get it wrong.
| Path | Status |
|---|---|
/llms.txt |
The location in Howard’s original proposal. Root of the domain. |
/.well-known/llms.txt |
The RFC 8615 well-known directory some tools and CMSs default to. |
The proposal specifies the root. A growing number of publishing tools drop the file into /.well-known/ because that is where machine-readable metadata is supposed to live. Neither is wrong, but a crawler looking in one place will not find a file sitting in the other. When we built the checker for this, we made it look in both, because in practice we found sites that had written a perfectly good llms.txt and parked it at the path the reader was not checking. A file nobody fetches is a file nobody reads.
Does any AI engine read it?
This is the part the vendors selling llms.txt generation packages tend to skip.
Google has said, on the record and more than once, that it does not use llms.txt, comparing it to the keywords meta tag that search engines ignored for two decades. There is no confirmation from OpenAI, Anthropic, or Perplexity that their retrieval systems consume the file to rank or cite pages. The most-shared critique in the field is bluntly titled “The llms.txt is dead”, and its core point holds: a file that no major consumer reads cannot move your visibility, no matter how well-formed it is.
So why write one? Two honest reasons remain. First, some documentation tools and smaller agents do fetch it, and if your product is developer-facing, that audience is disproportionately in the file’s user base. Second, the act of writing a good llms.txt forces you to state, in one page, what your most important content is and how to describe it. That clarity is worth something even if no robot ever reads the artifact.
The cargo-cult version is the trap: generate a llms.txt, put it in a slide, and call your site AI-ready. It is not a ranking signal. Treat it as low-cost hygiene, not a growth lever, and spend the real effort on the thing engines demonstrably do read: whether their crawlers can reach your pages at all.
The practical next step: if you already have a sitemap and a clear content hierarchy, writing llms.txt is twenty minutes of work and safe to do. Just do not expect it to show up in your traffic. To see what actually moves your AI visibility, run a free scan and start with the checks that map to engine behavior, not proposals.