llms.txt
llms.txt is a proposed convention in which a website publishes a Markdown file, usually at /llms.txt, that summarises the site and links to its most useful pages in a form language models can read easily.
Also called: /llms.txt
By Divya Sharma · Checked Nov 15, 2026 · How we review
Web pages are built for browsers: menus, scripts, cookie banners and layout code surround the text. A model or agent that fetches a page has to dig through all of that, often within a limited context window. llms.txt offers a shortcut: one plain file that says what the site is and where the important content lives.
How it works
The file is Markdown. The only required part is an H1 with the name of the site or project. After that it can have a short summary in a blockquote, some detail paragraphs, and H2 sections listing links with a one-line note on each. The proposal also suggests offering clean Markdown versions of pages. The file sits at the site root, or under a path to cover just that section.
What it is not
llms.txt is not robots.txt. robots.txt tells crawlers what they may fetch; llms.txt only suggests what is worth reading and grants or blocks nothing. It is also a community proposal, not a formal standard, and publishing one does not guarantee that any AI crawler or search product will use it.
Example
A developer tools company lists its quickstart, API reference and pricing pages in llms.txt, so a coding agent pointed at the site can find the reference docs without crawling the marketing pages. ReviewRamble publishes its own file at /llms.txt. See also web crawling and generative engine optimization.
Related terms
- Web crawling for AI
- Web crawling for AI is the automated fetching of web pages, by following links or a list of URLs, to collect text for training models, building search indexes or giving an application fresh content to retrieve.
- AI crawler
- An AI crawler is an automated bot run by an AI company that fetches web pages to collect training data, build an AI search index, or retrieve a page live when a user asks an assistant about it.
- Generative engine optimization
- Generative engine optimization (GEO) is the practice of shaping web content so that AI search engines and chat assistants find it, quote it and cite it in the answers they generate.
- Context window
- A context window is the maximum amount of text, counted in tokens, that a language model can consider at once, covering the system prompt, conversation, documents, tool results and the response it writes.
- AI search engine
- An AI search engine is a search service that runs web searches for a query, reads the results with a language model and returns a written answer with links to its sources, instead of only a list of links.