Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

llms.txt

llms.txt is a proposed convention in which a website publishes a Markdown file, usually at /llms.txt, that summarises the site and links to its most useful pages in a form language models can read easily.

Also called: /llms.txt

By Divya Sharma · Checked Nov 15, 2026 · How we review

Web pages are built for browsers: menus, scripts, cookie banners and layout code surround the text. A model or agent that fetches a page has to dig through all of that, often within a limited context window. llms.txt offers a shortcut: one plain file that says what the site is and where the important content lives.

How it works

The file is Markdown. The only required part is an H1 with the name of the site or project. After that it can have a short summary in a blockquote, some detail paragraphs, and H2 sections listing links with a one-line note on each. The proposal also suggests offering clean Markdown versions of pages. The file sits at the site root, or under a path to cover just that section.

What it is not

llms.txt is not robots.txt. robots.txt tells crawlers what they may fetch; llms.txt only suggests what is worth reading and grants or blocks nothing. It is also a community proposal, not a formal standard, and publishing one does not guarantee that any AI crawler or search product will use it.

Example

A developer tools company lists its quickstart, API reference and pricing pages in llms.txt, so a coding agent pointed at the site can find the reference docs without crawling the marketing pages. ReviewRamble publishes its own file at /llms.txt. See also web crawling and generative engine optimization.

Related terms

Web crawling for AI
Web crawling for AI is the automated fetching of web pages, by following links or a list of URLs, to collect text for training models, building search indexes or giving an application fresh content to retrieve.
AI crawler
An AI crawler is an automated bot run by an AI company that fetches web pages to collect training data, build an AI search index, or retrieve a page live when a user asks an assistant about it.
Generative engine optimization
Generative engine optimization (GEO) is the practice of shaping web content so that AI search engines and chat assistants find it, quote it and cite it in the answers they generate.
Context window
A context window is the maximum amount of text, counted in tokens, that a language model can consider at once, covering the system prompt, conversation, documents, tool results and the response it writes.
AI search engine
An AI search engine is a search service that runs web searches for a query, reads the results with a language model and returns a written answer with links to its sources, instead of only a list of links.

Sources