Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

AI crawler

An AI crawler is an automated bot run by an AI company that fetches web pages to collect training data, build an AI search index, or retrieve a page live when a user asks an assistant about it.

Also called: GPTBot, ClaudeBot, PerplexityBot, AI bot

By Divya Sharma · Checked Nov 15, 2026 · How we review

Site owners increasingly see bots such as GPTBot, ClaudeBot and PerplexityBot in their server logs. Each identifies itself with a user-agent string, and knowing which is which decides whether a site's content trains models, appears in AI answers, or neither.

Three kinds of bot

AI companies tend to split their bots by purpose. Training crawlers, such as OpenAI's GPTBot and Anthropic's ClaudeBot, collect pages that may go into AI training data. Search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, index pages so an AI search engine can cite them. User-triggered fetchers, such as ChatGPT-User and Claude-User, load a single page when a person asks the assistant to read it. OpenAI notes that robots.txt rules may not apply to user-initiated fetches.

Controlling access

The main control is robots.txt. A rule such as `User-agent: GPTBot` followed by `Disallow: /` asks that bot to stay away, and each bot is set separately, so a site can block training while allowing search. OpenAI says disallowing GPTBot signals that content should not be used for training. The Robots Exclusion Protocol is a request, not access control, so firewalls or bot management are needed for crawlers that ignore it. An llms.txt file is a separate idea: it points models at useful pages but blocks nothing.

Why it matters

Blocking search crawlers can keep a site out of AI answers, which affects generative engine optimization. Blocking training crawlers only affects future training runs. See web crawling for AI for how crawlers work in general.

Related terms

Web crawling for AI
Web crawling for AI is the automated fetching of web pages, by following links or a list of URLs, to collect text for training models, building search indexes or giving an application fresh content to retrieve.
llms.txt
llms.txt is a proposed convention in which a website publishes a Markdown file, usually at /llms.txt, that summarises the site and links to its most useful pages in a form language models can read easily.
AI training data
AI training data is the collection of examples a model learns from, such as web text, code, images, transcripts or labelled pairs. Its size, quality, mix and licensing shape what the model can do and where it fails.
AI search engine
An AI search engine is a search service that runs web searches for a query, reads the results with a language model and returns a written answer with links to its sources, instead of only a list of links.
Generative engine optimization
Generative engine optimization (GEO) is the practice of shaping web content so that AI search engines and chat assistants find it, quote it and cite it in the answers they generate.

Sources