Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Web crawling for AI

Web crawling for AI is the automated fetching of web pages, by following links or a list of URLs, to collect text for training models, building search indexes or giving an application fresh content to retrieve.

Also called: crawler, web scraping

By Divya Sharma · Checked Nov 15, 2026 · How we review

Many AI applications need current web content: a support bot that knows the product docs, a research agent, or a dataset for pretraining. Crawling is how that content gets collected at scale.

How it works

A crawler starts from seed URLs, fetches each page, extracts the links and adds new ones to a queue. Search crawlers discover most URLs this way, by following links on pages they have already fetched. For AI use, the crawler then strips navigation and ads, renders JavaScript where needed, and converts the page to clean text or Markdown ready for chunking. Tools such as Firecrawl and Crawl4AI bundle these steps.

Crawling and scraping overlap. Crawling usually means discovering and fetching many pages; scraping means pulling specific fields or text out of a page. Large public crawls also exist: Common Crawl publishes a free, open archive that many training datasets draw on.

Rules and etiquette

Site owners state what crawlers may fetch in robots.txt, defined by the Robots Exclusion Protocol. Those rules are requests, not access control, so honouring them is up to the crawler. Some sites also publish an llms.txt file pointing models at their most useful pages. Crawlers run by AI companies are covered under AI crawler. Check a site's terms and copyright position before reusing what you collect. The data ingestion guide compares crawling tools.

Related terms

llms.txt
llms.txt is a proposed convention in which a website publishes a Markdown file, usually at /llms.txt, that summarises the site and links to its most useful pages in a form language models can read easily.
AI crawler
An AI crawler is an automated bot run by an AI company that fetches web pages to collect training data, build an AI search index, or retrieve a page live when a user asks an assistant about it.
AI training data
AI training data is the collection of examples a model learns from, such as web text, code, images, transcripts or labelled pairs. Its size, quality, mix and licensing shape what the model can do and where it fails.
Chunking
Chunking is the step that splits long documents into smaller passages before they are embedded and indexed, so a retrieval system can find and return only the parts of a document that answer a query.
Retrieval-augmented generation
Retrieval-augmented generation (RAG) is a pattern where a system first searches a collection of documents for passages relevant to a question, then gives those passages to a language model so its answer draws on them.

Sources