Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Data preparation

Getting documents, web pages and files into a form an AI system can use.

AI crawler
An AI crawler is an automated bot run by an AI company that fetches web pages to collect training data, build an AI search index, or retrieve a page live when a user asks an assistant about it.
Chunking
Chunking is the step that splits long documents into smaller passages before they are embedded and indexed, so a retrieval system can find and return only the parts of a document that answer a query.
Document parsing
Document parsing converts files such as PDFs, scans, slides and spreadsheets into structured text that software can use, keeping headings, reading order, tables and images instead of a flat stream of characters.
Layout analysis
Layout analysis is the part of document parsing that finds the regions on a page, such as titles, paragraphs, tables, figures, captions and footers, labels each one, and works out the order a person would read them in.
llms.txt
llms.txt is a proposed convention in which a website publishes a Markdown file, usually at /llms.txt, that summarises the site and links to its most useful pages in a form language models can read easily.
Optical character recognition
Optical character recognition (OCR) is software that reads text from images, such as scanned pages, photos or screenshots, and turns it into machine-readable characters that can be searched, copied or fed to a model.
Table extraction
Table extraction is the task of finding tables in documents and rebuilding their rows, columns, headers and cells as structured data, such as HTML, Markdown or CSV, instead of loose lines of text.
Web crawling for AI
Web crawling for AI is the automated fetching of web pages, by following links or a list of URLs, to collect text for training models, building search indexes or giving an application fresh content to retrieve.