Data preparation
Getting documents, web pages and files into a form an AI system can use.
- AI crawler
- An AI crawler is an automated bot run by an AI company that fetches web pages to collect training data, build an AI search index, or retrieve a page live when a user asks an assistant about it.
- Chunking
- Chunking is the step that splits long documents into smaller passages before they are embedded and indexed, so a retrieval system can find and return only the parts of a document that answer a query.
- Document parsing
- Document parsing converts files such as PDFs, scans, slides and spreadsheets into structured text that software can use, keeping headings, reading order, tables and images instead of a flat stream of characters.
- Layout analysis
- Layout analysis is the part of document parsing that finds the regions on a page, such as titles, paragraphs, tables, figures, captions and footers, labels each one, and works out the order a person would read them in.
- llms.txt
- llms.txt is a proposed convention in which a website publishes a Markdown file, usually at /llms.txt, that summarises the site and links to its most useful pages in a form language models can read easily.
- Optical character recognition
- Optical character recognition (OCR) is software that reads text from images, such as scanned pages, photos or screenshots, and turns it into machine-readable characters that can be searched, copied or fed to a model.
- Table extraction
- Table extraction is the task of finding tables in documents and rebuilding their rows, columns, headers and cells as structured data, such as HTML, Markdown or CSV, instead of loose lines of text.
- Web crawling for AI
- Web crawling for AI is the automated fetching of web pages, by following links or a list of URLs, to collect text for training models, building search indexes or giving an application fresh content to retrieve.