Step 01 of 22
Ingestion & crawling
Pull content from websites, files, apps and databases. A model is only as current as the data you feed it. Ingestion decides what your system knows and how fresh it stays.
How to choose
- Crawling public sites: a crawler that returns clean Markdown saves a parsing step.
- Internal apps and drives: prefer connectors with incremental sync over full re-imports.
- Respect robots.txt and site terms; crawl only what you are allowed to.
What to look for
The questions that separate the options, in order of how much they matter.
01
Match the tool to where the data lives
Public websites, files on a drive and records inside SaaS apps need different tools. Crawlers (Firecrawl, Crawl4AI, Jina Reader) read web pages; connector platforms (Airbyte) call app and database APIs; document libraries (Unstructured) read files you already have. Most RAG systems need two of these, so list your sources before comparing features.
02
Clean output, not just fetched HTML
What matters is how much navigation, cookie banners and footer text survives into the Markdown, because every leftover line is embedded and can be retrieved. Run each candidate on ten of your own target pages and read the output. Selector options (keep this element, drop that one) usually fix more than switching tools.
03
Incremental updates and change detection
Re-crawling or re-importing everything each night is slow, costly and churns your vector index. Prefer tools that sync only changed records (Airbyte's incremental syncs) or let you schedule and diff crawls. Ask how deletions are handled: a page or record removed at the source should disappear from your index too.
04
JavaScript rendering and blocking
Many modern sites render content with JavaScript and block unfamiliar traffic. Open-source crawlers render pages with a headless browser but leave proxies and anti-bot handling to you; hosted services (Firecrawl Cloud, Apify) sell that layer. If your targets are your own docs or friendly sites, you do not need to pay for it.
05
The cost model, at your volume
Pricing units differ: credits per page (Firecrawl), output tokens (Jina Reader), compute units plus per-Actor fees (Apify), LLM tokens per page (ScrapeGraphAI) or sync credits (Airbyte). Multiply by your real page count and refresh rate. LLM-based extraction costs far more per page than plain Markdown conversion, so use it only for pages where you need typed fields.
06
Licence and self-hosting limits
Open-source licences here vary in ways that matter commercially. Firecrawl is AGPL-3.0 and keeps its anti-bot features in the cloud; Airbyte mixes MIT and ELv2, which bars reselling it as a hosted service; Crawl4AI and ScrapeGraphAI are permissive. Check which features the self-hosted edition actually includes before you plan to run it yourself.
07
Permission and provenance
Respect robots.txt and site terms, and keep the source URL and fetch date on every record so you can cite, refresh or delete it later. There is no neutral public benchmark for crawlers; speed and success-rate charts on vendor sites are the vendor's own, so treat them as claims.
How the options differ
The options fall into four families. Web crawlers and readers (Firecrawl, Crawl4AI, Jina Reader) fetch public pages and return Markdown; they differ mainly in whether you run them yourself and who handles proxies and blocking. Scraping platforms (Apify) run ready-made, site-specific scrapers in their cloud and return structured records, which suits large defended sites but bills per run.
Connector platforms (Airbyte) pull from SaaS apps and databases through their APIs with scheduled, incremental syncs, the right choice when your knowledge sits in a CRM or ticketing system rather than on the web. Document and extraction libraries sit at the edges: Unstructured turns files you already hold into typed elements, while ScrapeGraphAI uses an LLM to pull specific fields from pages, which is accurate for small jobs and expensive for bulk ingestion.
7 tools, compared
Ranked by our rating. Open any review for sources, prices and the full verdict.
| Tool | Rating | Our verdict | Best for | Price |
|---|---|---|---|---|
| Firecrawlrepo | 4.2/5 | A hosted web layer for agents that earns its fee; the open-source edition is a starting point, not a substitute. | Developers who want a managed API to turn websites into LLM-ready data. | AGPL-3.0 open source. Cloud: Free 1,000 credits/month; Hobby $19, Standard $99, Growth $399, Scale $749 a month; Enterprise custom. |
| Crawl4AIrepo | 4.2/5 | Still the strongest open crawler for LLM pipelines, but its Docker server's 2026 security record rules out exposing it without hardening. | Python teams that want a self-hosted crawler for RAG or agents without per-page costs. | Library and Docker server free (Apache 2.0 plus an attribution clause). Crawl4AI Cloud: 1,000 free credits a month; Starter $10 a month for 10,000 credits. |
| Apifytool | 4.1/5 | Apify turns scraping big, defended platforms into configuration rather than engineering, at the price of a bill nobody can predict before the job runs. | Teams that need structured data from specific large websites and would rather pay than maintain scrapers. | Free plan with $5 usage/month; Starter $19, Scale $199, Business $999 per month as prepaid usage (about 10% less yearly); compute $0.13–$0.20 per unit; residential proxy $7–$8/GB; Store Actors extra. |
| Jina Readertool | 4.0/5 | Jina Reader turns one URL into clean Markdown for fractions of a cent, which makes it the right default for agents that read a page at a time and the wrong tool for anyone who needs to crawl a site. | Developers who need an agent or pipeline to read individual web pages as Markdown at minimal cost. | Free without a key at 20 requests/minute; keys include free tokens, then usage is billed by output tokens from a shared Jina balance (Jina's 2025 guide listed $50 for 1 billion tokens; current rates sit behind the dashboard). Search costs at least 10,000 tokens per request. Commercial and on-premise licences sold annually by Elastic. |
| Airbyterepo | 3.9/5 | Airbyte has the widest open catalogue for pulling business data into a RAG index, but its AI destinations lag its ELT core and self-hosting it means owning its security patches. | Data teams feeding a RAG system from many SaaS apps and databases who want incremental syncs they can self-host. | Self-hosted Core is free (MIT and ELv2). Cloud Standard from $20/month with 5 credits; Plus from $189/month for 40 credits up to $4,999 for 2,000; Pro and Enterprise Flex custom. |
| Unstructuredrepo | 3.8/5 | Still the widest net for odd file formats, but its makers now publish the numbers showing why you would pay them instead. | Teams building ETL pipelines that ingest many document formats for RAG. | Library free under Apache 2.0. Hosted platform: 10,000 free pages, then $0.015 a page; Business plan with VPC or dedicated deployment priced on request. |
| ScrapeGraphAIrepo | 3.6/5 | Good at pulling a few typed fields from messy pages, but it ships your scraped content to its makers by default and pays a model call for every page. | Developers extracting typed fields from a modest number of varied web pages. | Library free under MIT; you pay for LLM tokens. Hosted API: 500 one-off free credits, then $20, $100 or $500 a month; Enterprise custom. |
Side by side
The specifications that matter for this step. Checked 2026-10-19; confirm on the vendor’s site before you commit.
| Tool | Type | Reads from | Output | Proxies / anti-bot | Hosting | Licence |
|---|---|---|---|---|---|---|
| Firecrawl | Crawl and scrape API | Websites; web search | Markdown, JSON | Cloud only | Cloud; limited self-host | AGPL-3.0 |
| Crawl4AI | Crawler library | Websites | Markdown; CSS/XPath/LLM extraction | Weak; bring your own proxies | Self-host only | Apache 2.0 |
| Apify | Scraper platform and Actor store | Specific platforms; any website | JSON datasets | Proxies included (billed) | Cloud | Proprietary; Crawlee Apache 2.0 |
| Jina Reader | Single-page reader API | URLs, PDFs; web search (top 5) | Markdown, JSON | Optional proxy header | Cloud API | Apache 2.0 code; hosted service |
| Airbyte | Connector / ELT platform | 600+ apps, databases, files | Rows; chunks and vectors | Not applicable | Self-host or cloud | MIT and ELv2 |
| Unstructured | Document ETL library | Files: PDF, Office, HTML, email | Typed elements (JSON) | Not applicable | Self-host; paid platform | Apache 2.0 |
| ScrapeGraphAI | LLM extraction library | Web pages, local files | JSON fields per prompt | None in library; paid API has it | Self-host; paid API | MIT |
Connect it to your agent
MCP servers that let assistants like Claude, ChatGPT and Cursor use these tools directly.
Where this stage fits
- Chat with your documents: we would start with Firecrawl.
- Customer support agent: we would start with Apify.
- Coding agent: we would start with GitHub MCP Server.
- Research & knowledge graph: we would start with Crawl4AI.
- Assistant with long-term memory: we would start with Unstructured.
- Image generation pipeline: we would start with Firecrawl.
- Video production: we would start with Firecrawl.
- Voice agent: we would start with Firecrawl.
- Private, on-device AI: we would start with Unstructured.
- Document processing: we would start with Unstructured.
- Your own fine-tuned model: we would start with Crawl4AI.
- Workflow automation: we would start with Apify.