Firecrawl指南2026：把任意网页变成LLM友好的干净Markdown
把原始HTML喂给LLM会浪费大量token在菜单和脚本上。Firecrawl提取正文并返回干净Markdown。
Firecrawl: Web Scraping Built for LLMs, Not for Humans
Firecrawl is the web scraping API designed for AI workloads - it takes a URL and returns clean, structured markdown instead of raw HTML. It went open source in April 2025 and has since become one of the fastest-growing scraping projects ever, reaching 158,989 GitHub stars by August 2026.
Why Raw HTML Is the Wrong Format for LLMs
A typical news article page is 300-800KB of HTML. The actual article text is maybe 20-40KB. The rest is navigation, scripts, ads, and tracking. Feeding the whole page into an LLM wastes tokens, hurts retrieval quality, and often exceeds context limits. Firecrawl solves this by rendering the page (including JavaScript), extracting the main content, and returning it as clean markdown - typically 90-95% smaller than the raw HTML.
What It Does Beyond Simple Extraction
- Crawl mode - start from a sitemap or a seed URL and get every page as markdown, with depth and path filters.
- Search mode - run a search query and return the top results as markdown, which is how AI search apps build their indexes.
- Structured extraction - ask for specific fields (prices, names, dates) and it returns JSON.
- Screenshot and PDF endpoints - for pages that need visual capture.
- MCP support - it exposes itself as a Model Context Protocol server, so Claude or any MCP client can scrape directly.
Self-Hosting vs Cloud
The repo is AGPL-3.0 licensed. Self-hosted, you control costs and data flow - important when scraping sensitive or internal sites. The hosted API handles proxy rotation, CAPTCHA, and scale for you, which is worth it when you need thousands of pages reliably. Many teams start with the free tier, then self-host once volume grows.
A Typical RAG Pipeline Use
Scrape company docs with Firecrawl crawl mode, chunk the markdown, embed it into a vector store (Qdrant, Milvus, Chroma), and you have a support chatbot grounded in live documentation that updates when the site updates.
FAQ
Is Firecrawl free? Open source and self-hostable; the cloud has a free tier with monthly credits.
Does it handle JavaScript-heavy sites? Yes - it renders with a real browser before extracting.
Can it bypass CAPTCHAs? The hosted service handles common anti-bot measures; self-hosted depends on your proxy setup.
Is it legal to scrape with it? Scraping legality depends on the site's terms and your jurisdiction; respect robots.txt and rate limits.
