Stop AI Scraping: Protect Your Site with robots.txt and ai.txt
AI crawlers now scrape the web at scale. Learn how to stop AI scraping with robots.txt, ai.txt, and server rules - and when not to.
💡 What You Will Learn
AI crawlers now scrape the web at scale. Learn how to stop AI scraping with robots.txt, ai.txt, and server rules - and when not to.
📜 Table of Contents
GPTBot, ClaudeBot, PerplexityBot and dozens of AI crawlers hit your server daily - some politely, some hammering bandwidth. Stopping AI scraping is a policy decision with three levers: robots.txt, ai.txt, and server-level rules.
The Three Levers
1. robots.txt: disallow specific AI user agents (GPTBot, Claude-Web, PerplexityBot, Google-Extended). Standard, respected by compliant crawlers. 2. ai.txt (2025+ standard): a machine-readable file declaring what AI crawlers may index - newer and understood by AI search engines. 3. Server rules: nginx or Cloudflare blocks for abusive crawlers that ignore robots.txt, rate-limit by user agent.
When NOT to block: if you want AI search visibility (Perplexity, ChatGPT Search citations), blocking GPTBot and friends removes you from AI answers entirely. Many sites now allow Google-Extended and PerplexityBot while blocking the rest. The nuance: blocking all AI crawlers protects bandwidth but costs AI-referral traffic - decide based on whether AI search sends you visitors.
Comparison
| Agent | Block? | Reason |
|---|---|---|
| GPTBot | Optional | OpenAI training + search |
| Claude-Web | Optional | Anthropic training |
| PerplexityBot | Optional | AI search citations |
| Google-Extended | Think twice | GEO visibility |
FAQ
Q: Will blocking AI crawlers hurt my Google ranking?
A: No - Google-Extended is separate from the main Googlebot index. Blocking training crawlers does not affect normal search ranking.
Q: What about crawlers that ignore robots.txt?
A: Block at the server level: nginx location rules or Cloudflare bot management, and rate-limit by user agent.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only โ no paid placements.
