The Robots.txt Mistake That Blocks AI Crawlers (And How to Fix It)
Many brands block AI crawlers in robots.txt without intending to. The fix is fifteen minutes of work. Without it, the brand is invisible to the models that drive buyer recommendations.
Key Highlights
- Many brands block AI crawlers in robots.txt without intending to. The fix is fifteen minutes of work. Without it, the brand is invisible to the models that drive buyer recommendations
- Common AI crawlers to explicitly allow include GPTBot (ChatGPT), ClaudeBot (Claude), Google-Extended (Gemini training), PerplexityBot (Perplexity), and CCBot (Common Crawl, used by many models)
- Blocking AI crawlers does not protect the brand from being cited. It prevents the brand from being cited well. The decision is a strategic one with a clear right answer for most B2B brands
- Brands that audit robots.txt monthly catch accidental blocks quickly. Brands that do not often lose citation rate for weeks before noticing
Why this article exists
Robots.txt is the AEO problem most often discovered the wrong way: by a citation rate drop that takes weeks to diagnose. The cause is almost always a robots.txt directive that blocks an AI crawler, often added by a CDN or a security tool without an explicit decision by the marketing team.
This article maps the AI crawlers that matter, the directives to allow or disallow, and the audit cadence that catches accidental blocks quickly. The fix is fast. The diagnostic cost of not having shipped the fix is large.
The AI crawlers that matter
Each major AI model uses one or more named crawlers. The names change occasionally; current names as of 2026 include:
- GPTBot. OpenAI's crawler for ChatGPT.
- OAI-SearchBot. OpenAI's search-related crawler.
- ChatGPT-User. OpenAI's on-demand fetcher for user prompts.
- ClaudeBot. Anthropic's crawler for Claude.
- Claude-Web. Anthropic's on-demand fetcher.
- Google-Extended. Google's crawler for Gemini training.
- PerplexityBot. Perplexity's crawler.
- CCBot. Common Crawl, used by many models.
Each crawler can be allowed or disallowed independently. Brands that want to be cited by the model should allow the model's crawler.
How brands accidentally block AI crawlers
Three common patterns produce accidental blocks:
-
CDN security rules. Cloudflare, Fastly, and similar services often ship default rules that challenge or block bot-like user agents. The default rules sometimes catch AI crawlers.
-
Catch-all disallow. A robots.txt with a Disallow: / under User-agent: * blocks every crawler that does not have a specific allow rule. Brands often add this for legacy reasons and forget to allow AI crawlers explicitly.
-
Subdomain inconsistency. The main site has a clean robots.txt, but a subdomain (docs., blog., kb.) inherits an old configuration with restrictive rules.
The fix
The fix is a minimal robots.txt addition that explicitly allows the AI crawlers that matter. The exact directives depend on the brand's existing robots.txt structure, but the pattern is:
- Audit current robots.txt on the main domain and every subdomain.
- Audit CDN security rules to confirm AI crawlers are not being challenged at the edge.
- Explicitly allow each AI crawler that matters (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot at minimum).
- Verify with each crawler's documented test endpoint that the crawler can access representative pages.
- Set up monthly monitoring that flags any robots.txt change.
AI crawlers and the model each one feeds
| Crawler | Model | Recommendation for most B2B brands |
|---|---|---|
| GPTBot | ChatGPT | Allow |
| OAI-SearchBot | ChatGPT search | Allow |
| ClaudeBot | Claude | Allow |
| Google-Extended | Gemini | Allow |
| PerplexityBot | Perplexity | Allow |
| CCBot | Many models via Common Crawl | Allow |
| Bytespider | TikTok / ByteDance | Per brand policy |
Should any brand block AI crawlers?
Some brands have legitimate reasons to block specific AI crawlers, including intellectual property concerns, paywall enforcement, or competitive sensitivity. The decision should be made deliberately, not by accident.
For most B2B brands, the right move is to allow the major AI crawlers and to compete on citation surface architecture rather than on access control. Blocking the crawler does not protect the brand; it just prevents the brand from being cited well.
Get your free AI visibility audit
OnlyAEO will audit your robots.txt across every subdomain and CDN, identify accidental blocks, and return a corrected configuration inside one week. No commitment.
Get Your Free AuditFrequently Asked Questions
Does allowing AI crawlers mean the model trains on the content?+
Can a brand block training but allow on-demand fetching?+
How often does robots.txt change accidentally?+
Is robots.txt the only place to block or allow AI crawlers?+
Does OnlyAEO offer a free robots.txt audit?+
OnlyAEO
Expert insights on Answer Engine Optimization and AI visibility strategy.
Related Articles
From Anonymous to Cited: The Entity Activation Playbook for New Brands
New brands face an entity problem: AI models do not know they exist. This is the OnlyAEO playbook for activating brand entity recognition inside ninety days.
Read article
Building Entity Authority Through Knowledge Graph Signals
Entity authority is the foundation of every AEO program. Knowledge graph signals (Wikidata, Knowledge Panel, sameAs, Organization schema) build it. This guide maps the six moves OnlyAEO uses.
Read article
Entity Building for AI: How to Make LLMs Know Your Brand
Entity building is the AEO discipline of making sure LLMs know exactly what your brand is, what it does, and why it is authoritative. Learn the schema, knowledge base, and external footprint moves that lift entity strength.
Read article