robots.txt file · Robots Exclusion Protocol
robots.txt is a file at a site’s root that tells bots which areas they may crawl. It is the main lever to allow or block AI crawlers, but it controls crawling, not indexing.
Standalone definition · 33 words
How does robots.txt work?
The file declares groups by user-agent with Allow and Disallow rules, and can reference sitemaps. It is standardized in RFC 9309. A well-behaved bot reads the file before crawling. Compliance relies on the bot’s good faith: it is not a technical barrier.
How do you manage AI crawlers in robots.txt?
One group per user-agent: for example GPTBot for OpenAI training, OAI-SearchBot for ChatGPT search, Google-Extended for use in Gemini, PerplexityBot for Perplexity. Decide by purpose, test the rules and check in logs that bots follow them.
Which mistakes should you avoid?
A blanket Disallow left over from staging, blocking resources needed for rendering, using robots.txt to deindex (use noindex), contradictory rules between groups and a file returning a server error, which Google handles with caution.
Frequently asked questions
Does robots.txt deindex a page?
No. It blocks crawling. A blocked URL can stay indexed with no content.
Should I block AI bots?
Depending on the goal: refusing training while allowing search is possible.
Where do I place the file?
At the domain root, for example /robots.txt, for each subdomain.
Related terms
Further reading
- Search Central Live Deep Dive Barcelona 2026: what Google said over the 3 days
- Microsoft Clarity, Bing Webmaster Tools: the Only GEO Stack That Actually Measures LLM Visibility
- Google’s Geo checklist: how to optimize your site for AI agents ?
- ← Back to the GEO glossary (114 terms)