AI bot · LLM crawler · AI user-agent
An AI crawler is the bot of an LLM provider, such as GPTBot, ClaudeBot, PerplexityBot or Google-Extended. Blocking in robots.txt the bot that feeds an engine’s search excludes the site from the matching generative answers, whereas blocking a training bot does not have the same effect.
Standalone definition · 46 words
How do AI crawlers work?
Each provider declares one or more user-agents with distinct purposes. OpenAI documents, for example, GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for user-triggered requests. Google-Extended is not a separate bot: it is a control token in robots.txt, with no effect on inclusion in Google Search.
What are the risks of blocking an AI crawler?
Blocking every AI bot by reflex removes the brand from answers where it could be cited. The decision is made by purpose: refuse training, allow search bots if AI visibility is a goal. Control happens in robots.txt and is checked in server logs.
How do you check what AI bots actually do?
In server logs. User-agents can be spoofed: confirm with the IP ranges published by providers. A robots.txt test proves neither the visit nor compliance with the rule.
Frequently asked questions
Do AI crawlers respect robots.txt?
Major providers document compliance with robots.txt for their official bots. For an unofficial bot, nothing guarantees it.
Should I allow GPTBot?
It depends on the goal: GPTBot concerns training, OAI-SearchBot concerns search. Allowing the second without the first is a possible setup.
Does an AI crawler render JavaScript?
Most do not, according to the Vercel and MERJ study of December 2024. Hence the value of server-side rendering.
Related terms
Further reading
- Microsoft Clarity, Bing Webmaster Tools: the Only GEO Stack That Actually Measures LLM Visibility
- Profound vs Otterly vs Ahrefs Brand Radar: 2026 Query Fan-Out Comparison
- ← Back to the GEO glossary (114 terms)