Crawling · Site exploration
Crawling is the automated traversal of a site by a bot to discover and fetch its pages. It is an absolute prerequisite: a page that is not crawled is neither indexed nor cited by engines that rely on crawling to retrieve their sources.
Standalone definition · 43 words
How does crawling work?
The bot starts from known URLs (links, sitemaps), fetches them, extracts the links it finds and schedules new visits. Googlebot then decides what to render, index and revisit. AI bots follow a similar principle, with policies specific to each provider.
What blocks crawling?
An overly restrictive robots.txt, server errors (5xx), high response times, links that exist only in JavaScript, redirect chains, orphan pages and protections that filter bots (WAF, anti-bot). Each is checked in logs and in the Crawl Stats report.
Why monitor crawling for GEO?
AI bots have their own behavior and most do not render JavaScript. Checking in logs which bots come, which URLs they read and which codes they get is the only way to know whether content is really accessible.
Frequently asked questions
Crawling and indexing: what is the difference?
Crawling fetches the page. Indexing records it in the index after rendering. A crawled page is not necessarily indexed.
How do I encourage a page to be crawled?
An internal link from crawled pages, a sitemap entry, a fast response and useful content.
Can I control an AI bot’s crawling?
Through robots.txt, with the user-agents documented by each provider.
Related terms
Further reading
- Search Central Live Deep Dive Barcelona 2026: what Google said over the 3 days
- Google Search Central Live Paris recap: Google Is Betting big on Agentic Commerce
- Google Search Central 2025 Zurich
- ← Back to the GEO glossary (114 terms)