Crawl budget
Crawl budget is the number of pages a search engine agrees to crawl on a site within a given time. According to Google, it only concerns very large sites: over a million pages with moderately changing content, or over 10,000 pages updated every day.
Standalone definition · 44 words
What is crawl budget made of?
Two elements: crawl capacity (what the server can handle without slowing down) and crawl demand (the engine’s interest in URLs, based on popularity and freshness). A slow server reduces capacity, low-value URLs reduce useful demand.
Which waste should you fix?
Infinite facets and URL parameters, duplicate pages, redirect chains, recurring 4xx and 5xx errors, soft 404 pages and sitemaps containing non-indexable URLs. Making them uncrawlable with robots.txt or removing them focuses crawling on useful pages.
Crawl budget and AI bots
AI bots add request load on the server. Logs show their volume and distribution. Under pressure, you can limit their access by purpose rather than blocking everything.
Frequently asked questions
Does my site need to worry about it?
Probably not if it has fewer than a few thousand pages. Google’s thresholds target very large sites.
Does crawl-delay work?
Googlebot ignores crawl-delay. Other bots may respect it.
How do I measure it?
Search Console’s Crawl Stats report and log analysis.
Related terms
Further reading
- Search Central Live Deep Dive Barcelona 2026: what Google said over the 3 days
- Checklist GEO and LLM / AI 2026
- ← Back to the GEO glossary (114 terms)