Common Crawl · Open web corpus
Common Crawl is a non-profit foundation that maintains a free, open repository of web crawl data, used as a source of training data by many language models. Its crawler is called CCBot.
Standalone definition · 32 words
What does it contain?
The site states more than 300 billion pages over 15 years, 3 to 5 billion new pages each month and more than 10,000 research papers citing it. It also offers URL indexes and web graphs. The data is accessible to anyone.
Why is it a GEO topic?
LLMrefs argues that Common Crawl chooses what to crawl using a harmonic centrality measure and that optimizing for it can increase authority in training data. This is a vendor analysis to read with caution. What the corpus contains weighs on a model’s parametric memory, hence on what it knows about a brand without web search.
How do you handle it?
Check logs for CCBot, decide whether to allow its crawling in robots.txt and follow the foundation’s opt-out ledger. Blocking CCBot reduces presence in future corpora, with no effect on citations from live searches.
Frequently asked questions
Is Common Crawl a search engine?
No. It is a data corpus used by researchers and model providers.
Does being in Common Crawl guarantee being cited?
No. It can influence the model’s knowledge, not source selection during a live search.
Related terms
Further reading
- Profound vs Otterly vs Ahrefs Brand Radar: query fan-out comparison
- ← Back to the GEO glossary (114 terms)