Common Crawl
Common Crawl is a nonprofit organization, founded in 2007 by Gil Elbaz, that crawls the public web and releases the results as a freely available, openly downloadable archive. It is the most widely used single source of text for large language models, not because it is a clean corpus but because it is large, free, and regularly refreshed. Researchers and companies that cannot afford to run their own web-scale crawler can start from its archives and apply their own filtering.
Common Crawl publishes new crawls on a roughly monthly cadence, and each crawl contains billions of pages amounting to many terabytes of compressed data. Its crawler identifies itself as CCBot and obeys robots.txt, and the crawl is a sample of the web, not a complete copy: it samples broadly across domains, and different crawls overlap only partly. The archive is hosted on cloud storage, where it can be downloaded or processed in place, and the data stretches back to 2008, which makes it attractive for studying how the web changes over time as well as for training.
Three file formats serve different needs: WARC, WAT, and WET. WARC (Web ARChive) files hold the raw crawl: the HTTP request and response, headers, and full HTML of each page. WAT files hold extracted metadata in JSON, such as links and header fields. WET files hold only the plain text extracted from each page, which is the easiest starting point but also the lossiest, since the generic extractor often keeps navigation text and boilerplate and discards structure such as tables and code formatting. Many serious pipelines therefore go back to WARC and run their own text extraction, which gives noticeably better quality.
GPT-3 (Brown et al., OpenAI, 2020) made Common Crawl's role in language modelling famous. The paper's training mix drew most of its roughly 300 billion training tokens from a filtered Common Crawl subset of about 410 billion tokens covering 2016 to 2019, weighted at around 60 percent of the training samples. To get there, the authors trained a classifier to prefer pages resembling higher-quality reference text, applied fuzzy deduplication, and added curated sources such as an expanded WebText, two books corpora, and Wikipedia. Other well-known derivatives followed: C4 (Raffel et al., 2020), the cleaned English subset of one 2019 snapshot used for T5, and later datasets such as RefinedWeb (2023) and FineWeb (2024), all built by filtering Common Crawl.
The main pitfall is treating Common Crawl as a finished dataset. Unfiltered, it is dominated by spam, boilerplate, duplicated text, adult content, and personal information, and it skews toward English and toward well-linked sites. It also contains public test sets and answer keys, so it is an important route for benchmark-contamination, and it has become a focus in debates over copyright and consent, since site owners did not necessarily anticipate model training. Mistakes in filtering propagate: a bad classifier threshold affects every model trained downstream. Opt-out through robots.txt applies to future crawls, but does not remove pages already archived.
Common Crawl is a distinct thing from a web-crawl in the general sense, and from any single model's data. A web-crawl is the activity and its output; Common Crawl is one shared, public, nonprofit-maintained instance of it, whereas commercial labs also run private crawlers whose contents are not public. Because so many models share the same upstream source, their knowledge and biases are more correlated than the diversity of model brands suggests.
Read Common Crawl as the web's public raw material rather than its finished library: it is an enormous shared snapshot that made open language-model research possible, but its value depends entirely on the quality of the filtering applied between the archive and the training run.