Web Crawl
A web crawl is the result of systematically fetching web pages by following hyperlinks from a set of starting URLs, and a web crawler (also called a spider or bot) is the program that does the fetching. Search engines use crawls to build their indexes, archivists use them to preserve pages, and AI developers use them to collect text for training. Common Crawl is one specific public crawl dataset; "web crawl" is the general activity and artifact, of which Common Crawl is one prominent example.
The core loop of a crawler is a queue called the frontier: take a URL, fetch the page, extract the links, discard those already seen, and add the rest back to the queue. Even this simple loop raises hard engineering problems. URLs must be normalized so that equivalent addresses are not fetched twice; pages must be checked for near-duplicates; infinite "spider traps" such as calendars that generate endless links must be detected; and the frontier must be prioritized so that important or fresh pages are revisited more often than obscure ones. Large crawlers distribute this work over many machines, partitioning the frontier by host so that all requests to one site come from a controlled place.
Politeness is the set of conventions that stops a crawler from overwhelming or abusing the sites it visits. The most visible one is robots.txt, a plain text file at a site's root implementing the Robots Exclusion Protocol, proposed by Martijn Koster in 1994 and formalized as an IETF standard, RFC 9309, in 2022. It lets site owners name crawlers by user-agent string and list paths that are allowed or disallowed. A Crawl-delay directive is honored by some crawlers but is not part of the formal standard. Polite crawlers also limit requests per host, identify themselves with a user-agent and contact information, back off on server errors, and respect HTTP caching headers.
For AI, the crawl is the upstream source of most pretraining text, and that has made crawler etiquette a live policy question. Language-model developers either run their own crawlers or reuse public ones. OpenAI documented its GPTBot crawler and how to block it through robots.txt in August 2023, and other AI companies and datasets published their own user-agent names, such as Common Crawl's CCBot. Many publishers responded by adding disallow rules for such bots. Robots.txt is a voluntary signal rather than an access control: it is not a technical barrier, it carries no authentication, and it says nothing about whether copying content for training is lawful, which is a separate and still-contested legal question in several jurisdictions.
A web crawl is also a biased, partial snapshot, not "the internet". Crawlers cannot see content behind logins, paywalls, or forms, and pages rendered only by JavaScript are missed by simple fetchers. Link-following favors well-linked, popular, English-heavy sites, so what gets collected reflects link structure and the crawler's choices, not the distribution of human knowledge. The content is also full of spam, templated text, machine-generated pages, and personal data, which is why raw crawls need heavy filtering and deduplication before anything resembling training data emerges, and why decisions about robots.txt compliance and removal requests carry ethical weight.
Running a crawl yourself versus using an existing one is a real trade-off. A custom crawl offers control over freshness, domains, and compliance, but costs bandwidth, infrastructure, engineering time, and the reputational burden of hitting other people's servers. Reusing a public dataset such as Common Crawl is far cheaper and spares website operators from duplicate traffic, but it fixes the coverage and timing and offers little say over what was collected.
Read web crawl as automated, link-following reading of the public web under a code of good manners: the fetch loop is simple, the politeness rules are what make it sustainable, and the resulting snapshot is always partial, biased toward what is linked and open, and raw material rather than finished data.