Common Crawl

Common Crawl Foundation is a California-registered 501(c)(3) non-profit organization, founded in 2007 by Gil Elbaz with the mission of democratizing access to web data. The organization builds and maintains an open web crawl data repository available free of charge to anyone. Data is collected by the CCBot crawler (based on Apache Nutch), and the archive is stored on Amazon S3 through the AWS Open Data Sponsorship Program. The dataset comprises more than 250 billion pages and is expanded by several billion new pages each month. By 2024, Common Crawl data had been cited in more than 10,000 scientific papers. The organization has been and continues to be a key provider of training data for leading language models, including GPT-3 and GPT-4 (OpenAI), Gemini (Google DeepMind), LLaMA (Meta), and Anthropic's models. The registered office is located in Beverly Hills, California, USA. For a long time Common Crawl employed a single person; since 2023 the organization has been expanding its team and actively seeking grants. The Executive Director is Rich Skrenta and the Board Chair is Gil Elbaz.