Preparing Fineweb - A Finely Cleaned Common Crawl Dataset for Language Models

Explore the process of creating Fineweb, a refined Common Crawl dataset, in this 37-minute video presentation. Dive into the intricacies of data processing pipelines, dataset comparisons, and quality filtering techniques. Learn about the evolution of Common Crawl data and how it compares to other datasets like C4, RefinedWeb, and those used for Llama 3 and Phi-3. Discover the use of Datatrove for efficient data processing and various deduplication strategies. Gain insights into LLM-assisted dataset filtering and the training of classifiers for improved data quality. Understand the advantages of Fineweb Edu Latest Crawl and explore theories on why Llama 3 outperforms its predecessor. Access additional resources, including presentation slides, dataset links, and code examples to further enhance your understanding of large-scale web data processing for machine learning applications.

Common Crawl Data Processing Pipeline
Video Overview
Common Crawl Raw Dataset
Common Crawl improves over time?
Dataset Comparisons: C4, RefinedWeb, Fineweb, Llama 3, Phi-3
Data Processing Pipeline and Datatrove
Quality filters: Gopher, C4, Fineweb
Deduplication strategies
Fineweb edu: LLM assisted dataset filtering
Training a classifier for dataset filtering
My recommendation: Fineweb Edu Latest Crawl
Why is Llama 3 better than Llama 2?