Automated Data Quality Checks for Big Datasets Using Machine Learning
Offered By: Data Science Festival via YouTube
Course Description
Overview
Explore automated data quality checks for large-scale datasets in this 32-minute conference talk from the Data Science Festival. Discover how major organizations implement real-time data reliability checks using data profiling and machine learning techniques. Learn about the open-source library Deequ, Spark framework, and reporting & notification tools for proactive data issue enforcement. Gain insights into a framework developed at Amazon and Visa for validating customer-facing data and its integration with notification tools based on statistical methods. Suitable for technical practitioners, this talk by Akshay Dineshkumar Jain from Innovate UK offers valuable knowledge on maintaining data quality in big data environments.
Syllabus
Data quality for big datasets
Taught by
Data Science Festival
Related Courses
CS115x: Advanced Apache Spark for Data Science and Data EngineeringUniversity of California, Berkeley via edX Big Data Analytics
University of Adelaide via edX Big Data Essentials: HDFS, MapReduce and Spark RDD
Yandex via Coursera Big Data Analysis: Hive, Spark SQL, DataFrames and GraphFrames
Yandex via Coursera Introduction to Apache Spark and AWS
University of London International Programmes via Coursera