Speed is Not All You Need for Data Processing
Offered By: PyCon US via YouTube
Course Description
Overview
Explore the evolving landscape of data processing frameworks in this PyCon US talk by Kevin Kho and Han Wang. Delve into the comparison between traditional tools like Pandas and newer options such as Polars and DuckDB. Examine the common assumptions about speed and performance in data processing, particularly focusing on Pandas and Pandas UDFs in relation to native Spark. Through benchmarks across various data sizes, discover surprising insights that challenge conventional wisdom about tool selection. Learn why practitioners should prioritize tools that best serve their needs rather than solely focusing on perceived speed advantages. Gain valuable perspective on making informed decisions when choosing data processing frameworks for your projects.
Syllabus
Talks - Kevin Kho, Han Wang: Speed is Not All You Need for Data Processing
Taught by
PyCon US
Related Courses
CS115x: Advanced Apache Spark for Data Science and Data EngineeringUniversity of California, Berkeley via edX Big Data Analytics
University of Adelaide via edX Big Data Essentials: HDFS, MapReduce and Spark RDD
Yandex via Coursera Big Data Analysis: Hive, Spark SQL, DataFrames and GraphFrames
Yandex via Coursera Introduction to Apache Spark and AWS
University of London International Programmes via Coursera