PySpark in Apache Spark 3.3 and Beyond

Explore the latest advancements in PySpark introduced with Apache Spark 3.3 and get a glimpse of future developments in this 35-minute Databricks conference talk. Dive into the evolution of PySpark since Project Zen's inception in Apache Spark 3.0, including improved error messages, type hints for autocompletion, and visualization implementations. Learn about the popular Pandas API on Spark, introduced in Apache Spark 3.2, which allows running pandas API on Apache Spark. Discover the new features in Apache Spark 3.3, such as expanded API coverage, faster default indexing in Pandas API on Spark, datetime.timedelta support, new PyArrow batch interface, enhanced autocompletion, Python & Pandas UDF profiler, and new error classification. Gain insights into the current efforts and roadmap for PySpark beyond Apache Spark 3.3, covering aspects of functionality, productivity, usability, performance, and feature parity.

Intro
Who are you?
Project Zen
What is this talk about?
Pandas API on Spark
New Functionalities
Productivity
Usability
Performance
Feature parity
PySpark in Apache Spark 3.3
PySpark in future Apache Spark
DATA-AI SUMMIT 2022