Automating Spark Upgrades and Migrations at Netflix

Discover how Netflix automates Apache Spark upgrades and migrations in this 41-minute conference talk. Learn about open-source tools for rewriting Spark code, techniques for testing Spark jobs in production, and methods for tracking job states. Explore the process of migrating to a containerized environment and gain insights from user experiences. Acquire valuable skills for upgrading Spark pipelines without stress and validating them using the write-audit-publish pattern. Ideal for data scientists, ML engineers, and platform engineers managing Spark infrastructure, this talk provides practical strategies for handling legacy data products and evolving AI libraries. Presented by Holden Karau and Robert Morck from Netflix, the session offers expert guidance on streamlining Spark upgrades and migrations in large-scale data environments.

Stranger Triumphs: Automating Spark Upgrades & Migrations at Netflix