YoVDO

Best Practices for Unit Testing PySpark

Offered By: Databricks via YouTube

Tags

PySpark Courses Software Development Courses Software Testing Courses Databricks Courses GitHub Actions Courses Unit Testing Courses Data Engineering Courses Continuous Integration Courses Integration Testing Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Discover best practices for unit testing PySpark code in this 19-minute talk by Matthew Powers, Staff Developer Advocate at Databricks. Learn how to create PySpark unit tests that run locally and in CI via GitHub actions, structure PySpark code for easy unit testing, and implement integration tests with a cluster for staging datasets. Gain insights on reducing production bugs, simplifying code refactoring, and enhancing overall code safety. Explore additional resources like the Big Book of Data Engineering and The Data Team's Guide to the Databricks Lakehouse Platform to further expand your knowledge in this field.

Syllabus

Best Practices for Unit Testing PySpark


Taught by

Databricks

Related Courses

内存数据库管理
openHPI CS115x: Advanced Apache Spark for Data Science and Data Engineering
University of California, Berkeley via edX Processing Big Data with Azure Data Lake Analytics
Microsoft via edX Google Cloud Big Data and Machine Learning Fundamentals en Español
Google Cloud via Coursera Google Cloud Big Data and Machine Learning Fundamentals 日本語版
Google Cloud via Coursera