YoVDO

Best Practices for Unit Testing PySpark

Offered By: Databricks via YouTube

Tags

PySpark Courses Software Development Courses Software Testing Courses Databricks Courses GitHub Actions Courses Unit Testing Courses Data Engineering Courses Continuous Integration Courses Integration Testing Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Discover best practices for unit testing PySpark code in this 19-minute talk by Matthew Powers, Staff Developer Advocate at Databricks. Learn how to create PySpark unit tests that run locally and in CI via GitHub actions, structure PySpark code for easy unit testing, and implement integration tests with a cluster for staging datasets. Gain insights on reducing production bugs, simplifying code refactoring, and enhancing overall code safety. Explore additional resources like the Big Book of Data Engineering and The Data Team's Guide to the Databricks Lakehouse Platform to further expand your knowledge in this field.

Syllabus

Best Practices for Unit Testing PySpark


Taught by

Databricks

Related Courses

Fundamentals of Scalable Data Science
IBM via Coursera Data Science and Engineering with Spark
Berkeley University of California via edX Master of Machine Learning and Data Science
Imperial College London via Coursera Data Analysis Using Pyspark
Coursera Project Network via Coursera Building Machine Learning Pipelines in PySpark MLlib
Coursera Project Network via Coursera