YoVDO

Best Practices for Unit Testing PySpark

Offered By: Databricks via YouTube

Tags

PySpark Courses Software Development Courses Software Testing Courses Databricks Courses GitHub Actions Courses Unit Testing Courses Data Engineering Courses Continuous Integration Courses Integration Testing Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Discover best practices for unit testing PySpark code in this 19-minute talk by Matthew Powers, Staff Developer Advocate at Databricks. Learn how to create PySpark unit tests that run locally and in CI via GitHub actions, structure PySpark code for easy unit testing, and implement integration tests with a cluster for staging datasets. Gain insights on reducing production bugs, simplifying code refactoring, and enhancing overall code safety. Explore additional resources like the Big Book of Data Engineering and The Data Team's Guide to the Databricks Lakehouse Platform to further expand your knowledge in this field.

Syllabus

Best Practices for Unit Testing PySpark


Taught by

Databricks

Related Courses

Automated Visual Software Analytics
openHPI JavaScript Testing
Udacity Desarrollo de Aplicaciones Web: Prueba y Despliegue
University of New Mexico via Coursera 软件工程
Peking University via Coursera Building R Packages
Johns Hopkins University via Coursera