YoVDO

Best Practices for Unit Testing PySpark

Offered By: Databricks via YouTube

Tags

PySpark Courses Software Development Courses Software Testing Courses Databricks Courses GitHub Actions Courses Unit Testing Courses Data Engineering Courses Continuous Integration Courses Integration Testing Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Discover best practices for unit testing PySpark code in this 19-minute talk by Matthew Powers, Staff Developer Advocate at Databricks. Learn how to create PySpark unit tests that run locally and in CI via GitHub actions, structure PySpark code for easy unit testing, and implement integration tests with a cluster for staging datasets. Gain insights on reducing production bugs, simplifying code refactoring, and enhancing overall code safety. Explore additional resources like the Big Book of Data Engineering and The Data Team's Guide to the Databricks Lakehouse Platform to further expand your knowledge in this field.

Syllabus

Best Practices for Unit Testing PySpark


Taught by

Databricks

Related Courses

Web Engineering III: Quality Assurance
Technische Hochschule Mittelhessen via iversity Introduction to Cloud Infrastructure Technologies
Linux Foundation via edX DevOps for Developers: How to Get Started
Microsoft via edX Accelerate Software Delivery using DevOps
Microsoft via edX Building R Packages
Johns Hopkins University via Coursera