Unit Tests - Overcoming Challenges in Structured Streaming

Explore strategies for overcoming challenges in unit testing structured streaming jobs in this 31-minute conference talk by Bartosz Konieczny, a freelance data engineer. Learn how to improve control over your codebase and increase confidence in your CI/CD pipeline releases. Discover solutions to common pain points data engineers face when writing unit tests for streaming jobs, including how to control assertions for long-running processes and represent third-party data stores and APIs in code. While focused on streaming jobs, gain insights applicable to batch pipelines as well. Enhance your data quality and processing layer through improved unit testing techniques for data engineering projects.

Unit Tests: How to Overcome Challenges in Structured Streaming