Designing Resilient Data Pipelines

Explore strategies for designing resilient data pipelines in this 30-minute conference talk from SREcon19 Americas. Learn how to quickly recover from failures, ensure data accuracy, and minimize risks associated with pipeline updates. Discover techniques for reducing operational complexity, implementing observability, and enhancing data pipeline reliability. Gain insights into metrics collection, status monitoring, immutable inputs, declarative pipelines, and data validation. Address common concerns and best practices for maintaining robust data pipelines in complex environments.

Intro
What are we going to cover today?
Data Pipelines Crash Course
Observability
Metrics Collection
Pipeline Latency
Status Monitoring
Immutable Inputs
Benefits of Idempotency and immutability
Declarative Pipeline Example
Benefits of Pipelines as Code
Usable Data
Pros and cons of Data Validation
Adding Validators to Data Pipelines
Common Concerns