Delta from a Data Engineer's Perspective - Creating Robust Big Data Solutions

Explore the daily challenges faced by data engineers in this 41-minute presentation that delves into the essential components for building robust end-to-end Big Data solutions. Gain insights into Delta architecture from a data engineer's perspective, covering topics such as Spark streaming, processing rates, handling corrupted data, and schema enforcement. Learn about the intricacies of Delta operations, including deletes, transaction logs, and time travel capabilities. Discover how to leverage Spark DataFrames and utilize powerful commands like vacuum for optimizing Delta tables. Walk away with a comprehensive understanding of Delta's features and their practical applications in data engineering workflows.

Introduction
Delta Demo
Getting Started
Spark Stream
Processing Rate
Corrupted Data
Schema Enforcement
Recap
Absurd
New Records
Spark DataFrame
Results
Delete
Explanation of Delete
History Command
Transaction Log
Delta Log
Transaction Data
Time Travel
Timestamp
Vacuum