Building a Streaming Microservice Architecture with Apache Spark Structured Streaming

Explore a technical deep dive into building a streaming microservice architecture using Apache Spark Structured Streaming and related technologies. Learn how to handle large-scale data processing, real-time aggregation, and event distribution in modern data platforms. Discover techniques for addressing data reliability issues in expanding systems and teams. Gain insights into a new mental model for constructing massive, interconnected services using Kafka, Google Protocol Buffers, gRPC, Parquet, Delta Lake, and Spark Structured Streaming. Benefit from lessons learned while developing a real-time insights platform at Twilio, focusing on data integrity and stream fault-tolerance. Cover topics including the big picture, Protocol Buffers, gRPC implementation, structured streaming, and more in this 39-minute presentation from Databricks.

Introduction
The Big Picture
Protocol Buffers
GRPC
GRPC Example
GRPC Messages
Service Definitions
Code
Protocol Stream
AdTrack
Structured Streaming
Scala PD
Recap