Dive into Streams with Brooklin

Explore a comprehensive overview of Brooklin, LinkedIn's managed data streaming service, in this 41-minute Strange Loop Conference talk. Learn how Brooklin addresses the challenges of scaling up to handle increasing data volume and supporting the proliferation of new data systems at LinkedIn. Discover the architecture and use cases of this centralized, extensible solution for continuously delivering data to nearline applications. Examine Brooklin's support for multiple pluggable sources and destinations, including data stores and messaging systems. Delve into its applications in change data capture (CDC), data propagation between different systems and environments, and Kafka data mirroring. Gain insights into Brooklin's production implementation, its role in replacing Kafka MirrorMaker at LinkedIn, and future development plans for this open-source project.

Intro
Nearline Applications
Heterogeneous Data Systems
Building the Right Infrastructure
Pluggable Sources & Destinations
Capturing Live Updates
Change Data Capture (CDC)
Streaming Bridge
Mirroring Kalka Data
Brooklin Kalka Mirroring
Application Use Cases
Client makes REST call to create datastream
Datastream is written to Zookeeper
Leader coordinator is notified of new datastream
Leader coordinator calculates work distribution
Leader coordinator writes the assignments to ZK
Zookeeper is used to communicate the assignments
Coordinators hand task assignments to consumers
Consumers start streaming data from the source
Consumers propagate data to producers
Producers write data to the destination
Destinations can be shared by apps
Brooklin Architecture
Brooklin in Production
Brooklin is now open-source!