YoVDO

How We Used Kafka to Scale Database Infrastructure

Offered By: USENIX via YouTube

Tags

SREcon Courses NoSQL Databases Courses

Course Description

Overview

Explore a comprehensive conference talk from SREcon19 Asia/Pacific that delves into scaling database infrastructure using Kafka. Learn about LinkedIn's home-grown NoSQL document store, its initial MySQL-based design, and the challenges faced. Discover the major changes implemented to replace MySQL replication with Kafka-based replication, including in-depth coverage of the implementation. Gain insights into Kafka configuration options for reliable delivery and application logic ensuring "exactly once delivery" and "rejection of out-of-band messages." Understand the workings of the NoSQL datastore, challenges with MySQL-based replication, how Kafka helped scale the infrastructure, and overall strategies for using Kafka for database replication at scale. The talk covers topics such as Espresso's features, current scale, basic architecture, replication requirements, local and cross-colo replication, cluster expansion, implementing Kafka-based replication, global transaction identifiers, replication flow, message protocols, checkpointing, and handling producer failures.

Syllabus

Intro
Espresso: Features
Espresso: Current Scale
Espresso : Basic Architecture
Espresso: Replication Requirements
Espresso : Local Replication
Espresso: Cross Colo Replication Legacy
Limitations: Per Instance Replication
Espresso: Replication Using Kafka
Cluster Expansion
Advantages: Per Partition Replication
Implementing Kafka based replication
Global Transaction identifier
Replication flow
Message protocol - Mastership Handoff
Checkpointing - Producer
Checkpointing - Consumer
Zombie Writes.
Conclusion
References
Producer Failure...


Taught by

USENIX

Related Courses

How to Not Destroy Your Production Kubernetes Clusters
USENIX via YouTube
SRE and ML - Why It Matters
USENIX via YouTube
Knowledge and Power - A Sociotechnical Systems Discussion on the Future of SRE
USENIX via YouTube
Tracing Bare Metal with OpenTelemetry
USENIX via YouTube
Improving How We Observe Our Observability Data - Techniques for SREs
USENIX via YouTube