MySQL Automation at Facebook Scale

Explore Facebook's massive MySQL database cluster management in this 50-minute SREcon15 talk. Discover how Facebook automates conventional DBA tasks to operate thousands of servers across multiple data centers. Delve into the design and architecture of their automation systems, including inventory management, automated provisioning, and central resource locking. Learn about server lifecycle management, handling full disks, RAID issues, and MySQL version updates. Examine the metadata database design and local triage processes. Understand how Facebook turns up new servers and manages maintenance at scale. Gain insights from real-world challenges, such as "The Case of the Helpful Janitor" and large-scale replacements. Prepare for an in-depth look at MySQL automation strategies employed by one of the world's largest database clusters.

Intro
What is this talk about?
Terminology
Inventory Management
Directory Service
Automated Provisioning
Automated Promotions
Central Resource Locking
Server's Lifecycle at Facebook
Full Disk
Bad RAID
Old MySQL Version
Metadata Database
Design Overview
Local Triage -MPS' Agent
Main Components
mps, examples
Picking a destination
Chosen Algorithm
Result: Pretty Graphs
Turning Up New Servers
Maintenance at Scale
The Case of the Helpful Janitor
REPLACE ALL THE THINGS!
Prepare for takeoff!
What happened?