Managing FoundationDB at Scale

Explore Apple's approach to managing hundreds of FoundationDB clusters with high availability requirements in this 21-minute conference talk. Delve into core workflows for cluster lifecycle management, high-level design considerations, and strategies for handling common operational challenges. Learn about installing binaries and configurations, adding new processes, changing database configurations, removing processes, and modifying coordinators. Examine bounce strategies, including rolling bounces and DR cutover alternatives. Gain valuable insights into the specialized tooling and careful design principles employed by Apple to effectively manage FoundationDB at scale.

Overview
Overall Design
Design Considerations
Core Operation Loop
Installing Binaries and Conf
Add New Processes
Change Database Configuration
Removing Processes
Change Coordinators
Bouncing Instances
Bounce Strategy
Wait, what? Come on.
Alternative: Rolling Bounces
Alternative: DR Cutover
Conclusion