Owl - Scale and Flexibility in Distribution of Hot Content

Explore a conference talk on Owl, a high-fanout distribution system for large data objects in Meta's private cloud. Learn how Owl combines a decentralized data plane using ephemeral peer-to-peer distribution trees with a centralized control plane for managing metadata and optimizing distribution. Discover how this innovative approach improves download speeds by 2-3 times compared to BitTorrent and previous static distribution trees. Understand the system's ability to handle over 800 petabytes of data daily, serving millions of client processes across 106 use cases with 55 different distribution policies. Gain insights into Owl's flexible policy interface, tracker sharding techniques, and the benefits of centralizing the control plane for scalability and customization.

Intro
Content distribution: a recurring
A lot of data to distribute
Exacting requirements
Prior approaches
Hierarchical caching
Owl: Centralization vs. decentrali
Owl: Ephemeral distribution tree
Scaling the control plane
Tracker sharding example
Tracker sharding with delegation
The Need for Flexibility
Emulation: Recording
Emulation: Replay
2021 Scaling: traffic vs. servers
BitTorrent vs. Owl in production
Conclusions