InfiniFS - An Efficient Metadata Service for Large-Scale Distributed Filesystems

Explore an innovative approach to managing metadata in large-scale distributed filesystems through this 15-minute conference talk from FAST '22. Dive into the challenges faced by modern datacenters in maintaining filesystem metadata for billions of files, including load balancing, locality preservation, long path resolution, and near-root hotspots. Learn about InfiniFS, an efficient metadata service that addresses these issues through three key techniques: decoupling access and content metadata of directories, implementing speculative path resolution, and utilizing optimistic access metadata cache on the client-side. Discover how these solutions significantly improve both latency and throughput in metadata operations, providing stable performance for extremely large-scale directory trees with up to 100 billion files.

Intro
Outline
Large-Scale Distributed Filesystem
Filesystem Metadata
Challenges of efficient Metadata Service
InfiniFS Architecture
Key Designs
1. Access-Content Decoupled Partitioning
2. Speculative Path Resolution
Scalability
Breakdown
Conclusion