Scalable Billion-point Approximate Nearest Neighbor Search Using SmartSSDs

Explore a groundbreaking conference talk on scalable billion-point approximate nearest neighbor search (ANNS) using SmartSSDs. Delve into the innovative SmartANNS solution, which leverages a hierarchical indexing methodology to overcome challenges in deploying ANNS algorithms on multiple SmartSSDs. Learn about the novel "host CPUs + SmartSSDs" cooperative architecture, dynamic task scheduling techniques, and a learning-based shard pruning algorithm that significantly enhance performance and scalability. Discover how this approach improves query per second (QPS) by up to 10.7 times compared to existing solutions, while achieving near-linear performance scalability for large-scale datasets using multiple SmartSSDs. Gain insights into the future of high-dimensional vector space searches and their applications in database and machine learning fields.

USENIX ATC '24 - Scalable Billion-point Approximate Nearest Neighbor Search Using SmartSSDs