Architectural Best Practices for Large-Scale Data Systems

Explore architectural best practices for large-scale data systems in this comprehensive conference talk from Conf42 CN 2024. Dive into various storage architectures and retrieval methods, comparing B-trees, quadtrees, LSM trees, R-trees, and inverted indexes. Learn why specific data structures are optimal for different scenarios, such as B-trees for distributed email services and quad-trees for proximity services. Examine database partitioning techniques, including local and global secondary indices, and understand their trade-offs. Discover conflict-free replicated datatypes (CRDTs) as a solution to write conflicts in multi-leader replication. Finally, contrast inside-out and outside-in architectural approaches for designing robust, scalable data systems.

intro
preamble
storage architecture & retrieval
b-trees vs quadtrees vs lsm trees vs r-trees vs inverted index
why b-trees in distributed email service
why quad-trees in proximity service
why lsm-trees in write-heavy system
why inverted indexx in search engines
partitioning database:- secondary indices
partitioning secondary index - local
partitioning secondary index - global
local vs global
conflict free replicated datatypes crdts
write conflict in multi-leader replication
solution: conflicat free replicated datatypes
inside-out vs outside-in architecture
inside out vs outside in