Data Locality in Kubernetes for Data-Intensive Workloads

Explore the challenges and solutions surrounding data locality in Kubernetes in this informative conference talk. Delve into the complexities of accessing data from cloud-native sources like AWS S3 and remote data warehouses when deploying data-intensive applications on Kubernetes. Examine the current practices of platform engineers, including data copying for I/O throughput optimization, and understand the associated risks and time constraints. Discover various approaches to emulate or introduce data locality in Kubernetes schedulers, weighing their advantages and disadvantages. Gain insights into the future of Kubernetes efficiency for data-intensive workloads and the critical role data locality will play in achieving higher performance.

On Data Locality in Kubernetes - Chen Wang, IBM Research & Shouwei Chen, Alluxio