Taming Data - State Challenges for ML Applications and Kubeflow

Explore the challenges and solutions for managing data and state in machine learning applications using Kubeflow on Kubernetes. Delve into the complexities of handling training data, library files, and models in large-scale AI/ML environments. Learn about various storage APIs, including POSIX/CSI solutions, NFS, S3, and HDFS, and their roles in addressing persistent storage challenges. Examine job operator patterns, model serving, data gravity, data locality, and data security issues in the context of Kubeflow applications. Gain insights into creating AI/ML environments capable of running thousands of pods and managing petabytes of training data efficiently.

Introduction
Data Scientists vs Data Science
State Challenges
Data Science Workflow
Job Operator Patterns
Model Serving
State
Data Gravity
Data Locality
Data Security