Scaling Machine Learning Pipelines in Cloud

Explore scaling machine learning pipelines in the cloud with this NDC Oslo 2021 conference talk by Salman Iqbal. Learn how to overcome common hurdles in developing end-to-end data science and machine learning models that operate at scale and in real-time. Discover the benefits of using Kubeflow, an open-source Kubernetes-based platform that abstracts away non-machine learning tasks while retaining control. Understand how Kubeflow enables easy development of ML models, repeatable and portable deployments across diverse infrastructures, and scalable operations based on demand. Gain insights into current use cases of Kubeflow and how teams from various industries leverage cloud technology to scale their machine learning operations. The talk covers topics such as Docker, Kubernetes, scaling pipelines, GPU workers, experimenting with pipelines, and considerations for platform management. Ideal for data scientists, machine learning engineers, and professionals looking to optimize their ML workflows and infrastructure.

Intro
About Salman
Giveaway
Agenda
Who is this talk for
Training Models
Docker
Where can we run Kubernetes
Benefits of Kubernetes
Scaling Pipelines with Kubernetes
Introducing Kubernetes
Data Science Machine Learning Engineers
Kubernetes tools
Operators
Fashion mnist
Neural Network
Hyper Parameters
Auto ML
Serving
kubeflow
Why Kubernetes
Data Science and Machine Learning
QFlow Pipeline
MiniCube
Python Pipeline SDK
GPU Pipeline
GPU Worker
Experimenting
Running the pipeline
Kale
Notebook
Kubernetes Extension
Adding Labels
Creating a Pipeline
Scaling Pipelines
Kubernetes Autoscaling
Considerations
Platform Management
OpenAI
Summary
References
Thanks
Outro