Discover How to Create Your Own Metadata-Driven ML Platform from Scratch

Learn how to build a metadata-driven machine learning platform from the ground up in this comprehensive 33-minute conference talk by Ted Chang and Yihong Wang from IBM. Explore the integration of a data lakehouse with a metadata-driven orchestration engine using open-source tools like Presto and Kubeflow. Gain insights into Presto's functionality within the open lakehouse concept for efficient data analytics, including setup, data source connections, and query execution. Dive into Kubeflow, a Kubernetes-based platform for ML workflows, covering setup, metadata-driven pipeline creation, and notebook server deployment. Discover distributed training operators, hyper-parameter tuning, and model serving within the Kubeflow ecosystem. Master the automation of the ML lifecycle, from data ingestion to model deployment, while implementing traceability and efficient workflow management. Explore techniques for integrating real-time data streams with your ML platform to ensure model relevance and effectiveness.

Discover How to Create Your Own Metadata-Driven ML Platform from Scratch - Ted Chang & Yihong Wang