Lessons Learned From Machine Learning Pipelines in Production

Explore lessons learned from utilizing machine learning pipelines in production environments through this conference talk. Gain insights into overcoming challenges when applying experimental logic to real-world scenarios, addressing gaps between experimental and production environments, and leveraging Kedro to transform pipelines. Discover practical solutions for project template generation, adding nodes outside of notebooks, connecting nodes to develop pipelines, and removing loops inside extracted notebook nodes. Learn valuable takeaways from a validation scenario and potential solutions for implementing machine learning projects in production settings.

Intro
Outline
1. Background: Machine learning in production
2. Assumption: Job specialization in machine learning projects
1-3. Issue for applying logics into production environment
1-4. Gaps between experimental and production environment
1-5. Challenges towards production environment
1-7. Overview of validation scenario and its target ML system
1. Utilizing Kedro to overcome challenges
3-1. Solution 1: Transforming pipelines in Kedro style
2-3-2. Step 1-A: Project Template Generation by Kedro
2-3-4. Step 1-C: Adding node not in notebook
2-3-6. Step 1-D: Connecting nodes to develop pipeline
2-5-2. Solution 3. Removing loop inside nodes extracted from Jupyter notebook
1. What we learned in validation scenario: good points
3. Possible solution