Airflow's Limitations for MLOps - Challenges and Alternatives

Explore a thought-provoking podcast episode featuring Stephen Bailey, who shares his insights on data platforms and MLOps challenges. Delve into topics such as training stakeholders, modular job design, and data classification for improved user understanding. Discover the importance of balancing speed with perfection in data engineering projects. Learn about Stephen's experience at Whatnot, a live shopping marketplace for collectibles, and his previous work in privacy tech. Gain valuable perspectives on orchestration layers, recommender systems, and the controversies surrounding Airflow in MLOps. Connect with the MLOps community through various channels and explore related resources, including Stephen's blog post "Airflow's Problem" and the "Fundamentals of Data Engineering" book.

[] Stephen's preferred coffee
[] Introduction to co-host Joe Reis
[] Takeaways
[] Subscribe to our newsletters!
[] Shout out to our sponsor, Wallaroo!
[] Whatnot
[] Stephen's side hustle
[] Stephen's work breakdown at Whatnot
[] Fundamental tensions in the data world
[] Initial questions to answer that you were on the right path
[] Recommender systems
[] Coordinating with ML teams
[] Daxter
[] Too advanced, more challenging
[] Orchestration layer
[] Decision criteria
[] Human design aspect of Daxter
[] Orchestration layer centralization and sharing knowledge with stakeholders
[] Airflow's Problem and the reception it got on Hacker News
[] Java of the Data World
[] Absence of DAGS
[] Interaction with the software engineering team
[] Different services
[] Modernization
[] Stephen's Philosophies
[] Wrap up