Testing Machine Learning Models

Explore the critical importance of testing machine learning and AI models in this 31-minute PyCon US talk by Carlos Kidman. Discover why traditional validation metrics are insufficient and learn how to assess quality attributes such as model behaviors, usability, and fairness. Gain insights into the risks and biases present throughout the MLOps pipeline, and master techniques for testing model behaviors and fairness using both exploratory and automated strategies. Apply these concepts to real-world scenarios and state-of-the-art models. Walk away with practical ideas and techniques to test ML/AI systems from a user's perspective, enhancing your ability to ensure the quality and reliability of AI-driven solutions.

Intro
Head of Engineering
Define the Problem
Define Success and Assess Risk
Design the initial Architecture
4. Collect Data
Prepare Data
Train and validate Models
7. Test the Models
Lessons Learned
What does it look like?