Testing Stochastic AI Models with Hypothesis

Explore property-based testing for stochastic AI models using the Hypothesis library in this PyCon US talk. Dive into the challenges of testing advanced AI systems and learn how to generate random examples of plausible edge cases. Discover the theory behind property-based testing and see practical use cases demonstrating the implementation of the Hypothesis library. Cover topics such as example-based testing, properties like commutativity and invariant functions, metamorphic testing, and Hypothesis strategies. Learn to define custom strategies, transform data functions, debug Hypothesis strategies, and implement repeatable random testing with shrinking capabilities. Gain insights into additional components like image rotation to enhance your AI model testing approach.

Intro
About me
Table of Content
Example-based testing- example
Example-based testing - issues
Example-based testing - merge_sort
Property: Commutativity
Property: Invariant functions
Property: The test oracle
Property-based testing
What are the properties in the example?
Metamorphic Testing
Metamorphic Relations
Hypothesis Library
Hypothesis basic strategies
merge_sort test
Define you own strategy
Transforming data functions
Debug hypothesis strategies
Repeatable random testing
Shrinking
Additional Components
Rotate the image