What Can AI Do for Me - Evaluating Machine Learning Interpretations in Cooperative Play

Explore a 20-minute conference talk from the 24th International Conference on Intelligent User Interfaces that delves into evaluating machine learning interpretations in cooperative play. Discover how researchers propose a novel approach to assess the interpretability and utility of AI models, particularly in natural language processing. Learn about their experimental design using a question answering task called Quizbowl, where both trivia experts and novices team up with AI. Gain insights into three different interpretation methods used by the computer to communicate predictions, and understand the framework for measuring interpretation effectiveness based on improved human performance. Examine the detailed analysis of human-AI cooperation, including regression analysis, bias analysis, and fine-grained analysis of buzzing positions. Uncover valuable design guidance for creating effective human-in-the-loop settings in natural language processing applications.

Introduction
Machine learning success story
Limitations of machine learning
Humancentered
Traditional approach
Human performance
Communication
Evaluation framework
Quiz Bowl
Quiz Bowl online
Sentence order
Human component
Alternative answers
Explanation by examples
Explanation by features
Interface
Data
Regression analysis
Bias analysis
Fine grain analysis
Buzzing position
Failed interpretation
Conclusion
More detailed analysis