Explaining Model Decisions and Fixing Them Through Human Feedback

Explore the intricacies of explaining and improving AI model decisions through human feedback in this 58-minute conference talk by Ramprasaath Selvaraju, a Sr. Machine Learning Scientist at Artera. Delve into algorithms that provide explanations for deep network decisions, focusing on building user trust, incorporating domain knowledge, learning grounded representations, and correcting unwanted biases in AI models. Gain insights into visual explanations, interpretability in AI evolution, and applications in image captioning, visual question answering, and medical AI. Examine topics such as Grad-CAM, multi-modal transformer architectures, contrastive self-supervised learning, and techniques for making models resilient to background changes. Learn about innovative approaches like Human Importance-aware Network Tuning (HINT) and Sub-Question Importance-aware Network Tuning for improving AI performance and addressing biases in vision and language models.

Intro
Interpretability in different stages of Al evolution
Approaches for visual explanations
Visualize any decision
Visualizing Image Captioning models
Visualizing Visual Question Answering models
Analyzing Failure modes
Grad-CAM for predicting patient outcomes
Extensions to Multi-modal Transformer based Architectures
Desirable properties of Visual Explanations
Equalizer
Biases in Vision and Language models
Human Importance-aware Network Tuning (HINT)
Contrastive Self-Supervised Learning (SSL)
Why SSL methods fail to generalize to arbitrary images?
Does improved SSL grounding transfer to downstream tasks?
CAST makes models resilient to background changes
VQA for visually impaired users
Sub-Question Importance-aware Network Tuning
Explaining Model Decisions and Fixing them via Human Feedback
Grad-CAM for multi-modal transformers