BERTology Meets Biology - Interpreting Attention in Protein Language Models

Explore the intersection of natural language processing and biology in this 37-minute video explanation of the paper "BERTology Meets Biology: Interpreting Attention in Protein Language Models." Dive into how BERT models trained on protein sequence data can uncover non-trivial higher-order biological properties of proteins. Learn about the journey from DNA to proteins, the application of BERT to amino acid sequences, and protein structure. Discover how inspecting BERT reveals insights into amino acid substitution, contact maps, and binding sites. Gain understanding of linear probes and their significance in this context. Benefit from a comprehensive outline covering key topics, including an introduction to protein biology, the adaptation of NLP techniques to protein analysis, and the interpretation of attention mechanisms in protein language models.

- Intro & Overview
- From DNA to Proteins
- BERT for Amino Acid Sequences
- The Structure of Proteins
- Investigating Biological Properties by Inspecting BERT
- Amino Acid Substitution
- Contact Maps
- Binding Sites
- Linear Probes
- Conclusion & Comments