CM3- A Causal Masked Multimodal Model of the Internet

Explore a comprehensive video featuring a paper explanation and in-depth interview with first author Armen Aghajanyan on CM3: A Causal Masked Multimodal Model of the Internet. Delve into the innovative approach of using HTML structure for machine learning, including text, hyperlinks, and images. Learn about the new Causally Masked Language Modelling training strategy and its implications for autoregressive language models. Discover how CM3 addresses limitations in current models and enables various applications such as text generation, captioning, image creation, and entity linking. Gain insights into the training process of large-scale models, dataset collection, content filtering, and experimental results. Understand the potential impact of CM3 on the future of universal models in artificial intelligence.

- Intro & Overview
- Directly learning the structure of HTML
- Causally Masked Language Modelling
- A short look at how to use this model
- Start of interview
- Feeding language models with HTML
- How to get bi-directionality into decoder-only Transformers?
- Images are just tokens
- How does one train such giant models?
- CM3 results are amazing
- Large-scale dataset collection and content filtering
- More experimental results
- Why don't we use raw HTML?
- Does this paper contain too many things?