When Is Scale Enough?

Explore the impact of scaling in deep learning models and datasets in this comprehensive lecture from the Deep Learning Theory Workshop and Summer School. Delve into Ethan Dyer's analysis of performance predictability in relation to model, dataset, and compute scale, with a focus on deep learning and large language models. Examine scaling in linear models as a simplified system that demonstrates phenomena found in more complex networks. Investigate empirical research aimed at determining which problems can be effectively solved through scaling alone and which require alternative approaches. Gain valuable insights into the limitations and potential of scale in advancing artificial intelligence and machine learning capabilities.

When is Scale Enough?