Knowledge Distillation and Network Augmentation for Efficient Machine Learning - Lecture 10

Explore knowledge distillation techniques in this comprehensive lecture from MIT's TinyML and Efficient Deep Learning Computing course. Delve into self and online distillation methods, as well as distillation for various tasks. Learn about network augmentation, an innovative training technique for tiny machine learning models. Gain insights into deploying neural networks on resource-constrained devices like mobile phones and IoT devices. Discover efficient inference and training techniques, including model compression, pruning, quantization, neural architecture search, and on-device transfer learning. Apply these concepts to optimize models for videos, point cloud data, and natural language processing tasks. Get hands-on experience implementing deep learning applications on microcontrollers, mobile devices, and quantum machines through an open-ended design project focused on mobile AI.

Lecture 10 - Knowledge Distillation | MIT 6.S965