GPT-Fast - Blazingly Fast Inference with PyTorch

Dive into a comprehensive video lecture featuring Horace He discussing techniques for accelerating inference using PyTorch native operations. Explore the concept of GPT-Fast, delve into PyTorch compile, and understand int8 and int4 quantization methods. Learn about speculative decoding, tensor parallelism, and various bonus optimizations to enhance inference speed. Gain insights into putting these techniques together for blazingly fast inference in PyTorch, with a Q&A session concluding the talk.

00:00 - Intro
00:45 - HyperStack GPUs! sponsored
02:23 - What is GPT-Fast?
08:40 - PyTorch compile
28:15 - int8 quantization
32:15 - Speculative Decoding
40:12 - Int 4 quantization
42:05 - Putting it all together, tensor parallelism
45:25 - Bonus optimizations
58:10 - Outro, questions