Finding Sources of Latency on Your Linux System

Explore techniques for identifying and analyzing sources of latency in Linux systems in this comprehensive 51-minute conference talk by Steven Rostedt from VMware. Delve into the concept of latency and its origins, examining both hardware and kernel-related causes. Learn about the Hardware Latency Detector and various kernel-level latency sources, including interrupt handling and preemption. Discover how to measure and trace latency using tools like PREEMPT_RT and the Scheduling Latency Tracer. Gain insights into advanced techniques such as histogram triggers and synthetic events for creating custom latency analysis tools. Master the skills to diagnose and address performance issues in Linux environments, enhancing system responsiveness and efficiency.

Intro
What is Latency
Where does latency come from?
Latency from Hardware
The Hardware Latency Detector
Latency from the Kernel
Interrupts disabled!
Measuring latency from interrupts
Preemption disabled
Preemption and interrupt disabled latency tracers
Tracing Latency from Interrupts with PREEMPT_RT (5.4.14-rt7)
The Scheduling Latency Tracer
Issues with the Latency Tracers
Histogram Triggers and Synthetic Events!
Creating a synthetic event
Making their disabled historgram
Creating a wake up latency synthetic event
Making the wakeup latency historgram
Making the wake up latency historgram