Container Performance Analysis and Optimization

Explore container performance analysis techniques in this 42-minute talk by Brendan Gregg, Senior Performance Architect at Netflix. Dive into the challenges of monitoring and analyzing container performance, including resource-oriented analysis, hardware and soft limits, and container interactions. Learn about new methodologies and tooling required for effective container performance analysis. Discover how to navigate differences in privilege levels and namespaces when using performance tools within containers or on the host. Gain insights into Netflix's approach to analyzing and tuning containers for optimal speed and efficiency. Follow along as the speaker demonstrates successful performance analysis in a Docker container environment, covering topics such as control groups, Linux containers, CPU shares, and container OS configuration. Explore host analysis challenges, including physical resources, containers, and cgroups, as well as guest analysis challenges like disks and metrics namespaces. Learn about various analysis tools and techniques, including USE method, nsenter wrapping, CPU flame graphs, and built-in Linux tracers. Participate in interactive scenarios to practice reverse diagnosis and CPU bottleneck identification. Conclude with a summary of key takeaways for effective container performance analysis and debugging.

Intro
Current Titus Scale
Titus Use Cases
Container Performance @Netflix
Control Groups
Linux Containers
CPU Shares
Container OS Configuration
Analysis Strategy
Host Analysis Challenges
3.1. Host Physical Resources
Host Perf Analysis in 60s
USE Method: Host Resources
3.2. Host Containers & cgroups
Namespaces
docker stats
Host PID - Container ID
nsenter Wrapping
nsenter: Host - Container top
perf: CPU Profiling
CPU Flame Graphs
3.3. Let's Play a Game
Game Scenario 1
Methodology: Reverse Diagnosis
CPU Bottleneck Identification
Guest Analysis Challenges
Disks
Metrics Namespace
perf & Container Debugging
Built-in Linux Tracers
ftrace: Overlay FS Function Calls
ftrace: Overlay FS Function Tracing
BPF: Scheduler Latency 2
Docker Analysis & Debugging
Summary