Tyranny of the Clock

Explore a war story from the front lines of solving a git-over-ssh problem on GitLab.com in this 30-minute conference talk from linux.conf.au 2020. Follow the journey from initial error reports through diagnosis, experimentation, and resolution, gaining insights into both technical and social aspects of problem-solving. Learn valuable lessons in system administration, including packet captures, log analysis using BigQuery, SSH configuration, and error rate detection. Discover the importance of leaving breadcrumbs, visualizing data, and effective scheduling in resolving complex issues. Gain practical knowledge applicable to various open-source projects and system administration tasks while enjoying the speaker's use of graphs and real-world examples.

System Administration
Packet captures
Lesson #1
Conversations
Logs logs and more logs - BigQuery to the rescue
Tom down connections
Lesson #2
SSH Config
Bump it
How high could we go?
Lesson #4: leave crumbs
A delightful graph
Lesson W5
A bit of ret-con math
Error rates
Lesson W6
Followup: Why u no alert?
Detecting using mtail
Visualize the Tyranny
Make people like me happy or sad with your scheduling
Final thoughts