Uptime - Building Resilient Services with Go

Explore the development of Fog Creek's first production Go service in this GopherCon 2015 talk. Learn how to build resilient services using Go by examining the rewrite of a Git and Mercurial SSH reverse proxy from Python. Discover techniques for monitoring memory, tracking user connections and actions, implementing drain-and-die functionality, and creating detailed forensic logs. Gain insights into improving response times, extending uptime, and enhancing crash analysis. Delve into topics such as concurrency, error handling, channel usage, panic management, race condition avoidance, timeout implementation, and memory optimization. Understand the importance of proper logging, profiling with PPROF, and conducting game day exercises. Apply these lessons to create robust and efficient Go services in your own projects.

Intro
2014 Google I/O: What's Go?
Kiln's SSH Reverse Proxy
Why Rewrite?
Tons of Concurrency 1. Accepts SSH connection (1 goroutine) 2. Authenticates via public/private key 3. Connects to backend server 4. Proxies STDIN, STDOUT, and STDERR (3 goroutines)
Resiliency: The Process
Careful Coding: Error Handling & Clean-up
Not Necessarily.
Careful Coding: Channels
Careful Coding: Panics!
Careful Coding: Avoid Race Conditions!
goroutines Should be used in unit tests, development, and testing environments
Careful Coding: Implement Timeouts
Network Timeouts: • network dial timeout • network connection inactivity timeout • total connection timeout
Know Your Service: How Does It Use Memory?
How much memory per connection? • Does the system reclaim memory that's no longer used? • What's the garbage collector doing? CODEBUG=gctrace=1 • Where is memory allocated? (PPROF)
Know Your Service: PPROF
Don't leak goroutines!
PPROF: From the Command Line
Know Your Service: Watch It Run
Keep Good Logs! • Create a semi-unique string per request • Use this request string as a prefix in all log entries • Always log at least the start and end of a request
Drain and Die
Game Day.
Wolfram Alpha Tells Me: 4MB X 10 = 40MB
Prod Profiling Told Me: This memory will be reclaimed