Using Serverless and ARM64 for Real-Time Observability

Explore how a sub-second latency query engine inspired by Facebook's Scuba was extended to handle massive data volumes using cloud object storage and serverless functions in this conference talk from GOTO Copenhagen 2023. Dive into the challenges of managing AWS Lambda at scale, including issues with impatience, maximum concurrency, runtime and hardware architecture configuration, and the price/performance trade-offs of using thousands of parallel workers. Learn about interactive investigation of production behavior, dynamic aggregation across any time range, and the benefits of buying compute in 1ms units. Gain insights into considerations for implementing serverless and ARM64 architectures for real-time observability, and understand why it's not just about cost savings. Discover practical tips and best practices for applying these concepts in your own projects.

Intro
Agenda
What is lambda for?
What is retriever for?
Interactive investigation of production behavior
Emphasis: interactive
Dynamic aggregation of any fields across any time range
Buy compute in 1ms units
Considerations
It's not all about the money
Yes*, do this at home
Outro