LLMs and AWS: Observability Maturity from Foundation to AIOps

Explore the evolution of observability in Large Language Models (LLMs) and AWS applications in this comprehensive conference talk. Delve into the fundamentals of observability, its significance in LLM contexts, and the distinction between direct and indirect LLM observability. Examine the observability maturity model for AWS Bedrock integrated applications, covering key pillars such as LLM-specific metrics, prompt engineering properties, and performance metrics for RAG models. Learn about visualization tools, incident management, security compliance, and cost optimization strategies. Discover the importance of a maturity framework and how to implement foundation-level LLM observability. Gain insights on measuring progress through business outcomes, best practices to follow, and common pitfalls to avoid in your journey towards advanced observability and AIOps capabilities.

intro
preamble
agenda
quick intro about myself
amazon bedrock
overview of genai apps
what is observability?
observability in llms: what exactly is it?
insight into the core: direct llm observability
beyond the surface: indirect llm observability
llm observability: observability maturity model for aws bedrock integrated applications
why observability matters for llms
pillars shaping indirect llm observability
llm specific metrics
prompt engineering properties
performance metrics, logging, and tracing for rag models
tracing
visualization tools
alerting and incident management
security and compliance
cost optimization
aiops capabilities
why you need maturity model?
maturity framework
level 1: foundation llm observability
measure progress with business outcomes
best practices
pitfalls to avoid
thank you