Petabyte Scale Logging with Fluentd and Fluent Bit - A Use Case From Intuit

Explore petabyte-scale logging solutions using Fluentd and Fluent Bit in this conference talk from KubeCon + CloudNativeCon North America 2021. Discover how Intuit tackled challenges in scaling up logging for their Kubernetes-deployed applications, including decreased throughput, higher resource consumption, and unexpected backpressure. Learn about leveraging CNCF projects to configure reliable buffering for network outages, implement flexible configurations for backpressure management, and avoid common mistakes. Gain insights into Intuit's journey to achieve petabyte-scale logging, covering topics such as Fluentd basics, architecture patterns, log transport, offloading, cost breakdown, and test results. This presentation offers valuable knowledge for developers and end-users working with large-scale observability systems in financial and software companies.

Introduction
What is Fluentd
Challenges for Scale Logging
Fluentd Fluent Bit
Architecture Patterns
Use Case
Log Transport
Challenges
Architecture
Offloading
Cost Breakdown
Test Cluster
Test Results
Summary