Scaling Fluentd From 0 to Millions of Events - Performance Optimization and Architecture

Explore scaling Fluentd from zero to millions of events in this 22-minute conference talk by TK Kubota from ITOCHU Techno-Solutions America, Inc. Gain insights into performance characteristics, solution architecture design, and optimization techniques for large-scale Fluentd deployments. Learn about the impact of data formats, messaging sizes, event rates, and data sources on performance. Discover tips and tricks for tuning parameters in input, filter, buffer, and output pipelines. Examine benchmark results and explore strategies to enhance performance through additional compute, multi-workers, and deployment architectures. Delve into topics such as logging operations, IT Ops team challenges, implementation models, aggregator benefits, and the relationship between Fluent Bit and Fluentd. Understand key performance metrics, including CPU usage, flush intervals, and message size impacts. Conclude with valuable insights for optimizing your own Fluentd deployments in large-scale environments.

Intro
Considerations in production
Logging operations
Challenges of IT Ops team
Implementation models
Benefits of Aggregator
Fluent Bit and Fluentd
Aggregator Performance?
Characteristic: CPU usage
flush_interval
Performance measurement
Scale performance : # of workers
Scale performance : # of CPU
Scale performance : Message size
Conclusion