Data Flow Control in Cluster Logging Pipeline

Explore data flow control in cluster logging pipelines through this lightning talk presented by Pranjal Gupta and Eran Raichstein from IBM. Dive into the challenges of managing massive log volumes in production environments and learn about a new feature in the in_tail input plugin that uses group rules for rate-limiting log collection. Gain insights from a systematic study on log loss in Fluentd plugins using an open-source benchmarking framework. Discover the Log Flow Control framework, which enables users to define and enforce log rate limit policies for predictable log loss control. Understand the importance of prioritizing application logs and collecting data from high-priority workloads in a controlled manner. Cover topics such as Openshift, logging pipelines, data clogging, group-based throttling, and policy definition and application.

Introduction
What is Openshift
Logging Pipeline
Log Loss Data Clogging
Motivation
Open Source Benchmark Tool
Experimental Setup
Graphs
Group Based Throttling
Generic Workload Path
Grouping Pattern
Red Hat IBM Research
Defining Policies
Applying Policies
Summary
Questions