YoVDO

High Throughput with Low Resource Usage - A Logging Journey

Offered By: CNCF [Cloud Native Computing Foundation] via YouTube

Tags

Conference Talks Courses Data Processing Courses System Administration Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the challenges and solutions in high-throughput logging with low resource usage in this 35-minute conference talk from KubeCon + CloudNativeCon North America 2021. Dive into the journey of performance optimization in the Fluent Bit project, focusing on Network I/O + TLS, filesystem buffers, routing, and multiplexing. Learn how to scale logging services in distributed environments while managing increased data volumes efficiently. Discover techniques to improve processing speed from 5k/sec to over 30k/sec using a single-core CPU through design improvements and optimal use of Linux OS interfaces. Gain insights into balancing performance and cost, understanding logging workflows, data analysis, and various aspects of the logging pipeline including input/output, processing, routing, and network setup. Compare JSON vs Message Pack for data serialization and explore memory serialization and buffering techniques. Conclude with a practical demonstration of the concepts discussed.

Syllabus

Introduction
Performance vs Cost
Fluency
Logging
Debug Mode
Logging Workflow
Data Analysis
IO
Data Processing
Output
Pipeline
System Code
Processing
Routing
Network Setup
JSON vs Message Pack
Data serialization
Memory serialization
Memory buffering
Demo


Taught by

CNCF [Cloud Native Computing Foundation]

Related Courses

Coding the Matrix: Linear Algebra through Computer Science Applications
Brown University via Coursera
كيف تفكر الآلات - مقدمة في تقنيات الحوسبة
King Fahd University of Petroleum and Minerals via Rwaq (رواق)
Datascience et Analyse situationnelle : dans les coulisses du Big Data
IONIS via IONIS
Data Lakes for Big Data
EdCast
統計学Ⅰ:データ分析の基礎 (ga014)
University of Tokyo via gacco