Challenges of Logging on the Edge: Building an Observability Pipeline with FluentBit

Explore the challenges and solutions of implementing logging systems on edge devices in this conference talk from Mickey Pashov of Medtronic Digital Surgery. Discover the journey from arbitrary log collection to building an observability pipeline using FluentBit, including the rationale behind choosing this agent over alternatives. Learn about on-premises deployment hurdles such as firewall configurations, cloud authentication, offline mode, and unfamiliar toolchain packaging. Delve into the complexities of deploying an observability pipeline on Kubernetes with FluentD, covering topics like Helm charts, liveness probes, TLS termination, and autoscaling. Gain insights into advanced concepts such as exponential backoff, memory and file buffering, synthetic transactions, and the use of OpenSearch Prometheus Exporter. Understand the implementation of forwarder and aggregator setups, and explore future work involving S3 archives and MQTT output.

Intro
Medtronic Digital Surgery
Edge
Observability Stack
Logs
Find the traceback
Basic setup
Unfamiliar Toolchain
Takeaways
BitBake Recipe
Tip: Check out packaging
Retries
Exponential Backoff
Grace period
Grace timeout
Memory and File Buffering
Offline mode
Metrics
Blinking lights
Agent deployment
Forwarder and Aggregator
Observability Pipeline
Synthetic Transactions
OpenSearch Prometheus Exporter
Fluent Bit Devtools
Fluentd Helm Chart restarts
Future work
S3 Archives
Kafka Input PoC
MQTT Output