Best Practices to Manage AWS Streaming Data Pipeline at Scale

Explore best practices for managing AWS streaming data pipelines at scale in this 24-minute conference talk by Akshay Jain at Conf42 CN 2024. Dive into a real-world use case and learn about key requirements for efficient data handling. Discover techniques for optimal shard data distribution, processing large records, implementing schema registries, and performing crucial validations. Gain valuable insights to enhance your AWS streaming pipeline management skills and ensure scalability in cloud-native environments.

intro
preamble
use case
requirements
shard data distribution
process large records
schema registry
validations
thank you!