The Streaming Plane - Bridging Operational and Analytical Data

Explore the concept of the streaming plane in platform engineering through this conference talk by Hubert Dulay at Conf42 Platform Engineering 2024. Delve into key concepts including the data divide, operational and analytical data planes, and data gravity. Learn about synchronous vs asynchronous data serving, data locality and replication, materialized views, and query optimization. Discover strategies for balancing push and pull queries, implementing global replication with local consumption, and managing streaming data catalogs and governance. Gain insights into overcoming challenges in data movement and optimizing data flow within modern platform architectures.

intro
overview of key concepts
understanding the data divide
operational vs analytical data planes
challenges in data movement
introduction to data gravity
the streaming plane explained
synchronous vs asynchronous data serving
data locality and replication
materialized views and query optimization
balancing push and pull queries
global replication and local consumption
streaming data catalog and governance
conclusion and summary