The Data Dichotomy - Rethinking Data and Services with Streams

Explore the challenges and solutions in building service-based systems with large datasets in this 55-minute conference talk. Dive into the dichotomy between data accessibility and service encapsulation, and learn how it impacts system design at scale. Discover how leading companies like Uber, Netflix, and LinkedIn use shared, immutable record sequences to balance internal and shared data, enabling them to handle millions of events per second. Examine the potential of stream processors for joining event-driven datasets across multiple services, and understand how this approach offers benefits similar to data warehousing without the same level of centralization. Gain insights into designing more efficient and scalable service-based architectures that effectively manage data flow and processing.

The data dichotomy: Rethinking data and services with streams - Ben Stopford