Stream All Things - Patterns of Modern Data Integration

Explore patterns of modern data integration in this conference talk from Philly ETE 2017. Delve into the challenges of data integration and learn about design and architecture patterns used to address these issues. Discover how Apache Kafka can be implemented to create fast, scalable, and manageable data pipelines. Examine topics such as streams of events, Kafka architecture, timestamp architecture, schema registry, Kafka scaling, and Kafka Connect. Gain insights into enriching events, basic architecture, latency considerations, and caching strategies. Investigate advanced concepts like search relevance, window joints, time updates, and buffers. Conclude with a recap of Kafka's applications in New York City and participate in a Q&A session.

Intro
What happened to integration
Job titles changed
Why jobs changed
Tools of the trade
Example
Streams of Events
Pageview Events
Potential Applications
Messy Architecture
Kafka Architecture
Timestamp Architecture
Comfortability
Contracts
Compatibility
Schema Registry
Kafka Scaling
Kafka Connect
Calculus
CAFCO
enriching events
basic architecture
latency
fundamental level
cache
update cache
connect
database
Bonus content
Search relevance
Window Joints
Time
Update
Buffers
Recap
Kafka in NYC
Questions