Breaking Out of the Habit of Bulk Loading Everything - Lessons from Streaming Data Integration

Explore a 21-minute conference talk from SQLBits that delves into the challenges and solutions of integrating streaming data into existing data warehouse environments. Learn how a team accustomed to working with relational databases and bulk loading patterns adapted to handle AMQP/MQTT interfaces and near real-time updates. Discover architecture patterns for processing business events, combining real-time analysis with long-term storage requirements, and comparing costs of PaaS building blocks with custom-built solutions. Gain insights into leveraging Azure services like Event Hubs, Streaming Analytics, and Azure Functions to modernize data ingestion and processing. Follow the journey of a small team as they transition from an on-premises Microsoft data stack to cloud-based solutions, addressing resistance, acquiring new skills, and evaluating cost implications. Uncover key takeaways and lessons learned in this comprehensive exploration of evolving data integration strategies.

Intro
The situation as-is
Resistance is futile - let's go with the cloud...
What we built on the first day
Ingestion - Event Hubs
Processing - Stream Analytics
Storage - keeping our options open
Serve Azure SQL
Visualization - Grafana
What we built in the first round
Same team - New skills
What about cost?
Main cost generators
Lessons learned
Key takeaways