Bringing Transactionality to the Streaming Ecosystem - Webinar

Explore the evolution of message queues and learn how to bring transactionality to streaming ecosystems in this Linux Foundation webinar. Discover how Alpaca, a developer-first API for equities and crypto trading, re-engineered their order management system to process millions of orders per minute without data loss or performance sacrifice. Gain insights into the challenges of scaling traditional message queues and the need for a modern platform that serves as real-time data infrastructure. Examine the innovative use of the Redpanda streaming data platform for transaction logs, and understand its architectural overview. Delve into topics such as the pros and cons of including databases in the system, approaches to bring transactionality to queue systems, and overcoming tradeoffs between performance and data safety guarantees. Learn about the Seastar framework, Raft consensus protocol, and how modern hardware can be leveraged for improved performance. Witness the transformation from traditional solutions like RabbitMQ to cutting-edge architectures that handle high-throughput scenarios efficiently.

Intro
Alpaca Order Management System (OMS)
Common solution
History of Kafka
State of the art in 2011
Things have changed...
Taking advantage of modern hardware
Seastar framework
Redpanda - built on Seastar
Raft - modern consensus protocol
Performance
The past is the future and the future is the past
OMS v2 Architecture
Throughput