How WebAssembly Simplifies Streaming Data Pipelines

Explore how WebAssembly (Wasm) simplifies streaming data pipelines in this 45-minute Linux Foundation webinar. Dive into the challenges of integrating a Wasm runtime into Redpanda, a high-performance streaming storage engine. Learn about the unique constraints of the Seastar framework's execution environment and how to support arbitrary user functions while maintaining strict memory consumption and latency bounds. Discover the benefits of eliminating data ping-pong between systems, reducing latency and resource usage. Gain insights into Redpanda's architecture, including async thread-per-core design, reactor loops, and shared-nothing allocator. Understand the implementation of Wasm transforms, preallocated memory, and asynchronous host functions. Explore related technologies such as Envoy proxy and Open Policy Agent.

Intro
What is WebAssembly?
Pillars of WebAssembly
WebAssembly instructions
Module compilation Interpretation vs compilation
Classic stream processing
Runtime limits
How do I actually use it?
Maturing upstream tooling
WASM feature support is a moving target
Alternatives?
Redpanda in 60 seconds
Redpanda data transforms
Async thread-per-core What makes Redpanda fast and efficient?
Reactor loops
Shared-nothing allocator
Wasm transforms software development kit
Preallocating memory
Asynchronous host functions
Envoy proxy
Open policy agent