Heterogeneous Pipeline Processing with Kubernetes and Google Cloud Pub Sub

Explore a conference talk on building a flexible, asynchronous pipeline using Kubernetes and Google Cloud Pub/Sub for heterogeneous processing requirements. Dive into Spacemaker's journey of replacing their batch-oriented pipeline with a message-based system to handle diverse computational needs in real estate development and architectural AI. Learn about the challenges of managing CPU-intensive algorithms, memory-demanding simulations, and GPU-optimized techniques within a single platform. Discover how the new pipeline architecture leverages a central message broker, task queues, and auto-scaled Kubernetes clusters to achieve elasticity and scalability. Gain insights into handling burst workloads, implementing auto-scaling based on queue lengths, and enabling team autonomy in choosing programming languages and tools. Examine the details of the message broker, API design, and worker implementation. Conclude with key performance results and reflections on modularizing pipelines for broader applications in complex, resource-diverse computing environments.

Heterogeneous pipeline processing with Kubernetes and Google Cloud Pub Sub - Håkon Åmdal