Distributing Work with Queues and GCP PubSub for Flight Search Optimization - ElixirConf EU 2022

Explore the complexities of flight search and booking APIs in this ElixirConf EU 2022 talk by Johanna Larsson. Dive into the challenges of distributing work with queues and GCP PubSub, learning how to reduce processing time variance and improve service reliability. Discover the implementation of Broadway and GCP PubSub to handle CPU-heavy workloads with low latency expectations. Gain insights into flight search intricacies, queue system selection, message formatting, and architecture design. Examine the benefits of the new system, including performance improvements through Broadway tweaking. Address memory issues, Erlang VM memory management, and a bug in the GCP PubSub provider. Ideal for professional Elixir developers working with demanding workloads and seeking to enhance their scaling strategies.

Intro
Flight Search Crash Course
Searching Multiple Airlines
Selecting a queue system
GCP PubSub Basics
Message format
Enqueuing messages
Routing back to process
Architecture
Broadway tweaking
The first version was slow, tweaking made a huge impact
Benefits of the new design
Memory issues with Broadway
Erlang VM Memory Management
Bug in GCP PubSub Provider