From Built-in Concurrency Primitives to Large Scale Distributed Computing

Explore Python's asynchronous execution models and concurrency primitives in this 40-minute conference talk from EuroPython 2024. Delve into the 'Future' and 'Executor' interfaces within the 'concurrent.futures' module and the event-driven architecture of 'asyncio'. Learn how to process large datasets efficiently, transitioning from single CPU applications to complex distributed systems with thousands of cores. Discover practical applications of these concepts in large-scale, distributed execution frameworks like Dask or Ray through step-by-step examples. Gain insights into map-reduce operations, efficient collaboration between different concurrency models, and addressing common challenges such as data serialization in distributed environments. Leave with a solid understanding of asynchronous code execution fundamentals and the ability to make informed decisions about applying concurrency in data processing workflows, ensuring optimal development lifecycle, performance, and scalability.

From built-in concurrency primitives to large scale distributed computing — Jakub Urban