Parallel Computing in Python - Current State and Recent Advances

Explore the current state and recent advances in parallel computing with Python in this EuroPython 2019 conference talk. Gain insights into interfacing Python with parallelism, from leveraging C-extensions to using multiprocessing and multithreading APIs. Learn about high-level parallel processing libraries like concurrent.futures, joblib, and loky, and their applications in various use cases. Discover the latest improvements in the Python standard library, including shared-memory management and serialization enhancements for large Python objects. Understand how these advancements benefit distributed data science frameworks such as dask, ray, and pyspark, and how they address performance bottlenecks in multi-core and multi-machine processing.

Introduction
Why is parallel computing important
Parallelization on a single machine
Multiprocessing libraries
Problems with multiprocessing
Multiprocessing in Python
Disclaimer
Sterilization
Pickle
pickle limitations
pickle errors
pickle extensions
pythonicpickle
pickle module
pickle protocol 5
pickle buffer
conclusion
security