Ingesting 35 Million Hotel Images with Python in the Cloud

Learn how Skyscanner built a distributed architecture using Python to process and manage 35 million hotel images in the cloud. Explore the challenges and solutions involved in creating an incremental image processing pipeline that discards poor quality and duplicate images while optimizing for mobile devices. Discover the tools and techniques used, including Pillow, ImageHash, Kombu, and Boto, to handle tasks such as ingesting partner-provided images, detecting and removing bad quality and duplicates, resizing images, and ensuring scalability within time constraints. Gain insights into the technical stack, triggering mechanisms, downloading processes, fingerprinting techniques, and methods for choosing the best images in this informative conference talk from EuroPython 2016.

Intro
Processing 55 million images
Tech stack
Triggering
Downloading images
Fingerprinting
Duplication
Duplicator
Guarantee
Choosing the best images
Picking the best image
Resize images
Final result