Distributed Web Scraping in Python

Explore distributed web scraping techniques in Python through this 24-minute PyCon US talk. Learn how to build a scalable and robust distributed web scraper to optimize large batch scraping jobs, reduce processing times, and enhance code durability. Discover the evolution from single requests to distributed systems, understand the advantages and disadvantages of distributed scraping, and gain insights into useful Python packages and considerations for implementation. Follow the speaker's journey through various iterations, addressing issues and implementing improvements along the way. Access accompanying slides for a comprehensive overview of the distributed web scraping process, from mental models to practical implementation using controllers, scraping nodes, queues, and message brokers.

Intro
Outline
Introduction
Data Science Project Stages
What is Distributed Web Scraping
Setting the Stage
Iteration - A Single Request
Looping Requests
Iteration 1 - Issues
Intermediate improvements
Iteration 2 - Issues
Distributed - Mental Model
Distributed - Controller
Distributed - Scraping Node
Distributed - Advantages
Distributed - Disadvantages
Distributed - Queues
Distributed - Message Brokers
Code Management
Useful Python Packages
Considerations
Conclusion