Processing Data at Scale with Rails

Explore strategies for processing large-scale data using Ruby on Rails in this 35-minute conference talk. Learn how to handle massive datasets from various sources, including public records, shared resources, and IoT data. Discover techniques for ingesting, transforming, and storing data efficiently using familiar tools like ActiveRecord, Sidekiq, and Postgres. Gain insights into breaking down large data jobs into manageable tasks, turning data firehoses into structured pipelines. Dive into topics such as ETL processes, pipeline options, and practical examples using lobby data. Understand the importance of scraping, Redis usage, fanning out processes, load jobs, unique indexing, and data normalization. Address scaling concerns and explore opportunities for optimizing data processing in Rails applications.

Introduction
Data Pipelines
Use Cases
What is ETL
Pipeline options
Example
Lobby Focus
Familiar tools
Scrape
Redis
Fanning out
Fanning out view
Load job
Unique index
Normalizing
Stuff Changes
Will this scale
Options
Opportunities