Runaway Complexity in Big Data Systems and a Plan to Stop It

Explore a comprehensive plan to address runaway complexity in big data systems in this conference talk from GOTO Aarhus 2012. Learn how embracing immutability and moving away from the CRUD paradigm can simplify data systems. Discover the role of NoSQL in the big picture and understand the "Lambda Architecture," a generic approach combining batch and real-time processing. Delve into topics such as human fault tolerance, data loss prevention, normalization vs. denormalization, and the challenges of schemas. Gain insights on building better data systems, pre-computation techniques, and the CAP theorem. Examine the architecture of batch views, real-time views, and the overall Lambda Architecture. Conclude with a look at future trends and performance considerations in big data systems.

Introduction
What is a data system
Common sources of complexity
Human fault tolerance
Design for human error
Data loss
Mutability
Immutability
Normalization vs Denormalization
Denormalization
Schemas
Schemas are bad
Schemas are confusing
What is a schema
What is structural integrity
Preventing corruption
Detecting corruption
Preventing mistakes
Learning from experience
Why schemas are painful
My ideal schema tool
Apache Thrift
New Sequel
No Sequel
How would you build a better data system
What do we actually use data systems for
Data Systems
Example
Realtime Queries
Pre Computation
Pre Computation Example
Architecture
Functions
View
Batch Processing
MapReduce
BatchView Databases
BatchView Properties
BatchView Architecture
Batch Computation
RealTime Views
Lambda Architecture
Cap Theorem
Eventually Accurate
Maximizing Value
Tools
Land Architecture
Movement Mistakes
Normalization Personalization
The Future
Book
Performance