Tips and Tricks for Robust Big Data Applications

Discover essential strategies for building robust big data applications in this conference talk from NDC Sydney 2020. Learn how to optimize data storage using columnar files to reduce costs and enhance performance. Explore techniques for ensuring data quality through unit testing and quality assurance measures. Understand the importance of data governance and tracking, including how to maintain dataset lineage for auditing and compliance purposes. Gain insights into useful tools for automation and collaboration in big data projects. Master key concepts such as metadata management, skew prevention, and the significance of thorough data understanding. Equip yourself with practical knowledge to tackle the challenges of modern big data applications and stay relevant in an ever-evolving technological landscape.

Intro
Brief History
Agenda
Know your data and storage format
Metadata matters with slow file system
Watch out for skewness
Guard against bad data
Governance
Invest in tooling