Processing a Trillion Rows Per Day with Delta Lake at Adobe

Explore data patterns and practices adopted by Adobe for scaling across 8000+ self-managed Delta tables in this 21-minute conference talk. Dive into Adobe Experience Platform's approach to ingesting terabytes of data daily and managing petabytes of customer data for their Unified Profile offering. Learn about scaling writers, JVM agnostic locking, balancing multi-tenancy and single tenancy, transaction management, and using append-only DeltaTables for global history tracking. Discover anti-patterns, data manipulation using UDFs, and maintenance operation scaling challenges. Gain insights into powering various marketing scenarios across multiple platforms and channels, including email and advertisements.

Processing a Trillion Rows Per Day with Delta Lake at Adobe