Reproducible Data Science Over Data Lakes

Explore reproducible data science techniques for data lakes in this 12-minute conference talk by Ciro Greco, presented at DE4AI. Delve into the challenges of achieving reproducibility in Lakehouse architectures and discover recent advancements made at Bauplan to address these issues. Learn about a system that decouples compute from data management by utilizing a cloud runtime alongside Nessie, an open-source catalog with Git semantics. Gain insights into how this system offers time-travel and branching semantics on top of object storage, enabling full pipeline reproducibility with simple CLI commands. Understand the importance of overcoming challenges such as slow testing, complex debugging, and error susceptibility in large data pipelines. Benefit from Ciro Greco's expertise as the former VP of AI at Coveo and founder of Tooso.ai, bringing his background in linguistics, cognitive neuroscience, and experience in Information Retrieval and Natural Language Processing to this presentation on replayable data pipelines with Bauplan and Nessie.

Reproducible data science over data lakes // Ciro Greco // DE4AI