Automated Metadata Management in Data Lakes - A CI/CD Driven Approach

Explore a 28-minute conference talk on implementing automated metadata management in data lakes using a CI/CD-driven approach. Learn how Northwestern Mutual engineers developed a tool to balance rapid metadata changes with robust validation for downstream system stability. Discover the architecture and design of their centralized git-managed repository for data schemas, utilizing YAML structures and CI/CD capabilities. Gain insights into maintaining information on databases, tables, and views, including schema, ownership, PII, and descriptions. Watch a live demo of creating a new table with CI/CD promotion to production, and understand how this tool can be used effectively by individuals with minimal Spark knowledge.

Introduction
About Northwestern Mutual
Agenda
Need for metadata management
Ease of use
Design
Configuration Files
Demo
CICD
Wrap Up