YoVDO

Auditing Data and Answering the Life Long Question - Is It the End of the Day Yet?

Offered By: NDC Conferences via YouTube

Tags

NDC Conferences Courses SQL Courses Data Processing Courses Lambda Functions Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the intricacies of data auditing and end-of-day processing in this 37-minute conference talk from NDC Conferences. Dive into Nielsen's robust Kafka architecture and ETL processes, uncovering the challenges of tracking data flow and preventing loss or duplication. Learn about the design process behind the Data Auditing system, Life Line, including AVRO Audit headers, auditing heartbeats, metadata design, and table optimization. Discover how to create an alert-based monitoring system and tackle the perennial question of determining the end of the day. Gain insights into technologies such as Kafka, Avro, Spark, Lambda functions, and complex SQL queries used in data auditing. Follow the journey from tracking and producing to analyzing and storing audit information, optimizing PostgreSQL for audit queries, and implementing alerts and add-ons to enhance data management processes.

Syllabus

Intro
Nielsen's Architecture (AT THE TIME)
Data Arrival Pain Points
Recovering from failures
Is it the end of day yet? Legacy answers to a legacy problem
Auditing Window
Auditing Header Injection
Designing Out Output Table
Optimizing PostgreSQL for Audit Queries
Alerts and add-ons


Taught by

NDC Conferences

Related Courses

Coding the Matrix: Linear Algebra through Computer Science Applications
Brown University via Coursera
كيف تفكر الآلات - مقدمة في تقنيات الحوسبة
King Fahd University of Petroleum and Minerals via Rwaq (رواق)
Datascience et Analyse situationnelle : dans les coulisses du Big Data
IONIS via IONIS
Data Lakes for Big Data
EdCast
統計学Ⅰ:データ分析の基礎 (ga014)
University of Tokyo via gacco