Building a Petabyte-Scale Cybersecurity Data Mesh in Azure with Delta Lake - HSBC Case Study

Explore a 33-minute conference talk detailing HSBC's journey in constructing a petabyte-scale cybersecurity data mesh using Azure and Delta Lake. Discover how HSBC tackled unique cybersecurity challenges, including high data volumes and strict regulatory requirements, to transform disparate data silos into a unified, scalable platform. Learn about the innovative infrastructure and architecture employed, from landing zone concepts and secure access workstations to data lake structures and isolated data ingestion. Gain insights into the hybrid data mesh leveraging Delta Lake, resulting in a flexible, secure, self-service environment that empowers cyber analysts. Delve into topics such as Azure Landing Zone overview, data ingestion processes, core data processing, and the implementation of data mesh principles. Understand the limitations faced and solutions devised, including the creation of data product teams and the use of Terraform-managed Databricks workspaces. Examine the end-to-end data pipeline, ETL core code, and data quality measures implemented to ensure robust and reliable cybersecurity analytics.

Intro
The scale of the problem
Cybersecurity Data Challenges
Don't Break DIKW
Cyber Security Science & Analytics
Azure Landing Zone Overview
Landing Zone Pattern Landing zone subscriptions
Central Delta Lake Central Delta Lake subscriptions
Data Ingestion On-premises Data External Data Ingest Transfer subscriptions subscriptions
Core Data Processing
Data Mesh Core Principles
Limitations to Data Mesh Implementation
Data Product Teams
Hybrid Distributed Data Mesh
Scalable, collaborative ETL
Terraform Managed Databricks Workspaces
End to End Data Pipeline
ETL Core Code
DLT Pipeline Flow View
Data Quality Overview
Data Quality Details
Data Quality: File Arrival Checks Workflow