Scaling Security Threat Detection with Apache Spark and Databricks

Explore advanced security threat detection techniques using Apache Spark and Databricks in this 24-minute conference talk. Learn about Apple's innovative solutions for addressing scale complications, including notebook-based testing CI, self-tuning alerts, automated investigations, and DetectionKit. Discover how to reduce testing time, amplify signal from noise, automate incident containment, and formalize job configuration and testing. Gain insights into modular pre/post processor transform functions and stream-compatible exclusion mechanisms using foreach Batch. Understand the challenges of cyclical investigations, pattern finding, and the importance of document recommendations and automated suggestions in security threat detection.

Intro
Which Technologies?
Detection === Code That Finds Bad Stuff
Development Overhead Average time to write, test, and deploy a
Mo' Detections, Mo' Problems
No Support for Common Patterns
Components
Detection and Alert Abstraction
Config Inheritance
Modular Pre/Post Processing
Manual Tuning Lifecycle
Self-Tuning Alerts
Repetitive Investigations... What Happens?
Automated Investigation Templates
Automated Containment
Detection Testing
Detection Functional Tests
Databricks Stacks!
Deploy/Reconfigure Jobs with Single PR
Problem #1 - Cyclical Investigations
Problem #3 - Finding Patterns
Solution: Document Recommendations
Automated Suggestions
Anatomy of an Alert
Entity Tokenization and Enrichment
Suggestion Algorithm WHY CANTI