BinaryPig - Scalable Malware Analytics in Hadoop

Explore scalable malware analytics using Hadoop in this Black Hat USA 2013 conference talk. Learn how Endgame developed BinaryPig, an open framework built on Apache Hadoop, Apache Pig, and Python, to process and analyze massive amounts of malware data. Discover techniques for handling terabytes of binary data, extracting feature sets for machine learning, and performing large-scale malware studies. Gain insights into the challenges of processing millions of malware samples and how BinaryPig addresses issues of scalability, workflow development, and parallel processing. Examine the architecture, optimizations, and implementations of BinaryPig, including loaders, scripting, and web interface. Delve into general findings, feature extraction methods, clustering results, and icon analysis. Understand the lessons learned and future directions for scalable malware analytics in the face of ever-increasing data volumes.

Intro
Background
Malware data mining is useful
Pre-BinaryPig: Architecture
BinaryPig - Results Exploration
BinaryPig Loaders
Optimizations in BinaryPig
BinaryPig: Loader Implementations
BinaryPig: Scripting
Web Interface
General Findings
Feature Extraction
Feature Depth
Clustering Results **.
ICO Extraction
Icon Features
Lessons Learned
Future work
black hat USA 2013