Automating Bulk Intelligence Collection

Learn to automate bulk intelligence collection for cybersecurity analysis in this BSidesCharm 2017 conference talk. Explore when and what to automate in data mining processes, focusing on extracting valuable information from malware files, emails, and various document types. Discover techniques for finding critical data in large datasets, preparing result checklists, and considering database integration. Follow along as the speaker demonstrates manual analysis methods, extracts malicious domains from different file formats, and analyzes email campaigns with suspicious attachments. Gain insights into using known tools for analysis, including Foremost for file extraction. Get introduced to the Yalda framework for data storage and quality control in automated intelligence gathering. Access the code on the Fidelis GitHub repository to implement these automation techniques in your own cybersecurity workflows.

Intro
When Automation is Needed?
What to Automate?
Planning Automation for Data Mining
Finding Needle in a Hayshack
Prepare Checklist of Desired Results
Consider Inserting Data in a Database!
Start with Manual Analysis
Extracting Data from Malware Files
Analyze Body of Email
Analyze json File with mime
Extracting Malicious Domains from wsf file!
Process of Analyzing Json Mime File Download Json Mime
Detect the First Chain!
Email Campaign Featuring a PDF Attachment
Extract URL from PDF
CVE-2017-0199 Malicious RTF Document
Use Known Tools for Analysis
Apply Foremost on the File!
Malicious Extracted Files
Introducing Yalda!
Yalda Framework
Data Storage File Type
Applying Quality Control
Conclusion
How to use Yalda
Code is available at Fidelis gitHub