Garbage in Garbage Out - Generating Useful Log Data in Complex Environments

Explore effective log data generation techniques for complex environments in this 51-minute conference talk from Derbycon 2016. Learn why logging everything is ineffective and discover the importance of meaningful log data. Delve into using thresholds, frequency analysis, and noise balancing to improve log quality. Examine log management systems, including ingestion, parsing, and visualization. Gain insights into JSON standards, Python integration, and handling old services and web server logs. Follow along with a demo using Phobos and Squid, and see real-world scenarios of identifying proxy abuse. Understand how to set up alerts, trend on thresholds, and utilize tools like Splunk. Discover the benefits of good, reliable data for small companies and large enterprises alike.

Introduction
Audience Participation
Presentation Overview
Logging everything is useless
The importance of meaningful log data
Using thresholds
Frequency
Noise Balancing
Log Management Systems
ingestion
parsing
visualization
JSON
JSON Standards
Python
Old Services
Web Server Logs
Phobos
Demo
Squid
Meeting up
Real world scenario
Who is abusing the squid proxy
Ellen takes it away
Alerts
Trending on Threshold
Using Splunk
Using Squid
Using JSON
Good reliable data
Good yields
I got a small company
Clarifications