Using Apache Spark for Predicting Degrading and Failing Parts in Aviation

Explore how Apache Spark is utilized for predicting degrading and failing parts in aviation in this 28-minute conference talk from Databricks. Learn about the challenges of working with large, heterogeneous civilian and military aviation datasets and how Spark overcomes these obstacles in ETL pipelines. Discover how Spark facilitates ad-hoc and recurring reporting for aircraft component health checks at scale, and how Spark ML is employed to flag anomalous data using regression models. Gain insights into how a small team leverages Spark to handle vast volumes of data across hundreds of schemas, parallelize aircraft component health scoring algorithms, and significantly reduce model running times. Understand the role of Spark in official reporting architecture and its success in flagging parts prior to failure. Examine the shortcomings encountered, such as data visualization limitations, and explore future directions for this technology in aviation maintenance and readiness improvement.

Introduction
Disclaimer
Who are we
Heart
Outline
Subject Matter Experts
Collaboration
Data Complexity
ETL Pipeline
Data Science Cycle
Feature Development
Deep Learning Models
Feature Normalization
Deployment Process
Communication
Development Cycle
Lessons Learned