Lessons Learned from Big Data and AI/ML Collaboration for Giant Hogweed Eradication

Explore a conference talk detailing the application of deep learning and big data technologies in eradicating giant hogweed. Dive into the project overview, challenges faced, and architectural design of a system combining various specialties and large data volumes. Learn about data preparation, inference processing with Apache Spark, and data analysis phases. Gain insights into scaled machine learning systems, the balance between model development and operation, and strategies for creating dev-friendly and ops-friendly environments. Discover real-world use cases from Uber and Twitter, and examine patterns for selecting toolsets and software choices. Acquire valuable tips on function abstraction and detecting confidence deterioration in applications.

Intro
Self introduction
Outline
Our journey to apply deep learning for giant hogweed eradication
About giant hogweed
Project Overview
Data volumes
Various specialty
Challenges of the project
Architecture overview
Data flow-1/4: Data preparation phase
Assistance tool for data preparation
Inference processing by Apache Spark
Data flow-4/4: Data analysis phase
Lessons Learned from a consideration of architecture design
Common view in Machine learning system
Scaled-ML systems
ML Application with scalability on our architecture
Model development vs. Model operating
Dev-friendly to Dev-friendly
Dev-friendly to Ops-friendly
Use case: Uber implemented ML Ops on Spark
Ops-friendly to Ops-friendly Patterns that selected a toolset familiar to Model Operator for both model development and operation
Use case: Twitter leveraged Scala for feature engineering
Example software: BigDL by Intel
Patterns of software choices Reprint Four patterns of the workflow regarding combination of Model Development phase and Model Operating phase
Architecture and data pipeline ver.2.0
Tips: Abstraction of functions used in applications
Tips: Detecting and storing deterioration of confidence