Design and Practice of OPPO Big Data Spark and Flink Engine Diagnosis System

Explore the design and implementation of OPPO's Big Data Spark and Flink Engine Diagnosis System in this 17-minute conference talk. Learn how OPPO addresses challenges associated with EB-level data volumes, multiple data components, and diverse problem types by building an engine diagnosis platform. Discover the system's approach to problem triggering, rapid positioning, optimization, and convergence. Gain insights into performance optimization tools for big data Flink and Spark operations, aimed at improving problem-solving efficiency and reducing resource usage. Understand the open-source contribution made through the Compass project (https://github.com/cubefs/compass) and its impact on the big data community.

Design And Practice Of Oppo Big Data Spark And Flink Engine Diagnosis System