YoVDO

Bytedance Deep Learning: Batch Flow Integrated Training Practice

Offered By: The ASF via YouTube

Tags

Machine Learning Courses Deep Learning Courses Cloud Computing Courses Apache Kafka Courses Recommendation Systems Courses Stream Processing Courses Batch Processing Courses Apache Iceberg Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the evolution and practical implementation of ByteDance's batch flow integrated machine learning training framework in this 27-minute conference talk. Dive into the architecture, scheduling mechanisms, and innovative approaches that enable ByteDance to handle massive-scale AI model training across various business applications. Learn about the integration of Apache Iceberg, HDFS, and Kafka for efficient batch and streaming data processing. Discover how the framework supports flexible scheduling, multi-stage multi-source data orchestration, and heterogeneous elastic training. Gain insights into the challenges faced and solutions developed for global shuffle of streaming samples, full link Native implementation, and training data visualization. Understand the benefits of the new scheduling architecture, including improved resource utilization and unified resource management. Examine the Primus open-source project and its integration with Spark for enhanced pre-processing capabilities in training workflows.

Syllabus

Bytedance Deep Learning Batch Flow Integrated Training Practice


Taught by

The ASF

Related Courses

Mining Massive Datasets
Stanford University via edX
Nearest Neighbor Collaborative Filtering
University of Minnesota via Coursera
Practical Deep Learning For Coders
fast.ai via Independent
Data Mining: Theories and Algorithms for Tackling Big Data | 数据挖掘:理论与算法
Tsinghua University via edX
ความรู้พื้นฐานเกี่ยวกับบิ๊กดาตา | Big Data Concept
Sukhothai Thammathirat Open University via ThaiMOOC