YoVDO

Orchestrating Scalable Data Pipelines with Apache Toree, YuniKorn, Spark, and Airflow

Offered By: The ASF via YouTube

Tags

Apache Spark Courses Apache Airflow Courses Jupyter Notebooks Courses Data Processing Courses Scalability Courses Data Pipelines Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the integration of Apache Toree, YuniKorn, Spark, and Airflow for creating efficient, scalable data pipelines in this 24-minute conference talk. Discover how Apache Toree provides an interactive analysis environment with Spark via Jupyter Notebook, and learn about using Apache YuniKorn to manage and schedule computational resources for improved system efficiency. Delve into the role of Apache Spark in large-scale data processing, focusing on its integration with Toree and YuniKorn. Examine how Apache Airflow orchestrates complex workflows, manages dependencies, and provides end-to-end processing solutions. Gain insights from AI Platform Architect Luciano Resende and Software Engineer Hongyue Zhang on leveraging these Apache projects for optimized data processing, drawing from their extensive experience in open source, enterprise-grade AI platforms, and data science technologies.

Syllabus

Orchestrating Scalable Data Pipelines with Apache Toree, YuniKorn, Spark, and Airflow


Taught by

The ASF

Related Courses

CS115x: Advanced Apache Spark for Data Science and Data Engineering
University of California, Berkeley via edX Big Data Analytics
University of Adelaide via edX Big Data Essentials: HDFS, MapReduce and Spark RDD
Yandex via Coursera Big Data Analysis: Hive, Spark SQL, DataFrames and GraphFrames
Yandex via Coursera Introduction to Apache Spark and AWS
University of London International Programmes via Coursera