YoVDO

Integrate External Services with Apache Airflow

Offered By: Pluralsight

Tags

Apache Airflow Courses Data Management Courses Google Cloud Storage Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
This course will teach you how to connect Apache Airflow with Google Cloud Storage, Google Cloud BigQuery, and AWS S3 enabling data management and processing across cloud platforms.

Apache Airflow provides built-in operators to connect to external cloud services. In this course, Integrate External Services with Apache Airflow, you’ll gain the ability to connect and manage various external services within your Airflow workflows. You'll connect your Airflow workflow to Google Cloud Storage and Cloud BigQuery as well as to AWS S3, enabling data management and processing across cloud platforms. First, you’ll explore how to set up permissions on Google Cloud using a service account to enable Airflow to access Google Cloud Storage buckets and BigQuery datasets. Next, you'll discover how to read files from your local machine and upload to Google Cloud, create a BigQuery dataset, and create an external table in BigQuery. Then, you'll integrate your workflow with AWS S3 buckets to transfer data from S3 to Google Cloud Storage. Finally, you’ll use hooks to work with S3, Cloud Storage, and BigQuery - hooks give you much more control over how you access your data from external services. When you’re finished with this course, you’ll have the skills and knowledge of Apache Airflow needed to integrate and manage external services effectively within your data workflows.

Syllabus

  • Course Overview 1min
  • Integrate Cloud Services with Airflow 22mins

Taught by

Janani Ravi

Related Courses

Advanced Data Engineering
Duke University via Coursera
BI Foundations with SQL, ETL and Data Warehousing
IBM via Coursera
Cloud Composer: Copying BigQuery Tables Across Different Locations
Google via Google Cloud Skills Boost
Cloud Composer: Qwik Start - Command Line
Google via Google Cloud Skills Boost
Cloud Composer: Qwik Start - Console
Google via Google Cloud Skills Boost