Kubernetes for Modern Data Engineering: An End-to-End Data Engineering Project

Dive deep into the world of Kubernetes for modern data engineering in this comprehensive tutorial video. Learn to set up Kubernetes on Docker, master kubectl for cluster management, deploy the Kubernetes Dashboard, and run Apache Airflow using Helm charts. Explore essential topics including resource management, service accounts, Role-Based Access Control (RBAC), secrets, and tokens. Gain hands-on experience in configuring Apache Airflow on Kubernetes, connecting DAGs, working with multiple DAGs, and optimizing workflows. Perfect for data engineers, DevOps professionals, and anyone looking to leverage Kubernetes for scalable and efficient data processing in cloud environments.

Introduction
Setting up Docker on your machine
Managing Resources for Docker and Kubernetes
Enabling Kubernetes on Docker Desktop
Setting up Kubectl and other Kubernetes Management Tools
Setting up Helm charts
Working with Kubectl commands
Setting up Kubernetes Dashboard and Managing the cluster
Generating and managing Kubernetes dashboard Token and secrets
Kubernetes Service Accounts
Kubernetes Role Based Access Control RBAC
Kubernetes Secrets and Tokens
Kubernetes Dashboard
Setting up Apache Airflow on Kubernetes with Helm Charts
Accessing Apache Airflow on Kubernetes
Reconfiguring Apache Airflow on Kubernetes
Connecting Apache Airflow DAGs to Kubernetes
Working with Multiple DAGs on Kubernetes
Optimising Airflow DAGs on Kubernetes
Outro