YoVDO

Building Data Workflows with Luigi and Kubernetes

Offered By: EuroPython Conference via YouTube

Tags

EuroPython Courses Big Data Courses Python Courses Docker Courses Kubernetes Courses Data Pipelines Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore how to build complex data pipelines in Python using Luigi and Kubernetes in this EuroPython 2019 conference talk. Learn about Luigi's problem-solving capabilities for batch job management, including dependency resolution, workflow management, visualization, and failure handling. Discover techniques for packaging Luigi pipelines as Docker images for easier testing and deployment. Gain insights into deploying pipelines on Kubernetes clusters for scalable Big Data processing and cost-effective infrastructure management. Get tips and tricks for optimizing Luigi Scheduler's performance with Kubernetes batch execution. Benefit from a demo project and practical advice tailored for data scientists, data engineers, BI developers, and software developers working with batch jobs and Big Data.

Syllabus

Intro
About Nar
Luigi Implementation
Kubernetes Implementation
Setup
Kubernetes
Questions
Different Kubernetes
Why Luigi
Configuring Luigi
Wrapping up


Taught by

EuroPython Conference

Related Courses

A Brief History of Data Storage
EuroPython Conference via YouTube Breaking the Stereotype - Evolution & Persistence of Gender Bias in Tech
EuroPython Conference via YouTube We Can Get More from Spatial, GIS, and Public Domain Datasets
EuroPython Conference via YouTube Using NLP to Detect Knots in Protein Structures
EuroPython Conference via YouTube The Challenges of Doing Infra-As-Code Without "The Cloud"
EuroPython Conference via YouTube