Complete Guide to Python for Data Engineering: From Beginner to Advanced

Practice fundamental skills using Python for data engineering in this hands-on, interactive course with coding challenges in CoderPad.

Introduction

• Welcome to the course
• What you should know
• CoderPad tour

1. Python Basics

• Introduction to Python and data engineering
• Setting up your Python environment
• Explore the Google Colab worksheet
• Variables and data types
• Operators and expressions
• Control structures
• Functions
• Modules and packages
• String manipulation
• Error handling
• Solution: Conditions

2. Python Collections

• Collection overview
• Python collections: Tuples
• Python collections: Lists
• Python collections: Sets
• Python collections: Dictionaries
• Solution: Collections

3. Python File Handling

• File I/O overview
• Working with CSV files
• Working with JSON files
• Solution: File handling

4. pandas DataFrame API

• Introduction to pandas
• Read files as DataFrames
• Data cleaning and preprocessing
• Data manipulation and aggregation
• Data visualization
• Write a DataFrame to a file
• Solution: pandas

5. NumPy

• Introduction to NumPy
• Array creation and attributes
• Array operations
• Indexing and slicing
• Linear algebra and statistics
• Write a NumPy array to a file
• Solution: NumPy

6. OOP with Python

• Understanding classes and objects
• Implementation: Classes and objects in Python
• Understand OOP features: Abstraction, inheritance, and more
• Solution: OOP

7. Advanced Data Engineering

• Tips to write efficient Python code
• What is ETL in the data engineering world?
• Understand PySpark for data engineering
• What is Hadoop
• Importance of visualization tools in data engineering
• On-premises vs. cloud data engineering

8. Web Scraping with Python

• HTML basics
• HTML parents, children, and descendants
• Understand web scraping
• BeautifulSoup basics
• Installing BeautifulSoup
• Get HTML from a web page
• Scrape the web page
• Export data as a TXT file

9. Advanced Built-in Functions

• Generators in Python
• Python generator classes and iterators
• Iterables in Python
• filter() and map() functions
• any() and all() functions in Python

10. Logging in Python

• What is logging?
• Custom logging
• Logging best practices

11. Capstone Project

• Capstone Project: Retail sales analysis
• Solution: Capstone project

Conclusion

• Next steps