Linux and Bash for Data Engineering

In this course, you will:

• Gain proficiency in using Linux for data engineering tasks
• Learn to configure and customize your Bash shell environment
• Understand how to manipulate data using shell variables and standard streams
• Construct powerful Bash scripts and command-line tools to automate workflows
• Master file and data management using Linux utilities and shell scripting

Whether you're a data engineer, software developer, or system administrator, this course will equip you with the Linux and Bash skills necessary to tackle real-world data challenges efficiently and effectively. Enhance your productivity and streamline your data workflows with the power of Linux and Bash scripting.

Module 1: Using Linux (18 hours)

\- Welcome to Linux and Bash for Data Engineering (Video, 0 minutes, Preview module)

\- Meet your Course Instructor: Noah Gift (Video, 0 minutes)

\- Overview of Key Concepts (Video, 4 minutes)

\- Introduction to Working with Linux (Video, 0 minutes)

\- Introduction to the Linux Shell (Video, 2 minutes)

\- Installing and Accessing Linux (Video, 11 minutes)

\- Getting Started with Github Codespaces (Video, 9 minutes)

\- Comparing Linux Shell Environments (Video, 9 minutes)

\- Writing 'Hello World' Bash Script (Video, 6 minutes)

\- Popular Terminal Commands in Linux (Video, 14 minutes)

\- Introduction to Shell Piping (Video, 1 minute)

\- What is Shell Piping? (Video, 3 minutes)

\- Walkthrough of Shell Pipeline Examples (Video, 12 minutes)

\- Introduction to SSH (Video, 1 minute)

\- What is SSH? (Video, 4 minutes)

\- Creating SSH Keys and using with GitHub (Video, 4 minutes)

\- Using SSH to Connect to Remote Server in AWS Cloud (Video, 7 minutes)

\- Walkthrough of SSH Tunneling Example (Video, 4 minutes)

\- Key Terms (Reading, 10 minutes)

\- Meet your Supporting Instructors: Kennedy Behrman and Alfredo Deza (Reading, 10 minutes)

\- Course Structure and Discussion Etiquette (Reading, 10 minutes)

\- Key Terms (Reading, 10 minutes)

\- Amazon CodeWhisperer CLI (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Key Terms (Reading, 10 minutes)

\- Shell Pipelines (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Key Terms (Reading, 10 minutes)

\- Securely Connecting to a Linux Instance with SSH (Reading, 10 minutes)

\- Secure Remote Development with VSCode and SSH (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Using Linux (Quiz, 30 minutes)

\- Quiz-Using Linux (Assignment, 180 minutes)

\- Quiz-How Shell Piping Works (Assignment, 180 minutes)

\- Quiz-Using SSH (Assignment, 180 minutes)

\- Meet and Greet (optional) (Discussion Prompt, 10 minutes)

\- Practicing Common Linux Shell Commands (Ungraded Lab, 60 minutes)

\- Using Linux Lab Challenge (Ungraded Lab, 60 minutes)

\- Practicing Common Shell Piping Operations (Ungraded Lab, 60 minutes)

\- Creating SSH Keys Locally (Ungraded Lab, 60 minutes)

\- Creating a Shell Pipeline (Ungraded Lab, 60 minutes)

Module 2: Using Bash (14 hours)

\- Introduction to Working with Bash (Video, 0 minutes, Preview module)

\- Introduction to Configuring your Bash Shell Environment (Video, 2 minutes)

\- What are Shell Configuration Files? (Video, 5 minutes)

\- Configuring .bashrc (Video, 11 minutes)

\- Configuring .zshrc and Third-Party Tool oh-my-zsh (Video, 9 minutes)

\- Introduction to Working with Shell Variables (Video, 1 minute)

\- What are Shell Variables? (Video, 4 minutes)

\- Exporting Shell Variables in an Interactive Prompt (Video, 6 minutes)

\- Using Shell Variables in a Bash Script (Video, 7 minutes)

\- Introduction to Standard Streams (Video, 1 minute)

\- What are Standard Streams? (Video, 2 minutes)

\- Using Standard Out (Video, 5 minutes)

\- Using Standard In (Video, 2 minutes)

\- Using Standard Error (Video, 2 minutes)

\- Key Terms (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Key Terms (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Key Terms (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Using Bash (Quiz, 30 minutes)

\- Quiz-Configuring the Shell Environment (Assignment, 180 minutes)

\- Quiz-Working with Shell Variables (Assignment, 180 minutes)

\- Quiz-Standard Streams: Using Standard In, Standard Out, Standard Error (Assignment, 180 minutes)

\- Configuring the Bash Shell (Ungraded Lab, 60 minutes)

\- Sourcing Shell Variables from a Script (Ungraded Lab, 60 minutes)

\- Practicing with Bash Streams (Ungraded Lab, 60 minutes)

Module 3: Building Bash Scripts (15 hours)

\- Introduction to Building Bash Scripts (Video, 0 minutes, Preview module)

\- Introduction to Shell Logic and Control Flow (Video, 0 minutes)

\- What is Shell Control Flow? (Video, 2 minutes)

\- Using Shell Control Flow to Solve Problems in Bash (Video, 2 minutes)

\- Using Shell Loops in Bash (Video, 6 minutes)

\- Evaluating Conditions in Bash (Video, 5 minutes)

\- Introduction to Manipulating Data in the Bash Shell (Video, 1 minute)

\- Shell Techniques for Data in the Bash Shell (Video, 2 minutes)

\- Truncating Data in Bash (Video, 3 minutes)

\- Filtering Data in Bash (Video, 3 minutes)

\- Searching Data in Bash (Video, 4 minutes)

\- Introduction to Writing Scripts and Command-Line Tools in Bash (Video, 0 minutes)

\- What are Scripts and Command-Line Tools? (Video, 3 minutes)

\- Building a Bash Script (Video, 4 minutes)

\- Building a Bash Function (Video, 5 minutes)

\- Building a Bash Command-Line Tool (Video, 3 minutes)

\- Bash-Adjacent Tools: Makefiles and Dockerfiles (Video, 7 minutes)

\- Data Structures in Bash (Video, 4 minutes)

\- Key Terms (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Key Terms (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Key Terms (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Building Bash Scripts (Quiz, 30 minutes)

\- Quiz-Using Shell Logic and Control Flow (Assignment, 180 minutes)

\- Quiz-Manipulating Data with the Shell (Assignment, 180 minutes)

\- Quiz-Writing Bash Scripts and Command-Line Tools (Assignment, 180 minutes)

\- Writing Loops in Bash (Ungraded Lab, 60 minutes)

\- Truncating Large Files with Bash (Ungraded Lab, 60 minutes)

\- Building a Command-Line Tool for Data Processing (Ungraded Lab, 60 minutes)

\- Building a Bash CLI with Options (Ungraded Lab, 60 minutes)

Module 4: Composing File and Data Solutions (16 hours)

\- Introduction to Composing File and Data Management Solutions in Linux (Video, 0 minutes, Preview module)

\- Introduction to Searching a Filesystem in Linux (Video, 0 minutes)

\- Methods for Searching a Filesystem in Linux (Video, 4 minutes)

\- Using the Locate Command to Find Files in Linux (Video, 3 minutes)

\- Using the Find Command to Find Files in Linux (Video, 2 minutes)

\- Using xargs to Extend Search Capabilities in Linux (Video, 2 minutes)

\- Using mdfind on OS X (Video, 6 minutes)

\- Introduction to Modifying Files, Directories, Permissions and Archiving in Linux (Video, 0 minutes)

\- Overview of Modifying the Filesystem in Linux (Video, 3 minutes)

\- Moving Files and Directories in Linux (Video, 6 minutes)

\- Setting Permissions on Files and Directories in Linux (Video, 4 minutes)

\- Archiving Data in Linux (Video, 3 minutes)

\- Introduction to Processing Text in Linux (Video, 0 minutes)

\- Benefits of Processing Text in Linux (Video, 1 minute)

\- Using grep, cut, sort and unique in Linux (Video, 8 minutes)

\- Editing with Truncation, awk and sed in Linux (Video, 7 minutes)

\- Using Regular Expressions (regex) in Linux (Video, 5 minutes)

\- Key Terms (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Key Terms (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Key Terms (Reading, 10 minutes)

\- Lesson Reflection (Reading, 10 minutes)

\- Next Steps (Reading, 10 minutes)

\- Composing File and Data Management Solutions with Linux (Quiz, 30 minutes)

\- Quiz-Searching the Filesystem (Assignment, 180 minutes)

\- Quiz-Modifying Files, Directories, Permissions and Archiving (Assignment, 180 minutes)

\- Quiz-Processing Text (Assignment, 180 minutes)

\- Searching from the Bash Prompt (Ungraded Lab, 60 minutes)

\- Practicing the Basics of Shell Permissions (Ungraded Lab, 60 minutes)

\- Searching for Text with Bash (Ungraded Lab, 60 minutes)

\- Searching the filesystem with find (Ungraded Lab, 60 minutes)

\- Final Course Challenge: Extended Phrase Repeater (Ungraded Lab, 60 minutes)