Databricks Fundamentals & Apache Spark Core

Learn how to process big-data using Databricks & Apache Spark 2.4 and 3.0.0 - DataFrame API and Spark SQL

What you'll learn:

• Databricks
• Apache Spark Architecture
• Apache Spark DataFrame API
• Apache Spark SQL
• Selecting, and manipulating columns of a DataFrame
• Filtering, dropping, sorting rows of a DataFrame
• Joining, reading, writing and partitioning DataFrames
• Aggregating DataFrames rows
• Working with User Defined Functions
• Use the DataFrameWriter API

Welcome to this course on Databricks and Apache Spark 2.4 and 3.0.0

Apache Spark is a Big Data Processing Framework that runs at scale.
In this course, we will learn how to write Spark Applications using Scala and SQL.

Databricks is a company founded by the creator of Apache Spark.
Databricks offers a managed and optimized version of Apache Spark that runs in the cloud.

The main focus of this course is to teach you how to use the DataFrame API & SQL to accomplish tasks such as:

• Write and run Apache Spark code using Databricks

• Read and Write Data from the Databricks File System - DBFS

• Explain how Apache Spark runs on a cluster with multiple Nodes

Use the DataFrame API and SQL to perform data manipulation tasks such as

• Selecting, renaming and manipulating columns

• Filtering, dropping and aggregating rows

• Joining DataFrames

• Create UDFs and use them with DataFrame API or Spark SQL

• Writing DataFrames to external storage systems

List and explain the element of Apache Spark execution hierarchy such as

• Jobs

• Stages

• Tasks