Big Data, Hadoop, and Spark Basics
Offered By: IBM via edX
Course Description
Overview
Organizations need skilled, forward-thinking Big Data practitioners who can apply their business and technical skills to unstructured data such as tweets, posts, pictures, audio files, videos, sensor data, and satellite imagery, and more, to identify behaviors and preferences of prospects, clients, competitors, and others. ****
This course introduces you to Big Data concepts and practices. You will understand the characteristics, features, benefits, limitations of Big Data and explore some of the Big Data processing tools. You'll explore how Hadoop, Hive, and Spark can help organizations overcome Big Data challenges and reap the rewards of its acquisition.
Hadoop, an open-source framework, enables distributed processing of large data sets across clusters of computers using simple programming models. Each computer, or node, offers local computation and storage, allowing datasets to be processed faster and more efficiently. Hive, a data warehouse software, provides an SQL-like interface to efficiently query and manipulate large data sets in various databases and file systems that integrate with Hadoop.
Open-source Apache Spark is a processing engine built around speed, ease of use, and analytics that provides users with newer ways to store and use big data.
You will discover how to leverage Spark to deliver reliable insights. The course provides an overview of the platform, going into the different components that make up Apache Spark. In this course, you will also learn how Resilient Distributed Datasets, known as RDDs, enable parallel processing across the nodes of a Spark cluster.
You'll gain practical skills when you learn how to analyze data in Spark using PySpark and Spark SQL and how to create a streaming analytics application using Spark Streaming, and more.
Syllabus
Module 1 – What is Big Data?
___Introduction to Big Data_ *
o What is Big Data?
o Impact of Big Data
o Parallel Processing, Scaling, and Data Parallelism
o Tools of Big Data
o Beyond the Hype
o Big Data Use Cases
o Viewpoints about Big Data
Module 2 – Introduction to the Hadoop Ecosystem
___Introduction to the Hadoop Ecosystem_ *
o What is Hadoop
o An introduction to MapReduce
o The Hadoop Ecosystem/Common components: Introducing HDFS, Hive, HBase, and Spark, other modules
o Working with HDFS
o Working with HBase
o Lab: MapReduce
Module 3 – Introduction to Apache Spark
___Introduction to Apache Spark_ *
o Why use Apache Spark?
o Functional Programming Basics
o Parallel Programming using Resilient Distributed Datasets
o Scale-out / Data Parallelism in Apache Spark
o DataFrames and SparkSQL
o Lab: Practical examples with PySpark
Module 4 – DataFrames and SparkSQL
___DataFrames and SparkSQL_ *
o Introduction to Data-Frames & SparkSQL
o RDDs in Parallel Programming and Spark
o Data-frames and Datasets
o Catalyst and Tungsten
o ETL with Data-frames
o Lab: ETL with Data-frames
o Real-world usage of SparkSQL
o Lab: SparkSQL
Module 5 – Development and Runtime Environment options
___Development and Runtime Environment options_ *
o Apache Spark architecture
o Overview of Apache Spark Cluster Modes
o How to Run an Apache Spark Application
o Using Apache Spark on IBM Cloud
o Lab: Scale-out on IBM Spark Environment in Watson Studio
o Setting Apache Spark Configuration
o Running Spark on Kubernetes
o Lab: Spark on Kube
Module 6 – Monitoring & Tuning
___Monitoring and tuning Apache Spark_ *
o The Apache Spark User Interface
o Monitoring Jobs
o Debugging of parallel jobs
o Understanding Memory resources
o Understanding Processor resources
o Lab: Monitoring and Performance tuning
Module 7 – Final Quiz ****
Taught by
Romeo Kienzler
Tags
Related Courses
Big Data EssentialsA Cloud Guru Hadoop Quick Start
A Cloud Guru Big Data: adquisición y almacenamiento de datos
Universitat Autònoma de Barcelona (Autonomous University of Barcelona) via Coursera Análise de big data com o Hadoop (Português) | Analyze Big Data with Hadoop (Portuguese)
Amazon Web Services via AWS Skill Builder Análise de big data com o Hadoop (Português) | Analyze Big Data with Hadoop (Portuguese)
Amazon Web Services via AWS Skill Builder