Smart City End-to-End Realtime Data Engineering Project - AWS Data Engineer

Build a comprehensive Smart City end-to-end real-time data streaming pipeline in this extensive tutorial. Learn to create a robust system architecture encompassing data ingestion, processing, and storage using cutting-edge tools and technologies. Dive into hands-on implementation with Docker containers, coding IoT service producers for various city data streams, and setting up Kafka for data production. Explore AWS services including S3, IAM, Glue, Athena, and Redshift for cloud-based data management and analysis. Master Apache Spark real-time streaming from Kafka, tackle schema issues, and connect your data warehouse to visualization tools. Gain practical experience in solving common challenges like IAM permission issues and optimizing your data pipeline for real-world smart city applications.

Introduction
System Architecture
Project Setup
Docker containers setup and coding
IOT services producer
Vehicle information Generator
GPS Information Generator
Traffic information Generator
Weather information Generator
Emergency Incident Generator
Producing IOT Data to Kafka
AWS S3 setup with policies
AWS IAM Roles and Credentials Management
Apache Spark Realtime Streaming from Kafka
Fixing Schema Issues in Apache Spark Structured Streaming
AWS Glue Crawlers
Working with AWS Athena
Loading Data into Redshift from AWS Glue Data Catalog
Connecting and Querying Redshift DW with DBeaver
Connecting Redshift to AWS Glue Catalog
Fixing IAM Permission issues with Redshift
Outro