Stream Processing Design Patterns with Kafka Streams

Upskill as a data professional by learning how to build streaming pipelines using Snowflake, Kafka, and AWS.

Introduction

• Building streaming data pipelines
• What you should know
• Prerequisites

1. Introduction to Data Streaming, AWS, and Snowflake

• Why AWS and Snowflake?
• Introduction to streaming ingestion pipelines
• Streaming data pipelines: A deeper dive
• Overview of streaming pipeline project

2. Building a Streaming Pipeline

• Setting up your MSK cluster and EC2 instance
• Setting up your keys
• What is Snowpipe?
• Installing Snowpipe Kafka connector
• Set up config for provider-MSK connection and create topic
• Preparing your Snowflake environment
• Setting up your Snowpipe Kafka connect config
• Sending data to Snowflake
• Final considerations

Conclusion

• Course review and next steps