Kafka Streams API for Developers Using Java/Spring Boot 3.X

This course is meticulously crafted to guide you through the intricacies of Kafka Streams API using Java and Spring Boot 3.x. You’ll start by getting familiar with the core concepts of Kafka Streams, understanding how to build basic stream processing applications, and moving quickly into practical examples. The course structure ensures that you not only grasp the theoretical aspects but also gain hands-on experience by developing real-world applications such as an Order Management System. As you progress, you'll delve into more advanced topics, including serialization/deserialization, stateful operations like aggregation, joins, and windowing. You'll learn how to manage and query state stores interactively, handle errors efficiently, and test your Kafka Streams applications using TopologyTestDriver and JUnit5. The course emphasizes best practices and recommended approaches, ensuring that your applications are both scalable and maintainable. By the end of this course, you'll have a deep understanding of Kafka Streams API, be proficient in using Spring Boot for building and running Kafka Streams applications, and be well-prepared to tackle complex, real-time data processing challenges in your projects. Whether you're building microservices or handling big data pipelines, this course will elevate your expertise in distributed systems. This course is designed for Java developers with intermediate knowledge of Spring Boot. A basic understanding of Kafka and distributed systems will be beneficial but is not mandatory.

• Getting Started with the Course
• In this module, we will introduce you to the course and outline the key prerequisites. You’ll learn what you need to know before diving into Kafka Streams and get an overview of how the course is structured.
• Getting Started with Kafka Streams
• In this module, we will delve into the core concepts of Kafka Streams, covering essential terminologies and providing an overview of the KStreams API. This foundational knowledge will prepare you for more advanced topics in the following sections.
• Greetings Kafka Streams App Using KStreams API
• In this module, we will guide you through building a simple Kafka Streams application, focusing on setting up and coding a topology that transforms messages. You’ll also learn how to test your application locally to ensure it functions correctly.
• Operators in Kafka Streams Using KStreams API
• In this module, we will explore various operators provided by the KStreams API, demonstrating how to apply filtering, transformation, and merging operations. You’ll also learn techniques to debug your streams effectively.
• Serialization and Deserialization in Kafka Streams
• In this module, we will dive into the mechanics of serialization and deserialization within Kafka Streams. You’ll learn how to configure default SerDes, create custom serializers/deserializers, and apply them in your streams application.
• Reusable Generic Serializer/Deserializer (Recommended Approach)
• In this module, we will focus on creating a generic serializer/deserializer that can be reused across multiple Kafka Streams applications. You’ll learn how to integrate this generic SerDes into your applications to streamline data processing.
• Order Management Kafka Streams Application - A Real-Time Use Case
• In this module, we will build a real-time Kafka Streams application for order management. You’ll create a topology that processes and transforms orders, applying branching logic to handle different types of orders in a retail environment.
• Topology, Stream, and Tasks - Under the Hood
• In this module, we will take a deep dive into the internal architecture of Kafka Streams, examining how topology, streams, and tasks operate. You’ll also learn how adjusting stream threads can affect your application's performance.
• Error/Exception Handling in Kafka Streams
• In this module, we will focus on the various failure scenarios in Kafka Streams and how to handle them. You’ll learn to create custom error handlers for different stages of the stream processing pipeline, ensuring robust application behavior even under failure conditions.
• KTable and Global KTable
• In this module, we will introduce KTable and GlobalKTable, key components in Kafka Streams for handling stateful data. You’ll build topologies that use these components and explore how they function under the hood to manage state across distributed systems.
• StateFul Operations in Kafka Streams - Aggregate, Join, and Windowing Events
• In this module, we will cover the essential stateful operations in Kafka Streams, including aggregation, joining, and windowing events. You’ll learn how to use different aggregation operators and materialized views to manage state effectively in stream processing.
• StateFul Operation Results - How to Access Them?
• In this module, we will explore various strategies for accessing the results of stateful operations in Kafka Streams. You’ll learn how to expose these results to external systems, enabling broader use of your processed data.
• Aggregation in Order Management Application - A Real-Time Use Case
• In this module, we will apply aggregation techniques in a real-time Order Management application. You’ll learn how to count orders and calculate revenue by store, leveraging Kafka Streams for real-time data processing in a retail context.
• Rekeying Kafka Records for Stateful Operations
• In this module, we will dive into the concept of rekeying in Kafka Streams. You’ll learn how null keys affect stateful operations and how to effectively repartition your data using the selectKey operator to ensure accurate processing.
• StateFul Operations in Kafka Streams - Join
• In this module, we will focus on join operations in Kafka Streams, covering inner, left, and outer joins across KStreams, KTables, and GlobalKTables. You’ll learn how to implement these joins in real-world scenarios and understand the internal mechanics behind them.
• Join in Order Management Application - A Real-Time Use Case
• In this module, we will apply join operations to a real-time Order Management application. You’ll implement a join between aggregated revenue data and store details, demonstrating the practical use of joins in Kafka Streams.
• StateFul Operations in Kafka Streams - Windowing
• In this module, we will explore the concept of windowing in Kafka Streams, an essential technique for managing time-based stateful operations. You’ll learn how to implement various window types and control the timing of results using advanced operators.
• Widowing in Order Management Application - A Real-Time Use Case
• In this module, we will extend the Order Management application to include windowing functionality. You’ll implement custom timestamp extractors, aggregate data by windows, and learn how to perform joins on windowed data for real-time insights.
• Behavior of Records with Future and Older Timestamp in Windowing
• In this module, we will examine how Kafka Streams processes records with timestamps that deviate from the current time window. You’ll learn how to manage these out-of-order records and ensure accurate windowed operations in your streams.
• Build Kafka Streams Application Using Spring Boot
• In this module, we will guide you through building a Kafka Streams application using Spring Boot. You’ll learn how to configure your application, build a simple stream topology, and test it locally to ensure it meets your needs.
• Spring Boot Autoconfiguration of Kafka Streams
• In this module, we will delve into the internals of how Spring Boot automatically configures Kafka Streams applications. You’ll gain insights into the mechanisms that simplify application setup and learn how to leverage this feature for efficient development.
• JSON Serialization/Deserialization in Spring Kafka Streams
• In this module, we will focus on handling JSON data within Kafka Streams using Spring Boot. You’ll learn how to implement serialization and deserialization with JsonSerde, and how to customize the ObjectMapper for more complex JSON processing needs.
• Error Handling in Spring Kafka Streams
• In this module, we will explore various strategies for error handling in Spring Kafka Streams applications. You’ll learn how to manage deserialization errors, handle uncaught exceptions in topologies, and deal with production errors to ensure robust application performance.
• Build Orders Kafka Streams Application Using Spring Boot
• In this module, we will guide you through building a Kafka Streams application for managing orders using Spring Boot. You’ll set up the project and implement the OrdersTopology, which will handle and process real-time order data efficiently.
• Interactive Queries - Querying State Stores Using RESTFUL APIs
• In this module, we will cover how to query Kafka Streams state stores using RESTful APIs. You’ll learn to build endpoints for retrieving order counts and revenue, and how to implement global error handling to improve the client experience.
• Interactive Queries - Querying Window State Stores Using RESTFUL APIs
• In this module, we will explore querying windowed state stores via RESTful APIs in Kafka Streams. You’ll build endpoints to retrieve order counts and revenue within specified time ranges, providing a complete picture of windowed data in your application.
• Testing Kafka Streams Using TopologyTestDriver and JUnit5
• In this module, we will focus on testing Kafka Streams applications using TopologyTestDriver and JUnit5. You’ll learn to write unit tests for various scenarios, ensuring that your Kafka Streams application behaves as expected under different conditions.
• Testing Kafka Streams in Spring Boot Using TopologyTestDriver and JUnit5
• In this module, we will guide you through the process of testing Kafka Streams applications built with Spring Boot. You’ll learn how to integrate TopologyTestDriver for testing and ensure that your Spring Boot-based Kafka Streams topologies perform correctly.
• Integration Testing Spring KafkaStreams App Using @EmbeddedKafka
• In this module, we will cover the setup and execution of integration tests for Kafka Streams applications using @EmbeddedKafka. You’ll learn how to test critical functionalities such as order counting and revenue aggregation to validate your application's end-to-end performance.
• Grace Period in Kafka Streams
• In this module, we will introduce you to the concept of grace periods in Kafka Streams, explaining how they are used to handle late-arriving events within windows. You’ll learn when and how to apply grace periods to improve the accuracy of your stream processing.
• Build and Package the Spring Boot App as an Executable
• In this module, we will show you how to package your Spring Boot Kafka Streams application as an executable jar file. You’ll learn how to build the jar and run your application locally, preparing it for deployment.
• Exactly Once Processing/Semantics in Kafka Streams
• In this module, we will explore exactly-once processing semantics in Kafka Streams. You’ll learn how to implement these semantics, including managing transactions and idempotent producers, and assess the trade-offs involved in using exactly-once processing.
• Running Kafka Streams Applications as Multiple Instances (Spring Boot)
• In this module, we will delve into running Kafka Streams applications as multiple instances using Spring Boot. You’ll learn how to manage distributed Kafka Streams applications, implement key-based queries, and aggregate data across multiple instances for a scalable and robust streaming solution.