Kafka Streaming API: Introduction

A sample Streaming Application

Concept: While doing the Kafka stream client development, need to take step wise approach. Steps can be-

  • Understand the event stream type and NFR
  • Identify and model Event
  • Transport event
  • Process the event stream

Example:

  • Customer Check-in is an event stream
  • Customer check-in system generate the event
  • Model the event
  • Transport the event though HTTP, JMS, DB etc
  • Process and publish the event

A sample stream client application

Follow the below steps in order to build the Stream client application.

  1. Create the stream configuration

2. Create the Stream Builder

3. Open a Stream for topic

4. Process the stream

5. Create the topology

6. Create the Kafka Stream with topology and properties.

7. Add the Runtime

The Complete code:

Test the code:

Start the producer

Start the Stream application

Kafka DSL-Streaming

Event Stream — Continuous flow of events, unbounded dataset and immutable data records.

Streaming Operations — Stateless, State full and window based. Used for transform, aggregate, filter and enrich the stream.

Stream APIs
Stateful and Stateless APIs

filter() — Filters the KStream and returns the new KStream which will have only the filtered value as part of stream.

Example-1: In the below stream ks0 is created which will have the value that ends with “yan”.

Example-2:

Without lambda approach-

filterNot(): This works just reverse of filter DSL API. It returns the KStream a new KStream which says to not to filter the provided value as part of stream.

Example-1: In the below stream ks0 is created which will have the value that ends with “yan”.

Example-2:

Without lambda approach-

map() — Returns the new KStream with new structure means it can transform the key and value to new one. Re-partition happens based on the new key and value. Mapper function is applied on each function.

Without Lambda style

mapValues(): Used for the Transformation for a value in the stream but it does not impact the key.

flatMap(): This works similar to map() function. It transform each record of the input stream into zero or more records in the output stream. Both the key and values can be changed in output records.

Below example returns the collection of key and value group form from one stream. For example a key has values names-John, Murthi, Utaam. The below example will create (names, John),(names, Murthi) and (names, Utaam).

--

--

Middleware Expert in Design and Development, Working on Kafka, Eager to investigate and learn — Software Engineer.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Narayan Kumar

Middleware Expert in Design and Development, Working on Kafka, Eager to investigate and learn — Software Engineer.