Introducing the aggregation in Kafka and explained this in easy way to implement the Aggregation on real time streaming.

In order to aggregate the stream we need do two steps operations.

groupBy…


Store, KTable, GlobalKTable,

If application requires aggregation, Kafka uses the store in order to aggregate the stream for further processing. The use case like word count, live trend of any event and live voting can be consider for the candidate for aggregation.

The Stream processing basically requires to consider for…


About kafka Streaming

Kafka DSL-Streaming

Event Stream — Continuous flow of events, unbounded dataset and immutable data records.

Streaming Operations — Stateless, State full and window based. Used for transform, aggregate, filter and enrich the stream.


Kafka sends the events though network which requires the message to be serialize before sending over the network. Publisher API provides the serializer like IntegerSerializer, StringSerializer etc, same sense of deserializer.

Serializer is used by publisher while Deserializer by consumer.

If the default serializer may not full-fill the need specially…


A sample Streaming Application

Concept: While doing the Kafka stream client development, need to take step wise approach. Steps can be-

  • Identify and model Event
  • Transport event
  • Process the event stream

Example:

  • Customer check-in system generate the…


This story talks about the event and processing of events using Kafka.

Event- Shows the change in state of object. The unbounded form of flow of event is called event stream. It shows something happened in the source of the event.

Streaming- The unbounded form of source of event

Producer


This story talk about advance feature of Kafka like -

Interceptor

Transaction

Partitioning

Idempotent Producer

Compression

Batching

Interceptor

Helps to change the behaviour of Kafka client application without changing the code. …


This talks about the Apache Kafka Architecture, its components and the internal structure for Kafka for its components.

Why Kafka?

  • Multiple consumers
  • Disk based persistence
  • Offline messaging
  • Messaging replay
  • Distributed
  • Super Scalable
  • Low-latency
  • High volume
  • Fault tolerance
  • Real-time Processing

Apache Kafka Architecture

Single Cluster: Apache Kafka

Components:

  • Producer
  • Consumer
  • Message


Prerequisites

Producer Flow

Event is published as ProducerRecord to the topic which resides in and managed by broker. The ProducerRecord is formed and pushed by Producer. It is followed in a step of steps that is depicted as below.

  • Create the Producer as below
  • Send the message

How…


Messaging: Application Integration

Messaging is the backbone of Enterprise wherein the applications talks to each other over integration layer. There are two basic factors are considered as a main driving factors for implementation of the integration layer.

1. Protocol

2. Design Principle

Protocol are like HTTP/ HTTPS/JMS.

I will take…

Narayan Kumar

Middleware Expert in Design and Development, Working on Kafka, Eager to investigate and learn — Software Engineer.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store