Kafka Messaging — Introduction

Messaging: Application Integration

Messaging is the backbone of Enterprise wherein the applications talks to each other over integration layer. There are two basic factors are considered as a main driving factors for implementation of the integration layer.

1. Protocol

2. Design Principle

Protocol are like HTTP/ HTTPS/JMS.

I will take little more time to talk about the Design principle. It can be one of the following. The implementation for mentioned below are considered on case to case basis. There is no hard line can be drawn in the selection of the application integration design.

Only one type of Metric publisher

Single-Direct Metrics publisher.

Many types of publishers for different metrics, using direct connections

Many metrics publishers, using direct connections

Many Metrics publisher application with multiple subscriber

A metrics publish/subscribe system

Multiple publish/subscribe systems

Multiple publish/subscribe systems

Getting Started With Kafka

Kafka Architecture

Brokers and Clusters: Brokers are heart of Kafka body that are also called the Kafka server.

Topics and Partitions: Topic is a logical space between producer and consumer. Topic is held by broker that stores events and managed by broker.

Producers and Consumers: Producer is responsible to publish the message to Topic and Consumer are one which polls the message from Topic. Both are available as collection of logical APIs that are quite decoupled items from broker.
Replication of partitions in a cluster
Replication Across Data centre

Replication across DCs

Commercial Version Apache Kafka — Confluent

Message

Reference:

A Guide To The Kafka Protocol

Request Message:

RequestMessage => RequestMessage
ApiKey => int16
ApiVersion => int16
CorrelationId => int32
ClientId => string
RequestMessage => MetadataRequest | ProduceRequest | FetchRequest | OffsetRequest | OffsetCommitRequest | OffsetFetchRequest

ApiKey: This is a numeric id for the API being invoked, kind of metadata for request.
ApiVersion: This is a numeric version number for this API.
CorrelationId: This is a user-supplied integer. It will be passed back in the response by the server, unmodified.
ClientId:
This is a user supplied identifier for the client application. The user can use any identifier they like and it will be used when logging errors, monitoring aggregates, etc.

Response Message:

Response =>
CorrelationId => int32
ResponseMessage => MetadataResponse | ProduceResponse | FetchResponse | OffsetResponse | OffsetCommitResponse | OffsetFetchResponse

Network

Kafka uses a binary protocol over TCP. The client initiates a socket connection and then writes a sequence of request messages and reads back the corresponding response message.

No handshake is required on connection or disconnection.

TCP is happier if you maintain persistent connections used for many requests to amortise the cost of the TCP handshake, but beyond this penalty connecting is pretty cheap.

Kafka maintains the connection pool for a single client instance in order to communicate to the broker.

Message Processing

In a sequence
Batch mode
Stream Processing
Real time Stream Processing