Kafka 101 — In Depth Concepts of Kakfa

9 min readOct 12, 2024

Introduction

Kafka is a distributed event streaming platform.
Use cases — distributed logging, stream processing, pub sub messasging

Event

Event is anything that has happened. It stores info like what has happened
Event = {key, value, timestamp, headers}
Events are immutable. Internally it stores everything as sequence of bytes. key, value can be anything. but key is generally a string or empty. Key of message may not be unique. Serialization & Deserialization.
Event get stored as a Record in ordered fashion.

Brokers

A computer, instance or container running the kafka process.
Each broker manage some set of kafka partitions
Handle read & write requests and replication of partitions.
Kafka actually stores all of its records to disk.

Topic

Kafka topic has partitions. Each partition has its own log. so it has its own offset. When creating a topic, we have to provide a partition count and the replication factor.

The Log

The data in the system is stored in topics. The fundamental basis of a topic is the log — a simple ordered data structure which stores records sequentially.

Each replica is nothing more than a few files itself, each of which embody the log data structure and sequentially form a larger log. Each record in the log is denoted by a specific offset, which is simply a monotonically-increasing number.

Partitioning in Kafka

Each partition can live on a separate node in the Kafka cluster.
Messages are written to partitions based on their key. Messages with the same key are always written to the same partition. Messages are written to partitions in a round-robin fashion if they have no key.
Partitioning allows Kafka to scale horizontally.

How to choose a partitioning key

The partitioning key should be a field that uniquely identifies each message and is evenly distributed across messages.

Best practices for partitioning

Create a sufficient number of partitions to ensure that data is evenly distributed across the Kafka cluster.
Monitor the number of messages in each partition to ensure that partitions are not overloaded.
Kafka also supports partitioning by range and custom partitioner. This can be useful for partitioning data based on a numerical value or implementing custom partitioning logic.

Kafka Replication

Copies of data for fault tolerance.
One lead partition and N-1 followers.
Replication factor is configurable per topic basis. means same cluster with different topics can have different replication factor.

What are Kafka Partitions Leader and Replicas?

For a given topic-partition, one Kafka broker is designated by the cluster to be responsible for sending and receiving data to clients. That broker is known as the leader broker of that topic partition. Any other broker that is storing replicated data for that partition is referred to as a replica.

Therefore, each partition has one leader and multiple replicas.

A replica can be in two states — in-sync or out-of-sync. As the name suggests, out-of-sync replicas are ones that don’t have the latest data for the partition.

What are In-Sync Replicas (ISR)?

An ISR is a replica that is up to date with the leader broker for a partition. Any replica that is not up to date with the leader is out of sync.

As a general rule, for a replication factor of N, you can permanently lose up to N-1 brokers and still recover your data.

Kafka Producers

A Client application responsible for putting messages into topics. Reading a kafka message does not destroy it, it is still there .

Producer api accumates records and sends in batches. This increases performance. Writes are always written to the leader replica of that partition. Writes can only go to that leader, which then asynchronously replicates the data to the N-1 followers.

kakfa producer config

linger.ms — linger.ms, is the number of milliseconds a producer is willing to wait before sending a batch out. And by default, it’s zero. That means that the producer should send data to Kafka right away. Now if you introduce a little bit of lag, for example, it’s okay if we don’t get the data right away, we can wait maybe five or ten milliseconds more. So you set linger.ms=5. So, basically, we increase the chances of messages being sent together in a batch. And at the expense of this small delay, we can increase the throughput, compression, and efficiency of the producer. So overall, adding a small delay may actually increase the efficiency. Now, if your batch is full, which is the batch.size setting, before the end of the linger.ms period, then we’ll be sent to Kafka right away.
batch.size — batch.size is the maximum number of bytes that will be included in one batch. By default, it is 16 KB. But you can increase that batch size to something like 32 KB or 64 KB because that helps to have a bigger batch, so that helps the compression, the throughput, and that places fewer requests on to Kafka. And so, overall, this is a good setting. Now you should know that any message that is bigger than the batch.size, will not be batched. So if you have a message that’s 100KB, then it won’t be batched. The batch.size, is allocated per partition. So when you set the batch size to a very high number, then you may just completely outrun your producer memory or waste memory, so don’t set it to too high a number. The defaults are fine, if you double it to 32KB, or quadruple it to 64KB, that’s good, but don’t set it to a super high number.
bootstrap.servers — A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.
compression.type — The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).
acks — Kafka producers only write data to the current leader broker for a partition. Kafka producers must also specify a level of acknowledgment acks to specify if the message must be written to a minimum number of replicas before being considered a successful write.
— acks=0 — When acks=0 producers consider messages as "written successfully" the moment the message was sent without waiting for the broker to accept it at all.
— acks = 1 — When acks=1 , producers consider messages as "written successfully" when the message was acknowledged by only the leader.
— acks = all — When acks=all, producers consider messages as "written successfully" when the message is accepted by all in-sync replicas (ISR).

Themin.insync.replicas can be configured both at the topic and the broker-level. The data is considered committed when it is written to all in-sync replicas - min.insync.replicas.

Kafka Consumers

Consumers are applications that read data from Kafka topics. Consumers can be configured to consume data from one or more partitions of a topic, from the beginning of a topic or from the latest offset, in a group or individually. Consumers in a group share the work of consuming data from a topic. Kafka Consumers always live as consumer group. Multiple consumer with same groupId will form a consumer group.

Kafka also supports consumers that are built into the Kafka broker. These consumers are called internal consumers. Internal consumers can be used to implement features such as replication and compaction.
Adding or deleting consumer in consumer group, triggers rebalancing process
Kafka consumers read by default from the partition leader. But since Apache Kafka 2.4, it is possible to configure consumers to read from in-sync replicas instead (usually the closest).

Consumer groups

Consumers form so-called consumer groups, which are simply a bunch of consumers who are logically grouped and synchronized together. They persist their progress (up to what offset they’ve consumed from any given partition) in a particular partition of a special Kafka topic called `__consumer_offsets`. The broker that is the leader of the partition acts as the so-called Group Coordinator for that consumer group, and it is this Coordinator that is responsible for maintaining the consumer group membership and liveliness.

Protocol

Clients connect to the brokers via TCP using the Kafka protocol.

Consensus

Any distributed system requires consensus — the act of picking exactly one broker to be the controller at any given time is fundamentally a distributed consensus problem.

Kafka historically outsourced consensus to ZooKeeper. When starting the cluster up, every broker would race to register the `/controller` zNode and the first one to do so would be crowned the controller. Similarly, when the current Controller died — the first broker to register the zNode subsequently would be the new controller.

Kafka also used to heavily leverage ZooKeeper’s watch mechanism, which would notify a subscriber whenever a certain zNode changed.

For the last few years, Kafka has actively been moving away from ZooKeeper towards its own consensus mechanism called KRaft (“Kafka Raft”).

QnA

Can replication factor greater than number of brokers in a kafka cluster ?
In Apache Kafka, the replication factor of a topic cannot be greater than the number of brokers in the cluster. The replication factor defines how many copies of each partition are maintained across the Kafka brokers to ensure data redundancy and fault tolerance.
Here’s why the replication factor cannot exceed the number of brokers:
— Each replica must reside on a different broker: Kafka ensures that the replicas of a partition are stored on different brokers. If the replication factor were to exceed the number of brokers, Kafka would not have enough brokers to place each replica on a separate one, violating this rule.
— Fault tolerance: The replication factor is used to guarantee fault tolerance. If the replication factor were greater than the number of brokers, it would not enhance fault tolerance since Kafka would need to place multiple replicas on the same broker, which would defeat the purpose of replication (as losing that broker would mean losing multiple replicas at once).
If there are 3 broker, will there be 3 replica of each partition ?
- No. As replication is configured per topic basis, so if there are 3 brokers in a cluster, then replication factor can be 3 or less than 3.
In a kafka cluster with 3 broker, will 1 broker be leader for all the topics in the cluster ?
In a Kafka cluster with 3 brokers, one broker will not be the leader for all topics by default. Kafka is designed to distribute the leadership of partitions across multiple brokers to balance the load and ensure high availability. Here’s how it works:
Can we have different replication factor for different topics in a kafka cluster ?
yes since replication factor is configurable at topic level.
Scenario — We have 3 partition of 1 topic, then if we have
— 1 consumer => this consumer will read from all partitions.
— 3 consumer => 1 consumer per partitions. 1 consumer will read from 1 partition.
— 5 consumer => 2 consumer will remain idle

If you read uptill now, then I hope you liked this article and if you like this article then please Clap 👏, as it motivates me to help the community.

Please comment if you found any discrepancy in this article or if you have any question related to this article.

Suggestions on improving this article would be marvellous.

Thank You for your time.

Subscribe to my YouTube Channel 🔔

Follow me on GitHub ✌

Connect me on LinkedIn 🤝

References