Kafka Interview Questions and Answers Part 10 - Real-World Manager Scenarios

Kafka Interview Q&A · Part 10 of 10

In this post we are going to cover real-world manager scenarios for Kafka. These are 7 real Kafka interview questions for data engineers, asked the way they come up on a real project, and the last one is a scenario: designing a streaming platform for a retail company with stores, an app and a warehouse.

This is the last part of the series. These questions are less about one config and more about judgment: reprocessing safely, managed or self-managed, growing a topic, contracts between teams, pushing back on "real time", and migrations. Interviewers ask them in the later rounds to see how you think when a manager is waiting for a plan.

Part 10 of the Real Time Kafka Interview Questions and Answers series on TechBrothersIT.

On this page

1. The short answer
2. Let's set up the cluster and some sample data
3. Question 1: A consumer bug wrote wrong totals for six hours. How do you reprocess the data safely?
4. Question 2: Leadership asks whether to run Kafka yourselves or use a managed service. How do you decide?
5. Question 3: A topic needs more partitions, but ordering per customer matters. How do you change it safely?
6. Question 4: Two teams want to change the same event format. How do you manage event contracts between teams?
7. Question 5: The business asks for real time, but the data is only needed once a day. How do you push back?
8. Question 6: The cluster must move to new hardware or a new region with no downtime. How do you plan the migration?
9. Scenario 7: In the final round, the hiring manager asks you to design a streaming platform for a retail company with stores, an app and a warehouse. What is your design?
10. The full script

The short answer

Say this first

Reprocess with the consumers stopped, a dry run of --reset-offsets --to-datetime, and an idempotent sink. Drain before you add partitions, treat events as owned, versioned contracts, match freshness to the decision, and migrate clusters with MirrorMaker 2, consumers first.

Let's set up the cluster and some sample data

We use a local Kafka 4.4 cluster in KRaft mode with 1 broker. Here are the starting files: orders.py sends twelve orders with event times, totals.py is our buggy totals consumer, hold.py keeps the totals service running, and keys.py shows where customer keys land.

orders.py

from datetime import datetime
from confluent_kafka import Producer

# (event time on 2026-10-26, partition): twelve orders across three partitions
orders = [('07:10', 0), ('07:40', 1), ('08:20', 2), ('08:50', 0),
          ('09:05', 1), ('09:30', 0), ('10:15', 2), ('11:00', 1),
          ('12:30', 0), ('14:45', 2), ('15:20', 1), ('16:00', 0)]
p = Producer({'bootstrap.servers': 'localhost:9092', 'enable.idempotence': True})
for i, (t, part) in enumerate(orders):
    ts = datetime.fromisoformat('2026-10-26T%s:00+00:00' % t)
    p.produce('orders', value='order-%d' % i, partition=part,
              timestamp=int(ts.timestamp() * 1000))
p.flush()
print('sent', len(orders), 'orders')

totals.py

from confluent_kafka import Consumer

c = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'totals',
              'auto.offset.reset': 'earliest'})
c.subscribe(['orders'])
n, idle = 0, 0
while idle < 5:
    m = c.poll(1.0)
    if m is None or m.error():
        idle += 1
        continue
    n += 1
c.close()          # commits the last offsets
print('processed', n, 'orders')

hold.py

import signal, sys
from confluent_kafka import Consumer

# the totals service, still running
c = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'totals',
              'enable.auto.commit': False})
c.subscribe(['orders'])
signal.signal(signal.SIGTERM, lambda *args: sys.exit(0))
try:
    while True:
        c.poll(0.5)
finally:
    c.close()                      # leave the group on shutdown

keys.py

from confluent_kafka import Producer

keys = ['c1', 'c2', 'c3', 'c4', 'c5', 'c6', 'c7', 'c8']
placed = {}

def on_delivery(err, msg):
    placed[msg.key().decode()] = msg.partition()

p = Producer({'bootstrap.servers': 'localhost:9092',
              'partitioner': 'murmur2_random'})   # same as the Java client
for k in keys:
    p.produce('customer-orders', key=k, value='order', on_delivery=on_delivery)
p.flush()
print('  '.join('%s:p%d' % (k, placed[k]) for k in keys))

Question 1: A consumer bug wrote wrong totals for six hours. How do you reprocess the data safely?

First, fix the bug and deploy it, so the damage stops. Then find when the bug went live, here 09:00 UTC, and check that retention still covers that time.

Let's send twelve orders and let the totals consumer process them.

Terminal

$ python orders.py
sent 12 orders
$ python totals.py
processed 12 orders
$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group totals | cut -c1-80
Consumer group 'totals' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
totals          orders          0          5               5               0
totals          orders          1          4               4               0
totals          orders          2          3               3               0

It processed all 12 orders, so its lag is 0 on every partition. Now let's try to reset offsets while the totals service is still up.

Terminal

$ python hold.py > hold.log 2>&1 &
$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group totals --topic orders --to-datetime 2026-10-26T09:00:00.000 --execute
Error: Assignments can only be reset if the group 'totals' is inactive, but the current state is Stable.

GROUP           TOPIC           PARTITION  NEW-OFFSET

The reset is refused: the group must be inactive, but the state is Stable. Kafka won't move offsets under a running group. So we stop every consumer in the group first.

Terminal

$ kill $(cat hold.pid); while kill -0 $(cat hold.pid) 2>/dev/null; do sleep 0.5; done

Then we reset by time. The dry run shows the new offset per partition: the first record at or after 09:00, found by timestamp. Then we execute it.

Terminal

$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group totals --topic orders --to-datetime 2026-10-26T09:00:00.000 --dry-run
GROUP           TOPIC           PARTITION  NEW-OFFSET
totals          orders          0          2
totals          orders          1          1
totals          orders          2          1
$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group totals --topic orders --to-datetime 2026-10-26T09:00:00.000 --execute
GROUP           TOPIC           PARTITION  NEW-OFFSET
totals          orders          0          2
totals          orders          1          1
totals          orders          2          1
$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group totals | cut -c1-80
Consumer group 'totals' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
totals          orders          0          2               5               3
totals          orders          1          1               4               3
totals          orders          2          1               3               2

The new offsets are 2, 1 and 1, and the lag shows the 8 orders to process again (3, 3 and 2). If we need one partition a little further back, --shift-by moves it by a few offsets.

Terminal

$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group totals --topic orders:0 --shift-by -1 --dry-run
GROUP           TOPIC           PARTITION  NEW-OFFSET
totals          orders          0          1

Reprocessing writes those totals again, so the sink must be idempotent: upsert by key, or overwrite the affected window, never plain inserts. That lets us start a little early, since a record's timestamp isn't when the buggy code read it.

• Cannot stop the service? Replay with a new consumer group into a separate output, then swap.
• Tell downstream teams before the numbers change.

Question 2: Leadership asks whether to run Kafka yourselves or use a managed service. How do you decide?

Compare the total cost of ownership, not the price per hour. Running Kafka ourselves means upgrades, broker failures, partition rebalancing, security patches, monitoring, and an on-call rota of people who know Kafka.

Managed services, like Confluent Cloud, Amazon MSK, or Azure Event Hubs with its Kafka endpoint, take most of that away. We trade some control: versions, broker configs and new features arrive on the provider's schedule.

	Self-managed	Managed service
Upgrades, patches	your team	the provider
Broker failures, rebalancing	your on-call rota	mostly the provider
Versions and configs	full control	the provider's schedule
Fits when	residency, on-prem, huge steady volume, strong team	most teams, fast start
Watch out for	people and on-call cost	feature gaps, egress cost

Let's be precise about two of them. Azure Event Hubs is a Kafka-protocol endpoint, not Apache Kafka, and Kafka Streams and transactions are still in preview there. Amazon MSK chooses the Kafka versions it offers. And data leaving a cloud costs money.

Self-managed makes sense with strict data residency, on-premises systems, very large steady volumes, or a team that already runs Kafka well. A Kubernetes operator like Strimzi makes it easier, not free.

My answer to leadership

Start managed unless a hard requirement says otherwise, and keep the clients on standard Kafka APIs, so we can move later.

Question 3: A topic needs more partitions, but ordering per customer matters. How do you change it safely?

The partitioner takes a hash of the key, modulo the number of partitions. So changing the count moves keys.

Let's create customer-orders with three partitions and send eight customer keys with keys.py. It uses murmur2_random, the same hash as the Java client. The Python client needs that setting to match.

Terminal

$ kafka-topics.sh --bootstrap-server localhost:9092 --create --topic customer-orders --partitions 3
Created topic customer-orders.
$ python keys.py
c1:p2  c2:p1  c3:p2  c4:p2  c5:p0  c6:p1  c7:p2  c8:p0

Now let's grow the topic to six partitions and send the same keys again.

Terminal

$ kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic customer-orders --partitions 6
$ python keys.py
c1:p5  c2:p1  c3:p5  c4:p2  c5:p3  c6:p1  c7:p5  c8:p3

Five of the eight keys move, like c1 from p2 to p5. Kafka doesn't move old records, so for a while one customer's events sit in two partitions, and the order between them is lost.

And there's no way back. Let's ask for three partitions again.

Terminal (expected error)

$ kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic customer-orders --partitions 3 2>/dev/null
Error while executing topic command : The topic customer-orders currently has 6 partition(s); 3 would
  not be an increase.

It's rejected: "3 would not be an increase". Partitions can only grow. So the safe plan is one of these:

• Drain, then grow: stop the producers, drain the consumers to zero lag, add partitions, and restart the producers so they all use the new count.
• No skipped records: consumers on earliest (Java: by_duration), not latest, or they can skip the first records on new partitions.
• Or a new topic: with the right count, and a planned cut-over of producers and consumers.

Best of all

Size it once, with headroom, so we never have to do this under pressure.

Question 4: Two teams want to change the same event format. How do you manage event contracts between teams?

Treat an event like a public API. One team owns each topic and its schema, and the schema lives in a registry, with a compatibility rule that the registry enforces.

Changes go through review like code, in this order:

• The owning team: one owner per topic.
• The schema file in Git, with a pull request and review.
• A CI check for compatibility against the registry.
• The schema registry keeps every version and enforces the rule.

Now let's look at how each kind of change ships.

Change	Example	How it ships
Additive	a new optional field	same topic, new schema version
Breaking	rename or retype a field	orders.v2 next to orders.v1, consumers migrate
Different needs	another team wants another shape	a derived topic, built with stream processing

Additive changes, like a new optional field, ship without breaking anyone. A breaking change, like renaming or retyping a field, becomes a new version of the event, usually a new topic, and both run side by side until consumers move.

If the two teams need different shapes, don't fight over one event. Keep the core event stable, and let each team build its own derived topic with stream processing.

Write it down

A data contract has the owner, the schema, what each field means, the compatibility mode, and how long old versions are supported.

Question 5: The business asks for real time, but the data is only needed once a day. How do you push back?

Ask what decision the data drives, and how fast someone acts on it. If the report is read once each morning, a streaming pipeline adds cost and on-call work, and nobody sees the difference.

Then put numbers on it: the freshness the decision needs, the cost to build and run each option, and who gets paged at night. A daily batch, or an hourly one, usually wins.

Need	How fresh	Build
Finance report read each morning	once a day	daily batch
Operations dashboard	15 to 60 minutes	hourly or micro-batch
Stock shown in the app	seconds to minutes	streaming
Blocking a fraudulent payment	seconds	streaming

Offer a path, not a no:

• Land the events in a Kafka topic anyway.
• Load them into the lake or warehouse in batches today.
• If a real-time use case shows up later, add a streaming consumer on the same topic, with no rework.

Real time is worth it when someone acts within seconds or minutes: blocking fraud, showing stock in the app, or alerting operations.

In the interview

Show that you weigh value against cost, not that you avoid streaming.

Question 6: The cluster must move to new hardware or a new region with no downtime. How do you plan the migration?

New hardware in the same cluster is the easy case. We add the new brokers, move partitions with the reassignment tool and a replication throttle, then remove the old brokers. Clients follow the metadata, and we update their bootstrap lists. In KRaft, controllers can be added and removed too, with a dynamic quorum.

A new region means a new cluster. MirrorMaker 2 replicates the topics, and with group offset sync, it writes translated consumer offsets to the new cluster. The identity replication policy keeps the topic names. Let's look at a one-way config.

mm2.properties

# mm2.properties: one-way old -> new, run with connect-mirror-maker.sh
clusters = old, new
old.bootstrap.servers = old-kafka-1:9092,old-kafka-2:9092
new.bootstrap.servers = new-kafka-1:9092,new-kafka-2:9092
old->new.enabled = true
old->new.topics = orders.*, payments.*
old->new.sync.group.offsets.enabled = true
# keep the topic names (the default adds an "old." prefix)
replication.policy.class = \
    org.apache.kafka.connect.mirror.IdentityReplicationPolicy
replication.factor = 3

Then we move in order:

• Consumers first, from their translated offsets.
• Then producers: stop them on the old cluster, let MirrorMaker catch up, and start them on the new one, so per-key order holds.
• Check record counts and MM2 lag.
• Keep a way back.

Offsets differ between the clusters, which is why the translation matters. And translated offsets come from periodic checkpoints, not per record, so consumers must tolerate a few duplicates at the cut-over.

Rehearse

Try one low-risk topic first, and follow a written runbook with a go or no-go check at every step.

Scenario 7: In the final round, the hiring manager asks you to design a streaming platform for a retail company with stores, an app and a warehouse. What is your design?

Start from the business events, not the tools: sales, orders, payments, stock changes and shipments. Let's build it layer by layer.

First, the sources. Stores send sales from the tills, the mobile app sends orders and clicks, and the warehouse sends stock and shipments. Core databases join through CDC with Debezium.

Then Kafka is the backbone: a topic per event type, keyed by what needs ordering, like order ID, or store and product for stock. Schemas in a registry, replication factor three, min in-sync replicas two.

Next, on top, stream processing for the real-time cases: live stock, fraud checks and alerts, feeding stock in the app and the on-call team. Sink connectors land every topic in the lake as bronze tables, then silver, for batch analytics and the warehouse.

Then we plan for failure. Stores lose their connection, so they buffer locally and send late, and pipelines use event time with grace. Idempotent producers, dead letter topics, and lag alerts everywhere.

Finally, the operating model: an owner and a contract per topic, ACLs per team, a managed service unless a hard requirement says otherwise, and capacity reviews.

Start small

Start with one flow, like stock, prove it, then grow.

Key points to remember

• Reprocess: stop the group, dry run, --to-datetime, idempotent sink
• Managed vs self-managed: total cost and hard requirements
• More partitions: drain first, or a new topic
• Events are owned, versioned contracts with a registry
• Migrations: MirrorMaker 2, consumers first, a rehearsed runbook

The full script

This is a Python script that writes the files and runs every command in order against a local Kafka 4.4. Run pip install confluent-kafka first.

Show the full script that writes the files and runs every step

Full script - Kafka 4.4 running locally, pip install confluent-kafka, then run it with Python

# ========================================================================
# Kafka Interview Q&A #10 - Real-world manager scenarios
# TechBrothersIT. Sample data is invented for teaching.
# Every step below was run on Apache Kafka 4.4.0 (KRaft mode, 1 node) with confluent-kafka 2.16.0;
# the outputs in the video and the blog post are the real outputs. To run it yourself (Linux, macOS or WSL):
#   1. Download Kafka 4.4.0 from kafka.apache.org, extract it, and put its bin folder on your PATH.
#   2. Start a local cluster (see the Kafka quickstart; node 1 listens on localhost:9092).
#   3. pip install confluent-kafka   then   python Kafka_Interview_QA_10_Code.py   in an empty folder.
# Steps that stop or start a broker say so in a comment: do that by hand at that point.
# ========================================================================

import os, subprocess

def write(path, text):
    os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
    with open(path, "w", encoding="utf-8") as f:
        f.write(text.strip("\n") + "\n")

def run(cmd, expect_error=False):
    print("\n$ " + cmd, flush=True)
    rc = subprocess.run(cmd, shell=True).returncode
    if rc != 0 and not expect_error:
        raise SystemExit("stopped: this command was expected to succeed")

# Step 0: files used by the steps
write('orders.py', """\
from datetime import datetime
from confluent_kafka import Producer

# (event time on 2026-10-26, partition): twelve orders across three partitions
orders = [('07:10', 0), ('07:40', 1), ('08:20', 2), ('08:50', 0),
          ('09:05', 1), ('09:30', 0), ('10:15', 2), ('11:00', 1),
          ('12:30', 0), ('14:45', 2), ('15:20', 1), ('16:00', 0)]
p = Producer({'bootstrap.servers': 'localhost:9092', 'enable.idempotence': True})
for i, (t, part) in enumerate(orders):
    ts = datetime.fromisoformat('2026-10-26T%s:00+00:00' % t)
    p.produce('orders', value='order-%d' % i, partition=part,
              timestamp=int(ts.timestamp() * 1000))
p.flush()
print('sent', len(orders), 'orders')
""")
write('totals.py', """\
from confluent_kafka import Consumer

c = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'totals',
              'auto.offset.reset': 'earliest'})
c.subscribe(['orders'])
n, idle = 0, 0
while idle < 5:
    m = c.poll(1.0)
    if m is None or m.error():
        idle += 1
        continue
    n += 1
c.close()          # commits the last offsets
print('processed', n, 'orders')
""")
write('hold.py', """\
import signal, sys
from confluent_kafka import Consumer

# the totals service, still running
c = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'totals',
              'enable.auto.commit': False})
c.subscribe(['orders'])
signal.signal(signal.SIGTERM, lambda *args: sys.exit(0))
try:
    while True:
        c.poll(0.5)
finally:
    c.close()                      # leave the group on shutdown
""")
write('keys.py', """\
from confluent_kafka import Producer

keys = ['c1', 'c2', 'c3', 'c4', 'c5', 'c6', 'c7', 'c8']
placed = {}

def on_delivery(err, msg):
    placed[msg.key().decode()] = msg.partition()

p = Producer({'bootstrap.servers': 'localhost:9092',
              'partitioner': 'murmur2_random'})   # same as the Java client
for k in keys:
    p.produce('customer-orders', key=k, value='order', on_delivery=on_delivery)
p.flush()
print('  '.join('%s:p%d' % (k, placed[k]) for k in keys))
""")
run('kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3')

# Step 1: Twelve orders with event times; the buggy consumer processed all of them
run('python orders.py')
run('python totals.py')
run('kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group totals | cut -c1-80')

# Step 2: Resetting offsets while the consumers still run is refused
run('timeout 60 python hold.py > hold.log 2>&1 & echo $! > hold.pid; sleep 8')
run('kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group totals --topic orders --to-datetime 2026-10-26T09:00:00.000 --execute')

# Step 3: Stop the consumer
run('kill $(cat hold.pid); while kill -0 $(cat hold.pid) 2>/dev/null; do sleep 0.5; done')

# Step 4: Stop the consumers, dry-run the reset to 09:00, then execute
run('kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group totals --topic orders --to-datetime 2026-10-26T09:00:00.000 --dry-run')
run('kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group totals --topic orders --to-datetime 2026-10-26T09:00:00.000 --execute')
run('kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group totals | cut -c1-80')

# Step 5: shift-by on one partition (dry run)
run('kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group totals --topic orders:0 --shift-by -1 --dry-run')

# Step 6: Three partitions: where each customer key lands
run('kafka-topics.sh --bootstrap-server localhost:9092 --create --topic customer-orders --partitions 3')
run('python keys.py')

# Step 7: Six partitions: some keys move
run('kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic customer-orders --partitions 6')
run('python keys.py')

# Step 8: Partitions can only grow
run('kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic customer-orders --partitions 3 2>/dev/null', expect_error=True)   # EXPECTED TO FAIL: the video shows this error

That is the whole series

All 10 parts are listed on the Real Time Kafka Interview Questions and Answers page on this blog.

The sample data on this page is invented for teaching. Every terminal output is the real output of the command above it, run on Apache Kafka 4.4.0 in KRaft mode with confluent-kafka 2.16.0; topic IDs are masked. Part of the Real Time Kafka Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of October 2026; check the official Apache Kafka documentation (kafka.apache.org) for anything you quote in an interview.