← Systems Analysis: APIs, Integrations and Data
Lesson
Message brokers: queue, topic, Kafka and RabbitMQ
Design an exchange through a broker: choose a queue or a topic, a partition key that keeps events in order, and the contents of the event, bearing in mind that at-least-once delivery produces duplicates.
An intermediary between systems
An intermediary between systems
A broker is an intermediary: the sender (producer) puts a message in, and the receiver (consumer) takes it when ready. Two models (Enterprise Integration Patterns):
• queue, point-to-point — each message goes to one of the consumers; several consumers share the load;
• topic, publish-subscribe — each message goes to every subscribed group.
RabbitMQ: a message goes to an exchange, which routes it to queues by rules (bindings). The consumer acknowledges processing (ack), and the message is deleted. If the connection drops before the ack, the message returns to the queue and comes again.
Kafka: a topic is a log of partitions, shared among the consumers of a group. Messages aren't deleted after reading but kept for a set period (retention); each group remembers its position (offset) and can re-read. Order is guaranteed only within a partition. Messages with the same key go to one partition, so key = orderId keeps an order's events in order.
The usual delivery guarantee is at-least-once: a message won't be lost but may arrive twice. Kafka does exactly-once only internally (table below). So every event has a unique eventId, by which the consumer discards repeats.
An event is a past-tense fact (OrderPaid): the sender doesn't know who listens. A command is a request to a specific receiver (SendSms). Event contracts are described in AsyncAPI, the OpenAPI counterpart for brokers.
RabbitMQ and Kafka
| Property | RabbitMQ | Kafka |
|---|---|---|
| model | queues, routing through an exchange | a topic as a log of partitions |
| after processing | deleted after ack | kept for a set period, can be re-read |
| order | within a queue, as long as nothing is redelivered | only within a partition |
| who gets what | a queue splits messages among consumers | each group gets all messages |
| typical at Romashka | SMS and push jobs inside the notification service | order events for notifications, couriers and analytics |
Delivery guarantees
| Guarantee | What it means |
|---|---|
| at-most-once, “no more than once” | may be lost, but never repeated |
| at-least-once, “one or more times” | won't be lost, but may arrive twice; the usual mode of both Kafka and RabbitMQ with acknowledgments |
| exactly-once, “once and only once” | Kafka can do it internally (transactions); at the boundary with an external system — an SMS gateway, a payment service, someone else's database — it is only at-least-once plus a consumer that discards duplicates |
What an event consists of
| Field | Why | Example |
|---|---|---|
| eventId | to recognize and discard a repeat | e-7f3a… |
| type | what happened | OrderPaid |
| occurredAt | when it happened | 2026-10-19T14:32:00+03:00 |
| orderId | which order; the message key | 9001 |
| schemaVersion | to evolve the format without breaking anything | 1 |
| data | what the consumers need | totalAmount 17980 |
Lesson notes
An intermediary between systems
A broker is an intermediary: the sender (producer) puts a message in, and the receiver (consumer) takes it when ready. Two models (Enterprise Integration Patterns):
• queue, point-to-point — each message goes to one of the consumers; several consumers share the load;
• topic, publish-subscribe — each message goes to every subscribed group.
RabbitMQ: a message goes to an exchange, which routes it to queues by rules (bindings). The consumer acknowledges processing (ack), and the message is deleted. If the connection drops before the ack, the message returns to the queue and comes again.
Kafka: a topic is a log of partitions, shared among the consumers of a group. Messages aren't deleted after reading but kept for a set period (retention); each group remembers its position (offset) and can re-read. Order is guaranteed only within a partition. Messages with the same key go to one partition, so key = orderId keeps an order's events in order.
The usual delivery guarantee is at-least-once: a message won't be lost but may arrive twice. Kafka does exactly-once only internally (table below). So every event has a unique eventId, by which the consumer discards repeats.
An event is a past-tense fact (OrderPaid): the sender doesn't know who listens. A command is a request to a specific receiver (SendSms). Event contracts are described in AsyncAPI, the OpenAPI counterpart for brokers.