arrow_backBack to Phase 2
mail

Message
Brokers

Decoupling services so they don't crash each other. When Service A needs to tell Service B something, but B is busy/down/slow, you need a buffer.

Why Asynchronous?

Synchronous (REST/HTTP)

User waits. If Service B is down, the request fails. The failure propagates back to the user immediately.

Asynchronous (Broker)

User gets "Accepted". Service A drops message in queue. Service B picks it up whenever it can. No direct dependency.

Key Benefit

Traffic Spikes

If 10,000 users sign up in 1 second, your Email Service will crash if called synchronously.

With a broker, the queue fills up, but the Email Service churns through them at its own steady pace (e.g., 50/sec).The queue acts as a shock absorber.

Queue vs Log (The Big Split)

This is the most important distinction in modern messaging.

queue

Message Queue

RabbitMQ, SQS, ActiveMQ

  • • Ephemeral: Message is deleted once consumed.
  • • Smart Broker: Tracks who read what.
  • • Load Balancing: Multiple consumers share the work (Competing Consumers).
"I just want this job done by someone, once."
receipt_long

Log-Based

Kafka, Kinesis

  • • Persistent: Messages stay on disk (retention period).
  • • Dumb Broker: Consumer tracks its own offset.
  • • Fan-out: Multiple consumers can read the same history independently.
"I want a record of everything that happened."

RabbitMQ (The Mailman)

RabbitMQ is complex but flexible. It uses Exchanges to route messages to Queues.

  • Direct Exchange: Routing key matches exactly. (info → info queue)
  • Fanout Exchange: Broadcast to all queues. (1 publisher → N queues)
  • Topic Exchange: Wildcard matching. (log.* → log.error, log.info)

Best for: Complex routing, job queues, task processing.

Exchange
arrow_downwardarrow_downward
Queue A
Queue B

Kafka (The Journal)

Partitions & Offsets

Kafka stores data in Topics, which are split into Partitions. A partition is just an append-only file on disk.

Ordering Guarantee: Order is only guaranteed within a partition, not across the whole topic.

Consumer Group:

If you want to parallelize processing, you add more consumers to a group. Kafka assigns each partition to one consumer. Max concurrency = Number of Partitions.

Partition 0:
Msg 1Msg 2Msg 3...
Partition 1:
Msg AMsg B...
"Sequential writes make Kafka insanely fast."

Push vs Pull

Push Model

RabbitMQ (Default)

Broker pushes messages to consumer as fast as possible.

  • Pro: Low latency.
  • Con: Can overwhelm slow consumers (needs backpressure/prefetch limits).

Pull Model

Kafka, SQS

Consumer asks for messages when it's ready ("Give me next 10").

  • Pro: Consumer controls the rate. Easy batching.
  • Con: Might have empty polling loops (latency).