Picture an e-commerce checkout. The moment a customer clicks "Place Order," a dozen things need to happen: charge the card, update inventory, notify the warehouse, send a confirmation email, log the event for analytics. If your order service calls each of those systems directly and waits for every response, you've built a house of cards — one slow email provider and the whole checkout hangs.
This is the problem message brokers exist to solve.
What a Message Broker Actually Is
A message broker is a piece of infrastructure that sits between services and handles the passing of messages between them. Instead of Service A calling Service B directly, Service A drops a message onto the broker, and Service B picks it up whenever it's ready. Neither service needs to know the other exists, be online at the same time, or respond within any particular window.
That decoupling is the entire point. Producers (services that send messages) and consumers (services that read them) are freed from timing, location, and availability constraints. If the email service is down for maintenance, order confirmations simply queue up and get sent once it's back — the checkout flow itself never notices.
Core Concepts
A few pieces of vocabulary show up in every broker, regardless of vendor:
- Producer — the service that publishes a message.
- Consumer — the service that reads and processes a message.
- Queue — a buffer holding messages until a consumer is ready for them. Once read (and acknowledged), a message is typically removed.
- Topic / Exchange — a named channel that messages get published to, which can then be routed to one or many queues based on rules.
- Acknowledgment (ack) — a consumer's signal to the broker that a message was successfully processed, so it can be safely removed from the queue.
Messaging Patterns
Two patterns cover most real-world usage:
Point-to-point (work queues). One producer, one message, exactly one consumer processes it — even if there are multiple consumers listening. This is how you'd distribute a batch of image-resizing jobs across a pool of workers: each job goes to exactly one worker, and adding more workers increases throughput.
Publish/subscribe (pub/sub). One message, many consumers. When an order is placed, the "order placed" event might need to reach the inventory service, the email service, and the analytics pipeline simultaneously. Each subscriber gets its own copy and processes it independently, on its own schedule.
Delivery Guarantees
Brokers differ in the promises they make about message delivery, and this matters a lot for what you build on top of them:
- At-most-once — a message is delivered zero or one times. Fast, but messages can be lost if something fails mid-delivery.
- At-least-once — a message is delivered one or more times, guaranteed to arrive, but consumers might see duplicates. This is the most common default, which means your consumers need to be idempotent (safe to process the same message twice without side effects like double-charging a card).
- Exactly-once — the message arrives once and only once. It's the ideal, but it's expensive to implement correctly and usually only offered in specific contexts (like Kafka transactions within a single cluster).
Popular Brokers, and When to Reach for Each
- RabbitMQ — a traditional message queue implementing AMQP. Great for complex routing logic, task queues, and request/reply patterns. Battle-tested and easy to reason about.
- Apache Kafka — technically a distributed log rather than a classic queue. Built for high-throughput event streaming, message replay, and cases where multiple consumers need to read the same data independently at their own pace.
- Amazon SQS / SNS — fully managed queue and pub/sub services. Low operational overhead, a natural fit if you're already on AWS and don't want to run broker infrastructure yourself.
- Redis Streams — lightweight, in-memory, good for lower-durability use cases where speed matters more than guaranteed persistence.
When You Don't Need One
Message brokers add real value, but they also add a moving part you now have to operate, monitor, and reason about. If two services can simply call each other synchronously and the coupling isn't a problem — a request that must complete before responding to the user, for instance — a broker just adds latency and complexity for no benefit. Reach for one when you have genuine asynchronous work, need to smooth out traffic spikes, or want to decouple services that shouldn't need to know about each other's uptime.
The Takeaway
A message broker's job is to buy your system slack: slack in time (producers and consumers don't have to be online together), slack in load (a burst of requests gets smoothed into a steady stream of processing), and slack in coupling (services can change independently). Understanding queues vs. topics, and at-least-once vs. exactly-once, is what lets you pick the right tool instead of just the familiar one.