message queue
**Message Queues** are **asynchronous communication middleware that decouple producers (senders) from consumers (receivers) using persistent or transient queues** — enabling parallel processing, load leveling, and fault tolerance by allowing components to operate at different speeds without blocking each other, forming the backbone of distributed system architectures.
**Core Concepts**
- **Producer**: Sends messages to the queue.
- **Queue/Topic**: Buffer that stores messages until consumed.
- **Consumer**: Reads and processes messages from the queue.
- **Broker**: Server that manages queues and routes messages.
**Message Queue vs. Direct Communication**
| Aspect | Direct (RPC/HTTP) | Message Queue |
|--------|-------------------|---------------|
| Coupling | Tight (caller waits) | Loose (fire and forget) |
| Failure handling | Caller must retry | Queue retains message |
| Speed mismatch | Caller blocked by slow receiver | Queue buffers overflow |
| Scalability | 1:1 or load balanced | 1:N fan-out, N:1 fan-in |
**Popular Systems**
| System | Type | Throughput | Latency | Persistence |
|--------|------|-----------|---------|------------|
| Apache Kafka | Distributed log | Millions msg/sec | 2-10 ms | Persistent (disk) |
| RabbitMQ | Traditional broker | 100K msg/sec | < 1 ms | Optional |
| Redis Streams | In-memory log | Millions msg/sec | < 0.5 ms | AOF/RDB |
| Amazon SQS | Managed queue | Unlimited (scaled) | 1-10 ms | Persistent |
| ZeroMQ | Brokerless library | Millions msg/sec | < 0.1 ms | None |
| NATS | Cloud-native | Millions msg/sec | < 1 ms | JetStream |
**Patterns for Parallel Processing**
**Work Queue (Competing Consumers)**
- Multiple consumers pull from same queue → parallel processing.
- Load automatically balanced — faster consumers process more messages.
- Example: 100 image resize tasks queued → 10 workers process in parallel.
**Fan-Out (Pub/Sub)**
- Producer publishes to topic → all subscribers receive a copy.
- Example: New user signup → email service, analytics service, CRM all notified.
**Request-Reply**
- Producer sends request with reply-to queue → consumer sends result to reply queue.
- Enables async RPC with queue-based routing.
**Delivery Guarantees**
| Level | Meaning | Implementation |
|-------|---------|---------------|
| At-most-once | May lose messages | Fire and forget |
| At-least-once | May duplicate messages | Ack + retry |
| Exactly-once | No loss, no duplicates | Transactional (Kafka) |
Message queues are **essential infrastructure for building reliable, scalable distributed systems** — by decoupling components and buffering communication, they enable parallel processing at scale while providing fault tolerance that synchronous communication cannot offer.