One-to-Many Messaging - Publish-Subscribe Pattern | OneNoughtOne

0/319

System Design (HLD)Publish-Subscribe

Publish-Subscribe Pattern

LevelIntermediate

Duration75 mins

TopicPublish-Subscribe

1 / 5

One-to-Many Messaging

The Broadcasting Revolution

Imagine you're building a stock trading platform. When a trade executes, dozens of systems need to know: the real-time dashboard needs to update, the portfolio service needs to recalculate positions, the risk engine needs to reassess exposure, the audit log needs to record the transaction, the notification service needs to alert users, and the analytics pipeline needs to capture the event for reporting.

With point-to-point messaging, your trade execution service would need to know about every one of these consumers. It would need to send separate messages to each, maintain separate connections, and become increasingly fragile as consumers are added or removed. The trade service becomes a bottleneck that knows too much about the rest of the system.

The publish-subscribe pattern (often called pub-sub) fundamentally inverts this architecture. Instead of the producer pushing to specific consumers, the producer broadcasts to a topic, and interested consumers subscribe to receive the events. The producer doesn't know—or care—how many consumers exist, who they are, or what they do with the messages.

What You Will Learn

By the end of this page, you will understand the fundamental mechanics of one-to-many messaging, how pub-sub creates true decoupling between producers and consumers, and why this pattern is essential for building scalable, event-driven distributed systems.

From Point-to-Point to Broadcast

To truly appreciate pub-sub, we must first understand the limitations it addresses. In traditional point-to-point messaging (like message queues), a message is produced by one sender and consumed by exactly one receiver. This creates a fundamental constraint: tight coupling between producers and consumers.

Let's trace the evolution of messaging patterns through a real-world lens:

Evolution of Messaging Paradigms
Pattern	Message Flow	Coupling	Scalability Challenge
Direct API Calls	Producer → Consumer	Tight: Producer knows consumer's address and API	Producer blocked waiting for response; consumer must be available
Message Queue	Producer → Queue → Consumer	Medium: Producer knows queue; queue routes to single consumer	Single consumer bottleneck; adding consumers creates competition
Publish-Subscribe	Producer → Topic → Many Consumers	Loose: Producer only knows topic; any number of consumers subscribe	Independent scaling; consumers added/removed transparently

The Critical Insight

The fundamental shift in pub-sub is inverting the dependency direction. In point-to-point, the producer depends on the consumer (it needs to know where to send messages). In pub-sub, the consumer depends on the producer's events, but the producer has no runtime dependency on consumers.

Consider what happens when you need to add a new consumer in each paradigm:

Point-to-Point: Modify the producer to send to an additional destination. Deploy the producer. Coordinate timing with the consumer.
Pub-Sub: Deploy the new consumer with a subscription to the topic. No changes to the producer. No coordination required.

This asymmetry is why pub-sub enables teams to move independently. The team maintaining the trade execution service doesn't need to coordinate with every downstream team. They publish events; anyone who cares subscribes.

The Newspaper Analogy

Think of traditional messaging like sending personal letters—you need to know each recipient's address and send separate letters. Pub-sub is like publishing a newspaper: you print once, and anyone who subscribes receives a copy. The publisher doesn't need to track subscribers; the subscription mechanism handles it.

Anatomy of Pub-Sub Systems

Every pub-sub system shares common architectural elements, though implementations vary in their specifics. Understanding these components and their interactions is essential for designing effective event-driven architectures.

Core Components of Pub-Sub Architecture

•Publishers (Producers) — Services or applications that emit events. Publishers write messages to topics without knowledge of who (or whether anyone) will receive them. A single publisher can write to multiple topics, and a single topic can receive from multiple publishers.
•Topics (Channels) — Named destinations that organize events by category or domain. Topics serve as the rendezvous point between publishers and subscribers. They may be logical (a routing label) or physical (a persistent log), depending on the implementation.
•Subscriptions — The binding between a subscriber and a topic. Subscriptions define how messages flow to consumers, including delivery semantics, filtering rules, and acknowledgment requirements.
•Subscribers (Consumers) — Services that receive events from topics they subscribe to. Each subscriber independently processes messages at its own pace. Multiple subscribers to the same topic each receive their own copy of every message.
•Message Broker — The infrastructure that manages topics, routes messages, handles subscriptions, ensures delivery, and provides durability. This may be a dedicated message broker (Kafka, RabbitMQ), a cloud service (Google Pub/Sub, AWS SNS), or even a custom implementation.

Converting Mermaid diagram...

Key Observation: Notice how the Trade Service publishes to the trades topic without any knowledge that four different services are subscribed. If the Risk Engine goes down or a new Machine Learning service subscribes tomorrow, the Trade Service remains unchanged. This is true decoupling at the infrastructure level.

The Message Lifecycle in Pub-Sub

Understanding the complete lifecycle of a message is crucial for designing reliable pub-sub systems. Let's trace a message from production to consumption, examining what happens at each stage.

Message Lifecycle Stages

•Production: A publisher constructs a message containing event data (the payload), metadata (headers, timestamps, correlation IDs), and a target topic. The message is serialized using a schema (JSON, Protobuf, Avro) and transmitted to the broker.
•Ingestion: The broker receives the message, validates it (authentication, authorization, schema validation if enabled), and writes it to the topic. At this point, the broker typically returns an acknowledgment to the publisher.
•Persistence (in durable systems): The message is written to stable storage (disk, replicated log). This ensures the message survives broker restarts and can be replayed if needed.
•Fan-Out: The broker identifies all active subscriptions for the topic. Each subscription receives its own reference to the message—this is the 'one-to-many' in action. The message is not copied; it's the same stored message with multiple delivery targets.
•Delivery: For each subscription, the broker delivers the message according to the subscription's semantics (push or pull, in order or out of order, filtered or unfiltered).
•Processing: Each subscriber receives the message, deserializes it, and processes it according to its business logic. Processing happens independently and concurrently across subscribers.
•Acknowledgment: The subscriber confirms successful processing. The broker tracks acknowledgments per subscription—a message isn't 'done' until all subscriptions have acknowledged it (or it expires).
•Retention/Cleanup: After acknowledgment (or expiration), the message follows retention policies. Some systems delete immediately; others retain for replay capability.

Acknowledgment Independence

A critical property of pub-sub is that acknowledgments are per-subscription. If the Portfolio Service acknowledges a trade event but the Risk Engine is still processing, the broker tracks them independently. One subscriber's speed doesn't block another's—this is essential for allowing heterogeneous consumers with different processing speeds.

message-lifecycle-example.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
// Publisher: Trade Execution Service
interface TradeExecutedEvent {
  eventId: string;
  eventType: 'TRADE_EXECUTED';
  timestamp: string;
  correlationId: string;
  payload: {
    tradeId: string;
    symbol: string;
    side: 'BUY' | 'SELL';
    quantity: number;
    price: number;
    accountId: string;
    executedAt: string;
  };
}
 
async function publishTradeEvent(trade: Trade): Promise<void> {
  const event: TradeExecutedEvent = {
    eventId: generateEventId(),
    eventType: 'TRADE_EXECUTED',
    timestamp: new Date().toISOString(),
    correlationId: trade.orderId, // Links to original order
    payload: {
      tradeId: trade.id,
      symbol: trade.symbol,
      side: trade.side,
      quantity: trade.quantity,
      price: trade.executedPrice,
      accountId: trade.accountId,
      executedAt: trade.executedAt,
    },
  };
 
  // Publish to topic - fire and forget from publisher's perspective
  // The broker handles fan-out to all subscribers
  await messageBroker.publish('trades', event, {
    messageKey: trade.accountId, // Partition key for ordering
    headers: {
      'content-type': 'application/json',
      'schema-version': 'v1.2',
    },
  });
  
  // Publisher is done - doesn't wait for any subscriber
  logger.info(`Trade event published: ${trade.id}`);
}
 
// Subscriber: Portfolio Service
async function handleTradeEvent(message: Message): Promise<void> {
  const event = JSON.parse(message.payload) as TradeExecutedEvent;
  
  try {
    // Update portfolio position
    await portfolioService.updatePosition({
      accountId: event.payload.accountId,
      symbol: event.payload.symbol,
      quantityDelta: event.payload.side === 'BUY' 
        ? event.payload.quantity 
        : -event.payload.quantity,
      avgPriceDelta: event.payload.price,
    });
    
    // Acknowledge successful processing
    await message.ack();
    
  } catch (error) {
    // Negative acknowledgment - message will be redelivered
    await message.nack({ requeue: true });
    logger.error(`Failed to process trade ${event.payload.tradeId}`, error);
  }
}

The Fan-Out Guarantee

The defining characteristic of pub-sub is the fan-out guarantee: every subscriber to a topic receives every message published to that topic. This sounds simple, but its implications are profound.

What the Fan-Out Guarantee Means:

Complete Visibility: Each subscriber has a complete view of all events on its subscribed topics. There are no 'lost' messages due to competing consumers.
Independent Processing: Subscribers process messages independently. The Risk Engine can be 100 messages behind the Portfolio Service—both will eventually process all messages.
Replay Capability: In log-based systems (like Kafka), new subscribers can 'start from the beginning' and receive historical messages, achieving the same complete view retroactively.

Contrast with Message Queues:

In a traditional message queue, multiple consumers compete for messages. If three instances of a service consume from a queue, each message goes to exactly one instance. This is load-balancing, not broadcasting.

Pub-Sub: Fan-Out Behavior

•Each message delivered to ALL subscribers
•Subscribers: {A, B, C}; Message M → A gets M, B gets M, C gets M
•Use case: Multiple services need the same event
•Example: Trade event → Dashboard, Risk, Analytics all receive it
•Adding subscriber D: A, B, C, D all get future messages

Queue: Competing Consumer

•Each message delivered to ONE consumer
•Consumers: {A, B, C}; Message M → only A gets M (or B, or C)
•Use case: Distribute workload across workers
•Example: Image processing → one worker handles each image
•Adding consumer D: load spread across A, B, C, D

Combining Patterns: Consumer Groups

Real systems often need BOTH patterns. Apache Kafka introduced 'consumer groups' to solve this: messages fan-out to all consumer groups (pub-sub behavior), but within each group, messages are distributed to one consumer (queue behavior). This lets you have multiple services (each a group) receiving all messages, while each service scales horizontally with competing instances.

Decoupling Dimensions

Pub-sub provides decoupling along multiple dimensions simultaneously. Understanding each dimension helps you appreciate the architectural flexibility this pattern provides.

Dimensions of Decoupling in Pub-Sub
Dimension	Without Pub-Sub	With Pub-Sub	Engineering Benefit
Space (Location)	Producer needs consumer addresses	Producer knows only topic	Consumers can move, scale, or be replaced without producer changes
Time (Synchrony)	Producer waits for consumer response	Producer completes immediately	Producer performance independent of consumer speed; no blocking
Identity (Knowledge)	Producer maintains list of consumers	Producer unaware of subscribers	Teams deploy independently; no coordination overhead
Cardinality (Count)	Producer sends N messages for N consumers	Producer sends 1 message; broker fans out	O(1) producer work regardless of subscriber count
Reliability (Failure)	Producer handles each consumer's failures	Broker handles delivery; retries per subscription	Producer logic simplified; failure isolation per consumer

Space Decoupling is the most visible: producers don't need to know consumer addresses. But time decoupling is equally important—producers complete their work immediately without waiting for downstream processing.

Consider the contrast in a trade execution:

Without time decoupling: Execute trade, call Portfolio (200ms), call Risk (500ms), call Analytics (100ms), call Notification (150ms) = 950ms before returning to user
With time decoupling: Execute trade, publish event (5ms) = 5ms before returning to user. All downstream processing happens asynchronously.

This 190x improvement in response time fundamentally changes what's possible for user-facing systems.

The Organizational Parallel

Decoupling dimensions map to organizational benefits. Space decoupling means teams don't coordinate deploys. Time decoupling means SLAs are independent. Identity decoupling means backlog priorities aren't blocked. This is why pub-sub is foundational to microservices—it enables Conway's Law to work in your favor.

Push vs Pull Delivery Models

Pub-sub systems implement message delivery using two fundamentally different approaches: push and pull. Each has distinct characteristics that affect performance, reliability, and operational complexity.

Push Model: The broker initiates delivery, forwarding messages to subscribers as they arrive.

How it works:

Subscriber registers a callback endpoint (webhook, long-lived connection)
When message arrives, broker immediately pushes to all subscribers
Subscriber processes and acknowledges
Broker tracks acknowledgments, retries on failure

Advantages:

Low latency: subscribers receive messages immediately
Simple subscriber logic: no polling loop needed
Efficient for high-throughput, low-latency requirements

Disadvantages:

Subscriber must handle spike traffic (no backpressure)
Broker must track subscriber availability
Network issues cause immediate failures

Examples: Google Pub/Sub (push mode), AWS SNS → Lambda, WebSockets

Push Handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
// Push model: Broker calls subscriber's endpoint
// Subscriber exposes an HTTP endpoint
 
import express from 'express';
const app = express();
 
app.post('/webhooks/trades', async (req, res) => {
  const event = req.body;
  
  try {
    await processTradeEvent(event);
    res.status(200).send('OK'); // Acknowledge
  } catch (error) {
    res.status(500).send('Retry'); // Broker will retry
  }
});
 
// Or with a push subscription in client code:
const subscription = pubsub.subscription('trades-sub', {
  pushConfig: {
    pushEndpoint: 'https://my-service/webhooks/trades',
  },
});

Message Durability and Persistence

A critical design decision in pub-sub systems is whether messages are ephemeral (delivered and discarded) or persistent (stored for replay). This distinction has profound implications for system design.

Ephemeral (Fire-and-Forget)

•Messages exist only in transit
•If no subscriber online → message lost
•Cannot replay historical events
•Lower storage costs
•Simpler broker implementation
•Example: Real-time notifications, stock ticks where latest matters

Persistent (Log-Based)

•Messages stored in append-only log
•New subscribers can read from any point in history
•Full replay capability for recovery
•Higher storage costs (mitigated by tiering)
•More complex (offset tracking, compaction)
•Example: Event sourcing, audit logs, analytics pipelines

The Log-Based Revolution

Apache Kafka popularized the log-based pub-sub model, treating topics as append-only logs rather than message queues. This seemingly simple change enables powerful capabilities:

Replay: New consumers can read from the beginning, perfect for rebuilding state or adding new analytics
Time-travel: Debug issues by re-processing events from a specific point in time
Exactly-once semantics: Combine with idempotent consumers for reliable processing
Multiple read patterns: Same log supports real-time streaming AND batch processing
Disaster recovery: Replicate the log; recovery means replaying from replica

The trade-off is storage. Kafka logs can grow large, requiring careful retention policies and tiered storage strategies.

Retention Policies

Persistent systems require retention decisions: delete after X days? Delete after X bytes? Keep forever with compaction (keeping only the latest value per key)? The right policy depends on your use case—audit logs need years of history; real-time dashboards need hours.

Scalability Characteristics

Pub-sub enables remarkable scalability, but understanding how it scales helps you design systems that leverage it effectively. Let's examine the scalability characteristics along different dimensions.

Scaling Dimensions

•Publisher Throughput: Scales with broker capacity and partitioning. A single Kafka cluster can handle millions of messages per second by partitioning topics across brokers.
•Subscriber Count: Each additional subscriber to a topic increases fan-out work, but this scales well because subscribers process independently. Thousands of subscribers are feasible with proper infrastructure.
•Consumer Throughput: Within a consumer group, adding instances distributes partitions, scaling processing capacity horizontally. Limited by partition count.
•Topic Count: Moderate scaling—thousands of topics practical, but very high counts (millions) strain broker metadata management.
•Message Size: Small messages (KB) scale better than large (MB) due to network and serialization overhead. For large payloads, store data externally and pass references.

Scalability Limits and Strategies
Dimension	Typical Limit	Scaling Strategy	Warning Sign
Message throughput	1M+ msgs/sec per cluster	Add brokers, increase partitions	Broker CPU saturation, replication lag
Subscriber fan-out	1000+ subscribers per topic	Use hierarchical topics, shard by region	Broker memory pressure, slow fan-out
Message size	1 MB practical limit	Reference pattern: store blob, pass URI	Network congestion, slow serialization
Partition count	100k across cluster (Kafka)	Fewer partitions, more topics if needed	Long leader elections, ZK/controller strain
Consumer groups	1000+ groups per cluster	Optimize offset storage, use compact topics	Slow group rebalancing, offset lag

The Reference Pattern

For large payloads (images, documents, videos), don't send the content through pub-sub. Store it in object storage (S3, GCS), then publish a small event with the object reference. Subscribers fetch the blob directly. This keeps the message bus lean and fast.

Summary: One-to-Many Messaging

We've established the foundation of publish-subscribe messaging and its one-to-many delivery model. Let's consolidate the essential concepts:

Key Takeaways

•Pub-sub inverts dependencies: Producers broadcast to topics; consumers subscribe independently. No producer changes needed when consumers change.
•Fan-out is the defining characteristic: Every subscriber receives every message—independent complete views of the event stream.
•Multiple decoupling dimensions: Space, time, identity, cardinality, and reliability are all decoupled between producers and consumers.
•Push vs pull are implementation choices: Push offers lower latency; pull offers natural backpressure. Modern systems often combine both.
•Durability is a design decision: Ephemeral messaging suits real-time signals; persistent logs enable replay and event sourcing.
•Scalability is multi-dimensional: Message throughput, subscriber count, and consumer parallelism scale independently with proper partitioning.

What's Next:

Now that we understand the fundamental mechanics of one-to-many messaging, we'll dive deeper into topics and subscriptions—the organizational primitives that structure pub-sub systems. We'll explore how to design topic hierarchies, manage subscription lifecycles, and configure delivery semantics for different use cases.

Page Complete

You now understand the foundational concepts of one-to-many messaging in pub-sub systems. This paradigm shift from point-to-point to broadcast messaging is the cornerstone of scalable, event-driven architectures. Next, we'll explore how topics and subscriptions provide the organizational structure for this messaging pattern.