High-Level Design: Architecture, Services, and Data Flow - High-Level Design vs Low-Level Design | OneNoughtOne

0/319

System Design (HLD)HLD vs LLD — Understanding the Difference

High-Level Design vs Low-Level Design

LevelBeginner

Duration60 mins

TopicHLD vs LLD — Understanding the Difference

1 / 4

High-Level Design: Architecture, Services, and Data Flow

Thinking Beyond Code

When engineers first transition from writing code to designing systems, they encounter a profound shift in perspective. As a developer, your primary concern is how to implement a feature—which algorithm to use, how to structure classes, what data types to choose. As a system designer, your primary concern becomes what components to build and how they interact—where data lives, how services communicate, and what happens when things fail.

This shift from implementation thinking to architectural thinking is the essence of High-Level Design (HLD). It's the difference between asking "How do I write this function?" and asking "How do I structure this system so it can serve millions of users reliably?"

What You Will Learn

By the end of this page, you will understand what High-Level Design encompasses, why architectural decisions matter more than implementation details at this level, and how to think about systems in terms of components, services, and data flows. You'll gain the mental model that separates senior engineers from junior developers.

What Is High-Level Design?

High-Level Design (HLD) is the process of defining the overall architecture of a system. It answers the question: "What are the major building blocks of this system, and how do they work together?"

At this level, you're not concerned with the implementation details of any single component. Instead, you're focused on:

What components exist — Databases, services, caches, message queues, load balancers
How they interact — Synchronous APIs, asynchronous messaging, event-driven communication
Where data lives — Which databases store what data, how data flows between services
How the system scales — Horizontal vs. vertical scaling, partitioning strategies
How failures are handled — Redundancy, failover, graceful degradation

HLD is fundamentally about abstraction. You abstract away the implementation details to focus on the structure and behavior of the system as a whole.

The 30,000-Foot View

Think of HLD as looking at a city from an airplane. You can see the major districts, the highways connecting them, the river flowing through, and the airport on the outskirts. You can't see individual houses or streets—but you understand how the city is organized and how people move through it.

HLD is not about perfection—it's about trade-offs.

Every architectural decision involves trade-offs. Choosing a microservices architecture gives you independent deployability but increases operational complexity. Using a SQL database gives you ACID transactions but may limit horizontal scaling. Implementing a cache improves read performance but introduces consistency challenges.

A skilled system designer understands these trade-offs and makes informed decisions based on the specific requirements and constraints of the system.

The Three Pillars of High-Level Design

High-Level Design rests on three fundamental pillars: Architecture, Services, and Data Flow. Understanding each pillar is essential to mastering HLD.

The Three Pillars

•Architecture — The structural skeleton of the system. What are the major components? How are they organized? What constraints govern their relationships?
•Services — The functional units that do the work. What responsibilities does each service have? How are they deployed and scaled?
•Data Flow — The movement of information through the system. How does data enter? How is it transformed? Where does it ultimately reside?

Let's examine each pillar in depth.

Architecture: The Structural Skeleton

Architecture defines the fundamental organization of a system—the components, their relationships, and the principles governing their design and evolution.

When we talk about system architecture, we're answering questions like:

Is this a monolithic application or a distributed system?
Is it client-server or peer-to-peer?
Is it layered (presentation, business logic, data) or more complex?
What are the boundaries between components?
What are the communication patterns?

Common Architectural Patterns in HLD
Pattern	Description	When to Use	Trade-offs
Monolithic	Single deployable unit containing all functionality	Early-stage products, small teams, simple domains	Easy to develop initially; hard to scale and maintain as it grows
Microservices	Independent services with focused responsibilities	Large teams, complex domains, need for independent scaling	Operational complexity; requires mature DevOps practices
Event-Driven	Components communicate through asynchronous events	High-throughput systems, loose coupling needs	Eventual consistency; debugging complexity
Serverless	Functions as a Service (FaaS) with managed infrastructure	Variable workloads, rapid development, cost optimization	Cold starts; execution time limits; vendor lock-in
CQRS	Separate read and write models	High-read systems; complex domains; event sourcing	Increased complexity; eventual consistency between models

Architecture is about constraints.

A well-defined architecture establishes constraints that guide subsequent design decisions. For example:

A layered architecture constrains how layers can communicate (typically only adjacent layers)
A microservices architecture constrains services to own their data (no shared databases)
An event-driven architecture constrains communication to be asynchronous

These constraints aren't limitations—they're guardrails that prevent chaos as the system grows. They ensure that the system remains maintainable, scalable, and evolvable.

Architecture vs. Design

Architecture is about the decisions that are hard to change. If you can easily modify something later, it's a design decision, not an architectural one. The choice between SQL and NoSQL is architectural; the choice of primary key type is design. The distinction helps prioritize what to get right early.

Services: The Functional Units

Services are the building blocks that perform the actual work in a system. A service is a cohesive unit of functionality that exposes a well-defined interface and manages its own state.

In HLD, we define services by answering:

What does this service do? — Its responsibilities and boundaries
What does it need? — Dependencies on other services and data
What does it expose? — Its API and contracts
How is it deployed? — Independently or as part of a larger unit
How does it scale? — Horizontally, vertically, or both

Service Boundaries: The Art of Decomposition

Defining service boundaries is one of the most critical—and challenging—aspects of HLD. Poor boundaries lead to:

Tight coupling — Services that can't be modified independently
Data duplication — The same data managed in multiple places
Distributed monolith — The complexity of microservices without the benefits
Chatty communication — Services that constantly need to call each other

Principles for Service Decomposition

•Single Responsibility — Each service should do one thing well. If you describe a service with 'and' (e.g., 'handles users and payments'), consider splitting it.
•Domain-Driven Design — Align services with business domains (bounded contexts). A 'User Service' makes sense; a 'Utils Service' doesn't.
•Data Ownership — Each service owns its data exclusively. No shared databases. If two services need the same data, one owns it and exposes an API.
•Independent Deployability — Services should be deployable without coordinating with other teams. If deployment requires synchronization, boundaries are wrong.
•Failure Isolation — A failure in one service shouldn't cascade to others. Services should be designed to handle downstream failures gracefully.

The Distributed Monolith Anti-Pattern

If all your 'microservices' must be deployed together, if a change in one requires changes in many others, if they share a database—you've built a distributed monolith. You have all the complexity of distributed systems with none of the benefits. This is worse than a monolith.

Data Flow: The Movement of Information

Data Flow describes how information moves through the system—from its origin (user input, external systems, scheduled jobs) through processing and transformation to its final destination (storage, response, external systems).

Understanding data flow is essential because:

It reveals bottlenecks — Where does data accumulate? Where is processing slow?
It exposes dependencies — Which services must be available for a request to complete?
It identifies consistency needs — Where must data be synchronized? Where can it be eventually consistent?
It highlights failure modes — What happens when a step fails? How does the system recover?

Synchronous vs. Asynchronous Data Flow

One of the most fundamental decisions in HLD is whether data flows synchronously or asynchronously between components.

Synchronous Flow

•Request/response pattern
•Caller waits for result
•Strong consistency guarantees
•Simpler mental model
•Latency adds up across calls
•Availability couples with dependencies
•Example: REST API calls, gRPC

Asynchronous Flow

•Fire-and-forget or event-driven
•Caller continues immediately
•Eventual consistency
•Complex error handling
•Decouples sender from receiver
•Higher throughput potential
•Example: Message queues, event streams

Data Flow Patterns

Common patterns for organizing data flow include:

Request-Response — Client sends request, server returns response. Simple but creates synchronous coupling.
Publish-Subscribe — Publishers emit events; subscribers react. Decoupled but requires event bus infrastructure.
Streaming — Continuous flow of data records. Ideal for real-time processing but requires stream handling expertise.
Batch Processing — Large volumes processed periodically. High throughput but introduces latency.
Saga Pattern — Distributed transactions across services via compensating actions. Complex but enables consistency without distributed locking.

Follow the Data

When designing a system, trace the path of data from entry to exit. Draw it out. Ask: Where does it enter? Who touches it? How is it transformed? Where is it stored? How is it retrieved? This exercise reveals more about your system than any amount of abstract discussion.

Putting It Together: A Practical Example

Let's apply these concepts to a concrete example: designing a URL shortening service like bit.ly.

Requirements:

Users submit long URLs and receive short URLs
Short URLs redirect to original long URLs
Analytics: track click counts
High availability and low latency

Architecture Decision:

We'll use a distributed architecture with:

API Gateway — Entry point for all requests
URL Service — Handles URL creation and resolution
Analytics Service — Tracks and reports click data
Distributed Cache — For fast URL lookups
NoSQL Database — For URL storage
Message Queue — For async analytics processing

Service Definitions:

Service	Responsibility	Data Owned	Exposed API
URL Service	Create short URLs, resolve to long URLs	URL mappings	POST /shorten, GET /{shortCode}
Analytics Service	Record clicks, generate reports	Click events, aggregations	POST /click (internal), GET /stats

Data Flow:

Create Short URL:
- User → API Gateway → URL Service → Generate short code → Store in DB → Cache → Return short URL
Redirect:
- User → API Gateway → URL Service → Check Cache → (miss: check DB) → Return redirect → Async: emit click event → Message Queue → Analytics Service → Update stats

Notice What's Missing

In this HLD, we haven't specified: which programming language, which exact database product, how the hash algorithm works, what the database schema looks like, or how the cache eviction policy works. Those are LLD concerns. HLD focuses on the components and their interactions.

Common HLD Artifacts

HLD produces several artifacts that communicate the design to stakeholders. Understanding these artifacts is crucial for effective communication.

Key HLD Artifacts

•System Context Diagram — Shows the system as a box, its users, and external systems it interacts with. The 30,000-foot view.
•Container Diagram — Shows the major containers (applications, databases, etc.) and how they communicate. Think: what would you deploy?
•Component Diagram — Zooms into a container to show its major components. Bridge between HLD and LLD.
•Sequence Diagrams — Show how components interact over time for specific scenarios. Essential for complex flows.
•Data Flow Diagrams — Show how data moves through the system. Crucial for understanding data lifecycle.
•Non-Functional Requirements Document — Captures latency, throughput, availability, and other quality attributes that shape the architecture.

C4 Model Recommended

The C4 model (Context, Containers, Components, Code) provides a hierarchical way to visualize architecture at different levels of abstraction. It's widely adopted and provides a common vocabulary for architectural diagrams.

Summary: The Essence of High-Level Design

We've covered the foundational concepts of High-Level Design. Let's consolidate the key takeaways:

Key Takeaways

•HLD is about structure, not implementation — Focus on components, their responsibilities, and their interactions—not on code.
•Architecture establishes constraints — Good architectural decisions create guardrails that prevent chaos as the system grows.
•Services are cohesive functional units — Each service has clear responsibilities, owns its data, and can be deployed independently.
•Data flow reveals system behavior — Tracing data from entry to exit exposes bottlenecks, dependencies, and failure modes.
•HLD is about trade-offs — Every decision has pros and cons. The goal is to make informed choices based on requirements.
•Abstraction is the key skill — The ability to think at the right level of abstraction, ignoring irrelevant details, is what makes HLD possible.

What's Next:

Now that we understand High-Level Design, we'll explore its counterpart: Low-Level Design. While HLD focuses on the forest, LLD zooms into individual trees—classes, interfaces, methods, and the implementation details that bring architectural decisions to life.

Page Complete

You now understand what High-Level Design encompasses—architecture as structural skeleton, services as functional units, and data flow as the movement of information. Next, we'll dive into Low-Level Design to understand the complementary perspective of implementation details.