Loading system design…
Loading system design...
Design a web search engine like Google that crawls the entire internet, builds an inverted index of hundreds of billions of pages, and returns the most relevant results for any text query in under 500ms — at a scale of 8.5 billion searches per day. This encompasses web crawling, indexing, query understanding, multi-signal ranking (BM25 + PageRank + ML), snippet generation, and serving with massive scatter-gather fan-out.
| Metric | Value |
|---|---|
| Indexed web pages | 100+ billion |
| Queries per day | 8.5 billion |
| Peak queries per second | 200,000 |
| Index size (compressed) | ~100 petabytes |
| Index shards | 10,000+ |
| Index shard replicas | 3× per data centre |
| Data centres | 20+ globally |
| Crawler pages fetched per day | 1+ billion |
| Unique terms in index | 1+ trillion |
| Average query latency | < 500ms |
| Result cache hit rate | ~30% (head queries) |
Web search: given a text query, return a ranked list of the most relevant web pages from a corpus of hundreds of billions of indexed pages, in < 500ms
Web crawling: continuously discover and fetch web pages across the entire internet; handle billions of pages; respect robots.txt; prioritise important/fresh pages; detect and avoid crawler traps
Indexing: parse crawled HTML, extract text/metadata, build an inverted index mapping every word to the list of documents containing it; support phrase queries, boolean operators, and field-specific search (title, URL)
Ranking: rank search results by relevance using hundreds of signals — link analysis (PageRank), content relevance (TF-IDF / BM25), user engagement (click-through rate), freshness, page quality, mobile-friendliness, and more
Query understanding: interpret user intent — spell correction ('did you mean'), synonym expansion, entity recognition ('Apple' = company vs fruit based on context), query rewriting, and knowledge graph lookups
Snippet generation: for each result, display a title, URL, and a contextual snippet (excerpt from the page highlighting the query terms); featured snippets for direct-answer queries
Autocomplete: as the user types, suggest completions based on popular queries, personalised history, and trending topics (refer H0013 — Search Autocomplete)
Image / video / news search: support vertical search across different content types; image search with visual similarity; news search with recency ranking; video search with thumbnail previews
Knowledge panel: for entity queries (people, places, companies), display a structured knowledge card with summary, facts, images, and related entities sourced from the knowledge graph
Personalisation and localisation: personalise results based on user location, language, search history, and device; localise results (local businesses, maps, currency) based on geo-IP
Non-functional requirements define the system qualities critical to your users. Frame them as 'The system should be able to...' statements. These will guide your deep dives later.
Think about CAP theorem trade-offs, scalability limits, latency targets, durability guarantees, security requirements, fault tolerance, and compliance needs.
Frame NFRs for this specific system. 'Low latency search under 100ms' is far more valuable than just 'low latency'.
Add concrete numbers: 'P99 response time < 500ms', '99.9% availability', '10M DAU'. This drives architectural decisions.
Choose the 3-5 most critical NFRs. Every system should be 'scalable', but what makes THIS system's scaling uniquely challenging?
The complete explanation is included with Career and All Access. Compare it with your approach once you’ve worked through the problem.
Your system-design submission history is available after you sign in.
Sign In