Scale From Zero to Millions of Users

The question

You have a web app on one machine. It works. Now it needs to serve millions of people. Walk me through how the system grows, one change at a time, and tell me at each step what broke and why the change fixes it.

This is chapter one of Alex Xu’s System Design Interview book, and it is the question I secretly ask inside every other question. Nobody says “scale from zero to millions” out loud any more. They say “design Instagram” and then watch whether you climb this ladder in the right order.

Explain it to a ten-year-old

You open a lemonade stand in your front garden. One table, one jug, one notebook to write down who paid. On day one, three neighbours come and it is easy.

Then the whole street comes. The queue is long, so you ask a friend to run a second table, and your mum stands at the gate pointing people to whichever table is free. That is the load balancer. The notebook is now the slow part, because both tables keep reaching for it, so you give the notebook its own helper who does nothing but write things down. That is the database. People keep asking the same question, “how much is a cup?”, so you write it on a sign so nobody has to ask. That is the cache. Kids from the next town want lemonade too, so you open a stand there with a photocopy of the price list. That is the CDN and the second data centre. And when the notebook itself gets too fat to flip through, you split it into two notebooks, A to M and N to Z. That is sharding.

Nobody plans a lemonade empire on day one. You add the next helper when the queue tells you to. Watch the ladder grow:

The trick

Never add a box because the book has it. Add a box because you can name the thing that broke. The order is not arbitrary either. Each rung fixes the bottleneck the previous rung exposed:

• One machine breaks because web and database fight over the same CPU and disk. So split them.
• One web server breaks because it is a single point of failure and has a ceiling. So add a load balancer and a second one. That is horizontal scaling, and it beats vertical scaling because a bigger machine still dies alone and there is always a biggest machine.
• The database breaks next, on reads first, because reads outnumber writes. So add replicas, then a cache, then a CDN. Three flavours of the same idea: answer the question closer to the person asking.
• Then you cannot add web servers freely because sessions live on them. So make the web tier stateless and put sessions in a shared store.
• Slow work, image resizing, emails, clogs the request path. So put it on a queue with workers.
• One building breaks when the building breaks. So run a second data centre and route with geoDNS.
• Finally the database breaks on writes and on size, and no replica helps with that. So shard.

If you can say the “because” for each rung, you understand the ladder. If you can only recite the rungs, an interviewer will find out in one follow-up.

The steps

Run it as the 45-minute plan. This question is unusual because the interviewer usually wants the whole ladder, not one deep dive, so keep each rung to two or three sentences and save the depth for whichever rung they poke.

1. Requirements. Ask what “millions” means: total accounts or daily active users, and reads versus writes. Ten million accounts with a hundred thousand daily readers is a small system. A million people writing at once is not. Ask whether it is global. That decides whether rungs six and nine matter.
2. One machine. DNS gives the browser an IP, the browser asks the web server, the server reads its own database and returns HTML or JSON. Say it in one breath. It shows you know what a request is.
3. Split the database out. Web tier and data tier on separate machines so each scales on its own. Pick relational unless you have a reason: unstructured data, very low latency, or a data volume one relational box cannot hold. “NoSQL because it scales” is not a reason. Say which one and why.
4. Load balancer plus more web servers. Users hit the balancer’s public IP. Web servers sit on private IPs behind it. If one dies, traffic moves. If load grows, add a server. This is the load balancer lesson in one paragraph.
5. Replicate the database. One primary takes writes. Replicas take reads. If a replica dies, reads go to the others. If the primary dies, a replica is promoted, and be honest that promotion is the messy bit: data still in flight on the old primary may be lost, and something has to pick the new primary.
6. Add a cache. Cache-aside, sometimes called read-through: check the cache, on a miss read the database and fill the cache. Use it for data read often and changed rarely. Set an expiry that is neither seconds nor days. Evict with LRU unless you can say why not. Spread cache nodes across machines so one dying does not stampede the database.
7. Add a CDN. Static files, images, CSS, JavaScript, served from a node near the user. The book’s numbers are 30 ms from the CDN against 120 ms from the origin. You pay per byte, so set the TTL with care, version files to invalidate them, and have a fallback when the CDN is down.
8. Make the web tier stateless. Sessions go into a shared store, Redis or a database. Now any server can take any request, sticky sessions go away, and autoscaling is just adding a server to the pool. This is the rung candidates most often skip and the one that makes everything after it possible.
9. Queue the slow work. Producers put messages on a queue, workers take them off. The web server answers “got it” in milliseconds and the resize happens later. Producers and workers scale independently, and if the workers are down the queue simply gets longer.
10. Second data centre. geoDNS sends each user to the nearest one. If one goes dark, everyone goes to the other. The hard problems are data: replicate asynchronously between sites, decide what happens to writes made in the dead site, and deploy the same thing to both.
11. Shard the database. Split rows across databases by a key, user_id % 4 in the book. Choose a key that spreads evenly and matches the queries. Then name the three problems before the interviewer does: resharding when a shard fills or gets hot, the celebrity problem when one key gets all the traffic, and joins across shards, which you solve by denormalising.
12. Logging, metrics, automation. Not a rung, a floor under every rung. Host metrics, aggregated metrics, business metrics. Errors in one place. Deploys by a machine, not a person.

The template

Memorise the ladder as a picture, and memorise the one word that pushes you up each rung.

  WHAT HURTS                 WHAT YOU ADD
  ─────────────────────      ─────────────────────────────────────
  CPU shared by web+db  ──►  separate DATABASE machine
  one web box           ──►  LOAD BALANCER + N web servers
  db reads              ──►  READ REPLICAS
  hot reads, still slow ──►  CACHE (cache-aside, TTL, LRU)
  static files, far away──►  CDN
  cannot add servers    ──►  STATELESS web tier, SESSION STORE
  slow work in request  ──►  QUEUE + WORKERS
  one building          ──►  SECOND DATA CENTRE + geoDNS
  db writes, db size    ──►  SHARDS by KEY
  cannot see anything   ──►  LOGS, METRICS, AUTOMATION

What changes from problem to problem is where the pain shows up first and how far up the ladder you need to go. Design Pastebin and you stop at the cache. Design a chat app and you climb to the second data centre on the first day, because the requirement says global. Design a payments ledger and you fight to stay off the sharding rung as long as you can, because sharded transactions are misery.

What must be understood, not memorised, is the left column. In an interview I will give you a number and ask where it hurts. The candidates who have only learned the right column reach for a box. The ones who learned the left column reach for a bottleneck, and then the box follows.

In GPU infrastructure

An inference service climbs exactly this ladder, with a GPU box where the web server was. One node with the model on it, then a load balancer across replicas whose health check is a tiny forward pass rather than a TCP ping, then a cache that is a prefix cache of already-computed KV blocks instead of Redis rows. The CDN rung becomes a regional mirror of model weights so a new node pulls a hundred gigabytes from next door rather than across the country. The queue rung is the batch scheduler in front of the GPUs, and the second data centre is a second region with its own copy of the fleet. Sharding is the odd one out: on the web the database outgrows one machine, and in AI the model outgrows one GPU, so the shard key becomes a tensor dimension and the join problem becomes an all-reduce. The floor under all of it, logs, metrics and automation, is the health-check pipeline that tells me a rack is sick before a customer does.

What I am listening for

• Whether you name the bottleneck before you name the box. “The database is getting hammered by reads, so replicas” beats “then we add replicas”.
• Whether you ask what “millions” means before you draw anything.
• Whether you put stateless before autoscaling. If you autoscale a stateful web tier, I ask where the sessions went.
• Whether you know that replicas fix reads and sharding fixes writes, and never mix them up.
• The follow-ups, which is where the round is actually decided:
  • Ten times more users overnight. What breaks first? The answer depends on your design and you should know it.
  • The primary dies. Who promotes the replica, how long does it take, and what happened to the last second of writes?
  • One user has fifty million followers. The celebrity problem. Their shard and their cache key are on fire. What now?
  • The cache cluster restarts cold. Every request is a miss at once. Does the database survive? Say “warm it up gradually” and “request coalescing” and you are fine.
  • A whole data centre goes dark. Reads are easy. What about the writes made there in the last minute?
  • Shard three is full and shard one is empty. How do you reshard without downtime? Consistent hashing is the answer, and it is its own lesson.

Where this question shows up

You will rarely be asked it by name. You will be asked one of its costumes:

• Design Instagram, Twitter, or a news feed. The ladder plus a fan-out decision. Hello Interview and Design Gurus both frame it this way.
• Design Pastebin or a URL shortener. The bottom half of the ladder. My URL shortener lesson stops at the cache on purpose.
• “How would you handle a million concurrent users?” Usually a bare version of rungs three to eight, common in screening rounds. EZ Tech Learn’s walkthrough is a typical example of the expected answer.
• “Your app just went viral, what do you do this weekend?” The same question, with the added constraint that you cannot rewrite anything. Cache, CDN and read replicas, in that order, because they need no code changes.
• The scaling follow-up inside any other design. “Now make it ten times bigger.” Design Gurus’ piece on follow-ups and a DEV Community post on pre-empting them are both honest about this being where the grade is set.

Go deeper

• Scale From Zero to Millions of Users on ByteByteGo is the chapter these notes follow. Read it once, then try to redraw the final diagram from memory.
• Noah Tigner’s chapter notes and CodeJeet’s summary are two shorter retellings if you want to compare against mine.
• The Excited Engineer on scaling to a million users adds the numbers at each rung.
• Amazon Builders’ Library for what each rung looks like when it is real and on fire, especially the pieces on caching and avoiding fallback.
• The follow-ups above each have a lesson here: load balancer, distributed cache, message queue, multi-region service, consistent hashing.

With AI on the table. An assistant draws the final diagram, all ten rungs, in a second, and it draws them for every problem whether they are needed or not. So I hand it a Pastebin clone with a thousand users and the assistant’s full diagram, and ask you to delete boxes until it is the right size. Knowing what to remove is the whole skill now.