Teach / GPU and AI
GPU and AI
The round that did not exist ten years ago. What a GPU is, how thousands of them talk, and how you know a cluster is healthy.
This is the track I am asked about most now, and the one with the fewest good books. A modern AI cluster is a distributed system where the network is the computer, the failure domain is a rack, and one bad cable can slow down a thousand GPUs. These lessons are the vocabulary and the mental pictures you need before someone asks you to design one.
- Design a Fleet Health Score and Repair LoopThousands of GPUs, each with fifty signals. Boil them into one number per node that says 'schedule on me' or 'do not', and a loop that takes sick nodes out, fixes them, proves they are fixed, and puts them back. The metric that matters is usable compute.
- Design Model Weight Distribution to a Thousand HostsA 500 GB model sits in one repository behind a 10 Gbps link. A thousand GPU hosts need it, each with a 10 Gbps card, and a few of them will die while you copy. Say the lower bound, then design the swarm that gets close to it, then the part that actually matters in production: verifying every copy and never serving from a half-loaded host.
- Design a Rack as One ComputerSeventy-two GPUs wired so tightly they behave like one. The unit you launch, monitor, and repair is now the rack. Design the control plane for that, and decide what happens when one tray out of eighteen dies.
- Design an Inference APIThousands of people send prompts, a pool of GPUs answers them, and the GPU only earns its keep when it works on many prompts at once. Design the layer in between: the batcher, the queues, the maths for how many GPUs, what happens when one dies, and how two models share eight GPUs when one of them needs all eight.
- Mixture of Experts and All-to-All TrafficA mixture-of-experts model does not send every token through every part of the network. A router picks a few experts per token, and the experts live on different GPUs. That turns a tidy all-reduce into an all-to-all, and the fabric feels the difference.
- Continuous Batching for InferenceGenerating text is one token at a time, and one token uses a fraction of the GPU. So you serve many conversations at once and swap them in and out as they finish, every single step. This is why a serving engine is not just 'call the model in a loop'.
- Size the KV CacheThe model has to remember every word said so far, for every user, for every layer. That memory is the KV cache. Six numbers multiplied together tell you how many users one GPU can serve. Do the multiplication before you buy the GPUs.
- Power and Cooling for GPU RacksA rack that used to draw ten kilowatts now draws over a hundred. Air cannot carry that heat away. The rack is plumbed, the floor is load-rated, and the power feed is a design decision before a single GPU is bought.
- Design a Burn-In PipelineBrand new hardware fails. Find the bad parts on your time, not the customer's. Test one GPU, then one server, then one rack, then the whole cluster, with a gate at every step. Fail fast, log everything, never let a stage run without a stop rule.
- Straggler Detection in Training JobsIn a synchronous job, the whole cluster runs at the speed of its slowest GPU. One card throttling at 80 percent makes a thousand cards run at 80 percent. Find it in minutes, automatically, and take it out.
- Measure GPU Utilisation and MFU'GPU utilisation 100 percent' can mean the chip is doing useful maths, or that one thread is spinning. Model FLOPs utilisation is the honest number: useful work divided by what the chip could do. Know how to compute it and what a good value is.
- Design the Training NetworkTwo networks, not one. East-west is where GPUs talk to GPUs and must never drop a packet. North-south is everything else. Draw the rails, keep it lossless, and never let the two share a wire.
- Diagnose a Slow NCCL JobThe job runs. Nothing errors. It is just half as fast as last week. This is the ticket every GPU infrastructure engineer gets, and it has a fixed order of questions. Learn the order and you solve it in an hour instead of a week.
- Design a Topology-Aware GPU SchedulerWhich GPUs a job lands on decides how fast it runs. A scheduler that hands out any free GPU is a scheduler that puts one job's ring across three spines. Give it a map of the cluster and make it place jobs the way the network wants.
- All-Reduce ExplainedEvery child has a number. Everyone needs the total. Pass a piece to your neighbour around the ring. This is how a thousand GPUs agree on one gradient, and why one slow link slows all of them.
- Number Formats: FP32 to FP8Every number in a model takes bits. Halve the bits and you double the speed, halve the memory, and risk the maths going wrong. Know which format keeps range, which keeps precision, and where you must not cut.
- Checkpointing at ScaleA thousand-GPU job will be interrupted. The only question is how much work you lose when it is. Save often enough that a failure costs minutes, cheaply enough that saving does not cost more than the failures.
- Why GPUs for AIA CPU is eight chefs who can cook anything. A GPU is ten thousand kids who can each only stir one pot, but all at the same time and all the same way. AI is mostly stirring pots.
- GPU Faults: XID, ECC, and What They MeanA GPU tells you it is unhappy in a few specific ways. Some mean 'retry'. Some mean 'reboot'. Some mean 'this card goes back in the box'. The job is to read the message and sort it into the right bin fast, without a human staring at logs.
- Data, Model, and Pipeline ParallelismThree ways to split a job that is too big for one GPU. Split the data, split the model's layers across machines, or split each layer across machines. Each one puts a different kind of traffic on the network, and that decides your cluster design.
- RDMA ExplainedNormal networking hands every packet to the operating system, which copies it, thinks about it, and copies it again. RDMA lets one machine write straight into another machine's memory with nobody in the middle. That is why GPU clusters use it.
- The GPU Memory HierarchyRegisters, shared memory, HBM, then the CPU across the bus. Each level is bigger and slower than the last by a lot. Most slow GPU code is fast GPU code waiting for the wrong level.
- NVLink versus PCIeTwo roads out of every GPU. PCIe is the public road to the CPU and the network. NVLink is the private motorway between GPUs in the same box. Know the speed of each, and know which one your traffic is on.