https://www.rik-kisnah.ai/teach/gpu-ai/Recent content in GPU and AI on Rik Kisnah - BlogHugoenTue, 09 Jun 2026 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/design-a-fleet-health-score/Tue, 09 Jun 2026 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/design-a-fleet-health-score/Thousands of GPUs, each with fifty signals. Boil them into one number per node that says ‘schedule on me’ or ‘do not’, and a loop that takes sick nodes out, fixes them, proves they are fixed, and puts them back. The metric that matters is usable compute.https://www.rik-kisnah.ai/teach/gpu-ai/design-model-weight-distribution/Tue, 17 Mar 2026 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/design-model-weight-distribution/A 500 GB model sits in one repository behind a 10 Gbps link. A thousand GPU hosts need it, each with a 10 Gbps card, and a few of them will die while you copy. Say the lower bound, then design the swarm that gets close to it, then the part that actually matters in production: verifying every copy and never serving from a half-loaded host.https://www.rik-kisnah.ai/teach/gpu-ai/design-a-rack-as-one-computer/Tue, 03 Mar 2026 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/design-a-rack-as-one-computer/Seventy-two GPUs wired so tightly they behave like one. The unit you launch, monitor, and repair is now the rack. Design the control plane for that, and decide what happens when one tray out of eighteen dies.https://www.rik-kisnah.ai/teach/gpu-ai/design-an-inference-api/Tue, 11 Nov 2025 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/design-an-inference-api/Thousands of people send prompts, a pool of GPUs answers them, and the GPU only earns its keep when it works on many prompts at once. Design the layer in between: the batcher, the queues, the maths for how many GPUs, what happens when one dies, and how two models share eight GPUs when one of them needs all eight.https://www.rik-kisnah.ai/teach/gpu-ai/mixture-of-experts-traffic/Tue, 16 Sep 2025 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/mixture-of-experts-traffic/A mixture-of-experts model does not send every token through every part of the network. A router picks a few experts per token, and the experts live on different GPUs. That turns a tidy all-reduce into an all-to-all, and the fabric feels the difference.https://www.rik-kisnah.ai/teach/gpu-ai/continuous-batching-for-inference/Tue, 15 Apr 2025 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/continuous-batching-for-inference/Generating text is one token at a time, and one token uses a fraction of the GPU. So you serve many conversations at once and swap them in and out as they finish, every single step. This is why a serving engine is not just ‘call the model in a loop’.https://www.rik-kisnah.ai/teach/gpu-ai/size-the-kv-cache/Tue, 18 Feb 2025 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/size-the-kv-cache/The model has to remember every word said so far, for every user, for every layer. That memory is the KV cache. Six numbers multiplied together tell you how many users one GPU can serve. Do the multiplication before you buy the GPUs.https://www.rik-kisnah.ai/teach/gpu-ai/power-and-cooling-for-gpu-racks/Tue, 13 Aug 2024 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/power-and-cooling-for-gpu-racks/A rack that used to draw ten kilowatts now draws over a hundred. Air cannot carry that heat away. The rack is plumbed, the floor is load-rated, and the power feed is a design decision before a single GPU is bought.https://www.rik-kisnah.ai/teach/gpu-ai/design-a-burn-in-pipeline/Tue, 09 Apr 2024 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/design-a-burn-in-pipeline/Brand new hardware fails. Find the bad parts on your time, not the customer’s. Test one GPU, then one server, then one rack, then the whole cluster, with a gate at every step. Fail fast, log everything, never let a stage run without a stop rule.https://www.rik-kisnah.ai/teach/gpu-ai/straggler-detection/Tue, 16 Jan 2024 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/straggler-detection/In a synchronous job, the whole cluster runs at the speed of its slowest GPU. One card throttling at 80 percent makes a thousand cards run at 80 percent. Find it in minutes, automatically, and take it out.https://www.rik-kisnah.ai/teach/gpu-ai/measure-gpu-utilisation-and-mfu/Tue, 14 Nov 2023 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/measure-gpu-utilisation-and-mfu/‘GPU utilisation 100 percent’ can mean the chip is doing useful maths, or that one thread is spinning. Model FLOPs utilisation is the honest number: useful work divided by what the chip could do. Know how to compute it and what a good value is.https://www.rik-kisnah.ai/teach/gpu-ai/design-the-training-network/Tue, 15 Aug 2023 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/design-the-training-network/Two networks, not one. East-west is where GPUs talk to GPUs and must never drop a packet. North-south is everything else. Draw the rails, keep it lossless, and never let the two share a wire.https://www.rik-kisnah.ai/teach/gpu-ai/diagnose-a-slow-nccl-job/Tue, 14 Mar 2023 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/diagnose-a-slow-nccl-job/The job runs. Nothing errors. It is just half as fast as last week. This is the ticket every GPU infrastructure engineer gets, and it has a fixed order of questions. Learn the order and you solve it in an hour instead of a week.https://www.rik-kisnah.ai/teach/gpu-ai/design-a-topology-aware-gpu-scheduler/Tue, 11 Oct 2022 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/design-a-topology-aware-gpu-scheduler/Which GPUs a job lands on decides how fast it runs. A scheduler that hands out any free GPU is a scheduler that puts one job’s ring across three spines. Give it a map of the cluster and make it place jobs the way the network wants.https://www.rik-kisnah.ai/teach/gpu-ai/all-reduce-explained/Tue, 10 May 2022 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/all-reduce-explained/Every child has a number. Everyone needs the total. Pass a piece to your neighbour around the ring. This is how a thousand GPUs agree on one gradient, and why one slow link slows all of them.https://www.rik-kisnah.ai/teach/gpu-ai/number-formats-fp32-to-fp8/Tue, 15 Feb 2022 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/number-formats-fp32-to-fp8/Every number in a model takes bits. Halve the bits and you double the speed, halve the memory, and risk the maths going wrong. Know which format keeps range, which keeps precision, and where you must not cut.https://www.rik-kisnah.ai/teach/gpu-ai/checkpointing-at-scale/Tue, 07 Dec 2021 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/checkpointing-at-scale/A thousand-GPU job will be interrupted. The only question is how much work you lose when it is. Save often enough that a failure costs minutes, cheaply enough that saving does not cost more than the failures.https://www.rik-kisnah.ai/teach/gpu-ai/why-gpus-for-ai/Tue, 14 Sep 2021 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/why-gpus-for-ai/A CPU is eight chefs who can cook anything. A GPU is ten thousand kids who can each only stir one pot, but all at the same time and all the same way. AI is mostly stirring pots.https://www.rik-kisnah.ai/teach/gpu-ai/gpu-fault-taxonomy/Tue, 13 Apr 2021 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/gpu-fault-taxonomy/A GPU tells you it is unhappy in a few specific ways. Some mean ‘retry’. Some mean ‘reboot’. Some mean ’this card goes back in the box’. The job is to read the message and sort it into the right bin fast, without a human staring at logs.https://www.rik-kisnah.ai/teach/gpu-ai/data-model-and-pipeline-parallelism/Tue, 15 Sep 2020 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/data-model-and-pipeline-parallelism/Three ways to split a job that is too big for one GPU. Split the data, split the model’s layers across machines, or split each layer across machines. Each one puts a different kind of traffic on the network, and that decides your cluster design.https://www.rik-kisnah.ai/teach/gpu-ai/rdma-explained/Tue, 10 Mar 2020 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/rdma-explained/Normal networking hands every packet to the operating system, which copies it, thinks about it, and copies it again. RDMA lets one machine write straight into another machine’s memory with nobody in the middle. That is why GPU clusters use it.https://www.rik-kisnah.ai/teach/gpu-ai/gpu-memory-hierarchy/Tue, 12 Nov 2019 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/gpu-memory-hierarchy/Registers, shared memory, HBM, then the CPU across the bus. Each level is bigger and slower than the last by a lot. Most slow GPU code is fast GPU code waiting for the wrong level.https://www.rik-kisnah.ai/teach/gpu-ai/nvlink-vs-pcie/Tue, 16 Apr 2019 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/nvlink-vs-pcie/Two roads out of every GPU. PCIe is the public road to the CPU and the network. NVLink is the private motorway between GPUs in the same box. Know the speed of each, and know which one your traffic is on.