https://www.rik-kisnah.ai/tags/inference/Recent content in Inference on Rik Kisnah - BlogHugoenTue, 11 Nov 2025 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/design-an-inference-api/Tue, 11 Nov 2025 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/design-an-inference-api/Thousands of people send prompts, a pool of GPUs answers them, and the GPU only earns its keep when it works on many prompts at once. Design the layer in between: the batcher, the queues, the maths for how many GPUs, what happens when one dies, and how two models share eight GPUs when one of them needs all eight.https://www.rik-kisnah.ai/teach/gpu-ai/mixture-of-experts-traffic/Tue, 16 Sep 2025 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/mixture-of-experts-traffic/A mixture-of-experts model does not send every token through every part of the network. A router picks a few experts per token, and the experts live on different GPUs. That turns a tidy all-reduce into an all-to-all, and the fabric feels the difference.https://www.rik-kisnah.ai/teach/gpu-ai/continuous-batching-for-inference/Tue, 15 Apr 2025 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/continuous-batching-for-inference/Generating text is one token at a time, and one token uses a fraction of the GPU. So you serve many conversations at once and swap them in and out as they finish, every single step. This is why a serving engine is not just ‘call the model in a loop’.https://www.rik-kisnah.ai/teach/gpu-ai/size-the-kv-cache/Tue, 18 Feb 2025 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/size-the-kv-cache/The model has to remember every word said so far, for every user, for every layer. That memory is the KV cache. Six numbers multiplied together tell you how many users one GPU can serve. Do the multiplication before you buy the GPUs.