https://www.rik-kisnah.ai/tags/batching/Recent content in Batching on Rik Kisnah - BlogHugoenTue, 11 Nov 2025 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/design-an-inference-api/Tue, 11 Nov 2025 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/design-an-inference-api/Thousands of people send prompts, a pool of GPUs answers them, and the GPU only earns its keep when it works on many prompts at once. Design the layer in between: the batcher, the queues, the maths for how many GPUs, what happens when one dies, and how two models share eight GPUs when one of them needs all eight.https://www.rik-kisnah.ai/teach/gpu-ai/continuous-batching-for-inference/Tue, 15 Apr 2025 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/continuous-batching-for-inference/Generating text is one token at a time, and one token uses a fraction of the GPU. So you serve many conversations at once and swap them in and out as they finish, every single step. This is why a serving engine is not just ‘call the model in a loop’.