https://www.rik-kisnah.ai/tags/kv-cache/Recent content in KV Cache on Rik Kisnah - BlogHugoenTue, 15 Apr 2025 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/continuous-batching-for-inference/Tue, 15 Apr 2025 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/continuous-batching-for-inference/Generating text is one token at a time, and one token uses a fraction of the GPU. So you serve many conversations at once and swap them in and out as they finish, every single step. This is why a serving engine is not just ‘call the model in a loop’.https://www.rik-kisnah.ai/teach/gpu-ai/size-the-kv-cache/Tue, 18 Feb 2025 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/size-the-kv-cache/The model has to remember every word said so far, for every user, for every layer. That memory is the KV cache. Six numbers multiplied together tell you how many users one GPU can serve. Do the multiplication before you buy the GPUs.