https://www.rik-kisnah.ai/tags/memory/Recent content in Memory on Rik Kisnah - BlogHugoenTue, 18 Feb 2025 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/size-the-kv-cache/Tue, 18 Feb 2025 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/size-the-kv-cache/The model has to remember every word said so far, for every user, for every layer. That memory is the KV cache. Six numbers multiplied together tell you how many users one GPU can serve. Do the multiplication before you buy the GPUs.https://www.rik-kisnah.ai/teach/gpu-ai/gpu-memory-hierarchy/Tue, 12 Nov 2019 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/gpu-memory-hierarchy/Registers, shared memory, HBM, then the CPU across the bus. Each level is bigger and slower than the last by a lot. Most slow GPU code is fast GPU code waiting for the wrong level.