All-to-All on Rik Kisnah - Bloghttps://www.rik-kisnah.ai/tags/all-to-all/Recent content in All-to-All on Rik Kisnah - BlogHugoenTue, 16 Sep 2025 09:00:00 -0700Mixture of Experts and All-to-All Traffichttps://www.rik-kisnah.ai/teach/gpu-ai/mixture-of-experts-traffic/Tue, 16 Sep 2025 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/mixture-of-experts-traffic/A mixture-of-experts model does not send every token through every part of the network. A router picks a few experts per token, and the experts live on different GPUs. That turns a tidy all-reduce into an all-to-all, and the fabric feels the difference.