Collectives on Rik Kisnah - Bloghttps://www.rik-kisnah.ai/tags/collectives/Recent content in Collectives on Rik Kisnah - BlogHugoenTue, 10 May 2022 09:00:00 -0700All-Reduce Explainedhttps://www.rik-kisnah.ai/teach/gpu-ai/all-reduce-explained/Tue, 10 May 2022 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/all-reduce-explained/Every child has a number. Everyone needs the total. Pass a piece to your neighbour around the ring. This is how a thousand GPUs agree on one gradient, and why one slow link slows all of them.