Stragglers on Rik Kisnah - Bloghttps://www.rik-kisnah.ai/tags/stragglers/Recent content in Stragglers on Rik Kisnah - BlogHugoenTue, 16 Jan 2024 09:00:00 -0800Straggler Detection in Training Jobshttps://www.rik-kisnah.ai/teach/gpu-ai/straggler-detection/Tue, 16 Jan 2024 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/straggler-detection/In a synchronous job, the whole cluster runs at the speed of its slowest GPU. One card throttling at 80 percent makes a thousand cards run at 80 percent. Find it in minutes, automatically, and take it out.