https://www.rik-kisnah.ai/tags/fleet-health/Recent content in Fleet Health on Rik Kisnah - BlogHugoenTue, 09 Jun 2026 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/design-a-fleet-health-score/Tue, 09 Jun 2026 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/design-a-fleet-health-score/Thousands of GPUs, each with fifty signals. Boil them into one number per node that says ‘schedule on me’ or ‘do not’, and a loop that takes sick nodes out, fixes them, proves they are fixed, and puts them back. The metric that matters is usable compute.https://www.rik-kisnah.ai/teach/gpu-ai/design-a-burn-in-pipeline/Tue, 09 Apr 2024 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/design-a-burn-in-pipeline/Brand new hardware fails. Find the bad parts on your time, not the customer’s. Test one GPU, then one server, then one rack, then the whole cluster, with a gate at every step. Fail fast, log everything, never let a stage run without a stop rule.https://www.rik-kisnah.ai/teach/gpu-ai/straggler-detection/Tue, 16 Jan 2024 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/straggler-detection/In a synchronous job, the whole cluster runs at the speed of its slowest GPU. One card throttling at 80 percent makes a thousand cards run at 80 percent. Find it in minutes, automatically, and take it out.https://www.rik-kisnah.ai/teach/gpu-ai/gpu-fault-taxonomy/Tue, 13 Apr 2021 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/gpu-fault-taxonomy/A GPU tells you it is unhappy in a few specific ways. Some mean ‘retry’. Some mean ‘reboot’. Some mean ’this card goes back in the box’. The job is to read the message and sort it into the right bin fast, without a human staring at logs.