Fault Tolerance on Rik Kisnah - Bloghttps://www.rik-kisnah.ai/tags/fault-tolerance/Recent content in Fault Tolerance on Rik Kisnah - BlogHugoenTue, 12 May 2026 09:00:00 -0700Design LLM Training Infrastructurehttps://www.rik-kisnah.ai/teach/systems/design-llm-training-infrastructure/Tue, 12 May 2026 09:00:00 -0700https://www.rik-kisnah.ai/teach/systems/design-llm-training-infrastructure/Twenty thousand GPUs, jobs that need four thousand of them at once, and hardware that fails every nine hours. The whole design falls out of one number: how often something breaks, and what that costs you.