https://www.rik-kisnah.ai/tags/reliability/Recent content in Reliability on Rik Kisnah - BlogHugoenTue, 10 Feb 2026 09:00:00 -0800https://www.rik-kisnah.ai/teach/systems/design-an-on-call-paging-system/Tue, 10 Feb 2026 09:00:00 -0800https://www.rik-kisnah.ai/teach/systems/design-an-on-call-paging-system/When the alert fires, wake exactly the right person, make sure someone acknowledges it, and escalate if nobody does. The system that pages must be the last thing standing when everything else is down, so it must not depend on anything else.https://www.rik-kisnah.ai/teach/gpu-ai/checkpointing-at-scale/Tue, 07 Dec 2021 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/checkpointing-at-scale/A thousand-GPU job will be interrupted. The only question is how much work you lose when it is. Save often enough that a failure costs minutes, cheaply enough that saving does not cost more than the failures.