https://www.rik-kisnah.ai/tags/checkpointing/Recent content in Checkpointing on Rik Kisnah - BlogHugoenTue, 07 Dec 2021 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/checkpointing-at-scale/Tue, 07 Dec 2021 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/checkpointing-at-scale/A thousand-GPU job will be interrupted. The only question is how much work you lose when it is. Save often enough that a failure costs minutes, cheaply enough that saving does not cost more than the failures.