https://www.rik-kisnah.ai/tags/storage/Recent content in Storage on Rik Kisnah - BlogHugoenTue, 07 Dec 2021 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/checkpointing-at-scale/Tue, 07 Dec 2021 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/checkpointing-at-scale/A thousand-GPU job will be interrupted. The only question is how much work you lose when it is. Save often enough that a failure costs minutes, cheaply enough that saving does not cost more than the failures.https://www.rik-kisnah.ai/teach/systems/design-an-object-store/Tue, 09 Nov 2021 09:00:00 -0800https://www.rik-kisnah.ai/teach/systems/design-an-object-store/Put a file in, get the same bytes back ten years later, from any of a hundred million files, after a hundred disks have died. Separate the names from the bytes, cut every file into pieces with spares, and check the pieces forever.https://www.rik-kisnah.ai/teach/systems/design-a-log-search-system/Tue, 17 Nov 2020 09:00:00 -0800https://www.rik-kisnah.ai/teach/systems/design-a-log-search-system/Every server writes thousands of lines a second. An engineer at 3am wants every line containing one request id from the last hour, in under two seconds. Ship, index, tier, and throw away. The hard part is the volume, not the search.https://www.rik-kisnah.ai/teach/systems/design-a-key-value-store/Tue, 07 Feb 2017 09:00:00 -0800https://www.rik-kisnah.ai/teach/systems/design-a-key-value-store/A giant dictionary spread across many machines. Put a value under a key, get it back later, even when machines die. Partition, replicate, and decide what ’later’ means when two copies disagree.https://www.rik-kisnah.ai/teach/systems/design-a-url-shortener/Tue, 17 May 2016 09:00:00 -0700https://www.rik-kisnah.ai/teach/systems/design-a-url-shortener/Turn a long web address into a short one and back again. One table, one counter, one cache, one redirect. The whole design is deciding how to make the short code so two people never get the same one, then making the read path fast because reads outnumber writes a thousand to one.