https://www.rik-kisnah.ai/tags/observability/Recent content in Observability on Rik Kisnah - BlogHugoenTue, 16 Jan 2024 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/straggler-detection/Tue, 16 Jan 2024 09:00:00 -0800https://www.rik-kisnah.ai/teach/gpu-ai/straggler-detection/In a synchronous job, the whole cluster runs at the speed of its slowest GPU. One card throttling at 80 percent makes a thousand cards run at 80 percent. Find it in minutes, automatically, and take it out.https://www.rik-kisnah.ai/teach/systems/design-a-log-search-system/Tue, 17 Nov 2020 09:00:00 -0800https://www.rik-kisnah.ai/teach/systems/design-a-log-search-system/Every server writes thousands of lines a second. An engineer at 3am wants every line containing one request id from the last hour, in under two seconds. Ship, index, tier, and throw away. The hard part is the volume, not the search.https://www.rik-kisnah.ai/teach/systems/design-a-metrics-pipeline/Tue, 09 Jun 2020 09:00:00 -0700https://www.rik-kisnah.ai/teach/systems/design-a-metrics-pipeline/Every host writes one line in its diary every minute. A hundred thousand hosts. Get the lines to a place where you can ask ‘which hosts are sad right now?’ in under a second. The write path must be dumb and cheap. The read path can be clever.