Crawling on Rik Kisnah - Bloghttps://www.rik-kisnah.ai/tags/crawling/Recent content in Crawling on Rik Kisnah - BlogHugoenTue, 16 Aug 2022 09:00:00 -0700Design a Web Crawlerhttps://www.rik-kisnah.ai/teach/systems/design-a-web-crawler/Tue, 16 Aug 2022 09:00:00 -0700https://www.rik-kisnah.ai/teach/systems/design-a-web-crawler/Start with a few pages, follow every link, and visit the whole web without visiting anything twice or knocking any site over. A giant breadth-first search with a to-do list on disk, a seen-set that fits in memory, and a rule to be nice.