How to crawl a quarter billion webpages in 40 hours

michaelnielsen.org