Userdriven Distributed Spidering...

"What if you could get information as to what pages are new and what pages are changes by just observing where people are surfing? This is the space that the [Google] accelerator occupies. Sitting neatly between your web browser and the Google architecture is a mini proxy that keeps checking if it can find a way to give you pages at a faster rate from the Google index than it does from the actual existing site. Along the way, Google finds out what pages are missing from its index (and gets a chance to add them) and what pages in its index are not up to date. Imagine a million people downloading the Google Web Accelerator and all of a sudden, you have an infrastructure that finds out about a lot of pages very quickly." (Google Accelerates Search)