Software is pretty much like cathedrals: first we build them, then we pray
Cirrus Europe is a research group specialized on these areas:
There are several projects under way, among them Cirrus Explorer a web spider that aims at making a partial archive of the Web.
The main goal is to make a copy of different website and later compare the metrics and try to find interactions between them that may not be obvious at first.
For more information about the crawler please follow the link cirrus explorer.