Is there a list of known web crawlers?
- by J. Pablo Fernández
I'm trying to get accurate download numbers for some files on a web server. I look at the user agents and some are clearly bots or web crawlers, but many for many I'm not sure, they may or may not be a web crawler and they are causing many downloads so it's important for me to know.
Is there somewhere a list of know web crawlers with some documentation like user agent, IPs, behavior, etc?
I'm not interested in the official ones, like Google's, Yahoo's, or Microsoft's. Those are generally well behaved and self-indentified.