Is there a list of known web crawlers?
- by J. Pablo Fernández
I'm trying to get accurate download numbers for some files on a web server. I look at the user agents and some are clearly bots or web crawlers, but many for many I'm not sure, they may or may not be a web crawler and they are causing many downloads so it's important for me to know.
Is there somewhere a list of know web crawlers with some…