Evidence for automatic browsing - Log file analysis
- by Nilani Algiriyage
I'm analyzing web server logs both in Apache and IIS log formats. I want to find the evidence for automatic browsing, like web robots, spiders, bots, etc. I used python robot-detection 0.2.8 for detecting robots in my log files, but I know there may be other robots (automatic programs) which have traversed through the web site but robot-detection can not identify.
So I want to ask:
Are there any specific clues that can be found in log files that human users do not leave but automated software would?
Do they follow a specific navigation pattern?
I saw some requests for favicon.ico - does this implicate that it is a automatic browsing?.
I found this article and this question with some valuable points.