Logic behind crawling an webpages like that of Screaming Frog? [on hold]
- by sree
I would like to know what is the parameters to be considered while developing a crawler like that of Screaming Frog.
Am looking forward for information on do's and dont's of webpage crawling. What are the problems the crawler may infuse on the webpages like loadtime (maybe?) or anything that effects webpage during crawling.
What are the rules the crawler needs to follow etc. Basically anything info that makes the crawler look good and accurate. Just point me in a right direction to achieve it..
Hope my requirement is clear this time.. :)