Guidelines for good webcrawler 'Etiquette'

Posted by Harry on Stack Overflow See other posts from Stack Overflow or by Harry
Published on 2009-06-09T13:33:12Z Indexed on 2010/05/10 8:44 UTC
Read the original article Hit count: 301

Filed under:

webcrawling

|

best-practices

I'm building a search engine (for fun) and it has just struck me that potentially my little project might wreak havok by clicking on ads and all sorts of problems.

So what are the guidelines for good webcrawler 'Etiquette'?

Things that spring to mind:

Observe Robot.txt instructions
Limit the number of simultaneous requests to the same domain
Don't follow ad links?

Stopping the crawler from clicking on ads - This one is particularly on my mind at the moment... how do i stop my bot from 'clicking' on ads? if it is going straight to the url in the ad is it counted as a click?

© Stack Overflow or respective owner

Related posts about webcrawling

Asynchronous Webcrawling F#, something wrong ?

as seen on Stack Overflow - Search for 'Stack Overflow'
Not quite sure if it is ok to do this but, my question is: Is there something wrong with my code ? It doesn't go as fast as I would like, and since I am using lots of async workflows maybe I am doing something wrong. The goal here is to build something that can crawl 20 000 pages in less than an hour… >>> More
WebCrawling Dynamic Links

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi Everyone, Anybody has any idea on crawling websites that have dynamic pages/queries? I mean if I click a certain link, it has different values every I try to reload it in a web browser. Now my webcrawler could not download the contents of these pages. Please advise. >>> More
Crawling engine architecture - Java/ Perl integration

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, I am looking to develop a management and administration solution around our webcrawling perl scripts. Basically, right now our scripts are saved in SVN and are manually kicked off by SysAdmin/devs etc. Everytime we need to retrieve data from new sources we have to create a ticket with business… >>> More
Building an automatic web crawler

as seen on Stack Overflow - Search for 'Stack Overflow'
I am building a web application crawler that's meant not only to find all the links or pages in a web application, but also perform all the allowed actions in the app (such as pushing buttons, filling forms, notice changes in the DOM even if they did not trigger a request etc.) Basically, this is… >>> More
Getting web page after calling DownloadStringAsync()?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello I don't know enough about VB.Net yet to use the richer HttpWebRequest class, so I figured I'd use the simpler WebClient class to download web pages asynchronously (to avoid freezing the UI). However, how can the asynchronous event handler actually return the web page to the calling routine… >>> More

Related posts about best-practices

Batch Best Practices and Technical Best Practices Updated

as seen on Oracle Blogs - Search for 'Oracle Blogs'
The Batch Best Practices for Oracle Utilities Application Framework based products (Doc Id: 836362.1) and Technical Best Practices for Oracle Utilities Application Framework Based Products (Doc Id: 560367.1) have been updated with updated and new advice for the various versions of the Oracle… >>> More
Partial template specialization of free functions - best practices

as seen on Stack Overflow - Search for 'Stack Overflow'
As most C++ programmers should know, partial template specialization of free functions is disallowed. For example, the following is illegal C++: template <class T, int N> T mul(const T& x) { return x * N; } template <class T> T mul<T, 0>(const T& x) { return T(0); } //… >>> More
Object Oriented PHP Best Practices

as seen on Stack Overflow - Search for 'Stack Overflow'
Say I have a class which represents a person, a variable within that class would be $name. Previously, In my scripts I would create an instance of the object then set the name by just using: $object->name = "x"; However, I was told this was not best practice? That I should have a function set_name()… >>> More
Good place to look for example Database Designs - Best practices

as seen on Stack Overflow - Search for 'Stack Overflow'
I have been given the task to design a database to store a lot of information for our company. Because the task is rather big and contains multiple modules where users should be able to do stuff, I'm worried about designing a good data model for this. I just don't want to end up with a badly designed… >>> More
Notification Email Best Practices--From Server Setup to Programming

as seen on Stack Overflow - Search for 'Stack Overflow'
All, I'm in the process now of building a SaaS tool that allows network admins to generate notification emails to the members of the end-users of our platform (among many many other things). I'm running into a bit of an "out of my expertise" wall, as I know there are a lot of variables involved… >>> More