What does the arxiv.org anti-bot "search and destroy" actually do?

Posted by Brian Campbell on Server Fault See other posts from Server Fault or by Brian Campbell
Published on 2009-05-01T16:11:13Z Indexed on 2012/06/19 9:19 UTC
Read the original article Hit count: 327

The lanl.arxiv.org math and scientific preprint service (formerly known as xxx.lanl.gov) has a strict policy against bots that ignore its robots.txt, Robots Beware. On that page, the have a link labelled with "Click here to initiate automated 'seek-and-destroy' against your site", which is forbidden by their robots.txt but presumably badly behaved robots will follow it, and reap the consequences. The question, what are the actual consequences? I have never had the guts to actually click on that link to see what it does. What can they be doing that is both effective and legal?

© Server Fault or respective owner

Related posts about Web

Related posts about denial-of-service