What does the arxiv.org anti-bot "search and destroy" actually do?
Posted
by
Brian Campbell
on Server Fault
See other posts from Server Fault
or by Brian Campbell
Published on 2009-05-01T16:11:13Z
Indexed on
2012/06/19
9:19 UTC
Read the original article
Hit count: 327
The lanl.arxiv.org math and scientific preprint service (formerly known as xxx.lanl.gov) has a strict policy against bots that ignore its robots.txt
, Robots Beware. On that page, the have a link labelled with "Click here to initiate automated 'seek-and-destroy' against your site", which is forbidden by their robots.txt
but presumably badly behaved robots will follow it, and reap the consequences. The question, what are the actual consequences? I have never had the guts to actually click on that link to see what it does. What can they be doing that is both effective and legal?
© Server Fault or respective owner