Blocking yandex.ru bot

Posted by Ross on Server Fault See other posts from Server Fault or by Ross
Published on 2010-04-29T07:49:12Z Indexed on 2010/04/29 7:58 UTC
Read the original article Hit count: 784

Filed under:

robots.txt

|

search-engine

I want to block all request from yandex.ru search bot. It is very traffic intensive (2GB/day). I first blocked one C class IP range, but it seems this bot appear from different IP ranges.

For example:

spider31.yandex.ru -> 77.88.26.27 spider79.yandex.ru -> 95.108.155.251 etc..

I can put some deny in robots.txt but not sure if it respect this. I am thinking of blocking a list of IP ranges.

Can somebody suggest some general solution.

© Server Fault or respective owner

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More

Related posts about search-engine

Search Engine Optimisation - Google Replaces Yahoo As T-Mobile's Default Search Engine

as seen on Ezine Articles - Search for 'Ezine Articles'
Competition has been stoked in the mobile search engine optimisation (SEO) sector with the announcement that Google is to replace Yahoo as T-Mobile's default engine. While T-Mobile USA has replaced Yahoo with Google as its default search on mobile phones such as RIM BlackBerrys, some Yahoo-based services… >>> More
Search Engine Optimization - The Importance of Page Optimization in Search Engine Optimization

as seen on Ezine Articles - Search for 'Ezine Articles'
In order for your website to rank well, your internal linking structure is critical to your success. This is covered some of the theory for this in various articles and blogs about Page Structure of a website, which said how you should map out the physical linking structure, but in this guide I will… >>> More
How to Improve Search Engine Ranking by On-Page Search Engine Optimization

as seen on Ezine Articles - Search for 'Ezine Articles'
There are two types of search engine optimization (SEO) techniques: on-page and off-page SEO. With on-page SEO, you have full control of all the techniques and methods you can use on your own sites. With off-page SEO, you will try some techniques to get other site owners to refer to your site which… >>> More
Search Engine Optimization For Great Search Engine Placement!

as seen on Ezine Articles - Search for 'Ezine Articles'
Do you do enough search engine optimization to get the search engine placement that you want for the keywords that you want to rank for? If not then read on and I will give you information that you need to know to start getting those rankings that you want, and start receiving traffic! There are a… >>> More
Search Engine Optimization - Use of META Tag in Search Engine Marketing

as seen on Ezine Articles - Search for 'Ezine Articles'
Want your WEBPAGE to land in the 1st PAGES of Google and Yahoo search engines? Then, make use of this powerful META TAG in your webpage/s, and you will have the result you have been aching for ages. >>> More