Baidu spider is hammering my server and bloating my error_log file

Posted by Gravy on Server Fault See other posts from Server Fault or by Gravy
Published on 2013-10-19T23:28:54Z Indexed on 2013/10/20 3:57 UTC
Read the original article Hit count: 514

Filed under:

apache2

|

error-logging

|

ip-blocking

|

spider

I am getting the following errors in my /etc/httpd/logs/error_log file

[Sun Oct 20 00:04:15 2013] [error] [client 180.76.5.16] File does not exist: /usr/local/apache/htdocs/homes
[Sun Oct 20 00:08:31 2013] [error] [client 180.76.5.113] File does not exist: /usr/local/apache/htdocs/homes
[Sun Oct 20 00:12:47 2013] [error] [client 180.76.5.88] File does not exist: /usr/local/apache/htdocs/homes
[Sun Oct 20 00:17:07 2013] [error] [client 180.76.5.138] File does not exist: /usr/local/apache/htdocs/homes

These kinds of errors are so often, that my error log files are over 500MB! I have done an IP trace on the client address to find that it belongs to something called baidu. Beijing Baidu Netcom Science and Technology Co in China.

Is there a way that I can just get apache to deny any incoming requests from some crummy spider that is repeatedly hitting my site??? Is there a better way of dealing with the problem? I am happy to completely block out China if it means that I can actually track real errors.

© Server Fault or respective owner

Related posts about apache2

apache2: Could not open configuration file /etc/apache2/apache2.conf: Permission denied

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I recently upgraded Ubuntu to the latest LTS edition on my work laptop, which I use as a LAMP development platform. The upgrade was from 12.4 to 14.4. Now I'm having trouble getting apache up and running again. Here is the output from an attempt: antonc@antonc-laptop:/etc/apache2$ sudo service apache2… >>> More
Apache segfault glibc segfault

as seen on Server Fault - Search for 'Server Fault'
I keep getting (about every 5-6 hours) this segfault in apache: [Tue Jun 26 12:43:10 2012] [notice] child pid 26810 exit signal Aborted (6) *** glibc detected *** /usr/sbin/apache2: free(): invalid pointer: 0xb68c2628 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x6ff22)[0xb75aef22] /lib/i386-linux-gnu/libc… >>> More
Localhost not working after installing PHP on Mountain Lion

as seen on Server Fault - Search for 'Server Fault'
I've installed php using brew install php54 --with-mysql, I've set up all the path correctly. which php will give me /usr/local/bin/php php -v will give me PHP 5.4.8 (cli) (built: Nov 20 2012 09:29:31) php --ini will give me: Configuration File (php.ini) Path: /usr/local/etc/php/5.4 Loaded Configuration… >>> More
Apache 2 Virtual Hosts no working on OSX 10.6

as seen on Server Fault - Search for 'Server Fault'
This is my first MacBook and I'm trying to get virtual hosts up and running so as it's going to be my dev machine. I've got apache/php/mysql running fine, the problem is that what ever address I go to I just get one of the virtual hosts I've setup. I can't even get to the root site anymore. I had… >>> More
apache2 error Could not open configuration file /etc/apache2/conf.d/: No such file or directory

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have just upgraded my Ubuntu 13.10 and apache2 is not working. When I try to start the apache2 server it is printing following errors: * Starting web server apache2 * The apache2 configtest failed. Output of config test was: apache2: Syntax error on line 263 of /etc/apache2/apache2.conf: Could… >>> More

Related posts about error-logging

Error logging with PHP and mod_fcgid

as seen on Server Fault - Search for 'Server Fault'
I have a mediawiki install that is acting up. Whenever I try to save an article, it goes to a blank screen. All though if I refresh that blank screen, it will go on to work. I have no idea why it's doing that, but it seems to be that it's a problem with PHP. I guess by default errors don't get logged… >>> More
MailMan error logging in Apache

as seen on Server Fault - Search for 'Server Fault'
I am working on a MailMan implementation using Fedora 12. I have it set up and running but I was looking at what happens if someone tries to brute force a password on the web GUI and found that MailMan doesn't log these errors at all or I'm completely looking in the wrong place. I have tried both… >>> More
Suggest any Error Logging facility for Winforms application for writing error in a remote database

as seen on Stack Overflow - Search for 'Stack Overflow'
My team working in a project using Winforms application(c#) & MSSQL 2005 as database. When I searched the goggle,I found ELMAH, NLog & log4net.I didn't have enough knowledge to implement in Winforms application. Please suggest any reference/library for writing Error Logging for Winforms… >>> More
ELMAH error logging for Windows Service

as seen on Stack Overflow - Search for 'Stack Overflow'
We are using ELMAH to log the web application exception which works great!. Also we want to know how to customize ELMAH to log the windows service exception. I am not interested in using another application for logging only windows service exception. Any help would be appreciated. >>> More
error logging in zend

as seen on Stack Overflow - Search for 'Stack Overflow'
i am using zend for 1st time..i am facing problems in finding errors. when a setter and getter function name mismatch i get a an error...but the error is show sometimes and not all the time...how do i log such errors.is there any separate log for this? >>> More