Googlebot cant access my site webmaster tools reply Unreachable robots.txt
Posted
by
Ahmad Ahmadi
on Server Fault
See other posts from Server Fault
or by Ahmad Ahmadi
Published on 2013-10-27T06:15:43Z
Indexed on
2013/10/27
9:57 UTC
Read the original article
Hit count: 427
When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server: tcpdump | grep google it return that google can access my server with IP 66.249.81.172 or 66.249.75.111. but there is not any think in access log or error log or other apache logs. cat access_log | grep google or cat error_log | grep 66.249.81.172 Other bot (bing,...) can access apache but google cant. there is not any problem in my robots.txt or its permissions because as you know robots.txt is not necessary so I delete it but again webmaster tools returned Unreachable robots.txt not 404 not found! information about server: Server OS : CentOS 6 Web Server : Apache 2.x Firewall : IPTables is stoped SELinux is Disabled There is not any think else for security on my server. how can I investigate the problem and is there any other command that can help me to find the problem.
© Server Fault or respective owner