Google Webmaster Tools robots test not working

Posted by tracy_snap on Pro Webmasters See other posts from Pro Webmasters or by tracy_snap
Published on 2013-06-12T03:52:15Z Indexed on 2013/11/09 16:15 UTC
Read the original article Hit count: 238

Filed under:

google-webmaster-tools

|

robots.txt

Within Webmaster Tools I have supplied my test content:

User-agent: *
Disallow:/admin/
Disallow: /tag/

When I specify the URL to test against, for example:

http://www.site.com/tag/

It gives me this result:

"Allowed: Detected as a directory; specific files may have different restrictions"

As far as I know I have set this up correctly, shouldn't Google be saying that the /tag/ directory is "disallowed"?

© Pro Webmasters or respective owner

Related posts about google-webmaster-tools

Weird entry for robots.txt on a Naked Domain in Google Webmaster Tools

as seen on Pro Webmasters - Search for 'Pro Webmasters'
We own a .co.uk address and use an Internet hosting company that has made mistakes around DNS in the past. Our main site is hosted on www. and their reluctance to allow editing of AAAA records on-line means our naked domain does not resolve. Currently when we attempt to reach the naked version there… >>> More
How to submit sitemap when your website has partial https? - Error: "Not in Domain"

as seen on Pro Webmasters - Search for 'Pro Webmasters'
My website is an ecommerce that is set up to do http for the item browsing portion, but https for things like shopping cart, contact us, etc.. (anything that has forms on it). I've submitted my website a long time ago to google webmaster tools as http://www.mywebsite.com. I also submitted a sitemap… >>> More
Why Use Google Webmaster Tools

as seen on Ezine Articles - Search for 'Ezine Articles'
This article talks about why Google Webmaster Tools is an amazing program for optimizing and debugging your website. It provides many advantages for any website owner who uses it. >>> More
What Are Soft 404s in Google Webmaster Tools

as seen on Ezine Articles - Search for 'Ezine Articles'
There are two types of 404 response codes when a web page cannot be found. Google Webmaster Tools now displays both types. Find out why you do not want to have any soft 404 errors. >>> More
Revamped Google Webmaster Tools

as seen on Get blogged by JoKi - Search for 'Get blogged by JoKi'
With a positive surprise I realized today that Google's Webmaster Tools had some minor overhauling and provide some more details than before. Most obvious are the changes on the dashboard where the Top Search Queries now provide information about impressions and clicktroughs instead of the rankings… >>> More

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More