Robots.txt never downloaded but some blocked URLs in GWT
Posted
by
Zistoloen
on Pro Webmasters
See other posts from Pro Webmasters
or by Zistoloen
Published on 2012-12-13T16:55:25Z
Indexed on
2012/12/13
17:17 UTC
Read the original article
Hit count: 398
There is something I don't understand in Google Webmaster Tools (GWT) for my Wordpress site.
In menu "Blocked URLs", it mention that my robots.txt has never been downloaded but there are some blocked URLs. It's kind of weird and not logical. Am i missing something?
User-agent : *
Disallow: /*?
Disallow: /wp-login.php
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Allow: /wp-content/uploads
Disallow: */trackback
Disallow: /*/feed
Disallow: /*/comments
Disallow: /cgi-bin
Disallow: /*.php$
Disallow: /*.inc$
Disallow: /*.gz$
Disallow: /*.cgi$
Disallow: /author/*
I'm afraid my robots.txt doesn't block several URLs I want to block.
© Pro Webmasters or respective owner