Blocking Just the Parent Domain via robots.txt
Posted
by
Bryan Hadaway
on Pro Webmasters
See other posts from Pro Webmasters
or by Bryan Hadaway
Published on 2012-09-06T12:29:40Z
Indexed on
2012/09/06
15:50 UTC
Read the original article
Hit count: 447
seo
Let's say you have a parent domain:
parent.com
and children subdomains under that parent domain:
child1.com
child2.com
child3.com
Is there a way to use just the following within parent.com:
User-agent: *
Disallow: /
Considering each child has their own robots.txt stating:
User-agent: *
Allow: /
Or is the parent robots.txt still going to have to make an exception for every single subdomain:
User-agent: *
Disallow: /
Allow: /child1/
Allow: /child2/
Allow: /child3/
Obviously this is important and tricky territory SEO wise so I'm looking to learn the definitive and safe, best practice method here to sharpen my skills.
Thanks, Bryan
© Pro Webmasters or respective owner