How can I allow robots access to my sitemap, but prevent casual users from accessing it?

Posted by morpheous on Pro Webmasters See other posts from Pro Webmasters or by morpheous
Published on 2010-07-20T09:33:33Z Indexed on 2011/01/13 2:59 UTC
Read the original article Hit count: 347

Filed under:

security

|

apache

|

lamp

|

sitemap

I am storing my sitemaps in my web folder. I want web crawlers (Googlebot etc) to be able to access the file, but I dont necessarily want all and sundry to have access to it.

For example, this site (superuser.com), has a site index - as specified by its robots.txt file (http://superuser.com/robots.txt).

However, when you type http://superuser.com/sitemap.xml, you are directed to a 404 page.

How can I implement the same thing on my website?

I am running a LAMP website, also I am using a sitemap index file (so I have multiple site maps for the site). I would like to use the same mechanism to make them unavailable via a browser, as described above.

© Pro Webmasters or respective owner

Related posts about security

sudo apt-get update errors

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
Here is what I get on my terminal when running sudo apt-get update errors. I dont know if the issue is from my sources.list or my proxy setup(have not made any changes to proxies). Thank you for any help in advanced. Ign http://security.ubuntu.com oneiric-security Release.gpg Ign http://security… >>> More
The Security-Developer Gap: Why IT Security Can't Fix Your Security Problems Alone

as seen on Devx - Search for 'Devx'
Two well-known white hats explain how hackers take advantage of security holes left by enterprise application developers. >>> More
The Security-Developer Gap: Why IT Security Can't Fix Your Security Problems Alone

as seen on Internet.com - Search for 'Internet.com'
Two well-known white hats explain how hackers take advantage of security holes left by enterprise application developers. >>> More
StackOverFlowError while creating Mac object on AS400/Java

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello all, I am a newbie to AS400-Java programming. I am trying to create my first program to test the implementation of Message Authentication Code (MAC). I am trying to use the HMACSHA1 hash function. My (Java 1.4) program runs fine on a dev box (V5R4).But fails terribly on the QA box (V5R3). My… >>> More
Google App Engine - Spring Security Issue (java.security.AccessControlException)

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm currently getting the AccessControlException below when I deploy to app engine (I don't see it when I run in my local environment). I'm using GAE 1.3.1, Spring 3.0.1, and Spring Security 3.0.2. Any ideas how to get around this issue? It appears to be an issue with Spring Security trying to get… >>> More

Related posts about apache

Web site not responding

as seen on Super User - Search for 'Super User'
I have website working fine before. But now its not able to connect to the server(I believe that is the problem). But its strange that the message not able to connect to the server is not coming and its keep connecting... for infinite time. Here is the screenshot. Here are some of the useful… >>> More
SVN Error 403 Forbidden

as seen on Stack Overflow - Search for 'Stack Overflow'
I can't figure this out. I try to import a new project into a svn repository from Netbeans and get 403 Forbidden. I just setup svn on my serverbox today. I can get to it through a browser just fine, though its empty as I haven't imported my project yet. Apache's path for html files is /var/www I… >>> More
How can I setup dependencies for Axis2 / Axiom on Maven2

as seen on Stack Overflow - Search for 'Stack Overflow'
I've tried the following settings on pom.xml to use Axis2 wsdl2code: <dependencies> <dependency> <groupId>org.apache.axis2</groupId> <artifactId>axis2</artifactId> <version>1.5.1</version> </dependency> </dependencies> ... <build> … >>> More
Problem compiling hive with ant

as seen on Stack Overflow - Search for 'Stack Overflow'
I compiling with Solaris 10 SPARC, jdk 1.6 from Sun, Ant 1.7.1 from OpenCSW. I have no problem running hadoop 0.17.2.1 However, I have problem compiling/integrating hive with the error 'cannot find symbol', although I followed the tutorial. I have the hive source code from SVN exactly from tutorial… >>> More
War deployment error related to classloading

as seen on Stack Overflow - Search for 'Stack Overflow'
hello when i am deploying my war file and run it it will give error like org.springframework.instrument.classloading.tomcat.TomcatInstrumentableClassLoader Jan 6, 2011 3:16:04 PM org.apache.catalina.startup.HostConfig deployDescriptor INFO: Deploying configuration descriptor servlet.xml Jan… >>> More