My First robots.txt

Posted by Whitechapel on Pro Webmasters See other posts from Pro Webmasters or by Whitechapel
Published on 2013-10-20T08:26:34Z Indexed on 2013/10/20 10:18 UTC
Read the original article Hit count: 280

Filed under:

robots.txt

I'm creating my first robots.txt and wanted to get a second opinion on it. Basically I have a FTP setup on my board for some special users to transfer files between each other and I do NOT want that included in the search by the bots. I also want to point to my sitemap which gets auto generated by a PHP page. So here is what I have, what else should I include, and if I need to fix anything with it?

Also, it's linking to xmlsitemap.php because that generates the sitemap when called.

My goal is to allow any search bot crawl the forums to grab meta data.

User-agent: *
Disallow: /admin/
Disallow: /ali/
Disallow: /benny/
Disallow: /cgi-bin/
Disallow: /ders/
Disallow: /empire/
Disallow: /komodo_117/
Disallow: /xanxan/
Disallow: /zeroordie/
Disallow: /tmp/

Sitemap: http://www.vivalanation.com/forums/xmlsitemap.php

Edit, I'm not sure how to handle all the user's folders under /public_html/ since the robots.txt will be going in /public_html.

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More

Developer IT