Example robots.txt for WordPress and PHPBB3

I have just finished retooling the robots.txt file I will now be using. It includes both wordpress and phpBB3. I took wordpress’s suggested robots.txt file, added in a few more directories, added in the sitemap reference, and added in the phpbb3 section. I am not sure how good it will preform, however if changes are made, I will note them. If you have any comments, or suggestions, let me know.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-content/backup-db
Disallow: /wp-content/bte-wb
Disallow: /wp-content/upgrade
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Disallow: /auto_backup
Disallow: /cp
Disallow: /webalizer
Allow: /wp-content/uploads

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# digg mirror
User-agent: duggmirror
Disallow: /

# phpBB3 Section
User-agent: *
Disallow: /phpBB3/admin/
Disallow: /phpBB3/images/
Disallow: /phpBB3/includes/
Disallow: /phpBB3/language/
Disallow: /phpBB3/templates/
Disallow: /phpBB3/common.php
Disallow: /phpBB3/config.php
Disallow: /phpBB3/groupcp.php
Disallow: /phpBB3/memberlist.php
Disallow: /phpBB3/modcp.php
Disallow: /phpBB3/posting.php
Disallow: /phpBB3/profile.php
Disallow: /phpBB3/privmsg.php
Disallow: /phpBB3/viewonline.php
Disallow: /phpBB3/search.php
Disallow: /phpBB3/faq.php

Sitemap: http://www.psychcomp.com/sitemap.xml

Be Sociable, Share!

• 

About the author:  Psychcomp is owned and maintained by Nathan Driskell, a Licensed Professional Counselor - Intern specializing in Internet Addiction and Asperger's Disorder. Nathan is also a Web Designer and Network Administrator. Contact Nathan at the following locations: dami...@psychcomp.com">E-Mail Twitter

Related Posts

1. PHPBB3 – Private Messaging Error
2. Remove Old Links from Google
3. Blocking Search Engine Bots
4. How to Establish Yourself Online – Part 5: WordPress SEO
5. How to Establish Yourself Online – Part 4: Customizing WordPress
6. Get Listed in Google Faster
7. How to Establish Yourself Online – Part 3: Setting up WordPress
8. WordPress Attack: Upgrade Now
9. How to Establish Yourself Online – Part 2: Domain Name and Web Host
10. Why I Use Google Chrome

Tags: phpbb3, robots.txt, SEO, wordpress

This entry was posted on Friday, August 7th, 2009 at 7:03 PM and is filed under Computers, PHPBB3, SEO, Web Design. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.