I have just finished retooling the robots.txt file I will now be using. It includes both wordpress and phpBB3. I took wordpress’s suggested robots.txt file, added in a few more directories, added in the sitemap reference, and added in the phpbb3 section. I am not sure how good it will preform, however if changes are made, I will note them. If you have any comments, or suggestions, let me know.
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /wp-content/backup-db Disallow: /wp-content/bte-wb Disallow: /wp-content/upgrade Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Disallow: /auto_backup Disallow: /cp Disallow: /webalizer Allow: /wp-content/uploads # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* # Internet Archiver Wayback Machine User-agent: ia_archiver Disallow: / # digg mirror User-agent: duggmirror Disallow: / # phpBB3 Section User-agent: * Disallow: /phpBB3/admin/ Disallow: /phpBB3/images/ Disallow: /phpBB3/includes/ Disallow: /phpBB3/language/ Disallow: /phpBB3/templates/ Disallow: /phpBB3/common.php Disallow: /phpBB3/config.php Disallow: /phpBB3/groupcp.php Disallow: /phpBB3/memberlist.php Disallow: /phpBB3/modcp.php Disallow: /phpBB3/posting.php Disallow: /phpBB3/profile.php Disallow: /phpBB3/privmsg.php Disallow: /phpBB3/viewonline.php Disallow: /phpBB3/search.php Disallow: /phpBB3/faq.php Sitemap: http://www.psychcomp.com/sitemap.xml
About the author: Psychcomp is owned and maintained by Nathan Driskell, a Licensed Professional Counselor - Intern and Internet Addiction Specialist. Nathan has worked for the past five years creating and maintaining websites, and has experience managing small business networks.
Contact Nathan at the following locations:
ndri...@psychcomp.com">E-Mail
Twitter
Related Posts
- Remove Old Links from Google
- PHPBB3 – Private Messaging Error
- Blocking Search Engine Bots
- How to Establish Yourself Online – Part 3: Setting up WordPress
- How to Establish Yourself Online – Part 5: WordPress SEO
- How to Establish Yourself Online – Part 4: Customizing WordPress
- SEO Centro: Rank Checker
- Get Listed in Google Faster
- How to Establish Yourself Online – Part 2: Domain Name and Web Host
- PHP MYSQL Database Connection Function
Tags: phpbb3, robots.txt, SEO, wordpress
Loading ...
