Blocking Search Engine Bots

When you are beginning a new website, the best course of action is to block all bots until the site is complete. As a new domain is created, Google and many of the other search engines will begin to bring bots to your site and list all content. This is not good, as you are under construction, and may not be ready to be indexed yet. There are two solutions to this problem. The first solution is to create a robots.txt file and tell the search engines to not index your site. The second is a more powerful option: using a noindex meta tag.

The first option is simple. First, using notepad, create a new text file. Save it is robots.txt, exactly. Now, in your blank text file, add the following code:

User-agent: *
Disallow: /

The first line states that all search engines will use this rule. The second line will block all access to the root of your directory. This will also include all subdirectories of your site. After this, your site will no longer be indexed. Now, upload the file to the root directory of your website. Do note: once your site is live, you must remove this, or your site will never be indexed.

Now, the second option is to add a meta tag to your pages. It is best to add the meta tag to the header page of your website, assuming your site is using a template based website. Open the header page and within the section, add the following:

<meta name="robots" content="noindex">

This will tell all robots not to index your site. There are two advantages to this: if your site has been listed from a different site, it will no longer be indexed. Second, some web hosts do not allow root level access, which would disallow a robots.txt file.

Be Sociable, Share!

• 

About the author:  Psychcomp is owned and maintained by Nathan Driskell, a Licensed Professional Counselor - Intern specializing in Internet Addiction and Asperger's Disorder. Nathan is also a Web Designer and Network Administrator. Contact Nathan at the following locations: dami...@psychcomp.com">E-Mail Twitter

Related Posts

1. Remove Old Links from Google
2. Example robots.txt for WordPress and PHPBB3
3. Get Listed in Google Faster
4. How to Establish Yourself Online – Part 2: Domain Name and Web Host
5. SEO Centro: Rank Checker
6. How to Establish Yourself Online – Part 5: WordPress SEO
7. How to Establish Yourself Online – Part 3: Setting up WordPress
8. PHPBB3 – Private Messaging Error
9. How to Establish Yourself Online – Part 4: Customizing WordPress
10. PHP MYSQL Database Connection Function

Tags: google, meta tag, robots.txt, SEO

This entry was posted on Friday, August 7th, 2009 at 7:59 PM and is filed under Computers, SEO, Web Design. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.