<< Back to Index Step 4: Robots & .htaccess Files
Robots Meta Tag
One meta tag that is worth mentioning is the robots tag. This lets you specify that a particulr page should NOT be indexed by search engines To keep spiders out, simply add this text between your head tags on each page you don't want indexed. The format is shown below:
<HEAD>
<TITLE> Page I don't want In Search Engines </TITLE>
< META NAME="ROBOTS" CONTENT="NO INDEX">
</HEAD>
You do NOT need to use variation of the meta robots tag to help your pages get indexed. They are unneccessary. By default, a spider will try to index all your web pages and will try to follow links from one page to another.
Robots.txt File
Most major search engines support the meta robots tag. However, the robots.txt convention of blocking indexing is more efficient, as you don't need to add tags to each and every page. If you do use the robots.txt file to block indexing, then you do not need to also use the meta robots meta tag.
Here's a site where you can easily generate a robots.txt file:
http://www.webtoolcentral.com/webmaster/tools/robots_txt_file_generator/
<< Back | Next >> |
