This post is intended for everyone.

These are 2 little known files that the huge majority of webmasters forget to include in their web sites. I think the reason why most people forget about them is that they need to be created manually … none of the HTML editors that I know of create them automatically.

ROBOTS.TXT

The robots.txt is a small text file that you load into the root folder of your website ie. the same folder where your homepage is stored. Its purpose is to tell a search engine spider what it can and cannot index within your site.

Read the rest of this entry