How to keep robots out of your web site

THE ROBOTS.TXT FILE

You know that search engines have been created to help people find information quickly on the Internet, and the search engines acquire much of their information through robots (also known as spiders or crawlers), that look for web pages for them.

The spiders or crawlers robots explore the web looking for and recording all kinds of information. They usually start with URL submitted by users, or from links they find on the web sites, the sitemap files or the top level of a site.

Once the robot accesses the home page then recursively accesses all pages linked from that page. But the robot can also check out all the pages that can find on a particular server.

After the robot finds a web page it works indexing the title, the keywords, the text, etc. But sometimes you might want to prevent search engines from indexing some of your web pages like news postings, and specially marked web pages (in example: affiliates pages), but whether individual robots comply to these conventions is pure voluntary.

ROBOTS EXCLUSION PROTOCOL

So if you want robots to keep out from some of your web pages, you can ask robots to ignore the web...