The role of the robots.txt file to improve site ranking!
Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!
The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.
Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.
The robots.txt file should be placed in the directory where your main files for your hosting...
To view and download this full PLR article, you must be logged in. Registration is completely free. Once you create your account, you will be able to browse, search & downlod from our PLR articles database of over "1,57,897+" on 1,000's of niches and 200+ categories without paying a penny. Click here to signup...
