One of the most fundamental aspects of search engine optimisation (SEO) is ensuring that the pages within your website are as accessible as possible to the search engines. It’s not only the homepage of a website that can be indexed, but also the internal pages within a site’s structure. The internal pages of a site often contain important content such as products, services or general information, and therefore can be uniquely optimised for related terms. As a result, easy access to these pages is vital.
There are many do’s and don’ts involved in ensuring all of your pages can be found by search engines. However, it is important to first establish how the search engines find and index web pages.
Search engines use “robots” (also known as “bots” or “spiders”) to find content on the web for inclusion in their index. A robot is a computer programme that can follow the hyperlinks on a web page, which is known as “crawling”. When a robot finds a document it includes the contents within the search engine’s index, then follows the next links it can find and continues the process of crawling and...