Almost all net users search for solutions and information by using search engines. With search engines like Google, Yahoo, MSN, Ask Jeeves, and AOL Search, its difficult to think of any other way of finding information on the internet.
Before search engines and the internet, it was not so easy to get information. Today, search engines can instantly return uncannily accurate results instantly from a selection of millions of websites.
Search engines are able to catalog these sites using either crawler-based technology or human powered directories. Before October of 2002, Yahoos directory was entirely indexed by hand. Now its mainly only small portal sites that build their directories by hand based on human judgment.
The large search engines like Google use web crawlers that crawl through the internet indexing the content on the pages. Each time they encounter a link, the crawler splits and another crawls through the linked site.
Google is a pioneer in crawler-based technology. Its crawlers, called Googlebots, spider the web constantly and harvest web pages for the input of the main Google database. Once crawlers return the information, Google uses...