Could The New Google Spider Be Causing Issues With Websites?
Around the time Google announced “Big Daddy,” there was a new Googlebot roaming the web. Since then I’ve heard stories from clients of websites and servers going down and previously unindexed content getting indexed.
I started digging into this and you’d be surprised at what I found out.
First, let’s look at the timeline of events:
In Late September some astute spider watchers over at Webmasterworld spotted unique Googlebot activity. In fact, it was in this thread: http://www.webmasterworld.com/forum3/25897-9-10.htm that the bot was first reported on. It concerned some posters who thought that perhaps this could be regular users masquerading as the famous bot.
Early on it also appeared that the new bot wasn’t obeying the Robots.txt file. This is the protocol which allows or denies crawling to parts of a website.
Speculation grew on what the new crawler was until Matt Cutts mentioned a new Google test data center http://www.mattcutts.com/blog/good-magazines/#comment-5293. For those that don’t know, Matt Cutts is a senior engineer with...
To view and download this full PLR article, you must be logged in. Registration is completely free. Once you create your account, you will be able to browse, search & downlod from our PLR articles database of over "1,57,897+" on 1,000's of niches and 200+ categories without paying a penny. Click here to signup...
