|
Not everything that looks like Googlebot is the real thing. One example of a bot that claims to be Googlebot but isn't comes from 69.64.49.243 (air840.startdedicated.com), which is apparently a dedicated server hosted with Server4You Inc., Missouri.
My suggestion: Block that IP. There's no good reason to claim being something you aren't. Fake:
air840.startdedicated.com - - [21/May/2005:09:28:45 +0200] "GET / HTTP/1.1" 200 19079 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
air840.startdedicated.com - - [21/May/2005:22:49:58 +0200] "GET / HTTP/1.1" 200 19509 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
The real Googlebot:
crawl-66-249-64-44.googlebot.com - - [22/May/2005:04:41:56 +0200] "GET / HTTP/1.0" 200 19466 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
crawl-66-249-71-55.googlebot.com - - [22/May/2005:04:52:06 +0200] "GET /robots.txt HTTP/1.0" 200 218 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
|