For the last three weeks, I'm counting about a thousand requests from "Exabot/2.0", usually in bursts where it sends one request per second. However, I don't see a single request for our robots.txt from the bot.
In other words: This is yet another clueless bot made by a clueless company that you may want to block.
The requests all seem to come from 193.47.80.42, aka crawl6.exabot.com. Both the IP address and exabot.com are registered to a French company named Exalead, located in Paris.
In their FAQ they state that Exalead's robot engine conforms to the Robot Exclusion standard and the robots META rules
. However, as stated above, I can not find any evidence of that in our logfiles. What I did find, however, is another bot, "NG/2.0" coming from the same IP address. That bot does seem to check the robots.txt occasionally (only twice in all of January, to be exact).
Sorry guys, that's not how this works. Go searching for a clue ...
P.S. Ann Elisabeth wasn't too happy with a previous incarnation of that bot either.
Comments (0)
Damn Spam!
http://spam.tinyweb.net/article.php/stupid-bots-exabot-ng