Okay, so I can be slow sometimes but I've only noticed today that the spam that I've been getting since last December and the spam trying to infect a visitor's PC with trojans has quite a few similarities.

First of all, the form and the destination: Most of that spam comes in as a submission for this site's Links section. It's using "Good site", "Thank you", "Great work", and similar phrases as the title for the link target and then crams the link description full with more links. Okay, so the spambot in question has no idea that Links submissions are working differently from, say, comments here.

Other similarities: The spambot always uses "uk" in the HTTP Accept-Language header. Since this is a language header, the "uk" is for the Ukraine, not for the United Kingdom. Which may give us an indication for the source of at least the spambot (not necessarily the spam itself).

The other constant is the User-Agent header: It's always
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) read more

For the last three weeks, I'm counting about a thousand requests from "Exabot/2.0", usually in bursts where it sends one request per second. However, I don't see a single request for our robots.txt from the bot.

In other words: This is yet another clueless bot made by a clueless company that you may want to block.

The requests all seem to come from 193.47.80.42, aka crawl6.exabot.com. Both the IP address and exabot.com are registered to a French company named Exalead, located in Paris.

In their FAQ they state that Exalead's robot engine conforms to the Robot Exclusion standard and the robots META rules. However, as stated above, I can not find any evidence of that in our logfiles. What I did find, however, is another bot, "NG/2.0" coming from the same IP address. That bot does seem to check the robots.txt occasionally (only twice in all of January, to be exact).

Sorry guys, that's not how this works. Go searching for a clue ... read more

Yet another stupid bot that caught my attention: EverbeeCrawler. Now, what's wrong with this request?

213.251.151.102 - - [05/Oct/2005:03:28:20 -0400] "GET /forum/viewtopic.php%253%3?forum%253%3=12%26showtopic%253%3=57796 HTTP/1.1" 400 227 "http://www.geeklog.net:80/forum/" "EverbeeCrawler"

I have no idea what the %253%3 (or the 12%26, for that matter) is supposed to mean or even where it's coming from - certainly not from a link on our site. Also notice the 400 HTTP response code - this broken request caused our webserver to barf. Stupid bot ...

Originally, I noticed this bot because of its tendency to add ":80" to its referrers (as can be seen in the request above). Besides, the actual idea of using referrers (from external sites) is a bad one for a bot.

I've put it in our robots.txt. It took a while, but it did catch on eventually - it's only requesting the robots.txt and nothing else now.

Interestingly enough, you can't find much about that bot on the web. Only lots of sightings in other people's referrer stats (why are those openly on the web, btw?). The IP address (also: 213.251.151.103) belongs to an outfit called "Everbee Networks", located in France, and related to OVH (ovh.net, ovh.com). OVH seems to be a hoster so maybe Everbee Networks is one of their customers.

I should really have noticed this one much earlier. I stumbled across lots of requests from a single IP address, 192.55.214.54, drawn out over a long period of time. The earliest one I could find was back on July 21st. This is apparently some sort of bot and uses "Microsoft_Internet_Explorer_5.00.439rh (fjones@isd.net)" as its user agent string.

The IP address belongs to securecomputing.com, who apparently sell some sort of web filter. So I guess this bot collects data for that filter.

What's annoying about this thing is that it doesn't bother checking the robots.txt. Instead, it only looks for a zzrobots.txt. What sort of nonsense is that? If you want to crawl our sites for commercial reasons, please at least have the decency and follow the standards.

Others have seen this bot, too, and came to the same conclusion: If it doesn't play nicely, block it.

We see another of those fake bots crawling our sites. It claims to be either Googlebot or msnbot and comes from 212.124.92.58 which belongs to Internet Bulgaria, a Bulgarian ISP.

In its msnbot incarnation, it seems to be particularly interested in our RSS feeds, while its Googlebot incarnation just crawls the site, following every link it can find (and ignoring the robots.txt, of course).

Those running Bad Behavior can relax, as it already blocks these sort of fake bots.