|
Ann Elisabeth has started a Wiki page collecting information on OmniExplorer_Bot and the persons and company behind it.
Since we (i.e. www.geeklog.net) seemed to be the only ones so far to actually see referrer spam from OmniExplorer_Bot and the IP addresses it uses, I decided to revisit the evidence we had - and found that I was wrong. The bot is not doing referrer spam.
The problem is with the bot's unique (among bots) feature to include referrers in its requests, i.e. to show where it's actually coming from.
And here's how and why I came to the wrong conclusion: 65.19.150.210 - - [09/Jun/2005:23:42:06 -0400] "GET / HTTP/1.1" 403 26 "http://www.aahhh.org" "OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) personals Crawler"
65.19.150.210 - - [09/Jun/2005:23:42:07 -0400] "GET / HTTP/1.1" 403 26 "http://www.aahhh.org/help/advancedsearch.html" "OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) personals Crawler"
So this is the bot, obviously, coming from www.aahhh.org, which is running Geeklog. It's someone's personal site and the bot simply follows the links back to www.geeklog.net on those two pages. Fine.
This, however, is the bot, from the same IP address, apparently doing referrer spam:
65.19.150.210 - - [08/Jun/2005:05:02:24 -0400] "GET / HTTP/1.1" 403 26 "http://drunk-girls-at-mardi-gras-galleries.gfdo43.info" "OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) personals Crawler"
65.19.150.210 - - [08/Jun/2005:08:01:39 -0400] "GET / HTTP/1.1" 403 26 "http://aria-giovanni-bondage.prgnix.info" "OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) personals Crawler"
65.19.150.210 - - [08/Jun/2005:22:17:35 -0400] "GET / HTTP/1.1" 403 26 "http://long-lesbian-movies.fd6765.info" "OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) personals Crawler"
65.19.150.210 - - [10/Jun/2005:03:02:54 -0400] "GET / HTTP/1.1" 403 26 "http://amateur-lesbo.reh9ge.info" "OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) personals Crawler"
Those sites are not running Geeklog. Three of them have a JavaScript redirect to just-sex.us, while the other redirects to top-5-search.info (with an affiliate ID) and from there to only-best-results.com. So this looks like a perfect example of referrer spam, doesn't it? Unfortunately, this is where I was wrong ...
If you visit those sites with JavaScript turned off (or using a text browser, such as Lynx), you'll see what every bot (which usually doesn't interpret JavaScript) sees: Lots of links to porn sites. But further down the page, there are also some links to non-porn sites: The W3C XHTML specification, for example, Microsoft's Internet Explorer homepage - and www.geeklog.net!
So this is where the apparent referrer spam came from: OmniExplorer_Bot didn't see the JavaScript redirect and instead followed all the links on that site, including the one to www.geeklog.net. And since it's always sending referrers, it looked like it was referrer spamming itself. Whether this was a deliberate attempt to draw attention to the site or just to make the site look like it had at least some proper content is everybody's guess. Since Microsoft and the W3C probably don't check their referrer logs too often, I tend to think it was the latter.
Well, I guess this is why Googlebot and all the other bots of the established search engines don't send referrers (as much as webmasters would like it sometimes, to see who's linking to them): Because it can (and will!) be misused.
Instead of apologizing to the makers of OmniExplorer_Bot, I'd rather like to apologize to all those who read my previous rant for doing sloppy research (again).
I still think that this bot is "evil" (just not that evil) and should be blocked. It does ignore the robots.txt and can cause huge amounts of traffic. And while it's not referrer spamming itself, it can easily be misused to do that (as demonstrated above). This is not the way to run a search engine business.
In the meantime, the bot has begun to hide its traces:
65.19.150.210 - - [12/Jun/2005:00:46:01 -0400] "GET / HTTP/1.1" 403 26 "http://male-swimmers-naked.9032rd.info" "Mozilla/5.0 (X11; U; Linux i686; fr-FR; rv:1.5)"
65.19.150.210 - - [12/Jun/2005:01:15:04 -0400] "GET / HTTP/1.1" 403 26 "http://www.experimentalkitchen.org/help/advancedsearch.html" "Mozilla/5.0 (X11; U; Linux i686; fr-FR; rv:1.5)"
This is still the same IP address as above, but instead of using its own user agent string, it's now claiming to be a French version of Mozilla 1.5. Otherwise, it's still the same: experimentalkitchen.org is really running Geeklog, while that porn site has a JavaScript redirect to just-sex.us and a link to www.geeklog.net buried between porn links.
Let's have a look at just-sex.us then: It's registered to a Ronald Walthy, using an apparently false address (city: Ontario, state: ON, country: United States). The site is hosted with AiNET Hosting Operations (205.134.180.254). The server also appears to be hosting two unrelated domains (one of which has just expired).
The site itself is a standard porn site with "appetizers" and asking you to "join now". It also uses some classic JavaScript tricks, e.g. opening a popup for www.edfdfdf.cc on exit (interestingly enough, that domain does not seem to be registered).
More on those unpronouncable sites (gfdo43.info, 9032rd.info, etc.) can be found in my earlier post and in Ann Elisabeth's further analysis.
|