BecomeBot DOES ingore robots.txt

BecomeBot DOES ingore robots.txt | 3 comments | Create New Account

The following comments are owned by whomever posted them. This site is not responsible for what they say.

64.124.85.74 - 64.124.85.74.become.com - and I have them in my robots.txt file...

User-agent: BecomeBot
Disallow:

Off to block their IP's

Authored by: Anonymous on Monday, August 28 2006 @ 00:22 CEST

[ Reply to This | # ]

BecomeBot DOES ingore robots.txt

There should be a slash after the "Disallow:" (as quoted in the article above). Otherwise, you're inviting them to spider your site.

As I wrote above, the real BecomeBot seems to be behaving properly as long as you use the "long" form of the robots.txt, i.e. with one Disallow: statement for each bot you want to block.

Authored by: Dirk on Monday, August 28 2006 @ 09:37 CEST

[ Reply to This | # ]

Topics

User Functions

Damn Spam!
	Search