The MSNBot is eating bandwidth!

[Johnny]

At my site, the MSNBot uses approx 150mb(!) of bandwidth every month, while the Google one only use 2mb or so. Is this the way it's supposed to be? Can anything be done to stop it?

[tanfwc]

Block the MSNBot using robots.txt

[Johnny]

yes, but won't that block all bots??

[DeadEye686]

yes, but won't that block all bots??

Not if you use the msnbot user-agent:

Code: Select all

User-agent: msnbot
Disallow: /

I have the exact same problem with MSNbot, it is absolutely disgusting. But hey, not much more can be expected from Microsoft.

[olger901]

The Bot iss a Microsoft Product, hey what did you expect it must have seen phpBB, so hey opensource lets dos that site

[tanfwc]

MSNbot did not really give me much headache. There is one time MSNbot visited my site for more than 50 times.

[Johnny]

i'll try what DeadEye686 said. hope it works! thanks for the tip

the MSNBot visits my domain well over a 100 times a month (every month in 2004), and on an average uses approx 7000 hits each month

[battye]

It's always google on my site now, haven't seen msnbot in ages

[Davus]

Ahh I know what you mean, I got MSN bot caught in my /manual/ MANY times. I didn't even link to it!
So I iptables'd it out.

iptables -I INPUT -s 000.000.000.000 -j DROP
iptables -I OUTPUT -s 000.000.000.000 -j DROP

In case anyone with root on a box wants to kill it. Don't know its IP address, replace 000.000.000.000 accordingly. But yeah, robots method is a bit more orthodox.

[stubbers]

I'm to busy trying to get bots to visit my site, they don't like me. I just seem to be a bot reppelant