Search Bot Functionality

[jimmygoon]

I hope that this is within the right scope for discussion here:

(Roughly) how does the Search Bot functionality work? Please just tell me that there are more assurances than simply checking the User Agent string... Does it do some reverse DNS checking... and is that still secure as well? Sorry for the paranoia, but I'm curious.

[DocBoum]

Why are you worried about that? If a user fakes a bot agent, they will see less of the forums, not more. Or am I missing something (e.g. are you worried that a search engine spider might be "disguised" as a real user? In that case, they would be to blame - they get the "more machine readable stuff" if they use the standard UA)? As far as I've digged into the session code, it relies on the UA string. Full stop...

[jimmygoon]

No. Googlebot has more access to my forums than my "guest" user does... and I would rather not have someone fake their UA string to appear as Googlebot and have read access...

[the_dan]

As opposed to just looking at Google's cache?

[jimmygoon]

As opposed to just looking at Google's cache?

That's why I'm a bit confused. I did NOT specify somewhere that I wanted to LET google's search engine be able to see more than anonymous individuals (at least intentionally)... what is the default behavior in this/these circumstances?

[the_dan]

I have no idea how the permissions are assigned after creating a forum, but you certainly can modify them (see the 'Bots' usergroup).

[jimmygoon]

This was on the front page of digg....

http://quicksilverscreen.com/ipb/index. ... opic=29115

concerning no?

[jimmygoon]

Hm, give it a try: http://www.avivadirectory.com/bethebot/

It works, sadly...

[Kellanved]

Well, bots have less access than guests by default and get a stripped down version of the page. I don't really see a problem. You can also specify the IPs for bots in the ACP, if you really want to make it impossible.

I don't see the point, though.

[jimmygoon]

Well, bots have less access than guests by default and get a stripped down version of the page. I don't really see a problem. You can also specify the IPs for bots in the ACP, if you really want to make it impossible.

I don't see the point, though.

http://shcommunity.mickens.us
Try it as a guest, then try it as a bot. Guest have no access permissions, google bot can see everything, (though you have to enter URLs manually), but if I can specify the IP address I suppose that is adequate...