Search Bot Functionality

Discussion of general topics related to the new version and its place in the world. Don't discuss new features, report bugs, ask for support, et cetera. Don't use this to spam for other boards or attack those boards!
Forum rules
Discussion of general topics related to the new release and its place in the world. Don't discuss new features, report bugs, ask for support, et cetera. Don't use this to spam for other boards or attack those boards!
jimmygoon
Registered User
Posts: 75
Joined: Thu Jun 23, 2005 3:59 am

Search Bot Functionality

Post by jimmygoon »

I hope that this is within the right scope for discussion here:

(Roughly) how does the Search Bot functionality work? Please just tell me that there are more assurances than simply checking the User Agent string... Does it do some reverse DNS checking... and is that still secure as well? Sorry for the paranoia, but I'm curious.
DocBoum
Posts: 29
Joined: Thu Jul 20, 2006 10:43 am

Re: Search Bot Functionality

Post by DocBoum »

Why are you worried about that? If a user fakes a bot agent, they will see less of the forums, not more. Or am I missing something (e.g. are you worried that a search engine spider might be "disguised" as a real user? In that case, they would be to blame - they get the "more machine readable stuff" if they use the standard UA)? As far as I've digged into the session code, it relies on the UA string. Full stop...
jimmygoon
Registered User
Posts: 75
Joined: Thu Jun 23, 2005 3:59 am

Re: Search Bot Functionality

Post by jimmygoon »

No. Googlebot has more access to my forums than my "guest" user does... and I would rather not have someone fake their UA string to appear as Googlebot and have read access...
User avatar
the_dan
Registered User
Posts: 700
Joined: Thu Apr 01, 2004 7:36 pm

Re: Search Bot Functionality

Post by the_dan »

As opposed to just looking at Google's cache?
jimmygoon
Registered User
Posts: 75
Joined: Thu Jun 23, 2005 3:59 am

Re: Search Bot Functionality

Post by jimmygoon »

the_dan wrote:As opposed to just looking at Google's cache?
That's why I'm a bit confused. I did NOT specify somewhere that I wanted to LET google's search engine be able to see more than anonymous individuals (at least intentionally)... what is the default behavior in this/these circumstances?
User avatar
the_dan
Registered User
Posts: 700
Joined: Thu Apr 01, 2004 7:36 pm

Re: Search Bot Functionality

Post by the_dan »

I have no idea how the permissions are assigned after creating a forum, but you certainly can modify them (see the 'Bots' usergroup).
jimmygoon
Registered User
Posts: 75
Joined: Thu Jun 23, 2005 3:59 am

Re: Search Bot Functionality

Post by jimmygoon »

This was on the front page of digg....

http://quicksilverscreen.com/ipb/index. ... opic=29115

concerning no?
jimmygoon
Registered User
Posts: 75
Joined: Thu Jun 23, 2005 3:59 am

Re: Search Bot Functionality

Post by jimmygoon »

Hm, give it a try: http://www.avivadirectory.com/bethebot/

It works, sadly...
User avatar
Kellanved
Former Team Member
Posts: 407
Joined: Sun Jul 30, 2006 4:59 pm
Location: Berlin

Re: Search Bot Functionality

Post by Kellanved »

Well, bots have less access than guests by default and get a stripped down version of the page. I don't really see a problem. You can also specify the IPs for bots in the ACP, if you really want to make it impossible.

I don't see the point, though.
No support via PM.
Trust me, I'm a doctor.
jimmygoon
Registered User
Posts: 75
Joined: Thu Jun 23, 2005 3:59 am

Re: Search Bot Functionality

Post by jimmygoon »

Kellanved wrote:Well, bots have less access than guests by default and get a stripped down version of the page. I don't really see a problem. You can also specify the IPs for bots in the ACP, if you really want to make it impossible.

I don't see the point, though.
http://shcommunity.mickens.us
Try it as a guest, then try it as a bot. Guest have no access permissions, google bot can see everything, (though you have to enter URLs manually), but if I can specify the IP address I suppose that is adequate...
Post Reply