Will 2.2 be webspider-friendly?

Discuss features as they are added to the new version. Give us your feedback. Don't post bug reports, feature requests, support questions or suggestions here.
Forum rules
Discuss features as they are added to the new version. Give us your feedback. Don't post bug reports, feature requests, support questions or suggestions here. Feature requests are closed.
Patr547
Registered User
Posts: 4
Joined: Sat Mar 01, 2003 3:24 am
Contact:

Re: Will 2.2 be webspider-friendly?

Post by Patr547 »

my forum (before i removed it from my server) was indexed by google, and as far as i know, i didn't add any mods which made the site "spiderable".. 8O
Pat
User avatar
Lugie
Registered User
Posts: 163
Joined: Sat Nov 16, 2002 9:56 pm
Location: Wisconsin
Contact:

Re: Will 2.2 be webspider-friendly?

Post by Lugie »

None of mine, on a variaty of hosts, were ever indexed. :? Luck of the draw maybe?
I can't see the meaning of this life I'm leading,
I try forget you as you forgot me...
This time there is nothing left for you to take -
this is goodbye...
GrandiJoos
Registered User
Posts: 40
Joined: Mon Jun 23, 2003 3:20 pm
Location: Breda, The Netherlands
Contact:

Re: Will 2.2 be webspider-friendly?

Post by GrandiJoos »

my forum was indexed by google and I didn't install any mods for that purpose. I did add my URL to their database via the "add url" form ;) Maybe you should try that?

GrandiJoos :cool:
What will Christmas bring???
Roberdin
Registered User
Posts: 1546
Joined: Wed Apr 09, 2003 8:44 pm
Location: London, United Kingdom

Re: Will 2.2 be webspider-friendly?

Post by Roberdin »

Lugie wrote:None of mine, on a variaty of hosts, were ever indexed. :? Luck of the draw maybe?
Most search engines (google excuded) hate pages with GET form data (the page.php?variable=something&something=somethingelse) and ignore them.
Rob
craven de kere
Registered User
Posts: 7
Joined: Wed May 21, 2003 8:45 am

Re: Will 2.2 be webspider-friendly?

Post by craven de kere »

Gee Acyd, now I know who to blame for Google's pagerank penalization of some phpbb files. :evil:

My board gets spidered all day every day, the serverload is high but I will not bite the hand that feeds me.

Tight now i am trying to figure out how to get around the phpbb penalization Google seems to have started using a few days ago (the page rank of all viewtopic.[h[ [ages has sunk dramatically on all phpbb sites that I have checked including phpbb.com).
Tomba
Registered User
Posts: 17
Joined: Sat Apr 12, 2003 1:59 pm

Re: Will 2.2 be webspider-friendly?

Post by Tomba »

ah well... there is already a mod for phpBB 2.0.x to allow google spider the forums, but without generating all different session for the bots. Anyway, can't manage to make it work for me though... :(

The problem is that I now have a site-wide login like phpBB, such that Google can't even spider any of my pages further than the homepage.
Therefore it is a high necessity for a google-friendly version, but as concurrency increases, I'm not sure if I could wait that long :?

anyway, people who don't want their board spidered, use a robots.txt, then everybody is happy

The main solution is not to append the sid to the url string...
User avatar
psoTFX
Registered User
Posts: 1984
Joined: Tue Jul 03, 2001 8:50 pm
Contact:

Re: Will 2.2 be webspider-friendly?

Post by psoTFX »

Remember Google isn't the only search engine out there ;) As I've noted this is on the ideas list for M-3. There are issues of security we need to look at as well as basic implementation.
User avatar
Peter M Dodge
Registered User
Posts: 40
Joined: Tue Aug 26, 2003 2:12 pm

Re: Will 2.2 be webspider-friendly?

Post by Peter M Dodge »

psoTFX wrote:Yes, you're quite correct ... it will be. However, in doing so it will generate dozens of new sessions with the same (or similar) IP. The request, and something that is on the "ideas" list for M-3 is to introduce better support for spidering. In this case a method (yet to be determined) would be used to allow Google et al to spider a site without regenerating dozens of IPs, and to (potentially) limit what it can and cannot view (i.e. some kind of dedicated user/group).
Would it be possible to limit it to one session per IP? (this would be good for security reasons IMO too)

Wouldn't completely get rid of the problem, but would help reduce it, and secure the system a bit. It's a bit too easy if you let guests on boards for a person to spam. This is why my flood control is set to 2 minutes :?
Post Reply