Will 2.2 be webspider-friendly?

Patr547 · Post by **Patr547** » Mon Jul 21, 2003 2:37 am

my forum (before i removed it from my server) was indexed by google, and as far as i know, i didn't add any mods which made the site "spiderable".. 8O

Lugie · Post by **Lugie** » Mon Jul 21, 2003 4:24 am

None of mine, on a variaty of hosts, were ever indexed.

Luck of the draw maybe?

GrandiJoos · Post by **GrandiJoos** » Mon Jul 21, 2003 8:30 am

my forum was indexed by google and I didn't install any mods for that purpose. I did add my URL to their database via the "add url" form

Maybe you should try that?

GrandiJoos

Roberdin · Post by **Roberdin** » Mon Jul 21, 2003 9:48 pm

Lugie wrote:None of mine, on a variaty of hosts, were ever indexed. Luck of the draw maybe?

Most search engines (google excuded) hate pages with GET form data (the page.php?variable=something&something=somethingelse) and ignore them.

craven de kere · Post by **craven de kere** » Wed Aug 13, 2003 9:38 am

Gee Acyd, now I know who to blame for Google's pagerank penalization of some phpbb files.

My board gets spidered all day every day, the serverload is high but I will not bite the hand that feeds me.

Tight now i am trying to figure out how to get around the phpbb penalization Google seems to have started using a few days ago (the page rank of all viewtopic.[h[ [ages has sunk dramatically on all phpbb sites that I have checked including phpbb.com).

Tomba · Post by **Tomba** » Fri Aug 22, 2003 10:41 am

ah well... there is already a mod for phpBB 2.0.x to allow google spider the forums, but without generating all different session for the bots. Anyway, can't manage to make it work for me though... :(

The problem is that I now have a site-wide login like phpBB, such that Google can't even spider any of my pages further than the homepage.
Therefore it is a high necessity for a google-friendly version, but as concurrency increases, I'm not sure if I could wait that long :?

anyway, people who don't want their board spidered, use a robots.txt, then everybody is happy

The main solution is not to append the sid to the url string...

psoTFX · Post by **psoTFX** » Fri Aug 22, 2003 12:29 pm

Remember Google isn't the only search engine out there

As I've noted this is on the ideas list for M-3. There are issues of security we need to look at as well as basic implementation.

Peter M Dodge · Post by **Peter M Dodge** » Tue Aug 26, 2003 2:51 pm

psoTFX wrote:Yes, you're quite correct ... it will be. However, in doing so it will generate dozens of new sessions with the same (or similar) IP. The request, and something that is on the "ideas" list for M-3 is to introduce better support for spidering. In this case a method (yet to be determined) would be used to allow Google et al to spider a site without regenerating dozens of IPs, and to (potentially) limit what it can and cannot view (i.e. some kind of dedicated user/group).

Would it be possible to limit it to one session per IP? (this would be good for security reasons IMO too)

Wouldn't completely get rid of the problem, but would help reduce it, and secure the system a bit. It's a bit too easy if you let guests on boards for a person to spam. This is why my flood control is set to 2 minutes

Development Discussion Board

Will 2.2 be webspider-friendly?

Re: Will 2.2 be webspider-friendly?

Re: Will 2.2 be webspider-friendly?

Re: Will 2.2 be webspider-friendly?

Re: Will 2.2 be webspider-friendly?

Re: Will 2.2 be webspider-friendly?

Re: Will 2.2 be webspider-friendly?

Re: Will 2.2 be webspider-friendly?

Re: Will 2.2 be webspider-friendly?