Will 2.2 be webspider-friendly?

[njtom]

Will phpBB 2.2 be more compatible w/ google, etc? I know that the current version needs a small mod to allow them..?

[stiffler]

I also support a more webspider-friendly version of phpbb.

[Wert]

I'm hoping it won't be. These bots can really hit your forums hard and use up a lot of systems reasources. (not good if you're on shared hosting)

But if it is, no big deal as I'll use "Robots.txt" to tell search bots not to crawl my forums.

[Techie-Micheal]

Unless I'm missing something, I haven't had any trouble with bots spidering my board and without any modifications to my board ...

[Lugie]

njtom wants spiders to crawl his board. So you two have different problems.

Right now spiders don't crawl phpBB for a reason I am uneducated on.

[Acyd Burn]

My personal board was indexed quite havily by google, that havily i had to block the googlebot IP.

Submitting a bug report to them helped a little bit but not ironed out the main problems. The googlebot produced very high load on the server.

[Techie-Micheal]

njtom wants spiders to crawl his board. So you two have different problems.

Right now spiders don't crawl phpBB for a reason I am uneducated on.

I know. And I am saying my board is indexed without any MODs.

[psoTFX]

Yes, you're quite correct ... it will be. However, in doing so it will generate dozens of new sessions with the same (or similar) IP. The request, and something that is on the "ideas" list for M-3 is to introduce better support for spidering. In this case a method (yet to be determined) would be used to allow Google et al to spider a site without regenerating dozens of IPs, and to (potentially) limit what it can and cannot view (i.e. some kind of dedicated user/group).

[Wert]

My personal board was indexed quite havily by google, that havily i had to block the googlebot IP.

Submitting a bug report to them helped a little bit but not ironed out the main problems. The googlebot produced very high load on the server.

Same here. Before I implemented 'Robots.txt' to limit google, it wasn't uncommon for it to open 20-30 processes when spidering. Not optimal on a shared server.

[avis]

Hi Acyd!

Whats the address of your website which was heavily indexed by google ? Coz i want to check the meta tags etc. so i can get my forums in googleand other search engines too!

Thanks!