Improving phpBB performance for small installations

General discussion of development ideas and the approaches taken in the 3.x branch of phpBB. The current feature release of phpBB 3 is 3.3/Proteus.
Forum rules
Please do not post support questions regarding installing, updating, or upgrading phpBB 3.3.x. If you need support for phpBB 3.3.x please visit the 3.3.x Support Forum on phpbb.com.

If you have questions regarding writing extensions please post in Extension Writers Discussion to receive proper guidance from our staff and community.
TerryE
Registered User
Posts: 95
Joined: Sat May 23, 2009 12:24 am
Contact:

Improving phpBB performance for small installations

Post by TerryE »

This topic is really about a phpBB application design goal, and that is the issue of application performance and responsiveness. I feel that good performance should be a non-functional requirement (NFR) for this application, and I suspect that most of the developers would agree with this. However, where we might begin to differ is just what constitutes good performance and what use-cases should be used to baseline measures.

I am raising these issues (i) from a basis of some expertise and (ii) because I have implemented and benchmarked most of the points (that I discuss here in a series of detailed articles).

Key to these discussions is an observation that the phpBB installed community falls largely into two camps: large-scale forums running on dedicated servers or VMs and where the admin has root access; and the large bulk of forums (perhaps 95% by number) which run on a low-cost ISP shared-service templates. In the former case, such site admins usually tune up the system to use mod_php or FastCGI and PHP opcode caches such as APC or Xcache, etc.. In the latter “95%” case, the forum installers have none of these freedoms and have to take the Apache/IIS service as offered by their hosting supplier, have limited tuning through .htaccess files, etc.

The current phpBB implementation has some features that lead to unnecessarily poor performance in this second case that could easily be addressed within an evolving phpBB architecture, baring in mind the key characteristics of this type of service:
  • The PHP RTS is activated per HTTP request in the account’s UID.
  • No code caching is available, so the entire included module set needs reading in and compiling per request.
  • These systems are nearly always physical disk I/O bound, so any read requests which are not already in the file-system cache tend to kill performance. The rough ratios are in the time that it takes to read in one file from disk, the system can compile ~10-20K lines of source code or execute ~500K lines of PHP code.
Given this, I would observe that the following can result in up to a 2-3x speed-up forum performance for such systems :
  • Basic webserver tuning though the .htaccess file to ensure that all compression and local caching possible occurs. OK this isn’t coding issue, but we don’t even have a 101 article on this in the kBase.
  • The current file cache uses a one file-per-object policy and this performs terribly on shared service hosts. (It also performs terribly on larger systems as well, but at least memory based alternatives exist for these.) I’ve developed globfile cache which uses a simple single compressed data file. This can improve request response times by up to 30% on this type of system. (This is also 3.0.x compliant.)
  • Code globbing – that is dynamically compiling a set main-path includes into a single composite file – can generate similar response savings, and again I have a 3.0.8 demonstrator of this working.
These changes aren’t really about coding efficiency, since given the MIP rate of current PHP interpreters on current H/W, this isn’t a major issue for small installations, and also a significant percentage of the codebase is rarely executed. However going through this exercise, it is clear to me that:
  • The retention of support for PHP 4.3 constrains the implementation to the PHP 4 object model. Who will run PHP 4 for phpBB 3.1 let alone 3.2? This should be dropped and a proper PHP 5 standard be implemented.
  • the modulariation / include strategy is sloppy and this needs to be overhauled by a coherent and standard approach to encapsulation. This work should be done for 3.2.
  • the template engine implementation is a mess and really needs rewriting.
  • the session.php is “hot” yet it has debris and convolved coding in it. This is due for a thorough review.
For more details, see my articles, but if anyone wants to discuss it here then I can hoist the relevant material here. Comments gratefully received.
Oleg
Posts: 1150
Joined: Tue Feb 23, 2010 2:38 am
Contact:

Re: Improving phpBB performance for small installations

Post by Oleg »

This topic is interesting but it lacks "actionable items". To respond to your list of requested/proposed (?) changes:
TerryE wrote: The retention of support for PHP 4.3 constrains the implementation to the PHP 4 object model. Who will run PHP 4 for phpBB 3.1 let alone 3.2? This should be dropped and a proper PHP 5 standard be implemented.
phpbb 3.1 will require (and take advantage of features in) php 5.2.
TerryE wrote: the modulariation / include strategy is sloppy and this needs to be overhauled by a coherent and standard approach to encapsulation. This work should be done for 3.2.
Patches are welcome.
TerryE wrote: the template engine implementation is a mess and really needs rewriting.
This is currently scheduled for 3.1, and assistance/patches are again welcome. viewtopic.php?f=84&t=33423
TerryE wrote: the session.php is “hot” yet it has debris and convolved coding in it. This is due for a thorough review.
Please feel free to propose an RFC or a patch.
TerryE wrote:This topic is really about a phpBB application design goal, and that is the issue of application performance and responsiveness. I feel that good performance should be a non-functional requirement (NFR) for this application, and I suspect that most of the developers would agree with this. However, where we might begin to differ is just what constitutes good performance and what use-cases should be used to baseline measures.
My impression of phpbb is that the goal is to have good performance in the majority of cases, while retaining ease of installation and administration and working within constraints of shared hosting. We are aware of "tweaks for large forums" topics but we do not automatically incorporate all such improvements into the core.
TerryE wrote:I am raising these issues (i) from a basis of some expertise and (ii) because I have implemented and benchmarked most of the points (that I discuss here in a series of detailed articles).
I understand that you spent a good amount of time writing those articles, but it would be nice if you linked to specific relevant section(s) of them instead of asking the reader to process them all.

If you have patches against phpbb that will improve performance please post them and we can discuss specifics.
TerryE
Registered User
Posts: 95
Joined: Sat May 23, 2009 12:24 am
Contact:

Re: Improving phpBB performance for small installations

Post by TerryE »

We probably need separate topics for the separate aspects, otherwise this dialogue will get too convolved. As I said in one of my articles, I am not just talking about (small) percentage improvements in response time for small sites, I am talking factors. My fundamental questions for this topic are ones of principle:
  1. Does the development team understand that the runtime characteristics of phpBB on a shared service are fundamentally different to phpBB running on a dedicated system or VM?
  2. Should we not consider some relatively small changes or options which will make the application a lot more responsive for users of installations in this first category?
User avatar
DavidIQ
Customisations Team Leader
Customisations Team Leader
Posts: 1864
Joined: Thu Mar 02, 2006 4:29 pm
Location: Earth
Contact:

Re: Improving phpBB performance for small installations

Post by DavidIQ »

TerryE wrote:[*] Should we not consider some relatively small changes or options which will make the application a lot more responsive for users of installation in this first category?[/list]
since this first category (shared hosting) most likely consists the largest group of our users I would say definitely yes. Any sort of performance improvements in that category would be great.
Image
Oleg
Posts: 1150
Joined: Tue Feb 23, 2010 2:38 am
Contact:

Re: Improving phpBB performance for small installations

Post by Oleg »

This post is my personal opinion, not necessarily shared by other developers.
TerryE wrote:Does the development team understand that the runtime characteristics of phpBB on a shared service are fundamentally different to phpBB running on a dedicated system or VM?
I have never used shared hosting to host anything. Thus I have no personal experience with "the runtime characteristics of phpBB on a shared service". I will consider improvement proposals raised with respect to performance on shared hosts, same as any other improvement proposals.

At the same time I used a bunch of phpbb boards over the years, and I am not currently aware of any show-stopping performance issues with phpbb on shared hosting.
TerryE wrote: Should we not consider some relatively small changes or options which will make the application a lot more responsive for users of installations in this first category?
I will consider changes that make the application more responsive for any subset of users.
TerryE
Registered User
Posts: 95
Joined: Sat May 23, 2009 12:24 am
Contact:

Re: Improving phpBB performance for small installations

Post by TerryE »

nn- wrote:I have never used shared hosting to host anything.
I must admit that until I got my personal hobby domain. I was the same. Until then I only worked with dedicated servers and server farms -- and later VMs under VMware ESX, though I now only use VirtualBox for personal use. I suspect that most developers and admins are the same. But that -- by and large -- is not the user community which uses phpBB, and the reason for this is pretty simple (I'm quoting a hosting provider which is a market leader; other providers have comparable prices ratios):
  • You can buy a shared service(1) for $50 p.a. You don't need any knowledge of LAMP or IIS, though it does help to get to grips with .htaccess foibles. This is perfectly adequate to host a typical hobby forum.
  • You can buy an entry-level VM(2) for $320 p.a. This is a bare (virtual) server, so you do need a minimum sysadmin expertise in LAMP or W2K8/IIS, and yes, if properly tuned it can support a 1-5 page requests/sec forum and you can scale up the price range/performance getting better concurrencies / memory etc.
I've written some articles which drill down into more depth on the difference in characteristics of these two installation use cases ("Use cases for phpBB") and a fuller list in Keyword PHP where the titles are pretty self explanatory). But in short, the key issues are:
  • Every PHP script requires PHP image startup (in the accounts UID).
  • No opcode caches (APC etc.) are available so all included files / source lines need to be compiled on each request.
  • The actual execution of the PHP bytecode itself is a small percentage of the total request servicing timeline. (PHP image activation is about the same CPU time as the interpreter executing ~50K source lines).
  • About the only useful caching that occurs is the server's filesystem cache. This help avoid physical I/O overheads on adjacent requests, but this cache is rapidly churned on busy servers so that the first request in, say, 5 mins will involve a lot of physical I/Os.
  • It's essential to set up your .htaccess correctly to get your caching and compression correct.
I must underline this issue of physical I/Os and their avoidance. Most shared servers are I/O bound. In the last decade, servers have increased in straight CPU performance by perhaps 30x (more if you allow for the multi-core development), but physical I/0 throughput by at most 50-100%. The current 3.0.8 version requires R/W to 32 application-specific files (3) just to execute a basic viewforum (and double this if the style is not locally cached on the browser).

My alpha demonstrator based on 3.0.8 get this down to 8 (4) and if I rewrote the templating engine using the new 3.1 coding guideline / PHP 5.2 style, I could drop this to 5 files. This isn't theory. This was 3 days of coding changes on the 3.0.8 code base, (most was bulk tweaks using tokeniser based filters). I could send you the diffs, but my intention here was a demonstrator. Doing this robustly will be a LOT easier with the new Ascraeus coding standard compliant code base. But in simple terms, on first execution after cache purge, each function such as viewforum dynamically builds a compressed code glob of all included files which is then preloaded with a single require as part of the preamble to subsequent execution

Code: Select all

require( "compress.zlib:///$loader_module_file" );
With the classes now defined the autoloader is simply bypassed, except in the case of additional objects which are required non-main path. This approach significant gains and consistency in request performance for this shared service use case and no material application changes. This behaviour is enabled by a configuration setting.

I've attached my beta 3.0.8 code for both my ACM/globfile and phpbb_module_loader to show the sort of approach adopted here.

[edit:] Note that I have a new version of ACM globfile called ACM unifile which reflects nn-'s feedback on this later post in this topic
Attachments
phpbb_load_module.zip
phpbb_load_module
(2.3 KiB) Downloaded 495 times
acm_globfile.zip
ACM globfile (now superseded)
(2.62 KiB) Downloaded 505 times
Last edited by TerryE on Wed Mar 23, 2011 1:05 am, edited 2 times in total.
User avatar
callumacrae
Former Team Member
Posts: 1046
Joined: Tue Apr 27, 2010 9:37 am
Location: England
Contact:

Re: Improving phpBB performance for small installations

Post by callumacrae »

nn- wrote:I have never used shared hosting to host anything. Thus I have no personal experience with "the runtime characteristics of phpBB on a shared service". I will consider improvement proposals raised with respect to performance on shared hosts, same as any other improvement proposals.
I'm staff for a (free) shared hosting company who have an auto suspension script in place for people who use too much resources, and I have never seen a suspension for phpBB, while I have seen thousands (probably literally) for WordPress (admittedly, mostly the plugins such as sitemap generators) and SMF. phpBB is definitely one of the better performing scripts, but that doesn't mean there isn't room for improvement and I would always be happy to help if someone told me what to do :)

~Callum
Made by developers, for developers!
My blog
Oleg
Posts: 1150
Joined: Tue Feb 23, 2010 2:38 am
Contact:

Re: Improving phpBB performance for small installations

Post by Oleg »

Edit: I think there is one problem. On a moderately busy board the removal of data from cache may not work correctly since concurrent processes can put the old data back. If this affects things like authorization data consequences could be severe. With this in mind I'm not sure the proposed implementation should be endorsed by us.

Original post:

I think the acm implementation proposed here may be included as it essentially does not negatively affect anything.

I want to note here that due to additional compression and decompression performed the amount of CPU time credited to php processes will go up for boards choosing to use it. I have seen topics in .com support forums where hosts limit phpbb based on cpu time used, but I do not believe I have ever seen restrictions based on the amount of I/O done.

The file/module should be renamed to something like "single file" since glob stands for something else.

On the actual code:

- Indentation and whitespace needs to be fixed in a number of places to comply with coding guidelines
- Trailing whitespace needs to be removed on some lines.
- die() call needs to be split into two calls, one for each reason.
- Comments need to be switched from # to // and go on their own line.
- Some if bodies are on the same line as conditionals.
- Whitespace removal in queries needs to be taken out. You cannot know whether such whitespace is significant.
- ?> needs to be deleted.
- Adjustments may be needed for current develop-olympus and most probably current develop.

The load module stuff I do not completely understand yet.
TerryE
Registered User
Posts: 95
Joined: Sat May 23, 2009 12:24 am
Contact:

Re: Improving phpBB performance for small installations

Post by TerryE »

You have floated a "problem", but I think more in terms of requirements and whether the implementation addresses them. This one is a cache coherency requirement, that's all. When and why must cache coherency be maintained?

I think that it's reasonable for me to observe that phpBB already ignores such coherency issues across-the-board, and it is moot as to whether fileglobbing of the cache data materially increases the chances of a coherency failure happening in the use cases where it would be deployed.

Why do I say this? All cached data elements have a TTL which is established when the cached data is read. If the data is stale then the data is reacquired or the original source and a new copy written back to the cache. Whilst the application uses flock() over the write process, to maintain coherency you would need to flock over the end-to-end read + write, or at least use optimistic locking or collision detection.

It would be trivial to add optimistic locking or collision detection to any of the ACM methods such as the file or globfile modules, but there are one of two modes of response to collision: (i) ignore the write on collision; (ii) retry until successful on collision; and I suspect that we would want (ii) for any ACP request and (i) otherwise.

So let's establish the requirement, propose an implementation and decide which branch that we're going to regress the change into. Please let us avoid using the label "problem" to dismiss the opportunity offer a speed up performance for the vast majority of users. This is really a very specific issue so I've openned a new thread on this.

Re the coding standard conformance. Yup this needs to be done. Re the CPU overhead issues of compressing, I discussed this in depth in one of the articles that I referenced. The overhead of decompression is smaller that of the system overhead for accessing all the individual files -- put roughly IIRC, it's a case of -10mS + 1mS giving a net saving in CPU time and let's not forget the odd few seconds saving ontop if the files are not in the server's filesystem cache.

The reason that I called if glob is that it gathers "data*.php sql*.php" into a single file. Suggest another name and lets go with that. :-)
Oleg
Posts: 1150
Joined: Tue Feb 23, 2010 2:38 am
Contact:

Re: Improving phpBB performance for small installations

Post by Oleg »

TerryE wrote:You have floated a "problem", but I think more in terms of requirements and whether the implementation addresses them. This one is a cache coherency requirement, that's all. When and why must cache coherency be maintained?
The problem (and since you did not disagree with it I am going to assume you agree with the failure scenario I described) is that your proposed implementation does not work correctly. In particular requests to delete data from the cache may be essentially ignored. This is a problem specific to your proposal in this topic; the file acm does not suffer from it.

Until it is addressed I do not see much utility in discussing the rest of the proposal. I kept my original post because I typed it up before I realized the problem, and if the problem is addressed the changes I enumerated would have to be made.

In the case under discussion, compromising correctness for a 30% performance improvement for the first request on completely cold boards is, in my opinion, not something that we should do.

When you are referring to out of topic materials, please provide a direct link to the page and ideally the section in question. This is basic courtesy.

I suggested a name for the proposed acm implementation in the previous post: single file.
Post Reply