Inside: a benchmark of phpBB's performance under simulated load

[asperous]

I'm creating software to DDos phpBB. It's coming along nicely.



What does that chart mean? It means on my machine, without apc, while the benchmark is running, 90 active users at once would might the response time to be quite poor. Maybe.

There is so much to do to make this test mechanism better, but I think a tool like this would be vital for phpBB development to test the system under load.

Code is here: http://github.com/asperous/overload.

[asperous]

I should probably make it clear that this is not a simple apacheBench-like test.

Those are simulated clients that are creating accounts, sleeping, and reading posts on phpBB software concurrently. Cookies, Sessions, & all.

That's why I believe this, with a lot of work, could be a useful tool.

[Oleg]

The fact that your graph starts at 2.5 seconds for one? user suggests to me that 1) you have debug enabled which rebuilds a crazy amount of php code on every request and is entirely unsuitable for production, and 2) you have a slow machine.

Code: Select all

pie@reactor ~ % time fetch -o /dev/null 'http://func/index.php?sid=1b11c3c6b7e0287b93c93f9b391a6894'
fetch: http://func/index.php?sid=1b11c3c6b7e0287b93c93f9b391a6894: size of remote file is not known
/dev/null                                                9 kB 1060 kBps
fetch -o /dev/null   0.00s user 0.01s system 1% cpu 0.431 total

[asperous]

Actually you are right on both! I didn't realize the debug flag effected performance, I though that was just to ignore the install folder's presence.

I'll rerun a bit later and update the data. A couple other problems with this benchmark I want to enumerate here so I can repair later:

1. Both benchmark and server are running on the same machine. They will effect each other at high numbers of clients.
2. The horizontal axis measures "active clients" when a better measurement would be "current number of requests outstanding" since requests made recently don't effect performance (?).
3. phpBB doesn't support deployments without apc (!) so I'll have to enable that before the next run.
4. Deltas are more useful than numbers. I'll try to run this against 3.0 and 3.1 and show differences (if any).
5. Clients are reading the same page over and over. I'll have to add some posts to include uncached page rendering effects into the bench.

If you have any more critisims hit me up!

[EXreaction]

What do you mean by #3?

[Oleg]

3. phpBB doesn't support deployments without apc (!) so I'll have to enable that before the
next run.

It does and plenty of people run it that way. APC on seldom used applications is pointless or possibly a waste of memory.

5. Clients are reading the same page over and over. I'll have to add some posts to include uncached page rendering effects into the bench.

This will mostly provide i/o load, and you may want to simply fabricate an i/o load yourself.

[asperous]

I re-ran the test without debug. Nothing else about the test has changed. I ran the test twice this time, and there is a lot more data because to my knowledge not a single request came back empty this time, while before there was quite a few dropped.



The cloud around 150-250 I believe was the result of another, conflicting operation happening on my machine in the background. I make this guess because I heard the hard drive clicking a lot more than usual.

Keep in mind the resource implications of running the test and the server on the same machine. I believe that due to this factor and others, it becomes clear there isn't much to gain from this specific test run as is. More work will be needed.

Raw Data: https://docs.google.com/spreadsheet/ccc ... sp=sharing

[asperous]

• Separated test & server
• Now measuring outstanding requests, not active clients
• Box and whisker (quads at [0.1, 0.25, 0.75, 0.9]

Raw Data: http://asperous.us/temp/phpBB_test2_results.csv

[asperous]

3.0 (light blue) vs 3.1 (grey)

A comparison of two phpBB versions. The amount of seconds phpBB reports that it took to render a page on my machine while the server was handling number of simultaneous requests.



Apache crashed around 130 requests on 3.0. These results are very confusing to me and I'm going to be checking all the steps and assumptions I have about these numbers. I think there's something wrong here, but I can't figure out what it is. Any suggestions would be greatly appreciated.

EDIT: The (3.0) along the x-axis was a result of forgetting to remove that label when running the analysis on the data. I suspect there are other mistakes here. Another thing I should mention is that the data is analyzed differently on these charts than above. The box charts here are composed of a median, a standard deviation for the spread, and 0.1 and 0.9 percentile for the whiskers.

[EXreaction]

That is some pretty interesting data. Is Apache consistently crashing around 130 requests in the 3.0 branch?