Inside: a benchmark of phpBB's performance under simulated load

Discuss general development subjects that are not specific to a particular version like the versioning control system we use or other infrastructure.
asperous
Google Summer of Code Student
Posts: 21
Joined: Mon Apr 22, 2013 3:26 pm
Location: Tigard, Or
Contact:

Inside: a benchmark of phpBB's performance under simulated load

Post by asperous » Sat Apr 27, 2013 12:34 am

I'm creating software to DDos phpBB. It's coming along nicely.

Image

What does that chart mean? It means on my machine, without apc, while the benchmark is running, 90 active users at once would might the response time to be quite poor. Maybe.

There is so much to do to make this test mechanism better, but I think a tool like this would be vital for phpBB development to test the system under load.

Code is here: http://github.com/asperous/overload.

asperous
Google Summer of Code Student
Posts: 21
Joined: Mon Apr 22, 2013 3:26 pm
Location: Tigard, Or
Contact:

Re: Inside: a benchmark of phpBB's performance under simulated load

Post by asperous » Sat Apr 27, 2013 12:38 am

I should probably make it clear that this is not a simple apacheBench-like test.

Those are simulated clients that are creating accounts, sleeping, and reading posts on phpBB software concurrently. Cookies, Sessions, & all.

That's why I believe this, with a lot of work, could be a useful tool.

Oleg
Posts: 1150
Joined: Tue Feb 23, 2010 2:38 am
Contact:

Re: Inside: a benchmark of phpBB's performance under simulated load

Post by Oleg » Sat Apr 27, 2013 2:41 am

The fact that your graph starts at 2.5 seconds for one? user suggests to me that 1) you have debug enabled which rebuilds a crazy amount of php code on every request and is entirely unsuitable for production, and 2) you have a slow machine.

Code: Select all

pie@reactor ~ % time fetch -o /dev/null 'http://func/index.php?sid=1b11c3c6b7e0287b93c93f9b391a6894'
fetch: http://func/index.php?sid=1b11c3c6b7e0287b93c93f9b391a6894: size of remote file is not known
/dev/null                                                9 kB 1060 kBps
fetch -o /dev/null   0.00s user 0.01s system 1% cpu 0.431 total

asperous
Google Summer of Code Student
Posts: 21
Joined: Mon Apr 22, 2013 3:26 pm
Location: Tigard, Or
Contact:

Re: Inside: a benchmark of phpBB's performance under simulated load

Post by asperous » Sat Apr 27, 2013 3:11 am

Actually you are right on both! I didn't realize the debug flag effected performance, I though that was just to ignore the install folder's presence.

I'll rerun a bit later and update the data. A couple other problems with this benchmark I want to enumerate here so I can repair later:

1. Both benchmark and server are running on the same machine. They will effect each other at high numbers of clients.
2. The horizontal axis measures "active clients" when a better measurement would be "current number of requests outstanding" since requests made recently don't effect performance (?).
3. phpBB doesn't support deployments without apc (!) so I'll have to enable that before the next run.
4. Deltas are more useful than numbers. I'll try to run this against 3.0 and 3.1 and show differences (if any).
5. Clients are reading the same page over and over. I'll have to add some posts to include uncached page rendering effects into the bench.

If you have any more critisims hit me up!

User avatar
EXreaction
Registered User
Posts: 1555
Joined: Sat Sep 10, 2005 2:15 am

Re: Inside: a benchmark of phpBB's performance under simulated load

Post by EXreaction » Sat Apr 27, 2013 4:09 am

What do you mean by #3?

Oleg
Posts: 1150
Joined: Tue Feb 23, 2010 2:38 am
Contact:

Re: Inside: a benchmark of phpBB's performance under simulated load

Post by Oleg » Sat Apr 27, 2013 4:21 am

asperous wrote: 3. phpBB doesn't support deployments without apc (!) so I'll have to enable that before the
next run.
It does and plenty of people run it that way. APC on seldom used applications is pointless or possibly a waste of memory.
5. Clients are reading the same page over and over. I'll have to add some posts to include uncached page rendering effects into the bench.
This will mostly provide i/o load, and you may want to simply fabricate an i/o load yourself.

asperous
Google Summer of Code Student
Posts: 21
Joined: Mon Apr 22, 2013 3:26 pm
Location: Tigard, Or
Contact:

Re: Inside: a benchmark of phpBB's performance under simulated load

Post by asperous » Sat Apr 27, 2013 5:04 am

I re-ran the test without debug. Nothing else about the test has changed. I ran the test twice this time, and there is a lot more data because to my knowledge not a single request came back empty this time, while before there was quite a few dropped.

Image

The cloud around 150-250 I believe was the result of another, conflicting operation happening on my machine in the background. I make this guess because I heard the hard drive clicking a lot more than usual.

Keep in mind the resource implications of running the test and the server on the same machine. I believe that due to this factor and others, it becomes clear there isn't much to gain from this specific test run as is. More work will be needed.

Raw Data: https://docs.google.com/spreadsheet/ccc ... sp=sharing

asperous
Google Summer of Code Student
Posts: 21
Joined: Mon Apr 22, 2013 3:26 pm
Location: Tigard, Or
Contact:

Re: Inside: a benchmark of phpBB's performance under simulated load

Post by asperous » Sat Apr 27, 2013 5:57 pm

  • Separated test & server
  • Now measuring outstanding requests, not active clients
  • Box and whisker (quads at [0.1, 0.25, 0.75, 0.9]
Image

Image

Raw Data: http://asperous.us/temp/phpBB_test2_results.csv

asperous
Google Summer of Code Student
Posts: 21
Joined: Mon Apr 22, 2013 3:26 pm
Location: Tigard, Or
Contact:

Re: Inside: a benchmark of phpBB's performance under simulated load

Post by asperous » Sat Apr 27, 2013 11:24 pm

3.0 (light blue) vs 3.1 (grey)

A comparison of two phpBB versions. The amount of seconds phpBB reports that it took to render a page on my machine while the server was handling number of simultaneous requests.

Image

Apache crashed around 130 requests on 3.0. These results are very confusing to me and I'm going to be checking all the steps and assumptions I have about these numbers. I think there's something wrong here, but I can't figure out what it is. Any suggestions would be greatly appreciated.

EDIT: The (3.0) along the x-axis was a result of forgetting to remove that label when running the analysis on the data. I suspect there are other mistakes here. Another thing I should mention is that the data is analyzed differently on these charts than above. The box charts here are composed of a median, a standard deviation for the spread, and 0.1 and 0.9 percentile for the whiskers.
Attachments
threeoh.png
(92.96 KiB) Downloaded 2442 times
Last edited by asperous on Tue Aug 04, 2015 9:36 pm, edited 2 times in total.

User avatar
EXreaction
Registered User
Posts: 1555
Joined: Sat Sep 10, 2005 2:15 am

Re: Inside: a benchmark of phpBB's performance under simulated load

Post by EXreaction » Sun Apr 28, 2013 12:23 am

That is some pretty interesting data. Is Apache consistently crashing around 130 requests in the 3.0 branch?

Post Reply