[Please read]Cut down the memory consumption and CPU load

Discuss features as they are added to the new version. Give us your feedback. Don't post bug reports, feature requests, support questions or suggestions here.
Forum rules
Discuss features as they are added to the new version. Give us your feedback. Don't post bug reports, feature requests, support questions or suggestions here. Feature requests are closed.
User avatar
mikovchain
Registered User
Posts: 45
Joined: Mon Oct 16, 2006 5:16 am

[Please read]Cut down the memory consumption and CPU load

Post by mikovchain » Tue Nov 13, 2007 9:09 am

For CPU load, especially in search, if the input contains more than 5 words, will give huge load to mysql server, I can prevent it from code, but it' better adding some options in control panel. This will be a nightmare to the admins who are using sharing hosting space. I know there's a good algorithm in split all kinds of conditions from input, but 99% of the time it's useless.

For memory consumption, I don't know why phpBB3 can use such large memory in a single session, around 3~4 MB, and sometimes hits 5~6 MB in submitting posts. It's not so bad comparing with vBB, but I do know some forum softwares can limit the memory consumption into 800 KB in most of the pages with rich features and amazing response speed. Have you ever tried Discuz!?

This is important for some hosting service provider, they will tend to persuade user using a light load forum.
If not being binded in so many hosting machines, phpBB2 wouldn't be the most widely used forum software. If all you want phpBB3 to be the first choice of all forum software, you need optimize it.

The memory distribution in index page:

Code: Select all

after load Extensions 64 Bytes
after require acm/acm_ 94.85 KB
after require cache 175.81 KB
after require template 268.84 KB
after require session 755.38KB
after require auth 948.34 KB
after require function 1.72 MB
after require function_content 1.96 MB
after require constants 1.98 MB
after require db 2.2 MB
after require utf_tools 2.47 MB
at the end of common.php 2.57 MB
after include function_display 2.81 MB
after session_begin 2.85 MB
after acl(userdata) 2.86 MB
after setup() 3.04 MB
before page_footer() 3.06 MB
Total 3.07 MB
PHPBB CHINA For Chinese Users

User avatar
Kellanved
Former Team Member
Posts: 407
Joined: Sun Jul 30, 2006 4:59 pm
Location: Berlin

Re: [Please read]Cut down the memory consumption and CPU load

Post by Kellanved » Tue Nov 13, 2007 1:09 pm

Memory consumption depends largely on the infrastructure. PHP 5.2.x is the worst offender, especially so on windows servers. We use includes for the language, templates and the cache, which regretfully aren't very well handled in PHP. Future versions will change that, but there is no viable PHP4 compatible solution (even 5.0 compatible is a problem).
Another factor increasing the memory footprint is the UTF-8 support.


CPU-wise and for the database, the script is pretty friendly and heavily optimized (much lighter than phpBB2). Your issue sounds like a server misconfiguration. The search backends provide various load limiting options. In the worst case, a custom search backend can easily be done by the end user.
No support via PM.
Trust me, I'm a doctor.

User avatar
mikovchain
Registered User
Posts: 45
Joined: Mon Oct 16, 2006 5:16 am

Re: [Please read]Cut down the memory consumption and CPU load

Post by mikovchain » Wed Nov 14, 2007 2:27 pm

I checked all the functions in includes/functions.php. Following functions will never be used in 90% of the users' operations:

Code: Select all

function markread($mode, $forum_id = false, $topic_id = false, $post_time = 0, $user_id = 0)
function get_topic_tracking($forum_id, $topic_ids, &$rowset, $forum_mark_time, $global_announce_list = false)
function get_complete_topic_tracking($forum_id, $topic_ids, $global_announce_list = false)
function update_forum_tracking_info($forum_id, $forum_last_post_time, $f_mark_time = false, $mark_time_forum = false)
function tracking_serialize($input)
function tracking_unserialize($string, $max_depth = 3)
function phpbb_hash($password)
function phpbb_check_hash($password, $hash)
function _hash_gensalt_private($input, &$itoa64, $iteration_count_log2 = 6)
function _hash_encode64($input, $count, &$itoa64)
function _hash_crypt_private($password, $setting, &$itoa64)
function language_select($default = '')
function style_select($default = '', $all = false)
function tz_select($default = '', $truncate = false)
function reapply_sid($url)
function login_forum_box($forum_data)
function get_backtrace()
function phpbb_checkdnsrr($host, $type = '')
But they are included every time we build a session :shock:
Split them from functions.php into several small inc.php files, load them on call, in this way we can save about 300KB memory per session.

Wish PHP6 can impl the features caching code between sessions, that will make things simple
PHPBB CHINA For Chinese Users

User avatar
Highway of Life
Registered User
Posts: 1399
Joined: Tue Feb 08, 2005 10:18 pm
Location: I'd love to change the World, but they won't give me the Source Code
Contact:

Re: [Please read]Cut down the memory consumption and CPU load

Post by Highway of Life » Wed Nov 14, 2007 3:02 pm

I agree here with what Henry has said. -- And in-fact, on StarTrekGuide memory consumption on a rather large index page including an AJAX Chatbox, the Memory amounts to about 1.2MB
On my localhost fresh installation, it uses 486Kb, a 300Kb savings with removing those functions would seem highly unlikely at that rate.

I believe your high memory readings have something to do with a server misconfiguration.

EDIT: I should note that I’m running both on PHP 5.2.4 with eAccelerator/ZO on my localhost, and X-Cache/ZO (IIRC) on StarTrekGuide.
Image

APTX
Registered User
Posts: 680
Joined: Thu Apr 24, 2003 12:07 pm

Re: [Please read]Cut down the memory consumption and CPU load

Post by APTX » Wed Nov 14, 2007 4:46 pm

eAccelerator/X-Cache would be the reason for low memory use.
Don't give me my freedom out of pity!

User avatar
naderman
Consultant
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

Re: [Please read]Cut down the memory consumption and CPU load

Post by naderman » Wed Nov 14, 2007 4:50 pm

You also have to note that without an opcode cache including a lot of small files is rather slow because the parser has to be restarted and run on each of those files.

User avatar
mikovchain
Registered User
Posts: 45
Joined: Mon Oct 16, 2006 5:16 am

Re: [Please read]Cut down the memory consumption and CPU load

Post by mikovchain » Thu Nov 15, 2007 4:02 am

Thanks to Kellanved, Highway of Life and naderman. I learn a lot from you.
I run and test my phpBB3 on windows xp + apache + PHP5.2.3. Maybe it's why I got such bad performance. :geek:
I will do the same test on linux.
PHPBB CHINA For Chinese Users

jmpoure
Registered User
Posts: 81
Joined: Sun Mar 11, 2007 9:42 am

Re: [Please read]Cut down the memory consumption and CPU load

Post by jmpoure » Thu Jan 03, 2008 7:18 pm

Performance : good

If you want good performance, choose Linux + postrgeSQL.

On my server, PhpBB 3.0 performs very well with 200 users simultaneously.
I mean it gives instant results.

Memory usage : High
My server uses 1,5 Gb of memory (Apache+PostgreSQL) to manage 200 simultaneous users.

Some tricks:

1) Avoid page gzip compression.
For some strange reason, it was default on my configuration.

2) Avoid any SWAP.
SWAP can reduce the overall performance by a large factor.

3) Make sure that Apache/Php do not fork too much.
I configured 24 Mb in Php.ini and set sessions to 300 seconds.
After 5 minutes of inactivity, sessions are freed.

At design level, I can only regret that PhpBB 3.0 runs hundreds of SQL queries ran to build a page.
Inner joins give also very poor results.
They should be replaced with LEFT JOINS.
Also,there should be a couple of views instead of queries, because views are optimized by the database.

Of course, you should not use built-in full-text indexing.
It is complete "crap" and cannot work.
Large databases with more than 200.000 messages will explode when using built-in indexes.

Instead use MySQL or PostgreSQL full-text indexing.
The difference is that in PostgreSQL full-text indexes are managed in shared memory.
It will not eat up each process memory.

Kind regards,
Jean-Michel

User avatar
Highway of Life
Registered User
Posts: 1399
Joined: Tue Feb 08, 2005 10:18 pm
Location: I'd love to change the World, but they won't give me the Source Code
Contact:

Re: [Please read]Cut down the memory consumption and CPU load

Post by Highway of Life » Fri Jan 04, 2008 10:02 am

Those are good tips for large boards (though somewhat unnecessary for small to medium boards with powerful servers).
It’s always a good idea to watch your Memory and CPU usage and tune your settings to compensate.
Say you have a smaller board, but a decent server, you can leave the session length quite a bit longer. On huge boards, it should come down a bit, but in my tests, the session length has not effected the performance on enough of a level to be concerned over. -- Take phpBB.com for example, the server itself has no trouble handling sessions at 1 hour intervals (IIRC). -- Though the DB Server seems to have problems on occasion, it is not related to the number of sessions or session time.
jmpoure wrote:At design level, I can only regret that PhpBB 3.0 runs hundreds of SQL queries ran to build a page.
Inner joins give also very poor results.
They should be replaced with LEFT JOINS.
Also,there should be a couple of views instead of queries, because views are optimized by the database.
Hi Jean, I’m a bit confused by your post here.
hundreds of queries to run a build page? What do you mean by build page? :?

If you see areas where the Joins optimisations can be improved, please let us know which queries those are, -- and I know the Developers have their eyes on areas where the DBAL could get some optimisation enhancements. :)

--oOo--

Here is just a bit of trivia for those that are unfamiliar with the Database queries that go on behind the scenes on every page load in phpBB3.

You have 3 main queries:
  • Dynamic config data
  • User/Session Data -- Inner joined query.
  • Update session with current page
And 2 main cached queries...
  • Style, Imageset, Template and Theme data -- Inner Joined query.
  • Language imageset data
Then page-specific queries, for example, the index page has:
  • Forum Data and Forum Read Tracking -- Left joined query.
  • Forum moderators (cached) -- Left joined query
  • List of groups in legend (should be cached, IMO)
  • Birthday’s (if enabled)
  • And finally the who is online list. -- inner joined query.
Which totals 7 queries and 3 cached queries for a total of 10.
Image

roobarb
Registered User
Posts: 4
Joined: Sun Mar 02, 2008 1:50 am

Re: [Please read]Cut down the memory consumption and CPU load

Post by roobarb » Sun Mar 02, 2008 2:40 am

jmpoure wrote: Some tricks:

1) Avoid page gzip compression.
For some strange reason, it was default on my configuration.

2) Avoid any SWAP.
SWAP can reduce the overall performance by a large factor.

3) Make sure that Apache/Php do not fork too much.
I configured 24 Mb in Php.ini and set sessions to 300 seconds.
After 5 minutes of inactivity, sessions are freed.

At design level, I can only regret that PhpBB 3.0 runs hundreds of SQL queries ran to build a page.
Inner joins give also very poor results.
They should be replaced with LEFT JOINS.
Also,there should be a couple of views instead of queries, because views are optimized by the database.

Of course, you should not use built-in full-text indexing.
It is complete "crap" and cannot work.
Large databases with more than 200.000 messages will explode when using built-in indexes.

Instead use MySQL or PostgreSQL full-text indexing.
The difference is that in PostgreSQL full-text indexes are managed in shared memory.
It will not eat up each process memory.
Many thanks for this information. I've been looking for it all over the place. I have just over 100 users now and my board makes extensive use of the search facility. It was set to fulltext native and we got the MySQL server falling over with 25 simultaneous users.
phpMyAdmin had various non-indexed searches selected in red. I can post these if they are of any use. I'm using a commercial web service which is not short of horsepower.

Post Reply