[RFC|Merged] Improved template engine

These requests for comments/change have lead to an implemented feature that has been successfully merged into the 3.1/Ascraeus branch. Everything listed in this forum will be available in phpBB 3.1.
APTX
Registered User
Posts: 680
Joined: Thu Apr 24, 2003 12:07 pm

[RFC|Merged] Improved template engine

Post by APTX » Sun Jul 18, 2010 3:36 am

Improved template system for phpBB 3.1
This is a draft and subject to change

A more efficient implementation of phpBBs template engine ported from ascreus-experiment with some minor additions especially for the new hook feature.

Compatibility
- The template class API remains unchanged.
- Newlines in templates will be preserved. The current template engine preserves or removes newlines seemingly at random.

Technical details
The template compiler is implemented as a PHP stream filter. This means that templates are no longer read into memory as a whole, but compiled in chunks. This significantly reduces memory requirements. Due to changes in the compiler logic compilation is (insignificantly) faster.
The generated codes is a bit more optimized, for example it uses references to frequently used variables like $user->lang.

Tracker
Related Ticket: http://tracker.phpbb.com/browse/PHPBB3-9726
Current code in progress: https://github.com/p/phpbb3/compare/dev ... ate-engine
Pull request: https://github.com/phpbb/phpbb3/pull/171
Don't give me my freedom out of pity!

TerryE
Registered User
Posts: 95
Joined: Sat May 23, 2009 12:24 am
Contact:

Re: [RFC|Accepted] Improved template engine

Post by TerryE » Mon Feb 28, 2011 5:07 pm

I like the templating engine approach adopted in phpBB, but dislike some of the unnecessary inefficiencies in it's implementation. By way of an experiment, I decided to implement a templating engine for my own blog drawing on some my experience of using phpBB's and another engine. I discuss this in the article "My blog’s templating engine". The engine works well and is very efficient in compilation and run-time terms.

OK, I use more of a Smarty syntax and it's not worth the regression work changing the clunky phpBB syntax, but some of the issues that I have addressed are well worth considering including in the phpBB engine as these give good performance benefits:
  • Having a per language compilation process. This isn't an issue for most forums which only operate one or two languages. However, this allows the hoisting of all language-specific processing into the template compilation process and language strings such as {L_FORUM_RULES} simply become embedded HTML text in the compiled template.
  • The list of variables referenced in the template is collected and a simple 6 line preamble establishes reference synonyms for these to the template variable with a (blank) default if not set. This means that a reference {U_POST_NEW_TOPIC} now gets compiled to echo $varPOST_NEW_TOPIC;.
  • A global switch (in phpBB a board configuration setting) cases included templates to be compiled inline during the compilation process. OK, this doesn't make much sense if the installation is using an opcode accelerator such as APC, but it does have performance advantages for smaller boards installed on a service which doesn't have such caching available.
  • My language lookup is though a customisable callback and the convention that I have adopted is use the English text as the lookup key and I keep the language table in the database. (This isn't a runtime overhead since error paths are about the only language lookup that I do outside the compilation process). The major advantage is that the templates are a lot more readable.
I realise that the Smarty syntax the NL resource approach that I use isn't applicable to phpBB because of the change impacts on existing templates, but the other suggestions are. I am happy to rework the compilation module as a demonstrator assigning phpBB joint copyright for my contribution, and also to supply my existing code + examples for background. But only if you are interested: I don't want to do this work for it only to be casually discarded.

(Sorry if this post is in the wrong topic, but if so perhaps one of the mods can move it and let me know).

Oleg
Posts: 1150
Joined: Tue Feb 23, 2010 2:38 am
Contact:

Re: [RFC|Accepted] Improved template engine

Post by Oleg » Thu Mar 17, 2011 1:44 am

TerryE wrote: Having a per language compilation process. This isn't an issue for most forums which only operate one or two languages. However, this allows the hoisting of all language-specific processing into the template compilation process and language strings such as {L_FORUM_RULES} simply become embedded HTML text in the compiled template.
This is a good idea for most boards, but unfortunately for the few boards that do have many styles and many languages it may be a very negative change. Which means either the implementation has to be conditional or some other solution is needed.
TerryE wrote: [*]The list of variables referenced in the template is collected and a simple 6 line preamble establishes reference synonyms for these to the template variable with a (blank) default if not set. This means that a reference {U_POST_NEW_TOPIC} now gets compiled to echo $varPOST_NEW_TOPIC;.
I suppose we can do this, are there any downsides? Do you have a patch we can look at?
TerryE wrote: A global switch (in phpBB a board configuration setting) cases included templates to be compiled inline during the compilation process. OK, this doesn't make much sense if the installation is using an opcode accelerator such as APC, but it does have performance advantages for smaller boards installed on a service which doesn't have such caching available.
Inlining code is not automatically a good thing. What are the gains here precisely?
TerryE wrote: My language lookup is though a customisable callback and the convention that I have adopted is use the English text as the lookup key and I keep the language table in the database. (This isn't a runtime overhead since error paths are about the only language lookup that I do outside the compilation process). The major advantage is that the templates are a lot more readable.
The problem with this is it makes it impossible to edit the English language strings in minor releases as it will break all translations and will affect modifications as well.

TerryE
Registered User
Posts: 95
Joined: Sat May 23, 2009 12:24 am
Contact:

Re: [RFC|Accepted] Improved template engine

Post by TerryE » Thu Mar 17, 2011 2:28 am

Per language compilation process
nn- wrote:This is a good idea for most boards, but unfortunately for the few boards that do have many styles and many languages it may be a very negative change. Which means either the implementation has to be conditional or some other solution is needed.
It would only be a negative change in the scenario where the board has a lot of languages and is using a PHP accelerator such as APC, because of the code cache increase. However, it would be trivial to add an extra flag to the phpbb_config table, so the sysadmin could elect to do this or not.

Using reference synonyms
nn- wrote:I suppose we can do this, are there any downsides? Do you have a patch we can look at?
The major benefit is in reduction of the compiled template simplicity and size, both source and compiled opcode (for APC etc.) The runtime difference is in microseconds. See my article My blog’s templating engine where I explain this. It does work well.

I would be reluctant to propose a patch for something that is really beyond patching. It really needs a proper refactoring / rewrite.

Optional inlining of template includes during the compilation process
nn- wrote:Inlining code is not automatically a good thing. What are the gains here precisely?
This inlining is transparent at the template level and would only be apparent at the compiled .html.php level. This is really for the smaller installs running on a shared service. Less files = less I/O. The file space overhead of including the overall_header in all templates where it is referenced is minimal.

[Your point about the string keys decoupling translations from base English is valid, plus regression costs across all templates so forget this one :oops: ]

I am happy to look at a rewrite /refactoring of the engine over the next few weeks (initially 3.0.8 compatible but with PHP 5.2 coding) as long as I am not cutting across someone else who is also actively doing this. It's a bit of work, so I'd only do it if it stands a good chance of being used. I would assume that the invariants are the template language / syntax and the data access API.

User avatar
naderman
Product Manager
Product Manager
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

Re: [RFC|Accepted] Improved template engine

Post by naderman » Thu Mar 17, 2011 5:53 pm

Marek (APTX) has been working on a changes to the template engine, so some coordination would be good here. Personally I've been looking into maybe replacing the template engine entirely with twig to give us some more flexibility/extensibility to introduce new constructs for MOD hooking in templates, while keeping things 100% backward compatible. More on that once I actually get to it and can see if this is a viable option ;-)

TerryE
Registered User
Posts: 95
Joined: Sat May 23, 2009 12:24 am
Contact:

Re: [RFC|Accepted] Improved template engine

Post by TerryE » Thu Mar 17, 2011 6:04 pm

I see two main problems with templating engines like twig and smarty: (i) whilst they are extremely flexible and powerful, you do pay a performance penalty; (ii) their syntax and semantics are materially from the existing template language specification (TLS), and because of the package complexity, you would need to do a complex and costly-to-maintain fork to maintain compatibility with the existing TLS. If you want to move to another TLS then you've got a major change and regression test cost for existing templates out there.

I guess that you also thing that the current templating language sucks, but a proper refactoring of the existing engine based on a PHP 5-compliant object model would be far simpler and safer IMHO. :)

APTX
Registered User
Posts: 680
Joined: Thu Apr 24, 2003 12:07 pm

Re: [RFC|Accepted] Improved template engine

Post by APTX » Thu Mar 17, 2011 6:20 pm

Please look at https://github.com/APTX/phpbb3/tree/fea ... ngine-fate for the 3.1 template engine.

I'll look into implementing at least the "per language compilation".
Don't give me my freedom out of pity!

TerryE
Registered User
Posts: 95
Joined: Sat May 23, 2009 12:24 am
Contact:

Re: [RFC|Accepted] Improved template engine

Post by TerryE » Thu Mar 17, 2011 7:20 pm

APTX, I'll clone your APTX / phpbb3 branch and go through this, and give any specific feedback on this 3.1 snapshot, but at a first look you seem to have done most of the refactoring that I was thinking of. ;)

User avatar
naderman
Product Manager
Product Manager
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Karlsruhe, Germany
Contact:

Re: [RFC|Accepted] Improved template engine

Post by naderman » Thu Mar 17, 2011 7:53 pm

TerryE wrote:I see two main problems with templating engines like twig and smarty: (i) whilst they are extremely flexible and powerful, you do pay a performance penalty; (ii) their syntax and semantics are materially from the existing template language specification (TLS), and because of the package complexity, you would need to do a complex and costly-to-maintain fork to maintain compatibility with the existing TLS. If you want to move to another TLS then you've got a major change and regression test cost for existing templates out there.
I guess that you also thing that the current templating language sucks, but a proper refactoring of the existing engine based on a PHP 5-compliant object model would be far simpler and safer IMHO. :)
twig is actually significantly different from smarty in this regard. It's more of a framework to define your own templating language with a sensible default. Using OOP you can define the tokens and AST nodes for parser and lexer. We don't need to fork their code at all to make it support our current templating language. I haven't made detailed performance benchmarks yet, but based on how twig works it should operate at a similar speed as our current template engine. The reason for looking at it at all is to avoid having to spend a lot of time on building a new OOP template engine ourselves. phpBB in general suffers from a NIH mentality problem, and it's really not necessary to reinvent template engines yet again.

Apart from that I'm not saying this is going to happen, I'm purely looking at it to see if it can really be done that easily in which case I will propose this as an alternative. So we can get into a more detailed discussion on pros/cons of twig then ;-)

Oleg
Posts: 1150
Joined: Tue Feb 23, 2010 2:38 am
Contact:

Re: [RFC|Accepted] Improved template engine

Post by Oleg » Sun Apr 24, 2011 3:03 am

Work in progress: https://github.com/p/phpbb3/compare/dev ... ate-engine

There are some failing tests: https://gist.github.com/939261

Edit: tests are fixed.

I submitted the pull request: https://github.com/phpbb/phpbb3/pull/171

Let's get this tested and merged.

Post Reply