any UTF8 support ?

Discuss features as they are added to the new version. Give us your feedback. Don't post bug reports, feature requests, support questions or suggestions here.
Forum rules
Discuss features as they are added to the new version. Give us your feedback. Don't post bug reports, feature requests, support questions or suggestions here. Feature requests are closed.
extrabigmehdi
Registered User
Posts: 5
Joined: Fri May 26, 2006 2:25 am

any UTF8 support ?

Post by extrabigmehdi »

Hi,
more and more script are supporting by default utf8.
I've been searching around the forum , but as wonderfull is the upcoming phpbb3 , nowhere it is announced that it will officially support utf8.
Sorry but this looks like a blasphemy.

I've currently succesfully bridged phpbb 2 with my favorite CMS i.e Joomla (1.08)
In fact I'm the devellopper of the bridge ( here )
The upcoming Jooma version will support only utf8 (version 1.5)
I've been trying to bridge this Joomla version (alpha status) with phpbb 2
and got errors like : "Illegal mix of collations for operation"

It seems that switching between two operations would help:

Code: Select all

mysql_query( "SET NAMES 'utf8'", $joomla_connection );
mysql_query( "SET NAMES 'latin1'", $phpbb_connection);
But I really hope that next coming phpbb3 version will officially support utf8.
User avatar
Cheater512
Registered User
Posts: 245
Joined: Thu Mar 23, 2006 1:29 am
Location: Brisbane, Australia
Contact:

Re: any UTF8 support ?

Post by Cheater512 »

I didnt pay attention when I was looking at it but I belive it does support UTF8.
APTX
Registered User
Posts: 680
Joined: Thu Apr 24, 2003 12:07 pm

Re: any UTF8 support ?

Post by APTX »

When psoTFX was still around he said why phpBB doesn't use UTF-8.
viewtopic.php?f=3&t=12397&p=78456#p78456
I havent seen any new statements on this.
The PHP version which supports UTF fully is PHP6.
Don't give me my freedom out of pity!
extrabigmehdi
Registered User
Posts: 5
Joined: Fri May 26, 2006 2:25 am

Re: any UTF8 support ?

Post by extrabigmehdi »

When psoTFX was still around he said why phpBB doesn't use UTF-8.
That's sad. Whatever the reasons why the developper psoTFX refuse to work on utf8 support, this looks like a kind of selfish decision. (Hey, I speak English why would I bother with other people ?)
I guess we will have to apply some hacks to phpbb like the ones described here. Solution is not perfect, and more that one newbie will be discouraged in following these steps. They'll look into an alternative board.

If Joomla is able to make a succesfull move into full utf8 support I guess that phpbb
can also. However all other encoding support will be dropped in Joomla : utf8 and only utf8.
Also minimal php version supported by the CMS is 4.3
User avatar
SHS`
Registered User
Posts: 1628
Joined: Wed Jul 04, 2001 9:13 am
Location: The Boonies, Hong Kong
Contact:

Re: any UTF8 support ?

Post by SHS` »

extrabigmehdi wrote:
When psoTFX was still around he said why phpBB doesn't use UTF-8.
That's sad. Whatever the reasons why the developper psoTFX refuse to work on utf8 support, this looks like a kind of selfish decision. (Hey, I speak English why would I bother with other people ?)
I guess we will have to apply some hacks to phpbb like the ones described here. Solution is not perfect, and more that one newbie will be discouraged in following these steps. They'll look into an alternative board.
Had you actually read Paul's post, he said if PHP had proper support for Unicode (which neither PHP4 or PHP5 do), he'd jump right to it. The developers here are probably the most switched on regarding localisation... the actual issue was the platform (PHP). mbstring doesn't actually work that well, not to mention the fact it's a rather esoteric extension to have enabled. Your accusation of "cultural arrogance" might hold some water had phpBB have everything hardcoded in English, which frankly isn't even true for phpBB2.0.x.

PHP6 is actually the first version of PHP that'll support Unicode properly, though with it still being in development, the odds that it'll be stable before phpBB3 goes gold are slim. Interestingly, mbstring must be disabled for PHP6 to use the Unicode ICU library.

The changes required for phpBB2.0.x as described in the KB are trivial, and a similar process is need for phpBB3 as it currently stands.
extrabigmehdi wrote: If Joomla is able to make a succesfull move into full utf8 support I guess that phpbb
can also. However all other encoding support will be dropped in Joomla : utf8 and only utf8.
Also minimal php version supported by the CMS is 4.3
phpBB3 CVS works just fine in Unicode (I've tested UTF-7, UTF-8, UTF-16/UCS2, UTF-32/UCS4) on either PHP5.x or PHP6-dev, though it does require more entensions to be enabled (mbstring in PHP5, ICU in PHP6) which in most cases are off.

There is currently an issue regarding phpBB3's where there are language files of multiple encodings, which I've yet to bug report... the best fix is indeed to just run in a Unicode encoding and be done with it. However, if it unduly pushes up requirements, it might not.
User avatar
smithy_dll
Registered User
Posts: 461
Joined: Tue Jan 08, 2002 6:27 am
Location: Australia
Contact:

Re: any UTF8 support ?

Post by smithy_dll »

extrabigmehdi wrote: That's sad. Whatever the reasons why the developper psoTFX refuse to work on utf8 support, this looks like a kind of selfish decision. (Hey, I speak English why would I bother with other people ?)
If you had bothered to read the post you'd realise it was anything by a selfish decision. It was a technical decision and was backed up by fact.
extrabigmehdi
Registered User
Posts: 5
Joined: Fri May 26, 2006 2:25 am

Re: any UTF8 support ?

Post by extrabigmehdi »

to answer to SHS,
phpBB3 CVS works just fine in Unicode (I've tested UTF-7, UTF-8, UTF-16/UCS2, UTF-32/UCS4) on either PHP5.x or PHP6-dev, though it does require more entensions to be enabled (mbstring in PHP5, ICU in PHP6) which in most cases are off.
It's not reasonable to expect people to use PHP5, and PHP6. A big majority of hostings only support PHP4. And I think that people don't want to make the move to PHP5 because backward compatibility with PHP4 is somehow broken and thus some scripts may won't work any more.

Anyway , did you realize that Joomla 1.5 announce a full utf-8 support while using PHP 4.3 as a minimal requirement ?
the actual issue was the platform (PHP). mbstring doesn't actually work that well, not to mention the fact it's a rather esoteric extension to have enabled
Despite this issue, Joomla's devellopper still found a way fully suport utf8 with old PHP versions. There's an interesting article about this: implementation of UTF-8 in Joomla! 1.5
In short from the article:
[...]
The problem lies with the fact that until PHP 6 is released, there is no comprehensive native UTF-8 support in PHP. There is a multi-byte extension named ‘mbstring’ which exists from version 4.1 but it is not loaded by default. [...]
Here again, the core team decided to vote for full backward compatibility and for the solution to be transparent to the user. The solution is a combination of either using PHP provided functions, if they are present, or using a special library of UTF-8 aware string functions, if no PHP native functions are available. This provides the best performance (PHP functions available) together will complete backward compatibility. A Joomla String Class provides this functionality and it will be included in the API for third party developers.
Maybe phpbb developpers could re-use this "Joomla String Class" , or get in touch with Joomla's devellopers.

I also know an other script that fully support utf8. It is Coppermine Gallery.
What I've understood is that they use mb_ functions instead of regular ones.
If the mbstring extension is not available, then mb_ functions are emulated. This doesn't seem very complicated when I take a look in their code (file mb.inc.php in their distibution)

Now to answer to smithy_dll:

Code: Select all

If you had bothered to read the post you'd realise it was anything by a selfish decision. It was a technical decision and was backed up by fact.
Sorry if I was a bit provocative. But the lack of utf-8 support in phpbb , while this one available in sophisticated scripts like Joomla and Coppermine just made me upset. Also the requirement of PHP5 for utf8 suport doesn't satisfy me, and I guess it won't satisfy many people.
User avatar
SHS`
Registered User
Posts: 1628
Joined: Wed Jul 04, 2001 9:13 am
Location: The Boonies, Hong Kong
Contact:

Re: any UTF8 support ?

Post by SHS` »

extrabigmehdi wrote: to answer to SHS,
phpBB3 CVS works just fine in Unicode (I've tested UTF-7, UTF-8, UTF-16/UCS2, UTF-32/UCS4) on either PHP5.x or PHP6-dev, though it does require more entensions to be enabled (mbstring in PHP5, ICU in PHP6) which in most cases are off.
It's not reasonable to expect people to use PHP5, and PHP6. A big majority of hostings only support PHP4. And I think that people don't want to make the move to PHP5 because backward compatibility with PHP4 is somehow broken and thus some scripts may won't work any more.

Anyway , did you realize that Joomla 1.5 announce a full utf-8 support while using PHP 4.3 as a minimal requirement ?
the actual issue was the platform (PHP). mbstring doesn't actually work that well, not to mention the fact it's a rather esoteric extension to have enabled
Despite this issue, Joomla's devellopper still found a way fully suport utf8 with old PHP versions. There's an interesting article about this: implementation of UTF-8 in Joomla! 1.5
In short from the article:
[...]
The problem lies with the fact that until PHP 6 is released, there is no comprehensive native UTF-8 support in PHP. There is a multi-byte extension named ‘mbstring’ which exists from version 4.1 but it is not loaded by default. [...]
Here again, the core team decided to vote for full backward compatibility and for the solution to be transparent to the user. The solution is a combination of either using PHP provided functions, if they are present, or using a special library of UTF-8 aware string functions, if no PHP native functions are available. This provides the best performance (PHP functions available) together will complete backward compatibility. A Joomla String Class provides this functionality and it will be included in the API for third party developers.
Maybe phpbb developpers could re-use this "Joomla String Class" , or get in touch with Joomla's devellopers.
FFS. I've said in my very post that for proper Unicode support, the not that great mbstring library has to be enabled at server level when the PHP version used in =< 5.x. From version 6.0 onwards and when ICU is enabled, proper Unicode support finally arrives in PHP.

As nice as Unicode is, having to deal with it is slower than "default" charsets, that includes everything from string manipulation functions (particularly is emulating something like mbstring if it isn't available as an extension) to the fact PHP itself is slower when in fully Unicode mode.

There is no such thing as a free lunch, and if you just for one second stop being so blinkered in that phpBB3 not supporting Unicode by default is the end of the world, I've already hinted that this may well be a possibility anyway, for other reasons.

If for whatever reason phpBB3.0.0 comes out and is by default ISO-whatever... being Opensource, you're more than welcome to MOD it/fork it/whatever to have a "native Unicode" version.

Compris?
Jonathan “SHS`” Stanley • 史德信
phpBB™ 3.1.x, Bug/Security trackers
phpBB™ Bertie Bear 3.0 — prosilver Edition!Asking Questions The Smart Way
extrabigmehdi
Registered User
Posts: 5
Joined: Fri May 26, 2006 2:25 am

Re: any UTF8 support ?

Post by extrabigmehdi »

Hi,
As nice as Unicode is, having to deal with it is slower than "default" charsets
Not a big problem. I don't think I would notice the difference of speed.
I've already hinted that this may well be a possibility anyway, for other reasons
I hope so but preferably with the ability to use php 4.3
If for whatever reason phpBB3.0.0 comes out and is by default ISO-whatever... being Opensource, you're more than welcome to MOD it/fork it/whatever to have a "native Unicode" version.
I know, but I would prefer official support from phpbb...
and if you just for one second stop being so blinkered in that phpBB3 not supporting Unicode by default is the end of the world
I'll stop, but I still cross my fingers, and hope that.... ok I stop. :mrgreen:
mdshort
Registered User
Posts: 2
Joined: Sat Mar 04, 2006 5:39 pm

Re: any UTF8 support ?

Post by mdshort »

utf8 should be supported. I'm full english but I'm all about reaching everyone, and it is unfair to use incompatible encodings.
Post Reply