password md5s upon upgrade to phpbb3 - a corner case

Discussion of general topics related to the new version and its place in the world. Don't discuss new features, report bugs, ask for support, et cetera. Don't use this to spam for other boards or attack those boards!
Forum rules
Discussion of general topics related to the new release and its place in the world. Don't discuss new features, report bugs, ask for support, et cetera. Don't use this to spam for other boards or attack those boards!
to_allo
Registered User
Posts: 4
Joined: Sun May 13, 2007 7:35 pm

password md5s upon upgrade to phpbb3 - a corner case

Post by to_allo »

My phpbb 2.0.x-based forum uses a "foreign" encoding (iso-8859-7 in specific). I am willing to wait as long as necessary until a converter to a stable version of phpbb3 is available (absolutely no hurry for the upgrade) but I would like to know if the following corner case would be covered by the converter:

It is a fact that phpbb3 will be based on UTF-8 and this includes the database encoding. However, password hashes in our database (as it currently is) are stored as an md5 of the actual password as encoded in the original encoding (i.e. iso-8859-7). For passwords consisting of characters in the ASCII range, iso-8859-7 or UTF-8 would not make a difference. The byte representation would be the same under either encoding, hence also the md5. However, given that we are a foreign-language forum, it is likely that some of our users will have already chosen passwords in greek (thus consisting of characters falling outside the above mentioned range). In that case, byte representations will be different (as each character from the upper range of iso-8859-7 is encoded as multiple bytes under UTF-8), hence also the md5s - which will in turn cause authentication failures.

Is the developer team aware of the above issue? Will there be some mechanism in place in the converter to deal with this?

An outline of the first solution that sprung to my mind upon realisation of the issue:

The conversion of the hash could be defered until the first post-upgrade login. An md5 hash of the iso-8859-7 representation of the supplied password encoded icould be compared to the existing database record for authentication. Upon successful authentication, the md5 hash of representation in UTF-8 of the same password would then replace the record in the database. This in turn, however, necessitates that track of what the pre-upgrade, original encoding was be kept somewhere.

Thank you for your attention.
User avatar
naderman
Consultant
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Berlin, Germany
Contact:

Re: password md5s upon upgrade to phpbb3 - a corner case

Post by naderman »

As we don't keep the encoding we can't do this as well as you said. We check on first post-upgrade login whether the passwords match in ISO-8859-1 (default phpbb encoding) or in UTF-8 (used by a few language packs). If it's an ASCII password it will work, otherwise the user will be informed that his password could not be converted and that he needs to request a new one.

It's possible to check for a specific encoding by modifying includes/auth/auth_db.php but this requires, that you write a utf8_to_your_encoding function which does reverse mapping from the UTF-8 input on phpBB3 login into your old encoding. Currently it only checks utf8_to_cp1252 which can be found in includes/utf/data/recode_basic.php
to_allo
Registered User
Posts: 4
Joined: Sun May 13, 2007 7:35 pm

Re: password md5s upon upgrade to phpbb3 - a corner case

Post by to_allo »

Thank you for the clarification.

I checked includes/utf/data/recode_basic in beta 5 and found only functions converting to UTF-8 from 8-bit encodings (incl. iso-8859-7 however). It did not take me long to come up with a reverse converter (i.e UTF-8 to iso-8859-7). Do I have the option of posting the function somewhere (perhaps within this topic) ? Alternatively, would you or any other developer be interested in it? I am sure that it could be a useful resource to other people.

Regarding the code hack you suggest, please let me know whether I have indeed properly identified what should be changed within auth_db.php:

In line 108, I should replace the check

Code: Select all

md5($password_old_format) == $row['user_password']
with

Code: Select all

md5(utf8_to_iso_8859_7($password_old_format)) == $row['user_password']
to obtain the desired behavior - am I right?
User avatar
naderman
Consultant
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Berlin, Germany
Contact:

Re: password md5s upon upgrade to phpbb3 - a corner case

Post by naderman »

Yeah it'd be alright to post that function here. You should replace

Code: Select all

if (md5($password_old_format) == $row['user_password'] || md5(utf8_to_cp1252($password_old_format)) == $row['user_password'])
with

Code: Select all

if (md5($password_old_format) == $row['user_password'] || md5(utf8_to_iso_8859_7($password_old_format)) == $row['user_password'] || md5(utf8_to_cp1252($password_old_format)) == $row['user_password'])
to_allo
Registered User
Posts: 4
Joined: Sun May 13, 2007 7:35 pm

Re: password md5s upon upgrade to phpbb3 - a corner case

Post by to_allo »

Thank you. Nice to see that the issue can be dealt with as "cleanly" as that (in terms of code). My implementation of said function is provided at the end of this post.

(Note that I was looking at an earlier version of the code, it appears, as the cp1252-specific code was not in place).

A friendly suggestion: Would the developer team consider (perhaps for the stable release) making this encoding check default functionality (i.e. part of the "official" code, as with cp1252)?

Note that I am not asking for myself (since, with your help, I now have the solution to that) but for all other forum operators out there who are less knowledgeable with code. Most would be unaware of the issue until affected upon upgrading. In principle, they should not have to deal with upgrade side-effects since these would not be a result of any wrong-doing in their part in the first place. I also believe that it would spare you guys from having to deal with related complaints and support requests in the near future. In time, I am sure other friends of phpbb could submit converters for remaining major 8-bit encodings.

Once again thanks for your help and responsiveness. The code follows:

Code: Select all

function utf8_to_iso_8859_7($string)
{
	static $transform = array(
		"\xC2\x80"     =>   "\x80",
		"\xC2\x81"     =>   "\x81",
		"\xC2\x82"     =>   "\x82",
		"\xC2\x83"     =>   "\x83",
		"\xC2\x84"     =>   "\x84",
		"\xC2\x85"     =>   "\x85",
		"\xC2\x86"     =>   "\x86",
		"\xC2\x87"     =>   "\x87",
		"\xC2\x88"     =>   "\x88",
		"\xC2\x89"     =>   "\x89",
		"\xC2\x8A"     =>   "\x8A",
		"\xC2\x8B"     =>   "\x8B",
		"\xC2\x8C"     =>   "\x8C",
		"\xC2\x8D"     =>   "\x8D",
		"\xC2\x8E"     =>   "\x8E",
		"\xC2\x8F"     =>   "\x8F",
		"\xC2\x90"     =>   "\x90",
		"\xC2\x91"     =>   "\x91",
		"\xC2\x92"     =>   "\x92",
		"\xC2\x93"     =>   "\x93",
		"\xC2\x94"     =>   "\x94",
		"\xC2\x95"     =>   "\x95",
		"\xC2\x96"     =>   "\x96",
		"\xC2\x97"     =>   "\x97",
		"\xC2\x98"     =>   "\x98",
		"\xC2\x99"     =>   "\x99",
		"\xC2\x9A"     =>   "\x9A",
		"\xC2\x9B"     =>   "\x9B",
		"\xC2\x9C"     =>   "\x9C",
		"\xC2\x9D"     =>   "\x9D",
		"\xC2\x9E"     =>   "\x9E",
		"\xC2\x9F"     =>   "\x9F",
		"\xC2\xA0"     =>   "\xA0",
		"\xE2\x80\x98" =>   "\xA1",
		"\xE2\x80\x99" =>   "\xA2",
		"\xC2\xA3"     =>   "\xA3",
		"\xE2\x82\xAC" =>   "\xA4",
		"\xE2\x82\xAF" =>   "\xA5",
		"\xC2\xA6"     =>   "\xA6",
		"\xC2\xA7"     =>   "\xA7",
		"\xC2\xA8"     =>   "\xA8",
		"\xC2\xA9"     =>   "\xA9",
		"\xCD\xBA"     =>   "\xAA",
		"\xC2\xAB"     =>   "\xAB",
		"\xC2\xAC"     =>   "\xAC",
		"\xC2\xAD"     =>   "\xAD",
		"\xE2\x80\x95" =>   "\xAF",
		"\xC2\xB0"     =>   "\xB0",  
		"\xC2\xB1"     =>   "\xB1",  
		"\xC2\xB2"     =>   "\xB2",
		"\xC2\xB3"     =>   "\xB3",
		"\xCE\x84"     =>   "\xB4",
		"\xCE\x85"     =>   "\xB5",
		"\xCE\x86"     =>   "\xB6",
		"\xC2\xB7"     =>   "\xB7",
		"\xCE\x88"     =>   "\xB8",
		"\xCE\x89"     =>   "\xB9",
		"\xCE\x8A"     =>   "\xBA",
		"\xC2\xBB"     =>   "\xBB",
		"\xCE\x8C"     =>   "\xBC",
		"\xC2\xBD"     =>   "\xBD",
		"\xCE\x8E"     =>   "\xBE",
		"\xCE\x8F"     =>   "\xBF",
		"\xCE\x90"     =>   "\xC0",
		"\xCE\x91"     =>   "\xC1",
		"\xCE\x92"     =>   "\xC2",
		"\xCE\x93"     =>   "\xC3",
		"\xCE\x94"     =>   "\xC4",
		"\xCE\x95"     =>   "\xC5",
		"\xCE\x96"     =>   "\xC6",
		"\xCE\x97"     =>   "\xC7",
		"\xCE\x98"     =>   "\xC8",
		"\xCE\x99"     =>   "\xC9",
		"\xCE\x9A"     =>   "\xCA",
		"\xCE\x9B"     =>   "\xCB",
		"\xCE\x9C"     =>   "\xCC",
		"\xCE\x9D"     =>   "\xCD",
		"\xCE\x9E"     =>   "\xCE",
		"\xCE\x9F"     =>   "\xCF",
		"\xCE\xA0"     =>   "\xD0",
		"\xCE\xA1"     =>   "\xD1",
		"\xCE\xA3"     =>   "\xD3",
		"\xCE\xA4"     =>   "\xD4",
		"\xCE\xA5"     =>   "\xD5",
		"\xCE\xA6"     =>   "\xD6",
		"\xCE\xA7"     =>   "\xD7",
		"\xCE\xA8"     =>   "\xD8",
		"\xCE\xA9"     =>   "\xD9",
		"\xCE\xAA"     =>   "\xDA",
		"\xCE\xAB"     =>   "\xDB",
		"\xCE\xAC"     =>   "\xDC",
		"\xCE\xAD"     =>   "\xDD",
		"\xCE\xAE"     =>   "\xDE",
		"\xCE\xAF"     =>   "\xDF",
		"\xCE\xB0"     =>   "\xE0",
		"\xCE\xB1"     =>   "\xE1",   
		"\xCE\xB2"     =>   "\xE2",
		"\xCE\xB3"     =>   "\xE3",
		"\xCE\xB4"     =>   "\xE4",
		"\xCE\xB5"     =>   "\xE5",
		"\xCE\xB6"     =>   "\xE6",
		"\xCE\xB7"     =>   "\xE7",
		"\xCE\xB8"     =>   "\xE8",
		"\xCE\xB9"     =>   "\xE9",
		"\xCE\xBA"     =>   "\xEA",
		"\xCE\xBB"     =>   "\xEB",
		"\xCE\xBC"     =>   "\xEC",
		"\xCE\xBD"     =>   "\xED",
		"\xCE\xBE"     =>   "\xEE",
		"\xCE\xBF"     =>   "\xEF",
		"\xCF\x80"     =>   "\xF0",
		"\xCF\x81"     =>   "\xF1",
		"\xCF\x82"     =>   "\xF2",
		"\xCF\x83"     =>   "\xF3",
		"\xCF\x84"     =>   "\xF4",
		"\xCF\x85"     =>   "\xF5",
		"\xCF\x86"     =>   "\xF6",
		"\xCF\x87"     =>   "\xF7",
		"\xCF\x88"     =>   "\xF8",
		"\xCF\x89"     =>   "\xF9",
		"\xCF\x8A"     =>   "\xFA",
		"\xCF\x8B"     =>   "\xFB",
		"\xCF\x8C"     =>   "\xFC",
		"\xCF\x8D"     =>   "\xFD",
		"\xCF\x8E"     =>   "\xFE"
	);
	return strtr($string, $transform);
}
User avatar
jojobarjo32
Registered User
Posts: 164
Joined: Wed Jun 22, 2005 7:38 pm
Location: France

Re: password md5s upon upgrade to phpbb3 - a corner case

Post by jojobarjo32 »

Please excuse me to answer in place of a phpBB developer but I think it must not be done.
Indeed, if the developers begin to include that function for your encoding, they will have to do the same for all other encodings supported... and that's a bit too much for only the password conversion in my opinion :)

As naderman has said previously, if the password was not converted owing to an encoding issue, a message will tell the users to request a new password. Then there is no real issue (ok perhaps a small waste of time for those users... but nothing really important ;)).
to_allo
Registered User
Posts: 4
Joined: Sun May 13, 2007 7:35 pm

Re: password md5s upon upgrade to phpbb3 - a corner case

Post by to_allo »

jojobarjo32:

Like you said yourself, I was not looking for your opinion. However, since you participate in the conversation, I will reply to you:

I don't understand what is "too much". As demostrated by naderman, one line per additional encoding is all that needs addition to auth_db.php. I do not expect the developers to provide the implementation for each and every major encoding. This may easily be provided by the interested parties - in fact I came up with my own in just 5 minutes. Try having a look in includes/utf/data/recode_basic.php and see how straightforward it is: for each array cell initialisation a=>b, I only had to swap places for b and a. I thus am not arguing for something that would place burden on the developers.

Secondly, the code in consideration would only run once per user (on the first post-upgrade login and never, ever again until the end of time). Thus performance will not be degraded either.

Third, mails urging users to provide a new password (and activation emails) are >90% likely to be lost in transit, marked as spam etc. Definitely not a good solution given that the information would still be in the database.

Fourth, I do not like your insinuation that us speakers of non western european languages are second-class users and us being "collateral damage" is alright. Ok, my language is just spoken by 14-15 million people but then again think of the Russians, the Chinese, the Japanese, the Arabs...

Fifth (and possibly most important from the perspective of the developers), I believe that it is a simple mechanism that will prevent a ton of complaints from scorned forum operators (for something that wasn't their fault in the first place), burden on support and negative publicity for the project (as in "phpbb3 is buggy /sloppy etc", which I think the project deserves to be spared of). It will really be a pity if all the worthwhile features of the software and the hard work from the developers are overshadowed, to any extent, from such stuff. In the past I have submitted bugs etc, and the developers have been responsive and the software has served me well so I am saying this in good faith. Such details are often "decisive" in forming public opinion about what is "ready for production environments", "mature" etc and given that phpbb3 is an open source project which does not rely on spin-masters, promotion etc I am only saying "let's avoid self-inflicted damage where it is easy to do so".

Anyway, nowhere did I suggest that the password md5 issue raised should be an immediate priority for the developers. I was making a friendly suggestion and there is still time until the stable release anyway for them to look into the possibility later.
User avatar
naderman
Consultant
Posts: 1727
Joined: Sun Jan 11, 2004 2:11 am
Location: Berlin, Germany
Contact:

Re: password md5s upon upgrade to phpbb3 - a corner case

Post by naderman »

We won't include other encoding for two reasons. One was mentioned above, if we add all encodings that's just too much, and I don't want to be the one who decides which one goes in and which one doesn't. Additionally such a recoding makes hash collisions extremely much more likely. So if we test all encodings, the probability that an incorrect password will allow you to login becomes a whole lot more likely. So I don't like that idea at all.
User avatar
jojobarjo32
Registered User
Posts: 164
Joined: Wed Jun 22, 2005 7:38 pm
Location: France

Re: password md5s upon upgrade to phpbb3 - a corner case

Post by jojobarjo32 »

to_allo wrote: I do not like your insinuation that us speakers of non western european languages are second-class users and us being "collateral damage" is alright. Ok, my language is just spoken by 14-15 million people but then again think of the Russians, the Chinese, the Japanese, the Arabs...

Sorry if I offended you, it was not at all intended. I did not say that "speakers of non western European languages" must be left behind or are "second-class users" :? I was just saying that, in my opinion of (not professional) developer, the conversion for passwords does not need "as much code" as that. naderman said that maybe better than me :) iso-8859-1 and utf-8 must be managed because they are respectively phpBB 2.0's and phpBB 3.0's default encodings. All other encodings, though used by lots of people, can not (again, in my opinion) be managed in the password conversion because of the reasons naderman and I have given.
On the other hand, it's needed to manage them in all other places (the converter is after all very specific... at least in my opinion), as it's already done :)

Again, please excuse me if I offended you or someone else.
to_allo wrote: mails urging users to provide a new password (and activation emails) are >90% likely to be lost in transit, marked as spam etc. Definitely not a good solution given that the information would still be in the database.

You are right :) I must have thought this :? Indeed it can be an issue if those e-mails are marked as SPAM. I don't recall if it's already written or not but I think the messages appearing when a user requests a new password, registers or does something else which needs a confirmation by e-mail must clearly tell those users to check their SPAM folder. This is not perfect, I agree, but I do not see any other solution :?
Roberdin
Registered User
Posts: 1546
Joined: Wed Apr 09, 2003 8:44 pm
Location: London, United Kingdom

Re: password md5s upon upgrade to phpbb3 - a corner case

Post by Roberdin »

Is there no way of detecting what encoding phpBB 2 used during the phpBB 3 installation stage and making some kind of a note of it in the configuration table?
Rob
Post Reply