[Dovecot] Password authentication and character set
Hi,
I've searched in the wiki and in the mailinglist archives but haven't found anything about password character sets within the dovecot authentication deamon.
My problem is that we have users with passwords containing scandinavian characters (äöå, umlauts) and the debug log shows that different clients send the password in different charsets. The passwords are stored in a Mysql table, if that makes any difference.
Outlook Express with LATIN 1 (ISO-8859-1):
Nov 18 16:56:39 resilar dovecot: auth-worker(default): sql(fgr-1,193.64.206.190): Password mismatch Nov 18 16:56:39 resilar dovecot: auth-worker(default): sql(fgr-1,193.64.206.190): MD5(<E4><E4>kk<F6>si<E4>) != '$1$xMPPHRdL$I0mrlPi5FMtwauSf20vjz0'
MacMail UTF8:
Nov 18 17:23:37 resilar dovecot: auth-worker(default): sql(fgr-1,193.64.206.190): Password mismatch Nov 18 17:23:37 resilar dovecot: auth-worker(default): sql(fgr-1,193.64.206.190): MD5(ääkkösiä12) != '$1$xMPPHRdL$I0mrlPi5FMtwauSf20vjz0'
Is there a setting that "forces" the authentication daemon to convert the provided password to a specific charset before the comparison takes place, or how should one handle this?
dovecot -n
# 1.1.4: /etc/dovecot/dovecot.conf log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s managesieve ssl_cert_file: /etc/ssl/certs/mail.crt ssl_key_file: /etc/ssl/private/mail.key ssl_cipher_list: ALL:!LOW disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_executable(managesieve): /usr/lib/dovecot/managesieve-login login_greeting: mail ready. login_process_per_connection: no mail_privileged_group: mail mail_location: maildir:~/mail mail_debug: yes mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_executable(managesieve): /usr/lib/dovecot/managesieve mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve imap_client_workarounds(default): delay-newmail outlook-idle imap_client_workarounds(imap): delay-newmail outlook-idle imap_client_workarounds(pop3): imap_client_workarounds(managesieve): pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh pop3_client_workarounds(managesieve): sieve_storage(default): sieve_storage(imap): sieve_storage(pop3): sieve_storage(managesieve): ~/ sieve(default): sieve(imap): sieve(pop3): sieve(managesieve): ~/.dovecot.sieve auth default: mechanisms: plain login username_chars: abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@+ verbose: yes debug: yes debug_passwords: yes worker_max_count: 50 passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: prefetch userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 384 user: vmail plugin: quota: maildir
Chears
Fredrik
Fredrik Grönqvist
On Nov 18, 2008, at 5:32 PM, Fredrik Grönqvist wrote:
Is there a setting that "forces" the authentication daemon to
convert the provided password to a specific charset before the
comparison takes place, or how should one handle this?
Dovecot doesn't know the character set that the client is using, so it
can't do charset conversion reliably. So the possibilities would be:
a) UTF-8 vs. another 8bit charset can be detected pretty well by
checking if the input string is valid UTF-8 or not. So there could be
a setting to specify the fallback charset.
b) Just brute force through all the configured charsets and test the
password against all of them.
I don't really like either solution and I don't have much interest in
coding those myself. Feel free to do it yourself though, I might even
accept patches. :)
On Tue, Nov 18, 2008 at 05:51:05PM +0200, Timo Sirainen wrote:
On Nov 18, 2008, at 5:32 PM, Fredrik Grönqvist wrote:
Is there a setting that "forces" the authentication daemon to
convert the provided password to a specific charset before the
comparison takes place, or how should one handle this?Dovecot doesn't know the character set that the client is using, so it
can't do charset conversion reliably. So the possibilities would be:a) UTF-8 vs. another 8bit charset can be detected pretty well by
checking if the input string is valid UTF-8 or not. So there could be
a setting to specify the fallback charset.b) Just brute force through all the configured charsets and test the
password against all of them.I don't really like either solution and I don't have much interest in
coding those myself. Feel free to do it yourself though, I might even
accept patches. :)
It seems like this is a limitation in the IMAP protocol. From RFC 3501:
Characters are 7-bit US-ASCII unless otherwise specified. Other character sets are indicated using a "CHARSET", as described in [MIME-IMT] and defined in [CHARSET]. CHARSETs have important additional semantics in addition to defining character set; refer to these documents for more detail.
The SEARCH command has this optional CHARSET flag (to search in messages in non-ascii charsets) but it seems like the LOGIN command should accept only 7-bit ASCII arguments...
GeertOn Tue, 2008-11-18 at 17:26 +0100, Geert Hendrickx wrote:
On Tue, Nov 18, 2008 at 05:51:05PM +0200, Timo Sirainen wrote:
On Nov 18, 2008, at 5:32 PM, Fredrik Grönqvist wrote:
Is there a setting that "forces" the authentication daemon to
convert the provided password to a specific charset before the
comparison takes place, or how should one handle this?Dovecot doesn't know the character set that the client is using, so it
can't do charset conversion reliably. So the possibilities would be:It seems like this is a limitation in the IMAP protocol. From RFC 3501:
I remember reading something about using UTF-8 and stringprep in authentication strings, probably some SASL spec or something. Dovecot should implement it some day.. But that won't help in any way if the client doesn't send the password as UTF-8.
18.11.2008 19:03, Timo Sirainen wrote:
On Tue, 2008-11-18 at 17:26 +0100, Geert Hendrickx wrote:
On Tue, Nov 18, 2008 at 05:51:05PM +0200, Timo Sirainen wrote:
On Nov 18, 2008, at 5:32 PM, Fredrik Grönqvist wrote:
Is there a setting that "forces" the authentication daemon to
convert the provided password to a specific charset before the
comparison takes place, or how should one handle this?Dovecot doesn't know the character set that the client is using, so it
can't do charset conversion reliably. So the possibilities would be:It seems like this is a limitation in the IMAP protocol. From RFC 3501:
I remember reading something about using UTF-8 and stringprep in authentication strings, probably some SASL spec or something. Dovecot should implement it some day.. But that won't help in any way if the client doesn't send the password as UTF-8.
Ok, I see how this makes things problematic. One couldn't just encode it to UTF-8 anyway and do the comparison after that (provided there would be an option enabled)?
So basically a password containing any non 7-bit ASCII is only "correct" when provided by a client using the same charset as the password is stored in... If the RFC states that the password should be provided as 7-bit ASCII then I think I'll google for a reason why some clients send the password as something else.
Chears, Fredrik
--
Fredrik Grönqvist
On Tue, 2008-11-18 at 19:27 +0200, Fredrik Grönqvist wrote:
18.11.2008 19:03, Timo Sirainen wrote:
On Tue, 2008-11-18 at 17:26 +0100, Geert Hendrickx wrote:
On Tue, Nov 18, 2008 at 05:51:05PM +0200, Timo Sirainen wrote:
On Nov 18, 2008, at 5:32 PM, Fredrik Grönqvist wrote:
Is there a setting that "forces" the authentication daemon to
convert the provided password to a specific charset before the
comparison takes place, or how should one handle this?Dovecot doesn't know the character set that the client is using, so it
can't do charset conversion reliably. So the possibilities would be:It seems like this is a limitation in the IMAP protocol. From RFC 3501:
I remember reading something about using UTF-8 and stringprep in authentication strings, probably some SASL spec or something. Dovecot should implement it some day.. But that won't help in any way if the client doesn't send the password as UTF-8.
Ok, I see how this makes things problematic. One couldn't just encode it to UTF-8 anyway and do the comparison after that (provided there would be an option enabled)?
You can encode everything to UTF-8, but the result will be different depending on what the source character set is. If by "option" you mean that you'd have a single setting that specifies which the "non-utf8 charset" is that (hopefully) all your users are using, then sure that would be the a) choice in my previous reply.
So basically a password containing any non 7-bit ASCII is only "correct" when provided by a client using the same charset as the password is stored in... If the RFC states that the password should be provided as 7-bit ASCII then I think I'll google for a reason why some clients send the password as something else.
Most client programmers haven't even thought about the whole issue. The password is typically 7bit. So they just send the password using whatever charset that the OS by default happens to use.
In your case you're most likely not really seeing ISO-8859-1 charset, but rather Windows-1252. Although Windows-1252 is a superset of ISO-8859-1, but things like euro character is present in 1252 but not in 8859-1 (and euro in a different position in 8859-15).
18.11.2008 19:57, Timo Sirainen wrote:
Ok, I see how this makes things problematic. One couldn't just encode it to UTF-8 anyway and do the comparison after that (provided there would be an option enabled)?
You can encode everything to UTF-8, but the result will be different depending on what the source character set is. If by "option" you mean that you'd have a single setting that specifies which the "non-utf8 charset" is that (hopefully) all your users are using, then sure that would be the a) choice in my previous reply.
Yes, you're right. I suppose didn't think that through.
So basically a password containing any non 7-bit ASCII is only "correct" when provided by a client using the same charset as the password is stored in... If the RFC states that the password should be provided as 7-bit ASCII then I think I'll google for a reason why some clients send the password as something else.
Most client programmers haven't even thought about the whole issue. The password is typically 7bit. So they just send the password using whatever charset that the OS by default happens to use.
In your case you're most likely not really seeing ISO-8859-1 charset, but rather Windows-1252. Although Windows-1252 is a superset of ISO-8859-1, but things like euro character is present in 1252 but not in 8859-1 (and euro in a different position in 8859-15).
Yes, I see. So in light of this and the conversation on the imap-protocol -list
http://mailman2.u.washington.edu/pipermail/imap-protocol/2008-February/00082...
our current options seem to boil down to having the passwords ISO-8859-1 encoded (given the demographics of our users). Those using operating systems with native UTF-8 clients have to use passwords containing only 7-bit characters.
I didn't realise the specifications were so flexible on this password issue.
Chears, Fredrik
On Tue, Nov 18, 2008 at 10:00:00PM +0200, Fredrik Grönqvist wrote:
Yes, I see. So in light of this and the conversation on the imap-protocol -list
http://mailman2.u.washington.edu/pipermail/imap-protocol/2008-February/00082...
our current options seem to boil down to having the passwords ISO-8859-1 encoded (given the demographics of our users). Those using operating systems with native UTF-8 clients have to use passwords containing only 7-bit characters.
Actually I would do it the other way around. You can't really explain to your UTF-8 using users "you should use that older client instead of this newer one to make your login work". And some day you'll have to switch to UTF-8 anyway.
I didn't realise the specifications were so flexible on this password issue.
s/flexible/vague/ :-)
The consensus on the imap-protocol list, and particularly the message you refer to, seems to be "we should replace ASCII with UTF-8 in the spec".
Geert19.11.2008 01:34, Geert Hendrickx wrote:
On Tue, Nov 18, 2008 at 10:00:00PM +0200, Fredrik Grönqvist wrote:
Yes, I see. So in light of this and the conversation on the imap-protocol -list
http://mailman2.u.washington.edu/pipermail/imap-protocol/2008-February/00082...
our current options seem to boil down to having the passwords ISO-8859-1 encoded (given the demographics of our users). Those using operating systems with native UTF-8 clients have to use passwords containing only 7-bit characters.
Actually I would do it the other way around. You can't really explain to your UTF-8 using users "you should use that older client instead of this newer one to make your login work". And some day you'll have to switch to UTF-8 anyway.
Yes, I agree that it should be in UTF-8. My specific problem is that about 80% (a rough estimate) of our users are on either Windows or webmail. Those having passwords containing umlauts etc can log on, using their current client, if the passwords are kept ISO-8859-1 encoded instead of UTF-8.
I didn't realise the specifications were so flexible on this password issue.
s/flexible/vague/ :-)
The consensus on the imap-protocol list, and particularly the message you refer to, seems to be "we should replace ASCII with UTF-8 in the spec".
It does seem that way, and while I think it will be increasingly important in the future, I also got the feeling that this change will take a long time to filter down to the implementations (as both servers and clients need to change).
As Timo pointed out, the options to "fix" this on the server side are currently quite limited, so it seems I have to stick to the lowest common denominator in our password policy.
Chears, Fredrik
On Wed, Nov 19, 2008 at 08:44:21AM +0200, Fredrik Grönqvist wrote:
Yes, I agree that it should be in UTF-8. My specific problem is that about 80% (a rough estimate) of our users are on either Windows or webmail. Those having passwords containing umlauts etc can log on, using their current client, if the passwords are kept ISO-8859-1 encoded instead of UTF-8.
As Timo pointed out, the options to "fix" this on the server side are currently quite limited, so it seems I have to stick to the lowest common denominator in our password policy.
If you only have to support two or three different charsets, I think you could use a clever MySQL passdb query to match either of them.
Have a look at http://wiki.dovecot.org/AuthDatabase/SQL under "Password verification by SQL server", and expand the password match to something like (passwd = PASSWORD('%w') OR passwd = PASSWORD(CONVERT(_latin1'%w' USING utf8))) assuming you store the passwords as UTF-8 and assume the input is either UTF-8 or ISO-8859-1.
Geert19.11.2008 10:27, Geert Hendrickx wrote:
On Wed, Nov 19, 2008 at 08:44:21AM +0200, Fredrik Grönqvist wrote:
Yes, I agree that it should be in UTF-8. My specific problem is that about 80% (a rough estimate) of our users are on either Windows or webmail. Those having passwords containing umlauts etc can log on, using their current client, if the passwords are kept ISO-8859-1 encoded instead of UTF-8.
As Timo pointed out, the options to "fix" this on the server side are currently quite limited, so it seems I have to stick to the lowest common denominator in our password policy.
If you only have to support two or three different charsets, I think you could use a clever MySQL passdb query to match either of them.
Have a look at http://wiki.dovecot.org/AuthDatabase/SQL under "Password verification by SQL server", and expand the password match to something like (passwd = PASSWORD('%w') OR passwd = PASSWORD(CONVERT(_latin1'%w' USING utf8))) assuming you store the passwords as UTF-8 and assume the input is either UTF-8 or ISO-8859-1.
Geert
Thanks for the info, a setup like this is what I opted for eventually. I added a note to that wiki page that the query also needs to return the "nopassword" -field for Dovecot 1.1+ to accept the NULL password:
# NOTE: '\' line splitting is used only for readability, currently Dovecot doesn't support it password_query = SELECT NULL AS password, 'nopassword', userid AS user
FROM users WHERE userid = '%u' AND mysql_pass = password('%w')
In case someone else is looking for info about this, this workaround works for DB backed accounts, as you mention, but will probably not work with an LDAP (or other) backend.
Chears, Fredrik
On Wed, Nov 19, 2008 at 01:54:48PM +0200, Fredrik Grönqvist wrote:
Thanks for the info, a setup like this is what I opted for eventually. I added a note to that wiki page that the query also needs to return the "nopassword" -field for Dovecot 1.1+ to accept the NULL password:
Ok, good catch (I hadn't actually tested the example).
In case someone else is looking for info about this, this workaround works for DB backed accounts, as you mention, but will probably not work with an LDAP (or other) backend.
Right, it's just a hack that fits your particular setup. :-)
For a proper solution the IMAP specification would have to be updated to allow non-ASCII characters in logins/passwords, and apparantly they _are_ discussing that. But even if the IMAP protocol gets updated to allow it, this would also impact eg. POP3 and SMTP authentication which in many setups use the same passwords. And of course all IMAP (and other) client software (including webmails etc) would have to be adapted, too.
Geert19.11.2008 14:33, Geert Hendrickx wrote:
In case someone else is looking for info about this, this workaround works for DB backed accounts, as you mention, but will probably not work with an LDAP (or other) backend.
Right, it's just a hack that fits your particular setup. :-)
For a proper solution the IMAP specification would have to be updated to allow non-ASCII characters in logins/passwords, and apparantly they _are_ discussing that. But even if the IMAP protocol gets updated to allow it, this would also impact eg. POP3 and SMTP authentication which in many setups use the same passwords. And of course all IMAP (and other) client software (including webmails etc) would have to be adapted, too.
Geert
Yep, that would be the proper solution, and it's good that there is some progress towards this, but I'm not holding my breath for this to actually happen any time soon.
The need for backwards compatibility will probably make the adoption a slow (and possibly painful) transition.
Chears, Fredrik
On Tue, Nov 18, 2008 at 07:03:04PM +0200, Timo Sirainen wrote:
On Tue, 2008-11-18 at 17:26 +0100, Geert Hendrickx wrote:
It seems like this is a limitation in the IMAP protocol. From RFC 3501:
I remember reading something about using UTF-8 and stringprep in authentication strings, probably some SASL spec or something. Dovecot should implement it some day.. But that won't help in any way if the client doesn't send the password as UTF-8.
There has recently been a discussion about this on the imap-protocol mailing list:
http://mailman2.u.washington.edu/pipermail/imap-protocol/2008-February/threa...
(they talk about LOGIN as well)
Geert
participants (3)
-
Fredrik Grönqvist
-
Geert Hendrickx
-
Timo Sirainen