Fixing broken UTF-8 handling with MySQL driver
Hi,
I noticed that dovecot's MySQL driver doesn't play well with UTF-8 encoded strings. I presume this issue has been around for a while. However, it still fails with the current release candidate 2.2.16-rc1.
I have been using the sieve extdata plugin with a MySQL-backed dict to retrieve a folder name that is then passed to fileinto:
fileinto :create "${extdata.folder_name_drafts}";
Whenever this dict returned a string with non-7byte characters, a line such as the following popped up in sieve.log. (This example query should return "Entwürfe".)
error: folder name specified for fileinto command is not utf-8: Entw?rfe.
Although my MySQL installation is set up to use UTF-8 as the default charset on every level (config, database, table and field), and the character_set_* runtime variables all yield the value "utf8", it still seems that the mysql client library must be instructed to actually use UTF-8 explicitly. Adding the following statement to driver_mysql_connect() fixes the issue for me:
mysql_options(db->mysql, MYSQL_SET_CHARSET_NAME, "utf8");
I checked this against mysql 5.5.41.
Regards, Felix
On 11 Mar 2015, at 21:31, Felix Zandanel <felix@zandanel.me> wrote:
Although my MySQL installation is set up to use UTF-8 as the default charset on every level (config, database, table and field), and the character_set_* runtime variables all yield the value "utf8", it still seems that the mysql client library must be instructed to actually use UTF-8 explicitly. Adding the following statement to driver_mysql_connect() fixes the issue for me:
mysql_options(db->mysql, MYSQL_SET_CHARSET_NAME, "utf8");
I think you can also add to /etc/my.cnf :
[client] default-character-set = utf8
Am 12.03.2015 um 14:55 schrieb Timo Sirainen <tss@iki.fi>:
On 11 Mar 2015, at 21:31, Felix Zandanel <felix@zandanel.me> wrote:
Although my MySQL installation is set up to use UTF-8 as the default charset on every level (config, database, table and field), and the character_set_* runtime variables all yield the value "utf8", it still seems that the mysql client library must be instructed to actually use UTF-8 explicitly. Adding the following statement to driver_mysql_connect() fixes the issue for me:
mysql_options(db->mysql, MYSQL_SET_CHARSET_NAME, "utf8");
I think you can also add to /etc/my.cnf :
[client] default-character-set = utf8
Sorry for the late reply. You were so right, that simple line did the trick. My fault, I didn't read the whole charset documentation of MySQL. It's a shame that UTF-8 isn't the default setting.
Anyway, as dovecot's internals expect all input strings to be UTF-8, wouldn't it be useful to enforce UTF-8 in the database drivers? Using anything else than ASCII / UTF-8 for dovecot's MySQL connections doesn't really make sense, I think. Also, a "default-character-set = utf8" line in my.cnf is a system wide configuration, which might break other software interacting with MySQL—in theory.
participants (2)
-
Felix Zandanel
-
Timo Sirainen