[Dovecot] Found bug on doing IMAP search with UTF8 chars
Hi!
I'm using Roundcube 0.9git as mail frontend and have installed Dovecot 2.1.7 on Debian Squeeze. When searching for mail via Roundcube and the keyword contains "special" chars (like german umlauts), then Dovecot seems to die when the folder does not contain a message which contains this keyword, i.e.
folder contains message /w subject "Dies ist eine Nachricht" => search for "Nachricht" => Roundcube shows that messages folder contains message /w subject "Dies ist eine Nachricht" => search for "Nachricht123" => Roundcube shows error that no messages where found folder contains message /w subject "Auflösung der Frage" => search for "Lösung" => Roundcube shows that messages folder contains message /w subject "Auflösung der Frage" => search for "Lösung123" => Roundcube shows error saying: "Connection to storage server failed"
Looking into /var/log/mail.log shows:
Sep 5 11:43:32 mail dovecot: imap(abc@def.de): Panic: search key not utf8: L.sung123 dsr_sub_allocation Sep 5 11:43:32 mail dovecot: imap(abc@def.de): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x4478a) [0 x7f065ba8e78a] -> /usr/lib/dovecot/libdovecot.so.0(+0x447d6) [0x7f065ba8e7d6] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f065ba655ef ] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9a3cb) [0x7f065bd6f3cb] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9a8aa) [0x7f065bd6f8 aa] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x78641) [0x7f065bd4d641] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_fore ach+0x3c) [0x7f065bd4d7bc] -> /usr/lib/dovecot/libdovecot.so.0(message_parse_header+0x29) [0x7f065ba78eb9] -> /usr/lib/dovecot/libdovecot-st orage.so.0(+0x9a0cf) [0x7f065bd6f0cf] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9b160) [0x7f065bd70160] -> /usr/lib/dovecot/libdovecot- storage.so.0(index_storage_search_next_nonblock+0x48) [0x7f065bd706e8] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonbl ock+0x27) [0x7f065bd51a07] -> dovecot/imap() [0x41668f] -> dovecot/imap(imap_search_start+0xd8) [0x416ad8] -> dovecot/imap(cmd_sort+0x1e6) [ 0x40e636] -> dovecot/imap(command_exec+0x3d) [0x41157d] -> dovecot/imap() [0x4104fe] -> dovecot/imap(client_handle_input+0x135) [0x410805] -
dovecot/imap(client_input+0x5f) [0x41112f] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f065ba9a926] -> /usr/lib/dovecot/ libdovecot.so.0(io_loop_handler_run+0x9f) [0x7f065ba9b95f] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f065ba9a8c8] -> /usr/li b/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f065ba87053] -> dovecot/imap(main+0x2a4) [0x419354] -> /lib/libc.so.6(__libc_start_ma in+0xfd) [0x7f065b706c8d] -> dovecot/imap() [0x408e79]
Huh? This error only occurs when I search for a keyword, that contains probably non-ASCII-US chars (like umlauts, euro sign EUR ...) and no message was found in that folder. When there is any message which contains this keyword, then Roundcube is fine and there error above cannot be found in mail.log. Can anybody confirm this behaviour?
Regards
On 7.9.2012, at 10.30, Weinreich, Nico wrote:
I'm using Roundcube 0.9git as mail frontend and have installed Dovecot 2.1.7 on Debian Squeeze. When searching for mail via Roundcube and the keyword contains "special" chars (like german umlauts), then Dovecot seems to die when the folder does not contain a message which contains this keyword, i.e.
Sep 5 11:43:32 mail dovecot: imap(abc@def.de): Panic: search key not utf8: L.sung123 dsr_sub_allocation
This crash was fixed in v2.1.8 I think.
Huh? This error only occurs when I search for a keyword, that contains probably non-ASCII-US chars (like umlauts, euro sign EUR ...) and no message was found in that folder. When there is any message which contains this keyword, then Roundcube is fine and there error above cannot be found in mail.log. Can anybody confirm this behaviour?
The error is that Roundcube doesn't send the string to Dovecot as UTF8 (or alternatively specify the correct charset). Even with the crashfix the search won't work.
On 09/11/2012 06:19 PM, Timo Sirainen wrote:
The error is that Roundcube doesn't send the string to Dovecot as UTF8 (or alternatively specify the correct charset). Even with the crashfix the search won't work.
This is surprising. Can you provide more info about this issue? -- Aleksander 'A.L.E.C' Machniak LAN Management System Developer [http://lms.org.pl] Roundcube Webmail Developer [http://roundcube.net] --------------------------------------------------- PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
On 11.9.2012, at 19.29, A.L.E.C wrote:
On 09/11/2012 06:19 PM, Timo Sirainen wrote:
The error is that Roundcube doesn't send the string to Dovecot as UTF8 (or alternatively specify the correct charset). Even with the crashfix the search won't work.
This is surprising. Can you provide more info about this issue?
No idea if that's a misconfiguration or what .. or maybe the web browser says it's sending UTF8 but isn't. Anyway, with v2.1.7 it crashes if you send:
a SEARCH subject "8bit non-utf8 text"
I tested a few other possibilities, but that was the only way I could make it crash. v2.1.8 fixes that crash. The only strange thing in the original bug report was the "dsr_sub_allocation" text:
Sep 5 11:43:32 mail dovecot: imap(abc@def.de): Panic: search key not utf8: L.sung123 dsr_sub_allocation
Oh sorry, this seems to be an copy&paste failure. "dsr_sub_allocation" was one of the search terms (we run a data warehouse here)
Von: dovecot-bounces@dovecot.org im Auftrag von Timo Sirainen Gesendet: Di 11.09.2012 22:31 An: A.L.E.C Cc: dovecot@dovecot.org Betreff: Re: [Dovecot] Found bug on doing IMAP search with UTF8 chars
[...]
I tested a few other possibilities, but that was the only way I could make it crash. v2.1.8 fixes that crash. The only strange thing in the original bug report was the "dsr_sub_allocation" text:
Sep 5 11:43:32 mail dovecot: imap(abc@def.de): Panic: search key not utf8: L.sung123 dsr_sub_allocation
Yes, Roundcube had a similiar bug in past. But as I've seen in source code (of Roundcube), they encode the search string as UTF8. I'll give 2.1.8 a try and give feedback.
Thanks so far
I've installed 2.1.9 from source (and Pigeonhole also). The error in Roundcube is still the same:
-searching for a term containg UTF-8 chars and a mail is found in actual folder then all is fine -searching for an UTF-8 term and no mail is found then Roundcube shows still "Connection to storage server failed." -searching for a non-UTF-8 term and no mail is found then Roundcube shows correctly a not-found error
Until here all is exactly the same, except, that I cannot see the Dovecot crash in /var/log/mail.log anymore. May I assume, that this is now a problem of Roundcube? (But if this is related to Roundcube, why do I still get an connection error (see above)?
Von: dovecot-bounces@dovecot.org im Auftrag von Weinreich, Nico Gesendet: Mi 12.09.2012 09:24 An: dovecot@dovecot.org Betreff: Re: [Dovecot] Found bug on doing IMAP search with UTF8 chars
Yes, Roundcube had a similiar bug in past. But as I've seen in source code (of Roundcube), they encode the search string as UTF8. I'll give 2.1.8 a try and give feedback.
Thanks so far
On 09/12/2012 10:04 AM, Weinreich, Nico wrote:
I've installed 2.1.9 from source (and Pigeonhole also). The error in Roundcube is still the same:
-searching for a term containg UTF-8 chars and a mail is found in actual folder then all is fine -searching for an UTF-8 term and no mail is found then Roundcube shows still "Connection to storage server failed."
I think this patch fixes the issue. https://github.com/roundcube/roundcubemail/commit/c093dcc391ad886ab815e6e706... ps. Roundcube has a nice imap_debug option, use it for debugging. -- Aleksander 'A.L.E.C' Machniak LAN Management System Developer [http://lms.org.pl] Roundcube Webmail Developer [http://roundcube.net] --------------------------------------------------- PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
Yeah, this fixed it. Will this patch go into source?
Von: dovecot-bounces@dovecot.org im Auftrag von A.L.E.C Gesendet: Mi 12.09.2012 10:16 An: dovecot@dovecot.org Betreff: Re: [Dovecot] Found bug on doing IMAP search with UTF8 chars
I think this patch fixes the issue. https://github.com/roundcube/roundcubemail/commit/c093dcc391ad886ab815e6e706...
ps. Roundcube has a nice imap_debug option, use it for debugging.
On 09/12/2012 10:04 AM, Weinreich, Nico wrote:
I've installed 2.1.9 from source (and Pigeonhole also). The error in Roundcube is still the same:
I've found what's really going on here. This is initial search request with utf-8: C: A0003 UID SORT (DATE) UTF-8 ALL OR HEADER SUBJECT {10+} C: Lösung123 HEADER FROM {10+} C: Lösung123 S: * SORT S: A0003 OK Sort completed. Now, because of a bug, fixed in commit I posted before, Roundcube sends another request with search arguments converted to US-ASCII. It should do this only if the first command returns "unsupported charset" response. C: A0004 UID SORT (DATE) US-ASCII ALL OR HEADER SUBJECT {9+} C: L.sung123 HEADER FROM {9+} C: L.sung123 S: A0004 NO Invalid search key The problem is that on that convertion ö is converted to character with code 246 (hex F6), which is from extended ASCII table. This is because in PHP (iconv/mb_string) US-ASCII means extended ASCII. So, I'll fix this in Roundcube. -- Aleksander 'A.L.E.C' Machniak LAN Management System Developer [http://lms.org.pl] Roundcube Webmail Developer [http://roundcube.net] --------------------------------------------------- PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
On 09/12/2012 10:37 AM, A.L.E.C wrote:
The problem is that on that convertion ö is converted to character with code 246 (hex F6), which is from extended ASCII table. This is because in PHP (iconv/mb_string) US-ASCII means extended ASCII. So, I'll fix this in Roundcube.
Euh. Of course not PHP issue, the bug was in another place. Fixed. https://github.com/roundcube/roundcubemail/commit/40e1c2d14d11c679d97c00ce83... The fixes are applied to master and release-0.8 branches. ps. sorry for spamming your list, I think it's EOT now. -- Aleksander 'A.L.E.C' Machniak LAN Management System Developer [http://lms.org.pl] Roundcube Webmail Developer [http://roundcube.net] --------------------------------------------------- PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
participants (3)
-
A.L.E.C
-
Timo Sirainen
-
Weinreich, Nico