charset_to_utf8 assertion of remaining src_size

newer
dovecot 2.2.18 and...

older
"doveadm backup" doesn't work...

Xin Li

15 May 2015 15 May '15

8:58 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

Hi,

I have seen the following assertion fails on my own mail server (indexer-worker):

Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE)

However, when charset_to_utf8_try returns FALSE (e.g. iconv() got EINVAL or EILSEQ), the for loop in charset_to_utf8 may end earlier, and there may be additional pending characters, and the assertion doesn't seem always hold in the error case, perhaps we should assert that pos is non-zero in the error case?

Cheers,

Xin LI delphij@delphij.net https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.1.2 (FreeBSD)

iQIcBAEBCgAGBQJVVjO/AAoJEJW2GBstM+nsbKUP/29fYdZArCiUSStK0hCtRvC3 o4L35zq4iP4WUry7VqNtdMg2Uiythei9l0JokNOJAy2ZDktJ3ns9CArXHi/pJE7/ V8374Iu9ygb4tzDvcNY/353SFkN+Sw4pJzSGJTelVT8nHxahQMtN7E0u26l54zNx p1H8rgt2nVA8eRvKL9MrhudN3eSI2+Jq4HX+dR4/Tu4qLkSU1nqi8sopQxNXzW4C 4XG4haSbTtXLu0H1MGa8y5pmwrFmZEluNOBm2SOww+UqeteD/MwE/pgA7g+cQx8l /t3FWzSe4C9HROV1VGW5V5HDM7cDvNJZ+nxQGI6L6qInkCBqQx/muIb9nIe7dhro tiHIuX4h8X2zaga+iD+B9fkf4QpDUc11WYE+0DUArb2cNTTAgeTXQ68+g0Q+UC8s OxUVDA4KwOSzkpEifDgvVCdgPG7U87SGe9NlMGjSZvAIEPsww16A3f2IaM4fhFa2 lVuTh1wq8phdWXLbCbZ0F36hAyxVbbwC5Xyj4Ygy3QL4RiBC7CJJ8I4f1lRLnoFz 1Wg0/M7psXBLVN7vx2MKtK5Rj1OvMOKmL4VpkhKcnp2z/Y9I5Qq2CP4C5uOv3DLJ JjoCsnVDpWJKQo6EPleb9OxVJF+NmjHd63zimXbuUyDJ7AJCsx/0m9mJZ5QYf+jU U3OqESCU5e5q++QLd2Kl =ksoA -----END PGP SIGNATURE-----

Show replies by date

Xin Li

16 May 16 May

9:39 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, On 5/15/15 10:58, Xin Li wrote:

...

Hi,

I have seen the following assertion fails on my own mail server (indexer-worker):

Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE)

I've managed to narrow down the issue to one mail, and can reliably trigger the assertion with 'doveadm index -u user INBOX' with only that email. Maybe we should reset the iconv descriptor after each full conversion? i.e.: diff -r 91bbd6753689 src/lib-mail/message-decoder.c - --- a/src/lib-mail/message-decoder.c Thu May 14 23:49:12 2015 +0300 +++ b/src/lib-mail/message-decoder.c Fri May 15 22:52:41 2015 -0700 @@ -347,6 +347,8 @@ sizeof(ctx->translation_buf)); memcpy(ctx->translation_buf, data + pos, ctx->translation_size); + } else { + charset_to_utf8_reset(ctx->charset_trans); } output->data = ctx->buf2->data; output->size = ctx->buf2->used; Cheers, -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVVuYpAAoJEJW2GBstM+nsrr8P+wZRjznaTJuAE7eaKlmO5Zba m9bUHsHh/rZdjfTCfUiwYwU66+EI8O2WEgr0sXynMrRhRWp5BVWMKGTl2loJ/oNL O3qXrGoVmAJoWmnd+ZQ49s+zSJUgXk9rJneN1i4erId6vjN1pcWXfrLbmeKhreN8 EmOMDO8WtV/XBq20UerKu+kGnatC8tNzpTQ5PCTnMc9z1nvNytE/NxM3Gli3wTRG 8nhZY8fS+y6guJfi68RZK6Th1W6jKQQW8+sNPlNbCRPxDb5delXLGq58DnQVk4mM Y6VxzQF4ItTU5h+ikNOaY7v4LoC3uydUdhPkMVVb5JVH+wx9en7PefhV4dH2MNjJ a8dJ4gvsYNBLZ1r3QLFF0NqJMhPXtidKLy7iQkBW0Z1LqmQrEpwZpLVGrdDtWLpA E4HkOmTMEyIbTuK5hG5zNyoF2+XLHSCAELZBHxJAxoHhwakm3n4MhJJao+aJho8+ DbgbGMFk4dDgvOAD0gqANSrIfpZkWGAzUiNiMP015Trg4bQm7CSztuvM6+xhr8EE zzvYjbeCAl3A1+u2q2UFaw8Ze60kD345FyRmkkxLxgJY3r32fSaU+1VvazlbIFKa zq4R2aospgIIxwVQOH3drqGgXpxEe54QG++3IQxmZyNCh3qocb+FnatVupHIJzX/ vvqgdswgAsXP04duqKKn =QaAX -----END PGP SIGNATURE-----

Timo Sirainen

12:32 p.m.

On 16 May 2015, at 09:39, Xin Li wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

Hi,

On 5/15/15 10:58, Xin Li wrote:

...
Hi,

I have seen the following assertion fails on my own mail server (indexer-worker):

Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE)

I've managed to narrow down the issue to one mail, and can reliably trigger the assertion with 'doveadm index -u user INBOX' with only that email.

Can you send me some mail that triggers the crash? I'm having trouble reproducing.

...

Maybe we should reset the iconv descriptor after each full conversion? i.e.:

diff -r 91bbd6753689 src/lib-mail/message-decoder.c - --- a/src/lib-mail/message-decoder.c Thu May 14 23:49:12 2015 +0300 +++ b/src/lib-mail/message-decoder.c Fri May 15 22:52:41 2015 -0700 @@ -347,6 +347,8 @@ sizeof(ctx->translation_buf)); memcpy(ctx->translation_buf, data + pos, ctx->translation_size); + } else { + charset_to_utf8_reset(ctx->charset_trans); }

I think the fix probably belongs to charset-iconv.c

Cassidy Larson

22 May 22 May

1:47 a.m.

I've been seeing this same error as well now: Fatal: master: service(indexer-worker): child 20979 killed with signal 6 (core not dumped - set service indexer-worker { drop_priv_before_exec=yes }) Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE) It's only shown up on one particular user, and after tracking down the message(s) that caused it I found out that there were two identical messages with the same exact MD5/SHA hashes from our storage server in 2012 or so. Removing one of the duplicates fixed it for me. Not sure how that happened, but wonder if it's worth checking for? Thanks, -c On Sat, May 16, 2015 at 3:32 AM, Timo Sirainen wrote:

...

On 16 May 2015, at 09:39, Xin Li wrote:

...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

Hi,

On 5/15/15 10:58, Xin Li wrote:

...
Hi,

I have seen the following assertion fails on my own mail server (indexer-worker):

Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE)

I've managed to narrow down the issue to one mail, and can reliably trigger the assertion with 'doveadm index -u user INBOX' with only that email.

Can you send me some mail that triggers the crash? I'm having trouble reproducing.

...
Maybe we should reset the iconv descriptor after each full conversion? i.e.:

diff -r 91bbd6753689 src/lib-mail/message-decoder.c - --- a/src/lib-mail/message-decoder.c Thu May 14 23:49:12 2015 +0300 +++ b/src/lib-mail/message-decoder.c Fri May 15 22:52:41 2015 -0700 @@ -347,6 +347,8 @@ sizeof(ctx->translation_buf)); memcpy(ctx->translation_buf, data + pos, ctx->translation_size); + } else { + charset_to_utf8_reset(ctx->charset_trans); }

I think the fix probably belongs to charset-iconv.c

Timo Sirainen

1:59 a.m.

On 21 May 2015, at 18:47, Cassidy Larson wrote:

...

I've been seeing this same error as well now:

Fatal: master: service(indexer-worker): child 20979 killed with signal 6 (core not dumped - set service indexer-worker { drop_priv_before_exec=yes }) Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE)

It's only shown up on one particular user, and after tracking down the message(s) that caused it I found out that there were two identical messages with the same exact MD5/SHA hashes from our storage server in 2012 or so. Removing one of the duplicates fixed it for me. Not sure how that happened, but wonder if it's worth checking for?

What OS or Linux distro are you using?

...

Thanks,

-c

On Sat, May 16, 2015 at 3:32 AM, Timo Sirainen mailto:tss@iki.fi> wrote: On 16 May 2015, at 09:39, Xin Li mailto:delphij@delphij.net> wrote:

...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

Hi,

On 5/15/15 10:58, Xin Li wrote:

...
Hi,

I have seen the following assertion fails on my own mail server (indexer-worker):

Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE)

I've managed to narrow down the issue to one mail, and can reliably trigger the assertion with 'doveadm index -u user INBOX' with only that email.

Can you send me some mail that triggers the crash? I'm having trouble reproducing.

...
Maybe we should reset the iconv descriptor after each full conversion? i.e.:

diff -r 91bbd6753689 src/lib-mail/message-decoder.c - --- a/src/lib-mail/message-decoder.c Thu May 14 23:49:12 2015 +0300 +++ b/src/lib-mail/message-decoder.c Fri May 15 22:52:41 2015 -0700 @@ -347,6 +347,8 @@ sizeof(ctx->translation_buf)); memcpy(ctx->translation_buf, data + pos, ctx->translation_size); + } else { + charset_to_utf8_reset(ctx->charset_trans); }

I think the fix probably belongs to charset-iconv.c

3475

Age (days ago)

3481

Last active (days ago)

List overview

4 comments

3 participants

participants (3)

Cassidy Larson
Timo Sirainen
Xin Li