- dovecot - dovecot.org

Seeking Advice for data recovery
by Riccardo Bicelli 03 Apr '21

03 Apr '21

Hello, we had a drive failure and needed to restore our dovecot server to operational status in no time. So we restored an "old" snapshot backup and worked on the failed drive in order to recover the mails. Data backup didn't worked :-( Now the storage is updated with mails from beginning of time to snapshot, and from recovery day to now, and a copy of the storage of mails from beginning of time to failure. Is there a way to "inject" the mails from "snapshot to failure"? Which is the best way? We're using mdbox. Thanks, Riccardo

1 0

Emails Ending up in Junk
by Asai 03 Apr '21

03 Apr '21

Greetings, I'm trying to debug an issue with a user who is finding emails (user uses Thunderbird as client) ending up in Junk folder, and constantly marks email as Not Junk, but important email ends up in Junk folder. I have sieve set up, but am unable to determine if this is the result of a sieve problem (sieve will automatically move certain emails into Junk if certain headers are present, but these headers are not present on these emails.) I'm looking in Dovecot debug logs to try to figure this out, but all I can find is sometimes this, and I don't know if this is relevant. Can anyone point us in the right direction for solving this issue? Apr 01 11:51:05 imap(user(a)domain.tld)<102535><+kTsYd6+f83AqEbH>: Debug: Mailbox Junk: Mailbox opened because: UID move Apr 01 11:57:08 imap(user(a)domain.tld)<102735><P3dSa96+kM3AqEbH>: Debug: Mailbox Junk: UID 565: Opened mail because: prefetch Apr 01 11:57:08 imap(user(a)domain.tld)<102735><P3dSa96+kM3AqEbH>: Debug: Mailbox Junk: UID 565: Opened mail because: access Apr 01 11:57:08 imap(user(a)domain.tld)<102735><P3dSa96+kM3AqEbH>: Debug: Mailbox Junk: UID 565: Opened mail because: full mail Apr 01 12:00:20 imap(user(a)domain.tld)<71333><OX7P1+2+BfcKCgFy>: Debug: Mailbox Junk: Mailbox opened because: STATUS Apr 01 18:52:10 imap(user(a)domain.tld)<114066><WLt3mPO+z8QKCgFy>: Debug: Mailbox Junk: Mailbox opened because: SELECT

3 3

Virtual folders and mailbox_list_get_root_forced
by Joan Moreau 02 Apr '21

02 Apr '21

Hi The function mailbox_list_get_root_forced returns sometimes the first or the second value of the INDEX param for the same mailbox. How to make sure this returns only the correct one of the corresponding mailbox ? mail_location = maildir:/var/vmail/%d/%n:LAYOUT=fs:INDEX=/var/mailindex namespace { location = virtual:/nix/store/toto-virtual:INDEX=/var/vmail/%d/%n/virtual prefix = virtual/ separator = / subscriptions = no } Thank you

2 5

Re: Re: nginx configuration to pass x-originating-ip
by Henry 01 Apr '21

01 Apr '21

Hello, Does anyone has a solution about it? Thanks

1 0

Replicator: getting full resync requests to sync
by Norman King 31 Mar '21

31 Mar '21

Hello. I'm getting the following output. root@mail:~# doveadm replicator status Queued 'sync' requests 0 Queued 'high' requests 0 Queued 'low' requests 0 Queued 'failed' requests 0 Queued 'full resync' requests 61 Waiting 'failed' requests 4 How can i get the full resync requests to actually sync? Is there a configurable resync delay time that i can use to force a resync of these mailboxes? Also, is there a way using either doveadm replicator or doveadm sync that i can see all the full resync users? Thanks. -- regards Norman King Compass Foundation sysadmin/tech support Phone, 856-974-5335 ext 221 support(a)compassfoundation.io <mailto:support@compassfoundation.io> Website <https://compassfoundation.io>

1 0

RE: Dovecot 2.3.14 - Invalid mailbox name: Name must not have '/'
by Markus Valentin 31 Mar '21

31 Mar '21

> On 03/30/2021 8:25 PM Jürgen Edner <juergen(a)eisfair.org> wrote: > Hi @all, > can anyone please shed some light on the changes of v2.3.14 and what need to be done to get the shared folders working again as in v2.3.13. > I cannot see any invalid characters in the folder names. I would also appreciate any advise on how I can debug the program any further. Hi, can you provide us with the output of `doveconf -n` please? Markus

2 1

Temporary disable push notifications while restoring a mailbox
by Ralf Becker 31 Mar '21

31 Mar '21

We use "doveamd import" to restore mailboxes from a snapshot. Since we use push with Dovecot 2.3.13: mail_plugins = $mail_plugins mail_lua notify push_notification push_notification_lua plugin { push_notification_driver = lua:file=/etc/dovecot/dovecot-push.lua push_lua_url = http://172.17.0.1/ the import creates a flood of push-notifications. Therefore the questions: is there a possibility to disable push for "doveadm import" on the command line (not in general)? Kind regards Ralf -- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0

2 1

Re:dovecot Digest, Vol 215, Issue 77
by 刘莹莹 30 Mar '21

30 Mar '21

------------------ Original ------------------ From: "dovecot" <dovecot-request(a)dovecot.org>; Date: Tue, Mar 30, 2021 11:55 PM To: "dovecot"<dovecot(a)dovecot.org>; Subject: dovecot Digest, Vol 215, Issue 77 Send dovecot mailing list submissions to dovecot(a)dovecot.org To subscribe or unsubscribe via the World Wide Web, visit https://dovecot.org/mailman/listinfo/dovecot or, via email, send a message with subject or body 'help' to dovecot-request(a)dovecot.org You can reach the person managing the list at dovecot-owner(a)dovecot.org When replying, please edit your Subject line so it is more specific than "Re: Contents of dovecot digest..." Today's Topics:    1. special characters in FTS Solr fail (piuma(a)piumalab.org)    2. json_parse_number broken by compiler optimization       (Christian Ehrhardt)    3. Re: json_parse_number broken by compiler optimization (Aki Tuomi) ---------------------------------------------------------------------- Message: 1 Date: Tue, 30 Mar 2021 16:20:56 +0200 From: "piuma(a)piumalab.org" <piuma(a)piumalab.org> To: dovecot(a)dovecot.org Subject: special characters in FTS Solr fail Message-ID: <3c779646-e2fc-9f51-063c-a8816450dbfd(a)piumalab.org> Content-Type: text/plain; charset="utf-8" Hi, I want to report a bug about searching. My dovecot version is 2.3.10.1 (a3d0e1171) and use Solr 7.7.2 for full text search. The issue appear when try to search using solr special characters. For example if I have emails with the subject that contain the only character "(" open parenthesis, then try to search it I receive zero results. In the Solr log seems the "(" is correctly excaped as "\(": 2021-03-30 15:14:20.304 INFO  (qtp333683827-20) [   x:dovecot] o.a.s.c.S.Request [dovecot]  webapp=/solr path=/select params={q={!lucene+q.op%3DAND}subject:\(&fl=uid,score&sort=uid+asc&fq=%2Bbox:6fff5821c5a2365df4060000d0d97f83+%2Buser:titolare01@kendama.it&rows=631&wt=xml} hits=0 status=0 QTime=2 I provide dovecot-sysreport as attachment. Best regards    Danilo -------------- next part -------------- A non-text attachment was scrubbed... Name: dovecot-sysreport-pecbox02.par-tec.it-1617116347.tar.gz Type: application/gzip Size: 7395 bytes Desc: not available URL: <https://dovecot.org/pipermail/dovecot/attachments/20210330/08492366/attachm… ------------------------------ Message: 2 Date: Tue, 30 Mar 2021 17:53:27 +0200 From: Christian Ehrhardt <christian.ehrhardt(a)canonical.com> To: dovecot(a)dovecot.org Subject: json_parse_number broken by compiler optimization Message-ID: <CAATJJ0JB9e4cZrS=hiM0r=yF4-Fz693ZK3qLa41L-rYVfy8QbQ(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" Hi, the recent Ubuntu (re)builds uncovered an issue with dovecot 1:2.3.13+dfsg1-1 build log: https://launchpadlibrarian.net/529849650/buildlog_ubuntu-hirsute-amd64.dove… A coworker tried 2.3.14 but got the same result. What fails is the json_parser build time test like:   test-json-parser.c:161: Assert(#25) failed: null_strcmp(json_output[pos].value, value) == 0 I was looking into that a bit more and what I found is that it is dependent on the new toolchain of gcc 10.2.0-1. Not all calls to json_parse_* fail, e.g. the first one looks all good and passes I was iterating the tests using a report function defined like (gdb) define repcon >c >p pos >p json_output[pos].type >p type >p json_output[pos].value >p value >call null_strcmp(json_output[pos].value, value) >end The first one to be bad was: Breakpoint 2, test_json_parser_success (full_size=<optimized out>) at test-json-parser.c:161 161 test_assert_idx(null_strcmp(json_output[pos].value, value) == 0, pos); $84 = 25 $85 = JSON_TYPE_NUMBER $86 = JSON_TYPE_NUMBER $87 = 0x555555633b25 "-12.456" $88 = 0x555555693110 "" $89 = 45 Earlier and later parsing was happy, for example Breakpoint 2, test_json_parser_success (full_size=<optimized out>) at test-json-parser.c:161 161 test_assert_idx(null_strcmp(json_output[pos].value, value) == 0, pos); $90 = 27 $91 = JSON_TYPE_NUMBER $92 = JSON_TYPE_NUMBER $93 = 0x555555633b32 "12.456e9" $94 = 0x555555693110 "12.456e9" $95 = 0 (gdb) We have two things we compare here. 1. json_output[] which is a static define and for this value is    67 ????{ JSON_TYPE_NUMBER, "-12.456" }, 2. the return value that json_parse_next returns.    25 ????"  \"sub2\":-12.456,\n"   148 ????????????????ret = json_parse_next(parser, &type, &value); I tried a non negative number and got success which is suspicious Breakpoint 5, test_json_parser_success (full_size=<optimized out>) at test-json-parser.c:164 164 test_assert_idx(null_strcmp(json_output[pos].value, value) == 0, pos); $122 = 25 $123 = JSON_TYPE_NUMBER $124 = JSON_TYPE_NUMBER $125 = 0x555555633b2c "12.456" $126 = 0x555555693110 "12.456" $127 = 0 (gdb) Also the return value otherwise LGTM, it is recognized as a number: 540 } else if ((ret = json_parse_number(parser, value_r)) >= 0) { (gdb) n 541 *type_r = JSON_TYPE_NUMBER; So parsing worked one might think, but the value was not set. (gdb) p *value_r $833 = 0x555555693110 "" It turns out json_parse_number is too optimized. Too optimized for debugging further into json_parse_number. But also optimization is causing the issue. With -O0 or -O1 it works fine, even putting the lower optimization level just around json_parse_number it is enough to avoid the issue. Not a fix but "avoidance" for now: --- a/src/lib/json-parser.c +++ b/src/lib/json-parser.c @@ -386,6 +386,8 @@ static int json_parse_int(struct json_pa   return 1;  } +#pragma GCC push_options +#pragma GCC optimize ("-O0")  static int json_parse_number(struct json_parser *parser, const char **value_r)  {   int ret; @@ -415,6 +417,7 @@ static int json_parse_number(struct json   *value_r = str_c(parser->value);   return 1;  } +#pragma GCC pop_options  static int json_parse_atom(struct json_parser *parser, const char *atom)  { I was trying to see what happened and tracked each str_append_c that happened inside json_parse_number. A breakpoint on str_append_c and backtraces showed that we clearly do the "right" parsing. From the backtraces I've seen it does (just from positions): For example (gdb) bt #0  str_append_c (chr=<optimized out>, str=0x5555556930d0) at str.h:44 #1  json_parse_digits (parser=<optimized out>) at json-parser.c:366 => that is a digit It does "- digit digit dot digit digit digit" This perfectly matches the input value, but still the result stays empty: (gdb) p (char*)(str->data) $738 = 0x555555693110 "" In comparison - when parsing the latter 12.456e9 which works I see two odd things: 1. the value isn't cleared despite     str_truncate(parser->value, 0);    at the beginning of json_parse_number 2. the value (this time) is read exactly as I assumed Breakpoint 17, json_parse_number (value_r=0x7fffffffe1a0, parser=0x555555692ee0) at json-parser.c:394 394 str_truncate(parser->value, 0); Breakpoint 18, str_append_c (chr=<optimized out>, str=0x5555556930d0) at str.h:44 44 buffer_append_c(str, chr); (gdb) p (char*)(str->data) $741 = 0x555555693110 "sub3" ^^ not cleared, still has "sub3" even later on Breakpoint 18, str_append_c (chr=<optimized out>, str=0x5555556930d0) at str.h:44 44 buffer_append_c(str, chr); (gdb) p (char*)(str->data) $742 = 0x555555693110 "1ub3" ^^ reading left to right one char at a time "over the old value", which didn't update the data. So it seems in json_parse_number the ordering of a) str_truncate(parser->value, 0); b) the parsing itself is messed up by optimization. We already know from the above that reducing just this function to -O1 or less avoids the issue. But what is it really - I don't know? Maybe you have a better idea what is going on, the issue should be reproducible when building dovecot with the toolchain present on the latest Ubuntu being 21.04 (Hirsute) right now. -- Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd ------------------------------ Message: 3 Date: Tue, 30 Mar 2021 18:55:44 +0300 (EEST) From: Aki Tuomi <aki.tuomi(a)open-xchange.com> To: Christian Ehrhardt <christian.ehrhardt(a)canonical.com>, dovecot(a)dovecot.org Subject: Re: json_parse_number broken by compiler optimization Message-ID: <1219230463.3728.1617119744636(a)appsuite-dev-gw1.open-xchange.com> Content-Type: text/plain; charset=UTF-8 > On 30/03/2021 18:53 Christian Ehrhardt <christian.ehrhardt(a)canonical.com> wrote: > >  > Hi, > the recent Ubuntu (re)builds uncovered an issue with dovecot 1:2.3.13+dfsg1-1 > build log: https://launchpadlibrarian.net/529849650/buildlog_ubuntu-hirsute-amd64.dove… > A coworker tried 2.3.14 but got the same result. > > What fails is the json_parser build time test like: >   test-json-parser.c:161: Assert(#25) failed: > null_strcmp(json_output[pos].value, value) == 0 > > I was looking into that a bit more and what I found is that it is > dependent on the new toolchain > of gcc 10.2.0-1. > > Not all calls to json_parse_* fail, e.g. the first one looks all good and passes > I was iterating the tests using a report function defined like > > (gdb) define repcon > >c > >p pos > >p json_output[pos].type > >p type > >p json_output[pos].value > >p value > >call null_strcmp(json_output[pos].value, value) > >end > > The first one to be bad was: > Breakpoint 2, test_json_parser_success (full_size=<optimized out>) at > test-json-parser.c:161 > 161 test_assert_idx(null_strcmp(json_output[pos].value, value) == 0, pos); > $84 = 25 > $85 = JSON_TYPE_NUMBER > $86 = JSON_TYPE_NUMBER > $87 = 0x555555633b25 "-12.456" > $88 = 0x555555693110 "" > $89 = 45 > > Earlier and later parsing was happy, for example > > Breakpoint 2, test_json_parser_success (full_size=<optimized out>) at > test-json-parser.c:161 > 161 test_assert_idx(null_strcmp(json_output[pos].value, value) == 0, pos); > $90 = 27 > $91 = JSON_TYPE_NUMBER > $92 = JSON_TYPE_NUMBER > $93 = 0x555555633b32 "12.456e9" > $94 = 0x555555693110 "12.456e9" > $95 = 0 > (gdb) > > > We have two things we compare here. > 1. json_output[] which is a static define and for this value is >    67 ????{ JSON_TYPE_NUMBER, "-12.456" }, > 2. the return value that json_parse_next returns. >    25 ????"  \"sub2\":-12.456,\n" >   148 ????????????????ret = json_parse_next(parser, &type, &value); > > I tried a non negative number and got success which is suspicious > > Breakpoint 5, test_json_parser_success (full_size=<optimized out>) at > test-json-parser.c:164 > 164 test_assert_idx(null_strcmp(json_output[pos].value, value) == 0, pos); > $122 = 25 > $123 = JSON_TYPE_NUMBER > $124 = JSON_TYPE_NUMBER > $125 = 0x555555633b2c "12.456" > $126 = 0x555555693110 "12.456" > $127 = 0 > (gdb) > > > Also the return value otherwise LGTM, it is recognized as a number: > 540 } else if ((ret = json_parse_number(parser, value_r)) >= 0) { > (gdb) n > 541 *type_r = JSON_TYPE_NUMBER; > > So parsing worked one might think, but the value was not set. > > (gdb) p *value_r > $833 = 0x555555693110 "" > > It turns out json_parse_number is too optimized. > Too optimized for debugging further into json_parse_number. > But also optimization is causing the issue. > With -O0 or -O1 it works fine, even putting the lower optimization > level just around json_parse_number it is enough to avoid the issue. > > > Not a fix but "avoidance" for now: > > --- a/src/lib/json-parser.c > +++ b/src/lib/json-parser.c > @@ -386,6 +386,8 @@ static int json_parse_int(struct json_pa >   return 1; >  } > > +#pragma GCC push_options > +#pragma GCC optimize ("-O0") >  static int json_parse_number(struct json_parser *parser, const char **value_r) >  { >   int ret; > @@ -415,6 +417,7 @@ static int json_parse_number(struct json >   *value_r = str_c(parser->value); >   return 1; >  } > +#pragma GCC pop_options > >  static int json_parse_atom(struct json_parser *parser, const char *atom) >  { > > > I was trying to see what happened and tracked each str_append_c > that happened inside json_parse_number. > A breakpoint on str_append_c and backtraces showed that we clearly do the > "right" parsing. From the backtraces I've seen it does (just from positions): > > For example > (gdb) bt > #0  str_append_c (chr=<optimized out>, str=0x5555556930d0) at str.h:44 > #1  json_parse_digits (parser=<optimized out>) at json-parser.c:366 > => that is a digit > > It does "- digit digit dot digit digit digit" > > This perfectly matches the input value, but still the result stays empty: > (gdb) p (char*)(str->data) > $738 = 0x555555693110 "" > > In comparison - when parsing the latter 12.456e9 which works I see two > odd things: > 1. the value isn't cleared despite >     str_truncate(parser->value, 0); >    at the beginning of json_parse_number > 2. the value (this time) is read exactly as I assumed > > Breakpoint 17, json_parse_number (value_r=0x7fffffffe1a0, > parser=0x555555692ee0) at json-parser.c:394 > 394 str_truncate(parser->value, 0); > Breakpoint 18, str_append_c (chr=<optimized out>, str=0x5555556930d0) > at str.h:44 > 44 buffer_append_c(str, chr); > (gdb) p (char*)(str->data) > $741 = 0x555555693110 "sub3" > > ^^ not cleared, still has "sub3" even later on > > Breakpoint 18, str_append_c (chr=<optimized out>, str=0x5555556930d0) > at str.h:44 > 44 buffer_append_c(str, chr); > (gdb) p (char*)(str->data) > $742 = 0x555555693110 "1ub3" > > ^^ reading left to right one char at a time "over the old value", > which didn't update the data. > > So it seems in json_parse_number the ordering of > a) str_truncate(parser->value, 0); > b) the parsing itself > is messed up by optimization. > > We already know from the above that reducing just this function to -O1 or less > avoids the issue. But what is it really - I don't know? > > Maybe you have a better idea what is going on, the issue should be > reproducible when building > dovecot with the toolchain present on the latest Ubuntu being 21.04 > (Hirsute) right now. > > -- > Christian Ehrhardt > Staff Engineer, Ubuntu Server > Canonical Ltd Thanks! We'll look into this. Aki ------------------------------ Subject: Digest Footer _______________________________________________ dovecot mailing list dovecot(a)dovecot.org https://dovecot.org/mailman/listinfo/dovecot ------------------------------ End of dovecot Digest, Vol 215, Issue 77 ****************************************

1 0

RE: Dovecot 2.3.14 - Invalid mailbox name: Name must not have '/'
by Jürgen Edner 30 Mar '21

30 Mar '21

Hi @all, can anyone please shed some light on the changes of v2.3.14 and what need to be done to get the shared folders working again as in v2.3.13. I cannot see any invalid characters in the folder names. I would also appreciate any advise on how I can debug the program any further. Thanks Juergen -- Email: juergen(a)eisfair.org

1 0

special characters in FTS Solr fail
by piuma＠piumalab.org 30 Mar '21

30 Mar '21

Hi, I want to report a bug about searching. My dovecot version is 2.3.10.1 (a3d0e1171) and use Solr 7.7.2 for full text search. The issue appear when try to search using solr special characters. For example if I have emails with the subject that contain the only character "(" open parenthesis, then try to search it I receive zero results. In the Solr log seems the "(" is correctly excaped as "\(": 2021-03-30 15:14:20.304 INFO (qtp333683827-20) [ x:dovecot] o.a.s.c.S.Request [dovecot] webapp=/solr path=/select params={q={!lucene+q.op%3DAND}subject:\(&fl=uid,score&sort=uid+asc&fq=%2Bbox:6fff5821c5a2365df4060000d0d97f83+%2Buser:titolare01@kendama.it&rows=631&wt=xml} hits=0 status=0 QTime=2 I provide dovecot-sysreport as attachment. Best regards Danilo

1 0