November 2020 - dovecot

Recovering expunged but not purged e-mails from mdbox with zlib compression
by Daniel Schütze 10 Nov '20

10 Nov '20

Yesterday I had to recover an e-mail which a user had deleted. If I understand this correctly the message was expunged but not purged (doveadm purge had not been run). This e-mail was clearly still in the mdbox stored with zlib compression as I could tell using the doveadm dump command (doveadm dump -t dbox filename). I could however not reveal the e-mail with the normal doveadm fetch -u username "body" guid (guid from dump output) In the end I was able to recover the e-mail by cutting it out from the mdbox and running gunzip over it, but this method was very fiddly and would not have worked for a bulk job. I appreciate I could have just deleted the index files and gotten all the deleted messages back by having the index rebuilt but that would have been the proverbial "hammer to crack a nut". Fortunately this e-mail had no attachment for me to worry about, as those are also detached for sis by dovecot. Can anyone tell me of a quick and easy way to recover one or multiple e-mails marked as expunged but not purged which are stored in a mdbox with zlib compression? I'm sure I'll have to do this again the future and my method wouldn't work with a folder! Clearly if I was not using zlib compression I could just have read the contents of the mdbox without any complication. -- Daniel Schütze Director ------------------ CWA International Ltd 5th Floor, 42 Trinity Square London EC3N 4DJ (t) + 44 (0)20 7242 8444 (e) dms(a)cwa.uk.com (w) http://www.cwa.international/

2 2

Sieve body test
by ＠lbutlr 10 Nov '20

10 Nov '20

I have an email where I need to edit the body. I know this is generally a bad idea, but in this case I need to do it. The email comes in automatically every week or two, and so I thought that SIEVE would be the way to go. if header :contains "from" "theaddress@tehdomain" { if body :raw :contains "A string" { # Magic happens here } } It looks like what I need to do is enable and use vnd.dovecot.filter to pipe the message to a script. (Er. | not vnd.dovecot.pipe) So, if I wanted to transform the message so that the string "foobar13" was changed to "foo-bar-13" I would add filter :try "myfoobarfix.sh" After "magic happens here"> OK so far? Now, what are the requirements on the executable script? Would it literally be as simple as an executable file that simply said sed -e '|foobar13|foo-bar-13|g' Or does it need to be a shebang script and take input on $1 and … ? And lastly, where do the executable scripts need to be? https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bo… says "the external programs cannot be chosen arbitrarily; the available programs are restricted through administrator configuration", but no details on exactly how that is configured. -- "Are you pondering what I'm pondering?" Pinky: I think so, Brain. But if I put on two tutu's, would I really be wearing a four-by-four? Brain: Why do I even bother asking? Pinky: I dunno, Brain. Maybe it's all part of some huge, cosmic plot formula!

2 7

DOP-1414 sieve vacation address matching is case sensitive
by Mathieu Gosset 09 Nov '20

09 Nov '20

Hello, One year ago a bug was reported regarding pigeonhole vacation plugin https://www.mail-archive.com/dovecot@dovecot.org/msg78588.html a mail sent to User(a)domain.com instead of user(a)domain.com will generate a " vacation action: discarding vacation response for implicitly delivered message; no known (envelope) recipient address found in message headers (recipient=<user(a)domain.com>, and no additional `:addresses' are specified)" and no auto-response will be sent the problem seems to be located in /src/lib-sieve/plugins/vacation/cmd-vacation.c and the _contains_my_address_() function the comparison is done using dovecot's lib-smtp (src/lib-smtp/smtp-address.h) smtp_address_equals() function, which returns smtp_address_cmp(.) == 0; smtp_address_cmp() does a case insensitive comparison of the domain parts (using null_strcasecmp()) and a case sensitive comparison of the local parts (using null_strcmp()) There is probably a good reason why the local parts comparison is case sensitive within smtp_address_cmp(), but having vacation rely on that function causes the problem described above. Is there any plan to fix that ? Regards

2 1

HIGHESTMODSEQ replication issue
by Gustavo Ajeitos [Office365] 09 Nov '20

09 Nov '20

Hello, I'm using dovecot 2.3.10 in OpenSUSE and I have configured a master/master replication between two backends, it works well but I have found an issue with the replication of the HIGHESTMODSEQ. When I move a set of message from a mailbox to other, e.g. move 3 messages from INBOX to Junk, the HIGHESTMODSEQ get different on each server, with no errors in the log: [Borrador] 34:34 [Sent] 63:63 [Drafts] 12:12 [Trash] 146:146 [Junk] 219:218 [INBOX] 1133:1130 It can be fixed executing a full sync: $ sudo doveadm -D -v sync -u user(a)test.priv -d -l 30 -U But it is not a solution because it happens all the time that a user move messages. I'm using ActiveSync and the ActiveSync server use the HIGHESTMODSEQ as part of the synchronization key (sync_key), so in that situation the server detect changes on the mailbox if it connects to the backend with a different HIGHESTMODSEQ that it previously cached, causing a loop beetween the ActiveSync server and the the email client, where the server constantly inform email client about nonexistent changes. Could it be a bug in Dovecot replication? How can I solve it? Best regards, Gustavo Ajeitos

4 7

Re: vsz_limit
by Aki Tuomi 06 Nov '20

06 Nov '20

Duh... src/lib/restrict-process-size.h Should be in the installed include files as well, /usr/include/dovecot/restrict-process-size.h Aki > On 06/11/2020 15:56 Joan Moreau <jom(a)grosjo.net> wrote: > > > Hello > I can't find "src/lib/restrict.h" . Is it in dovecot source ? > > > On 2020-11-06 13:20, Aki Tuomi wrote: > > Seems I had forgotten that you can use src/lib/restrict.h, in particular, restrict_get_process_size() to figure out the limit. You can combine this with getrusage to find out current usage. > > > > Aki > > > > > > > On 06/11/2020 13:26 Joan Moreau <jom(a)grosjo.net> wrote: > > > > > > > > > yes, will do so. > > > It would be nice however to be able to access the actual dovecot config from the plugin side > > > > > > > > > > > > On 2020-11-04 06:46, Aki Tuomi wrote: > > > > You could also add it as setting for the fts_xapian plugin parameters? > > > > > > > > Aki > > > > > > > > > > > > > > > > > On 04/11/2020 08:42 Joan Moreau <jom(a)grosjo.net> wrote: > > > > > > > > > > > > > > > For machines with low memory, I would like to detect how much ram remains available before starting indexing a mail, so I can commit everything on disk before the ram is exhausted (and break the process) > > > > > I tried to put a "fake" allocation to test if it fails, (so it can fail separately, and I can "if ram remaining is above X") but the is really not clean > > > > > > > > > > > > > > > > > > > > On 2020-11-04 06:28, Aki Tuomi wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/11/2020 05:19 Joan Moreau <jom(a)grosjo.net> wrote: > > > > > > > > > > > > > > > > > > > > > Hello > > > > > > > I am looking for help around memory management > > > > > > > 1 - How to get the current value of "vsz_limit" from inside a plugin (namely https://github.com/grosjo/fts-xapian/ ) , especially for indexer-worker > > > > > > > 2 - Is there a macro or function in dovecot to get the remaining free memory from this vsz value ? > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Joan, > > > > > > > > > > > > I don't think there is a feasible way to access this setting as of now. Is there a reason you need this? We usually recommend setting vsz_limit=0 for indexer-worker. > > > > > > > > > > > > Aki > > > > > >

2 2

Re: vsz_limit
by Aki Tuomi 06 Nov '20

06 Nov '20

You could also add it as setting for the fts_xapian plugin parameters? Aki > On 04/11/2020 08:42 Joan Moreau <jom(a)grosjo.net> wrote: > > > For machines with low memory, I would like to detect how much ram remains available before starting indexing a mail, so I can commit everything on disk before the ram is exhausted (and break the process) > I tried to put a "fake" allocation to test if it fails, (so it can fail separately, and I can "if ram remaining is above X") but the is really not clean > > > > On 2020-11-04 06:28, Aki Tuomi wrote: > > > > > > > On 04/11/2020 05:19 Joan Moreau <jom(a)grosjo.net> wrote: > > > > > > > > > Hello > > > I am looking for help around memory management > > > 1 - How to get the current value of "vsz_limit" from inside a plugin (namely https://github.com/grosjo/fts-xapian/ ) , especially for indexer-worker > > > 2 - Is there a macro or function in dovecot to get the remaining free memory from this vsz value ? > > > Thank you > > > > > > > > > > Hi Joan, > > > > I don't think there is a feasible way to access this setting as of now. Is there a reason you need this? We usually recommend setting vsz_limit=0 for indexer-worker. > > > > Aki > >

2 3

doveadm sync usage ; root INBOX unsynced
by François Poulain 06 Nov '20

06 Nov '20

Hi, I am trying to import a mail IMAP account using doveadm. Following https://serverfault.com/questions/605342/migrating-from-any-imap-pop3-serve… I did it with: doveadm -D -v \ -o imapc_host=mail.oldserver.tld \ -o imapc_user=contact(a)domain.org \ -o imapc_password=xxxxxxx \ -o imapc_features=rfc822.size \ -o imapc_ssl=starttls \ -o ssl_client_ca_file=/etc/ssl/certs/ca-certificates.crt \ -o mail_fsync=never \ sync -1 -R -u contact(a)domain.org imapc: Btw I am facing with exactly the same error than: https://serverfault.com/questions/960264/doveadm-backup-doesnt-transfer-mai… I.e. it works fine for all subfolders, but the root INBOX stay empty on the new server. Those debug messages tell me something seems wrong with INBOX root but I am unable to understand them: dsync(contact(a)domain.tld): Debug: brain M: Local mailbox tree: INBOX guid=deadbeefdeabbeefdeadbeefdeabbeef uid_validity=1604137198 uid_next=1 subs=no last_change=0 last_subs=0 dsync(contact(a)domain.tld): Debug: brain S: Remote mailbox tree: INBOX guid=deadbeefdeabbeefdeadbeefdeabbeef uid_validity=1604137198 uid_next=1 subs=no last_change=0 last_subs=0 dsync(contact(a)domain.tld): Debug: brain M: Mailbox INBOX: local=deadbeefdeabbeefdeadbeefdeabbeef/0/1, remote=00000000000000000000000000000000/0/0: mailbox not selectable yet dsync(contact(a)domain.tld): Debug: brain S: Mailbox INBOX: local=00000000000000000000000000000000/0/0, remote=deadbeefdeabbeefdeadbeefdeabbeef/0/1: mailbox not selectable yet dsync(contact(a)domain.tld): Debug: brain S: Ignore nonexistent mailbox GUID deadbeefdeabbeefdeadbeefdeabbeef with -1 sync dsync(contact(a)domain.tld): Debug: brain S: We don't have mailbox deadbeefdeabbeefdeadbeefdeabbeef dsync(contact(a)domain.tld): Debug: brain M: Change during sync: Remote lost mailbox GUID deadbeefdeabbeefdeadbeefdeabbeef (maybe it was just deleted?) dsync(contact(a)domain.tld): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID deadbeefdeabbeefdeadbeefdeabbeef (maybe it was just deleted?) Does someone have on hint? Best regards. François -- François Poulain <fpoulain(a)metrodore.fr>

3 4

Re: Fatal: write(indexer) failed: Resource temporarily unavailable
by Aki Tuomi 06 Nov '20

06 Nov '20

> On 06/11/2020 13:37 Joan Moreau <jom(a)grosjo.net> wrote: > > > Hello > I have this issue for Xapian plugin: > https://github.com/grosjo/fts-xapian/issues/62 > > But I am not sure where can it comes from. > Is dovecot calling some specific function in the plugin after the init, that would create such error ? > In doveadm dealing differently with plugins that dovecot core does ? > Has there been some recent changes in the plugin framework that would lead to such error ? > Thank you https://wiki.dovecot.org/SocketUnavailable Aki

1 0

Fatal: write(indexer) failed: Resource temporarily unavailable
by Joan Moreau 06 Nov '20

06 Nov '20

Hello I have this issue for Xapian plugin: https://github.com/grosjo/fts-xapian/issues/62 But I am not sure where can it comes from. Is dovecot calling some specific function in the plugin after the init, that would create such error ? In doveadm dealing differently with plugins that dovecot core does ? Has there been some recent changes in the plugin framework that would lead to such error ? Thank you

1 0

Indexer error after upgrade to 2.3.11.3
by Alessio Cecchi 06 Nov '20

06 Nov '20

Hi, after the upgrade to Dovecot 2.3.11.3, from 2.3.10.1, I see frequently these errors from different users: Aug 18 11:02:35 Panic: indexer-worker(info(a)domain.com) session=<g71KISOttvS5LNVj:O3ahCyuZO18cYAAAEPCW+w>: file http-client-request.c: line 1232 (http_client_request_send_more): assertion failed: (req->payload_input != NULL) Aug 18 11:02:35 Error: indexer-worker(info(a)domain.com) session=<g71KISOttvS5LNVj:O3ahCyuZO18cYAAAEPCW+w>: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x2f) [0x7f0ee3c828bf] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x26) [0x7f0ee3c829d6] -> /usr/lib64/dovecot/libdovecot.so.0(+0xeb7ba) [0x7f0ee3c8d7ba] -> /usr/lib64/dovecot/libdovecot.so.0(+0xeb801) [0x7f0ee3c8d801] -> /usr/lib64/dovecot/libdovecot.so.0(+0x42ff1) [0x7f0ee3be4ff1] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_more+0x415) [0x7f0ee3c2ba25] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_connection_output+0x114) [0x7f0ee3c30994] -> /usr/lib64/dovecot/libdovecot.so.0(+0x115470) [0x7f0ee3cb7470] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x55) [0x7f0ee3ca4eb5] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xdc) [0x7f0ee3ca6ebc] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5c) [0x7f0ee3ca4fac] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f0ee3ca51f8] -> /usr/lib64/dovecot/libdovecot.so.0(+0x8a955) [0x7f0ee3c2c955] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_request_finish_payload+0x21) [0x7f0ee3c2cbd1] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(solr_connection_post_end+0x45) [0x7f0ee1c85d15] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x3fa0) [0x7f0ee1c81fa0] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x86cc) [0x7f0ee297f6cc] -> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_backend_update_deinit+0x2c) [0x7f0ee297f74c] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xfd04) [0x7f0ee2986d04] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xff3f) [0x7f0ee2986f3f] -> /usr/lib64/dovecot/lib10_quota_plugin.so(+0xf64b) [0x7f0ee2dc764b] -> /usr/lib64/dovecot/lib01_acl_plugin.so(+0xde43) [0x7f0ee2fdce43] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit_get_changes+0x54) [0x7f0ee3f91db4] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit+0x16) [0x7f0ee3f91e76] -> dovecot/indexer-worker [info(a)domain.com INBOX](+0x291c) [0x557584acb91c] -> dovecot/indexer-worker [info(a)domain.com INBOX](+0x2e54) [0x557584acbe54] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x55) [0x7f0ee3ca4eb5] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xdc) [0x7f0ee3ca6ebc] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5c) [0x7f0ee3ca4fac] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f0ee3ca51f8] Aug 18 11:02:35 Error: indexer: Indexer worker disconnected, discarding 1 requests for info(a)domain.com Aug 18 11:02:35 Error: imap(info(a)domain.com) session=<g71KISOttvS5LNVj>: indexer failed to index mailbox INBOX Aug 18 11:02:35 Fatal: indexer-worker(info(a)domain.com) session=<g71KISOttvS5LNVj:O3ahCyuZO18cYAAAEPCW+w>: master: service(indexer-worker): child 24604 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) I'm using FTS with Solr 6.6.5. What is it? Thanks -- Alessio Cecchi Postmaster @http://www.qboxmail.it https://www.linkedin.com/in/alessice

9 25