- dovecot - dovecot.org

Understanding the limitations of Dovecot Submission Service
by R. Diez 04 Nov '20

04 Nov '20

Hi all: The Dovecot Submission Service is very enticing for laz... I mean for busy people like me. A couple of lines in dovecot.conf , and you have a local SMTP server that does authentication. No SASL loops to jump through! Very cool people like me also appreciate cool new features like BURL. But now I want to understand its limitations. Say I just bought e-mail domain xxx(a)example.org . Say employee A has e-mail address a(a)example.org . It this employee turns evil, will he/she be able to send e-mails impersonating b(a)example.org ? Or does the Dovecot Submission Service check that he only has permission to send as a(a)example.org ? My requirements are pretty modest. Basically e-mail should just be sent all right. But are there any other reasons why I should avoid the Dovecot Submission Server anyway? The documentation does not really say much. Thanks in advance, rdiez

1 0

Re: API support for placing a message for a specific user
by Aki Tuomi 04 Nov '20

04 Nov '20

> On 03/11/2020 02:24 Oren Yehezkely <orenyny(a)gmail.com> wrote: > > > Hello, > > I am trying to find out if Dovecot API allows placing a message directly in a customer's mailbox, without needing to send the email via SMTP. > > Thanks for your help, > > Oren `doveadm save` is available from HTTP API, but seems to be missing from documentation. Example: curl --user doveadm:password http://localhost:1122/doveadm/v1 -d '[["save",{"user":"testuser","file":"Hello, world"},"c0d0"]]' note that despite being called 'file', is it actually the mail contents. Aki API for the SAVE command: { "command": "save", "parameters": [ { "name": "allUsers", "type": "boolean" }, { "name": "socketPath", "type": "string" }, { "name": "user", "type": "string" }, { "name": "transFlags", "type": "integer" }, { "name": "userFile", "type": "string" }, { "name": "mailbox", "type": "string" }, { "name": "file", "type": "string" } ] }

2 1

Re: Panic-Error (ctx->nested_parts_count > 0)
by Aki Tuomi 04 Nov '20

04 Nov '20

You should be able to workaround this by enabling fts. Aki > On 04/11/2020 09:22 Andreas Piper <piper(a)hrz.uni-marburg.de> wrote: > > > Hello, > > on my dovecot-replicator-backend I see lots of panic-messages "assertion > failed: (ctx->nested_parts_count > 0)", produced from a single user. I > assume that this is correlated with > https://github.com/dovecot/core/commit/a668d767a710ca18ab6e7177d8e8be22a6b0… > > How can I identify the incorrect formatted message in the user's > mailbox? I want to remove it, as the error-processes drive up the > backend's load incredibly. > > Will a fixed dovecot-package be out soon? > > Any help is appreciated. > > Best regards, > Andreas > > My system-data: > # 2.3.11.3 (502c39af9): /etc/dovecot/dovecot.conf > # Pigeonhole version 0.5.11 (6c69c917) > # OS: Linux 4.15.0-122-generic x86_64 Ubuntu 18.04.5 LTS > > An example error-message: > > Nov 04 08:04:25 vhrz311 dovecot[6163]: > imap(xxx(a)staff.uni-marburg.de)<63176><ad/jlkKzSs2sGQEp>: Panic: file > message-parser.c: line 174 (message_part_finish): assertion failed: > (ctx->nested_parts_count > 0) > Nov 04 08:04:25 vhrz311 dovecot[6163]: > imap(xxx(a)staff.uni-marburg.de)<63173><ORvjlkKzRs2sGQEp>: Fatal: master: > service(imap): child 63173 killed with signal 6 (core dumps disabled - > https://dovecot.org/bugreport.html#coredumps) > Nov 04 08:04:25 vhrz311 dovecot[6163]: > imap(xxx(a)staff.uni-marburg.de)<63176><ad/jlkKzSs2sGQEp>: Error: Raw > backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) > [0x7f1dd28d6d22] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) > [0x7f1dd28d6e3e] -> /usr/lib/dovecot/libdovecot.so.0(+0xed75e) > [0x7f1dd28e175e] -> /usr/lib/dovecot/libdovecot.so.0(+0xed801) > [0x7f1dd28e1801] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) > [0x7f1dd2836beb] -> /usr/lib/dovecot/libdovecot.so.0(+0xcd40e) > [0x7f1dd28c140e] -> > /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x15c) > [0x7f1dd28c2b2c] -> > /usr/lib/dovecot/libdovecot.so.0(message_search_msg+0xa0) > [0x7f1dd28c52a0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xc8f9e) > [0x7f1dd2c67f9e] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_foreach+0x45) > [0x7f1dd2be31e5] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xca111) > [0x7f1dd2c69111] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcb214) > [0x7f1dd2c6a214] -> > /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblock+0x10d) > [0x7f1dd2c6a9ed] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonblock+0x22) > [0x7f1dd2becdf2] -> dovecot/imap(+0x25eb1) [0x561689529eb1] -> > dovecot/imap(command_exec+0x64) [0x561689522bc4] -> > dovecot/imap(+0x25592) [0x561689529592] -> > /usr/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0x12f) > [0x7f1dd28fa3ff] -> > /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xbe) > [0x7f1dd28fbebe] -> > /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) > [0x7f1dd28fa75c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) > [0x7f1dd28fa988] -> > /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) > [0x7f1dd2867c23] -> dovecot/imap(main+0x339) [0x561689512d19] -> > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f1dd2424bf7] > -> dovecot/imap(_start+0x2a) [0x561689512f2a]

1 0

Panic-Error (ctx->nested_parts_count > 0)
by Andreas Piper 04 Nov '20

04 Nov '20

Hello, on my dovecot-replicator-backend I see lots of panic-messages "assertion failed: (ctx->nested_parts_count > 0)", produced from a single user. I assume that this is correlated with https://github.com/dovecot/core/commit/a668d767a710ca18ab6e7177d8e8be22a6b0… How can I identify the incorrect formatted message in the user's mailbox? I want to remove it, as the error-processes drive up the backend's load incredibly. Will a fixed dovecot-package be out soon? Any help is appreciated. Best regards, Andreas My system-data: # 2.3.11.3 (502c39af9): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.11 (6c69c917) # OS: Linux 4.15.0-122-generic x86_64 Ubuntu 18.04.5 LTS An example error-message: Nov 04 08:04:25 vhrz311 dovecot[6163]: imap(xxx(a)staff.uni-marburg.de)<63176><ad/jlkKzSs2sGQEp>: Panic: file message-parser.c: line 174 (message_part_finish): assertion failed: (ctx->nested_parts_count > 0) Nov 04 08:04:25 vhrz311 dovecot[6163]: imap(xxx(a)staff.uni-marburg.de)<63173><ORvjlkKzRs2sGQEp>: Fatal: master: service(imap): child 63173 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) Nov 04 08:04:25 vhrz311 dovecot[6163]: imap(xxx(a)staff.uni-marburg.de)<63176><ad/jlkKzSs2sGQEp>: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7f1dd28d6d22] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f1dd28d6e3e] -> /usr/lib/dovecot/libdovecot.so.0(+0xed75e) [0x7f1dd28e175e] -> /usr/lib/dovecot/libdovecot.so.0(+0xed801) [0x7f1dd28e1801] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f1dd2836beb] -> /usr/lib/dovecot/libdovecot.so.0(+0xcd40e) [0x7f1dd28c140e] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x15c) [0x7f1dd28c2b2c] -> /usr/lib/dovecot/libdovecot.so.0(message_search_msg+0xa0) [0x7f1dd28c52a0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xc8f9e) [0x7f1dd2c67f9e] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_foreach+0x45) [0x7f1dd2be31e5] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xca111) [0x7f1dd2c69111] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcb214) [0x7f1dd2c6a214] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblock+0x10d) [0x7f1dd2c6a9ed] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonblock+0x22) [0x7f1dd2becdf2] -> dovecot/imap(+0x25eb1) [0x561689529eb1] -> dovecot/imap(command_exec+0x64) [0x561689522bc4] -> dovecot/imap(+0x25592) [0x561689529592] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0x12f) [0x7f1dd28fa3ff] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xbe) [0x7f1dd28fbebe] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) [0x7f1dd28fa75c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f1dd28fa988] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f1dd2867c23] -> dovecot/imap(main+0x339) [0x561689512d19] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f1dd2424bf7] -> dovecot/imap(_start+0x2a) [0x561689512f2a]

1 0

Re: vsz_limit
by Aki Tuomi 04 Nov '20

04 Nov '20

> On 04/11/2020 05:19 Joan Moreau <jom(a)grosjo.net> wrote: > > > Hello > I am looking for help around memory management > 1 - How to get the current value of "vsz_limit" from inside a plugin (namely https://github.com/grosjo/fts-xapian/ ) , especially for indexer-worker > 2 - Is there a macro or function in dovecot to get the remaining free memory from this vsz value ? > Thank you > > Hi Joan, I don't think there is a feasible way to access this setting as of now. Is there a reason you need this? We usually recommend setting vsz_limit=0 for indexer-worker. Aki

2 1

Re: vsz_limit
by Aki Tuomi 04 Nov '20

04 Nov '20

> On 04/11/2020 05:19 Joan Moreau <jom(a)grosjo.net> wrote: > > > Hello > I am looking for help around memory management > 1 - How to get the current value of "vsz_limit" from inside a plugin (namely https://github.com/grosjo/fts-xapian/ ) , especially for indexer-worker > 2 - Is there a macro or function in dovecot to get the remaining free memory from this vsz value ? > Thank you > >

1 0

vsz_limit
by Joan Moreau 04 Nov '20

04 Nov '20

Hello I am looking for help around memory management 1 - How to get the current value of "vsz_limit" from inside a plugin (namely https://github.com/grosjo/fts-xapian/ ) , especially for indexer-worker 2 - Is there a macro or function in dovecot to get the remaining free memory from this vsz value ? Thank you

1 0

Feature Request: Redis support for username and TLS
by Mark Moseley 04 Nov '20

04 Nov '20

I was wondering if there was any imminent support in 2.3.12+ for using a username to log into Redis, as well as support for using TLS to connect to Redis. And if not, I'd like to put in a feature request for those two things (AUTH with username/password, and TLS connections to Redis). Specifically, I was looking at using a username/password combo to log into Redis for the quota_clone plugin. I found the 'password' param in the source (not documented at https://wiki.dovecot.org/Dictionary) There's no 'username' param (the 'username' in the source seems to refer to the mailbox, for the purpose of building the key name). Redis 6 supports authenticating with a username and password, as well as the ability to listen on a TLS-enabled port. Both of these significantly improve security, combined with the new ACL system. Obviously, these Redis 6 features are brand new, so I'd be shocked if they were already supported. But it'd be awesome if those were added to Dovecot :) Currently, I've got a localhost Envoy proxy doing TCP proxying from localhost+non-TLS to my Redis TLS port, which is a kludge at best. There's a neat Envoy Redis proxy that *almost* does the trick but the Envoy Redis proxy unfortunately doesn't support MULTI/EXEC, which Dovecot quota_clone uses, or I'd be using that instead of a plain TCP proxy (since the Envoy Redis proxy can use a username/password+tls to connect to the upstream Redis).

1 0

Delivering locally through the Submission Server
by R. Diez 03 Nov '20

03 Nov '20

Hi all: I am learning Dovecot step by step. I have enabled the Submission Server, in the hope that I would not need to learn other MTAs like Postfix. The Submission Server is very comfortable: it picks up the existing Dovecot configuration, so that you do not need to configure any user authentication separately. It is working fine on my test setup. My first thought was that, if the recipient is a local mailbox, the Submission Server would not need to relay the message to any external SMTP server, as it could just deliver it locally. After all, it is running on the same Dovecot. But it cannot do that, can it? Is there a work-around? I am still hoping that the mail server still works locally if the Internet connection fails. But I would need to learn some Postfix magic for that, would I? Thanks in advance, rdiez

5 7

Re: Delivering locally through the Submission Server
by Aki Tuomi 03 Nov '20

03 Nov '20

> On 03/11/2020 12:31 Piotr Auksztulewicz <dcml(a)hasiok.net> wrote: > > > On Mon, Nov 02, 2020 at 09:33:08PM +0100, R. Diez wrote: > > OK, so I gather that the Submission Server cannot do that (yet). > > And probably would never do. It isn't its job description. > > Actually, it is just a convenience/workaround feature, which comes handy > only if your own MTA cannot handle dovecot's SASL authentication (must be > something real strange) or there are some integration/security/policy > issue perceived (but I cannot think of any, actually). In this case you > can set up dovecot's submission server, which uses dovecot's authentication > settings, so you have single source of authentication, and whitelist > dovecot IP address in your MTA so it accepts anything that dovecot's > submission server lets through. But I don't think it is a good idea > personally, it is more open to exploitation this way, unless the > address is 127.0.0.1, in which case you can simply set up SASL over > Unix sockets, which is as secure as your host server is. > Submission service is not only a proxy, it - provides authentication natively from Dovecot - provides features like BURL, and maybe in future outbound Sieve but it does require real MTA behind. -- Aki

2 1

API support for placing a message for a specific user
by Oren Yehezkely 03 Nov '20

03 Nov '20

Hello, I am trying to find out if Dovecot API allows placing a message directly in a customer's mailbox, without needing to send the email via SMTP. Thanks for your help, Oren

1 0

Re: strange file .temp.1592374672.P11164Q21M692534.hostname - doveadm altmove
by Aki Tuomi 02 Nov '20

02 Nov '20

> On 01/11/2020 21:13 John Stoffel <john(a)stoffel.org> wrote: > > > >>>>> "Gionatan" == Gionatan Danti <g.danti(a)assyoma.it> writes: > > Gionatan> Il 2020-11-01 13:23 John Stoffel ha scritto: > >> How is your "mail_location" defined in your configuration? And what > >> is your search term? > > Gionatan> Hi, I just discovered having such strange .temp files on an > Gionatan> old server of mine. > > I honestly haven't a clue... I don't have these on my server at all. > > Gionatan> In some folders I have hundred of such files, but they > Gionatan> really are hardlinks to the same .temp file (for each > Gionatan> folder), so they do not consume much space. These files > Gionatan> seems much more common in the Trash folder, but they appear > Gionatan> even in other folder. > > I would probably just nuke these files, but maybe after confirming > what their contents really is. > > Gionatan> An example of a folder with 6 .temp files: > Gionatan> [root@mail dbox-Mails]# ls -al > Gionatan> /var/vmail/domain/user/dbox/mailboxes/INBOX/Trash/dbox-Mails > Gionatan> drwxr-x---. 2 vmail vmail 4096 Oct 31 12:42 . > Gionatan> drwxr-x---. 3 vmail vmail 4096 Mar 17 2020 .. > Gionatan> -rw-r-----. 1 vmail vmail 512 Oct 29 09:45 dovecot.index > Gionatan> -rw-r-----. 1 vmail vmail 392 Oct 28 13:59 dovecot.index.backup > Gionatan> -rw-r-----. 1 vmail vmail 55296 Oct 31 12:42 dovecot.index.cache > Gionatan> -rw-r-----. 1 vmail vmail 23536 Oct 31 12:42 dovecot.index.log > Gionatan> -rw-r-----. 6 vmail vmail 22215889 Oct 12 10:42 > Gionatan> .temp.1602492703.P47358Q0M194142.hostname > Gionatan> -rw-r-----. 6 vmail vmail 22215889 Oct 12 10:42 > Gionatan> .temp.1602494002.P50402Q0M898232.hostname > Gionatan> -rw-r-----. 6 vmail vmail 22215889 Oct 12 10:42 > Gionatan> .temp.1602494013.P50446Q0M771524.hostname > Gionatan> -rw-r-----. 6 vmail vmail 22215889 Oct 12 10:42 > Gionatan> .temp.1602494621.P51761Q0M119496.hostname > Gionatan> -rw-r-----. 6 vmail vmail 22215889 Oct 12 10:42 > Gionatan> .temp.1602494745.P51954Q0M387071.hostname > Gionatan> -rw-r-----. 6 vmail vmail 22215889 Oct 12 10:42 > Gionatan> .temp.1602495419.P53185Q0M671512.hostname > Gionatan> -rw-r-----. 1 vmail vmail 848909 Oct 29 21:12 u.1623 > > Gionatan> 6x 22M files = 128M... > Gionatan> [root@mail dbox-Mails]# du -hscl .temp.160249* > Gionatan> 22M .temp.1602492703.P47358Q0M194142.hostname > Gionatan> 22M .temp.1602494002.P50402Q0M898232.hostname > Gionatan> 22M .temp.1602494013.P50446Q0M771524.hostname > Gionatan> 22M .temp.1602494621.P51761Q0M119496.hostname > Gionatan> 22M .temp.1602494745.P51954Q0M387071.hostname > Gionatan> 22M .temp.1602495419.P53185Q0M671512.hostname > Gionatan> 128M total > > Gionatan> ...but only 22M really consumed (1 file, 6 hardlinks) > Gionatan> [root@mail dbox-Mails]# du -hsc .temp.160249* > Gionatan> 22M .temp.1602492703.P47358Q0M194142.hostname > Gionatan> 22M total > > Gionatan> I never run doveadm altmove; moreover, these .tempo files > Gionatan> are both old (>1 year) and new (~3 weeks ago). Below you can > Gionatan> find the output from dovecot -n. Any ideas? > > I really don't know, sorry! I'd have to got poke at the source to see > how they're created and where. I don't use dbox, I use Maildir for my > setup. > > So I suspect (but I'm not sure!) that these are just leftovers from > mailbox accesses which got left behind. Try googling for 'dovecot > dbox temp files' and maybe that will help. Sorry! > > John Those files are most likely leftovers due to failed operations, for reasons like crashes. Aki

2 1

v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed)
by PGNet Dev 02 Nov '20

02 Nov '20

I'm running, dovecot --version 2.3.11.3 (502c39af9) solr -version 8.6.3 uname -rm 5.8.13-200.fc32.x86_64 x86_64 grep _NAME /etc/os-release PRETTY_NAME="Fedora 32 (Server Edition)" CPE_NAME="cpe:/o:fedoraproject:fedora:32" Solr FTS plugin is enabled/configured, mail_plugins = virtual acl fts fts_solr plugin { fts = solr fts_autoindex = yes fts_solr = url=https://solr.example.com:8984/solr/dovecot/ fts_enforced = body fts_filters = normalizer-icu stopwords snowball fts_language_config = /usr/share/libexttextcat/fpdb.conf fts_languages = en es de fr it pt soft_commit = yes } IMAP capability returns, a OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY STATUS=SIZE SAVEDATE SPECIAL-USE LITERAL+ NOTIFY SPECIAL-USE QUOTA ACL RIGHTS=texk] Logged in I've got two messages in my IMAP store, cd /data/vmail/example.com/myuser/Maildir/cur/ ls -altr | grep S= | /bin/tail -n2 -rw------- 1 vmail vmail 1.3K Oct 11 14:05 1602450306.M393628P65260.mx.example.com,S=1278,W=1304:2,S -rw------- 1 vmail vmail 1.3K Oct 11 14:05 1602450353.M756184P65260.mx.example.com,S=1277,W=1303:2,S that differ in BODY CONTENT -- -- one message has ascii txt with NO character accents -- the other has the same text, but with ON character accent cat "1602450306.M393628P65260.mx.example.com,S=1278,W=1304:2,S" ... From: M User <myuser(a)example.com> Subject: test Reply-To: myuser(a)example.com To: "User, My" <myuser(a)example.com> Message-ID: <6fc7ac30-b460-7dd4-f85d-ca4403ad7188(a)example.com> Date: Sun, 11 Oct 2020 14:05:06 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.3.2 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit !!!! también cat 1602450353.M756184P65260.mx.example.com,S=1277,W=1303:2,S ... From: M User <myuser(a)example.com> Subject: test Reply-To: myuser(a)example.com To: "User, My" <myuser(a)example.com> Message-ID: <015b3fb4-46f9-87cc-d541-060db0a13086(a)example.com> Date: Sun, 11 Oct 2020 14:05:53 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.3.2 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit !!!! tambien i manually re-scan & index doveadm fts rescan -u myuser(a)example.com doveadm index -u myuser(a)example.com -q '*' ... ==> /var/log/dovecot/dovecot-info.log <== 2020-10-11 15:06:34 indexer-worker(myuser(a)example.com)<OyUmLeqBg18fDAEA+IOfAw>: Info: Indexed 21 messages in accts (UIDs 14399..130699) 2020-10-11 15:06:34 indexer-worker(myuser(a)example.com)<6NnOMuqBg18fDAEA+IOfAw>: Info: Indexed 16 messages in accts/v007132 (UIDs 13414..14778) ... with no errors. then search in mail client, here TBird 78, with [X] Run Search on Server for _un_accented "tambien", match is correctly -- and quickly -- returned. in logs, ==> /var/log/dovecot/dovecot-info.log <== 2020-10-11 14:57:05 imap-login: Info: Login: user=<myuser(a)example.com>, method=PLAIN, rip=10.0.1.7, lip=10.0.1.50, mpid=67743, TLS 2020-10-11 14:57:16 indexer-worker(myuser(a)example.com)<3ZUzQ2yx2JKsHgsH:9gu0MbF/g1+hCAEA+IOfAw>: Info: Indexed 4788 messages in INBOX (UIDs 135476..140263) BUT, repeating search for ACCENTED "también" returns *no* match/result. No errors in log, simply no match. Attempting to test/debug from from cmd line, doveadm fts lookup -u myuser(a)example.com body "tambien" causes a PANIC doveadm(myuser(a)example.com): Panic: file mail-storage.c: line 2112 (mailbox_get_open_status): assertion failed: (box->opened) doveadm(myuser(a)example.com): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x46) [0x7f3ee94accc6] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7f3ee94acde2] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10025b) [0x7f3ee94b625b] -> /usr/lib64/dovecot/libdovecot.so.0(+0x100297) [0x7f3ee94b6297] -> /usr/lib64/dovecot/libdovecot.so.0(+0x59bc6) [0x7f3ee940fbc6] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x4779e) [0x7f3ee95c379e] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x5849) [0x7f3ee9015849] -> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_backend_lookup+0x51) [0x7f3ee8c37491] -> /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so(+0x3280) [0x7f3ee8ba9280] -> doveadm(+0x343cd) [0x5637e99443cd] -> doveadm(+0x34fe0) [0x5637e9944fe0] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x22d) [0x5637e9945e2d] -> doveadm(doveadm_cmd_run_ver2+0x4e8) [0x5637e99568d8] -> doveadm(doveadm_cmd_try_run_ver2+0x3e) [0x5637e995692e] -> doveadm(main+0x1d4) [0x5637e9934cf4] -> /lib64/libc.so.6(__libc_start_main+0xf2) [0x7f3ee9071042] -> doveadm(_start+0x2e) [0x5637e99351ce] Aborted (1) What config -- dovecot &/or solr -- is needed to match on accented characters? (2) What add'l detail, if any, is needed for troubleshooting the panic?

5 37

removing 'use_libfts' from dovecot fts_solr config; docs/examples of solr-backend-only tokenizer config for dovecot?
by PGNet Dev 02 Nov '20

02 Nov '20

given existing problems with fts_solr = ... use_libfts ... i've rm'd it from dovecot config, and am moving all tokenization to solr config. iiuc, in dovecot + solr context, that's correctly config'd in solr/data/dovecot/conf/schema.xml dovecot release ships with 'starter configs', cp -af /usr/share/doc/dovecot/solr-config-7.7.0.xml /path/to/solr/data/dovecot/conf/solrconfig.xml cp -af /usr/share/doc/dovecot/solr-schema-7.7.0.xml /path/to/solr/data/dovecot/conf/schema.xml that^ schema.xml contains a limited solr.StandardTokenizerFactory config. it can of course be extended; lots of generic solr config info available. is there _documentation_, preferably a good example, of a dovecot-'friendly' solr config for non-"use_libfts" use case? specifically, moving any/all of the libfts front-end tokenizers @ https://doc.dovecot.org/configuration_manual/fts/tokenization/#fts-tokeniza… https://doc.dovecot.org/settings/plugin/fts-plugin/#plugin-fts-setting-fts-… fully to a backend solr instance? i suspect that solr config is fairly well insulated from dovecot's; what i'm not sure of is if dovecot's fts has specific requirements to get the backend-only tokenization working.

1 0

Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
by Aki Tuomi 02 Nov '20

02 Nov '20

> On 02/11/2020 15:11 PGNet Dev <pgnet.dev(a)gmail.com> wrote: > > > On 11/2/20 12:44 AM, Aki Tuomi wrote: > > you should try removing use_libfts from your config line and let solr do that part. > > sry, i'm a bit confused. > > you'd suggested I _add_ it, > > https://dovecot.org/pipermail/dovecot/2020-October/120258.html > > > I can reproduce your problem with the `fts lookup` command. Luckily it's equivalent to running `doveadm search`. I'll open a bug about this. > > Dovecot FTS tokenization is not done, unless you have `use_libfts` in fts_solr setting, in your case > > fts_solr = url=https://solr.example.com:8984/solr/dovecot/ use_libfts > > Without this, everything is sent to to solr as-is, which is then expected to do all the work. > > So what's the recommendation? use use_libfts, or not? It's a choice. You can let solr perform the tokenization etc. or you can let dovecot do it. There is no recommendation when using solr. It seems that use_libfts is broken with solr due to reasons, so I guess the only option for now is not to use it. Aki

2 1

Re: v2.3.11.3 solr plugin search via MUA fails to match accented ascii characters; cmd line exec of `doveadm fts lookup` PANICs (assertion failed) [proposed patch]
by Aki Tuomi 02 Nov '20

02 Nov '20

> On 01/11/2020 20:35 John Fawcett <john(a)voipsupport.it> wrote: > > > On 01/11/2020 15:20, PGNet Dev wrote: > > On 11/1/20 1:56 AM, John Fawcett wrote: > >> At the moment I don't see other corrections needed in dovecot apart from > >> command line doveadm fts which is not a show stopper. Via doveadm search > >> I confirm - on my simple config - that search for accented or non > >> accented characters works correctly as it does via imap connection. > > > > thx. hopefully it'll get considered for a next release soon. > > > >> Only thing I cannot vouch for is bringing dovecot fts library and config > >> into the equation because my setup delegates almost everything to solr. > > > > do i understand correctly that you're solr-indexing your dovecot mail > > store withOUT using dovecot fts plugin, and that -- with your > > aforementioned patch -- doveadm successfully uses the resulting indexes? > > > > i hadn't yet seriously considered _circumventing_ fts plugin; if this^ > > does get resolved soonish, then it's not a big deal. if not, an > > fts-plugin-less setup would be interesting to know more abt! > > > >> Can you get evidence of things not working? For example tests run with > >> soft_commit configured - that's important since without it the updates > >> don't show up immediately in searches, that do show that the update is > >> happening in solr via solr log, but then search is not working on > >> accented characters, despite it working on other text in the same > >> message? The solr logs also show whether the text was found or not via > >> the "hits=" value in the logged searches, for example: > >> > >> 2020-11-01 08:32:42.231 INFO (qtp24119573-21) [ x:dovecot] > >> o.a.s.c.S.Request [dovecot] webapp=/solr path=/select > >> params={q={!lucene+q.op%3DAND}body:también&fl=uid,score&sort=uid+asc&fq=%2Bbox:b1626f0fe8d9145e54100000c54a863a+%2Buser:john@voipsupport.it&rows=3202&wt=xml} > >> > >> hits=3 status=0 QTime=3 > >> > >> But if no hits are found, then dovecot cannot be expected to display > >> results. It still may be an indexing problem though. > > > > my current config has soft_commit enabled, > > > > fts_solr = url=https://solr.example.com:8984/solr/dovecot/ > > use_libfts soft_commit=yes batch_size=250 > > > > i'll see abt getting some clearer test results ... > > Yes, getting more data about any potential problem would be useful. > > Just to clarify: I have a fully working search setup for some time now > over various dovecot releases, so no patches needed to get it working. > > My setup does use fts plugin and fts-solr plugin, but it does not use > lib-fts functionality (that has many features for example it was > stopping you indexing excluded words like tambien). On my setup without > lib-fts everything goes to solr which does the work of indexing without > all the features of lib-fts. > > My setup is like this not because of issues in lib-fts, but because I > never had the need for it. There is no evidence at the moment however > that even with lib-fts enabled there are issues with dovecot indexing or > searching. > > What is currently nor working is "doveadm fts" command line utility. But > this is mitigated by being able to use a similar command line utility > "doveadm search". The issue on "dovecot fts" command line utility has > (so far as the available evidence suggests) no effect on indexing or > imap searches. > > fyi my working configuration includes fts and fts_solr plugins > > mail_plugins = quota notify replication fts fts_solr > > (and those are also recalled in the various specific plugin settings for > imap, lmtp ecc), The specific config I am using for fts and fts_solr is: > > fts = solr > fts_enforced = yes > fts_solr = url=https://user@server.example.com:443/solr/dovecot/ > batch_size=500 soft_commit=no > > BTW I use soft_commit=no because I have periodic soft commits setup on > solr and I accept that newly indexed text won't become searchable for up > to that interval, but for your testing purposes much more useful as you > have it. > > John PGNet, you should try removing use_libfts from your config line and let solr do that part. Aki

2 1

strange file .temp.1592374672.P11164Q21M692534.hostname - doveadm altmove
by Maciek Jackowski 01 Nov '20

01 Nov '20

dovecot 2.2.33.2 Ubuntu 18.04 I got strange file .temp.1592374672.P11164Q21M692534.hostname after doveadm altmove command in INBOX on alternate storage server What can I do with this file? Remove? I am using dbox regards

3 3

doveadm altmove not working for some folders
by Maciek Jackowski 01 Nov '20

01 Nov '20

dovecot 2.2.33.2 on Ubuntu 18.04 I am using dbox with altmove to save storage space on mail server I got strange behaviour with one email box where folder content Archives/2018 is not moving but other folders Archives/2017 and Archives/2019 are moved to the alternate storage server Well I dont know howto start to resolve this regards

1 0

sieve-extdata plugin breaks on pigeonhole 0.5 ish
by Ed W 01 Nov '20

01 Nov '20

Hi, I wonder if someone could give me a little help bringing the sieve-extdata plugin up to date to be usable with latest Dovecot/pigeonhole Trying to use both together at present I get an error: managesieve: Fatal: Couldn't load required plugin /usr/lib/dovecot/sieve/lib90_sieve_extdata_plugin.so: dlopen() failed: /usr/lib/dovecot/sieve/lib90_sieve_extdata_plugin.so: undefined symbol: sieve_sys_error This appears to be due to the change in error handling in pigeonhole 0.5 which appears to have removes the "sieve_sys_error()" and "sieve_sys_warning()" functions I'm unclear how this code should be change (I've commented it out for now - eek). I find this extension useful to pass in user specific parameters to my sieve scripts that are stored elsewhere in an sql database, eg spam score, size threshold for attachments, perhaps there is some other way to achieve this? Any hints on how to fixing this would be appreciated Thanks Ed W

3 2

Dovecot, sa-learn and sieve - where to save the Bayes DB
by Francis Augusto Medeiros-Logeay 31 Oct '20

31 Oct '20

Hi! I have dovecot, Spamassassin and postfix running. I started to use sa-learn to get better and more precise spam identification and filtering. I have not set the Bayes_path preference on my local.cf file, thus when I call sa-learn -u myuser(a)mydomain.org --spam, the Bayes DB gets updated at /root/.spamassassin. I decided then to create a sieve filter on dovecot, as described on Dovecot's wiki page. Basically, Dovecot's sieve plugin execute sa-learn with a -u argument, apparently the same way I do when I use it manually. BUT, the thing is that when Dovecot executes sa-learn, the bayes database is created on the virtual user's folder, ie., /var/mail/vmail/mydomain.org/user/.spamassassin. When I call it manually, the db used is /root/.spamassassin` I am now confused as I don't know if, when evaluating spam, spamd will use the Bayes DB on the user's Maildir, or if it will use the one under /root/.spamassassin. I would really like to have my spam evaluation done per user, not site wide, so I wonder: - How can I configure bayes_path for each user, individually, on a virtual users setup? - Does calling spamd with --virtual-config-dir have anything to do with this? I tried it, setting it up with my virtual users path, but it didn't seem to make any difference when it comes to what I describe in this post. With or without it set up, the results above are the same. -- Francis Augusto Medeiros-Logeay Oslo, Norway

1 0

Odd replication behaviour
by James Pattinson 31 Oct '20

31 Oct '20

Hi, I have just built a new pair of similar machines both running CentOS 8.2 (selinux disabled) and Dovecot 2.3.8 (9df20d2db). One machine is a VPS (host A) and one is on my home network (host B). The idea is that they are set up in a master/master config with Dovecot replication. I seem to have this 95% working but there is one strange issue I can’t work out. Currently B is a perfect replica of A. I have pointed an instance of Thunderbird at it, and I can see all my mails. If I delete any mails or change any flags, I see the same changes almost instantly on the A side. PROBLEM: if host A receives a new mail, I don’t see it on B until I do ‘something’ to change metadata, for example deleting any random email, or marking an email as read on EITHER side causes the new email to appear almost instantly on the B side. I would have expected emails on B to appear immediately. Am I doing something wrong? Extra info - my mailboxes are in Maildir format with single OS user (vmail). I have about 4000 emails in the Inbox and about 30k in other folders. There are only 5 users and I’m using passdb as the very simple backend. Replication is via doveadm on a specified port (not SSH). Some output from dovecot -n is below. Cheers James HOST A # 2.3.8 (9df20d2db): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.8 (b7b03ba2) # OS: Linux 4.18.0-193.28.1.el8_2.x86_64 x86_64 CentOS Linux release 8.2.2004 (Core) xfs # Hostname: hosta.domain auth_mechanisms = plain login doveadm_password = # hidden, use -P to show it doveadm_port = 4040 first_valid_uid = 1000 mail_debug = yes mail_home = /srv/vmail/%u mail_location = maildir:/srv/vmail/%u mail_plugins = notify replication managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = } passdb { args = scheme=BLF-CRYPT username_format=%u /etc/dovecot/users driver = passwd-file } plugin { mail_replica = tcp:b.b.b.b:4040 sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = /var/mail/SpamToJunk.sieve } protocols = imap lmtp service aggregator { fifo_listener replication-notify-fifo { group = root mode = 0660 user = vmail } unix_listener replication-notify { group = root mode = 0660 user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0600 user = postfix } } service doveadm { inet_listener { port = 4040 } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = vmail } } ssl = required ssl_cert = </etc/letsencrypt/live/mail2.pattinson.org/fullchain.pem ssl_cipher_list = PROFILE=SYSTEM ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_min_protocol = TLSv1.2 ssl_prefer_server_ciphers = yes userdb { args = username_format=%u /etc/dovecot/users default_fields = uid=vmail gid=mail home=/srv/vmail/%u driver = passwd-file } protocol lmtp { mail_plugins = sieve } protocol lda { mail_plugins = notify replication sieve } HOST B # 2.3.8 (9df20d2db): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.8 (b7b03ba2) # OS: Linux 4.18.0-193.28.1.el8_2.x86_64 x86_64 CentOS Linux release 8.2.2004 (Core) ext4 # Hostname: hostb auth_mechanisms = plain login doveadm_password = # hidden, use -P to show it doveadm_port = 4040 first_valid_uid = 1000 mail_debug = yes mail_home = /srv/vmail/%u mail_location = maildir:/srv/vmail/%u mail_plugins = notify replication managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = } passdb { args = scheme=BLF-CRYPT username_format=%u /etc/dovecot/users driver = passwd-file } plugin { mail_replica = tcp:a.a.a.a:4040 sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = /var/mail/SpamToJunk.sieve } protocols = imap lmtp service aggregator { fifo_listener replication-notify-fifo { group = root mode = 0660 user = vmail } unix_listener replication-notify { group = root mode = 0660 user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0600 user = postfix } } service doveadm { inet_listener { port = 4040 } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = vmail } } ssl = required ssl_cert = </etc/letsencrypt/live/mail2.pattinson.org/fullchain.pem ssl_cipher_list = PROFILE=SYSTEM ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_min_protocol = TLSv1.2 ssl_prefer_server_ciphers = yes userdb { args = username_format=%u /etc/dovecot/users default_fields = uid=vmail gid=mail home=/srv/vmail/%u driver = passwd-file } protocol lmtp { mail_plugins = sieve } protocol lda { mail_plugins = notify replication sieve }

1 1

Sieve filter script EXECUTION FAILED
by ＠lbutlr 31 Oct '20

31 Oct '20

So, I have the sieve working up to the point that it calls the script, and the script is called (I get different errors if the script is not there, for example). filter action execute program `darkmode.sh' [[EXECUTION ABORTED]] There's no other error logged (for example, when I had sieve_filter_bin_dir miss-set, I would get: sieve: action filter: failed to execute program `darkmode.sh': vnd.dovecot.filter extension is unconfigured In /var/log/messages, but I fixed that. I assume there is something wrong with what I am doing in the script, bit that is something I am not finding any examples on that are proving useful. The script file is simply: Darkmode.sh: #!/bin/sh echo $1 | sed -e '|</head>|<style>* {color:white !important; background-color: black !important; } </style></head>|' Which is somehwat similar to a working script for reporting spam: sa-learn-spam.sh: #!/bin/sh exec /usr/local/bin/sa-learn -u ${1} --spam -- "Are you pondering what I'm pondering?" "I think so, Brain, but won't it go straight to my hips?!"

5 8

Re: Sieve filter script EXECUTION FAILED
by Aki Tuomi 30 Oct '20

30 Oct '20

> On 30/10/2020 19:25 Bernd Petrovitsch <bernd(a)petrovitsch.priv.at> wrote: > > > On 30/10/2020 17:11, @lbutlr wrote: > [...] > > echo $1 | sed -e '|</head>|<style>* {color:white !important; background-color: black !important; } </style></head>|' > > What should the sed stuff do? > TTBOMK '|' is not known by sed ... > > MfG, > Bernd sed accepts | as alternative for / as boundary. But I think the sed here is missing 's' from start, so this does not actually do anything... Aki

5 5

Re: Sieve filter script EXECUTION FAILED
by Aki Tuomi 30 Oct '20

30 Oct '20

> On 30/10/2020 20:34 @lbutlr <kremels(a)kreme.com> wrote: > > > On 30 Oct 2020, at 11:57, Aki Tuomi <aki.tuomi(a)open-xchange.com> wrote: > > But I think the sed here is missing 's' from start, so this does not actually do anything... > > Copy/paste/edit error. The s is there in the file. > > darkmode.sh: > #!/bin/sh > echo $1 | sed -e 's|</head>|<style>* {color:white !important; background-color: black !important; } </style></head>|' > > I am not sure about the $1. I think filter just pipes the message (or part of the message. > > I will see what happens without the echo I suppose. > > Nope, still the same. > > 32: starting `:contains' match with `i;ascii-casemap' comparator: > 32: matching value `<!DOCTYPE html><html lang="en"><head><title>29-Oct-2020 ""</title> <style>?? ...' > 32: with key `</head>' => 1 > 32: finishing match with result: matched > 32: jump if result is false > 32: not jumping > 34: filter action > 34: execute program `darkmode.sh' > 34: [[EXECUTION ABORTED]] > > -- > They looked at the drinks. > They drank the drinks. I am pretty sure that you are holding filter wrong, it expects you to process the entire mail, not just $1. Aki

1 0

how to enable automatic sharing of folders added/created in an already shared folder hiearchy?
by PGNet Dev 30 Oct '20

30 Oct '20

I'm running dovecot --version 2.3.11.3 (502c39af9) folder sharing is enabled & working. as long as I manually enable enable sharing for each folder/subfolder. e.g., if I enable sharing for each of: sharedtopfolder subfolder1 subfolder2 then all's good -- shared-to users can access according to set permissions. BUT, if I drag any folder under that shared-hierarchy e.g., sharedtopfolder subfolder1 subfolder2 + subfolder3 + subfolder4 the just-added subfolder3 subfolder4 are NOT shared. They CAN be -- manually. And, as soon as they are, they're, again, usable by target users. I'd like to have any/all newly-created, or dragged-in, folders under the already-shared "sharedtopfolder" to be **automatically** enabled for sharing. how is that explicitly setup? is it 'just' Dovecot config? &/or does it require some Sieve?

1 0