- dovecot - dovecot.org

Server-side threading
by ohw.giles＠gmail.com 05 Dec '20

05 Dec '20

Hi list, Considering a message A in INBOX, where a reply B is sent and copied to the recipient's SENT folder, and to which a reply-to-reply C is received, again in INBOX, how should the list of threads in INBOX be (ideally) retrieved? Using THREADS=REFS when listing threads in INBOX to avoid matching unrelated threads unfortunately seems to miss the above example because B is only in SENT, not INBOX. REFERENCES allows A to be linked to C, but the sent message is missing from the returned thread. Of course this is by design. When listing messages in a selected mailbox, many modern clients provide the experience of listing the messages grouped by thread, showing all threads where *at least one message in the thread is in the selected mailbox*. IIUC, clients usually build their own indexes to achieve this, which seems wasteful. Does dovecot provide a server-side way to list or search messages and return threaded results in this manner?

1 0

doveadm sync return "failed: EOF (version not received)"
by Olivier Mars 04 Dec '20

04 Dec '20

Hi, I have 2 dovecot servers, server1.domain.tld and server2.domain.tld and I'm not able to do a simple doveadm backup command between them. Dovecot version 2.3.11.3, on debian buster, (installed from apt) I try to execute this cmd between to of my dovecot servers : doveadm sync -u postmaster(a)domain.tld remote:root@server2.domain.tld and it returns : doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): auth-master: userdb lookup(postmaster(a)domain.tld): Started userdb lookup doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): auth-master: conn unix:/var/run/dovecot/auth-userdb: Connecting doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): auth-master: conn unix:/var/run/dovecot/auth-userdb (pid=118292,uid=0): Client connected (fd=15) doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): auth-master: userdb lookup(postmaster(a)domain.tld): auth USER input: postmaster(a)domain.tld mail=maildir:/mails/2/domain.tld/postmaster/Maildir:INDEX=/var/lib/dovecot/indexes/domain.tld/postmaster@domain.tld home=/mails/2/domain.tld/postmaster quota_rule=*:storage=3145728000b:messages=500k doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): auth-master: userdb lookup(postmaster(a)domain.tld): Finished userdb lookup (username=postmaster(a)domain.tld mail=maildir:/mails/2/domain.tld/postmaster/Maildir:INDEX=/var/lib/dovecot/indexes/domain.tld/postmaster@domain.tld home=/mails/2/domain.tld/postmaster quota_rule=*:storage=3145728000b:messages=500k) doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): Added userdb setting: mail=maildir:/mails/2/domain.tld/postmaster/Maildir:INDEX=/var/lib/dovecot/indexes/domain.tld/postmaster@domain.tld doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): Added userdb setting: plugin/quota_rule=*:storage=3145728000b:messages=500k doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): Effective uid=65534, gid=65534, home=/mails/2/domain.tld/postmaster doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): Quota root: name=Mailbox backend=count args= doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): Quota rule: root=Mailbox mailbox=* bytes=3145728000 messages=512000 doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): Quota grace: root=Mailbox bytes=314572800 (10%) doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): Namespace inbox: type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/mails/2/domain.tld/postmaster/Maildir:INDEX=/var/lib/dovecot/indexes/domain.tld/postmaster@domain.tld doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): maildir++: root=/mails/2/domain.tld/postmaster/Maildir, index=/var/lib/dovecot/indexes/domain.tld/postmaster(a)domain.tld, indexpvt=, control=, inbox=/mails/2/domain.tld/postmaster/Maildir, alt= doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): quota: quota_over_flag check: quota_over_script unset - skipping doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): brain M: Namespace INBOX. has location maildir:/mails/2/domain.tld/postmaster/Maildir:INDEX=/var/lib/dovecot/indexes/domain.tld/postmaster@domain.tld doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): Namespace INBOX.: Using permissions from /mails/2/domain.tld/postmaster/Maildir: mode=0700 gid=default doveadm(postmaster(a)domain.tld): Error: remote(/var/run/dovecot/doveadm-server): read(root(a)server2.domain.tld) failed: EOF (version not received) doveadm(postmaster(a)domain.tld): Debug: remote(/var/run/dovecot/doveadm-server): auth-master: conn unix:/var/run/dovecot/auth-userdb (pid=118292,uid=0): Disconnected: Connection closed (fd=15) On the destination server, I can see that the destination server is well retrieving the user's info in userdb, before saying : server2 dovecot: doveadm: Error: read(<local>) failed: EOF (version not received) I tried to activate verbosity everywhere, but It does not show anything, like : ssh -v -lroot server2.domain.tld doveadm dsync-server -upostmaster(a)domain.tld -U debug1: Authentication succeeded (publickey). Authenticated to server2;domain.tld ([w.x.y.z]:22). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions(a)openssh.com debug1: Entering interactive session. debug1: pledge: network debug1: client_input_global_request: rtype hostkeys-00(a)openssh.com want_reply 0 debug1: Remote: /root/.ssh/authorized_keys:4: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding debug1: Remote: /root/.ssh/authorized_keys:4: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding debug1: Sending environment. debug1: Sending env LANG = fr_FR.UTF-8 debug1: Sending command: doveadm dsync-server -upostmaster(a)domain.tld -U debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: client_input_channel_req: channel 0 rtype eow(a)openssh.com reply 0 debug1: channel 0: free: client-session, nchannels 1 Transferred: sent 2800, received 2820 bytes, in 0.1 seconds Bytes per second: sent 25466.2, received 25648.1 debug1: Exit status 0 If somebody can tell me some hints to go further, it'll be very apreciate. Thanks ! Here is the doveconf -n output (the two servers have the same) # 2.3.11.3 (502c39af9): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.11 (6c69c917) # OS: Linux 4.19.0-12-amd64 x86_64 Debian 10.6 # Hostname: server1 auth_cache_negative_ttl = 0 auth_cache_size = 100 M auth_cache_ttl = 5 mins auth_master_user_separator = * default_client_limit = 2000 disable_plaintext_auth = no doveadm_api_key = # hidden, use -P to show it doveadm_worker_count = 50 listen = w.x.y.z mail_fsync = always mail_gid = nogroup mail_location = maildir:~/Maildir:INDEX=/var/lib/dovecot/indexes/%d/%n mail_max_userip_connections = 300 mail_plugins = quota mail_uid = nobody maildir_very_dirty_syncs = yes namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe autoexpunge = 90 days special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Trash { auto = subscribe autoexpunge = 90 days special_use = \Trash } prefix = INBOX. separator = . } passdb { args = /etc/dovecot/master-users driver = passwd-file master = yes result_success = continue } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { imapsieve_mailbox1_before = file:/var/lib/dovecot/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = INBOX.Junk imapsieve_mailbox2_before = file:/var/lib/dovecot/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = INBOX.Junk imapsieve_mailbox2_name = * quota = count:Mailbox quota_rule = *:storage=1G:messages=500k quota_vsizes = yes sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = /var/lib/dovecot/sieve/antispam.sieve sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment +spamtest +spamtestplus sieve_pipe_bin_dir = /var/lib/dovecot/sieve sieve_plugins = sieve_imapsieve sieve_extprograms sieve_spamtest_max_header = X-Spam-Status: [[:alnum:]]+, score=-?[[:digit:]]+\.[[:digit:]] required=([[:digit:]]+\.[[:digit:]]).* sieve_spamtest_status_header = X-Spam-Status: [[:alnum:]]+, score=(-?[[:digit:]]+\.[[:digit:]]) sieve_spamtest_status_type = score } pop3_fast_size_lookups = yes pop3_reuse_xuidl = yes pop3_save_uidl = yes pop3_uidl_format = %f protocols = imap pop3 lmtp service anvil { client_limit = 5000 unix_listener anvil-auth-penalty { mode = 00 } } service auth { client_limit = 0 } service doveadm { inet_listener http { port = 8080 } vsz_limit = 2 G } service imap-login { process_limit = 2000 service_count = 1 } service imap { client_limit = 1 process_limit = 5000 vsz_limit = 1 G } service indexer-worker { process_limit = 20 } service lmtp { client_limit = 1 inet_listener lmtp { port = 24 } process_limit = 200 process_min_avail = 20 vsz_limit = 1 G } service pop3-login { process_limit = 2000 service_count = 1 } ssl = no submission_host = smtp.domain.tld:25 userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_plugins = quota sieve } protocol imap { mail_plugins = quota imap_sieve imap_quota }

1 0

Any RPMs for aarch64 available?
by Peter Cooper Jr. 04 Dec '20

04 Dec '20

Hey, I really appreciate Dovecot being available with standard repos in repo.dovecot.org. I'm currently using the "CentOS 7" one on Amazon Linux 2 and it works great for x86_64. I'm wondering if there are any plans for also providing arm builds (aarch64) at some point? Or if they're available somewhere else already? I may play around with compiling things myself, but just downloading an official package is generally the easier way to go so I thought it couldn't hurt to ask. Thanks, Peter

3 4

follow up to my mail issues I posted about
by Christy S 03 Dec '20

03 Dec '20

Okay. A few days late, but I've gone through the replies I received from several of you and consolidated responses into one mail. Of course life gets crazy when I need to sit down and work on things. "It seems to me Thunderbird is struggling to write to the Sent mailbox, so disk space, and file permissions are the obvious ones to check. And yes, on the server rather than your local machine, as you're using IMAP" It's definitely not disc space. I wouldn't think an upgrade would change permissions, but it's a place to check. I'm showing my newbieness here but, I'm not even sure what account should have access to those files. I know, it's a miracle I got all this working in the first place. I should have taken notes, but I did so much fiddling with this and that to make it behave that I didn't know what to write down. "Anything interesting in the dovecot logs at the time when you check?" So I looked up dovecot logs on google, and what I'm seeing is that dovecot generally writes to mail logs under /var/log. The stuff I sent in my first email came from mail.err in that folder. The only other file I could find was mail.log. Using tail on that file, I see entries like these. Dec 2 20:53:07 kylesmith-music postfix/smtpd[396853]: warning: unknown[212.70.149.37]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 Dec 2 20:53:07 kylesmith-music postfix/smtpd[396853]: disconnect from unknown[212.70.149.37] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4 There are similar entries from a different ip address, but interestingly neither match the ip address of our fiber modem so I have no idea what's going on there. "First, do you have a backup prior to upgrading the server? You may want to refer to that to get a clean idea of how the configuration was set up initially. Sometimes the upgrade process can reset configuration files and its usually easier to work from a known working configuration. " I really wish I did, but I'm not sure how to effectively back up a VPS. "Second, can you describe how you set the mail stack you are using up?" I can tell you it's postfix and dovecot. That's what a lot of articles recommended, so I went with that. We use different devices so I chose to use imap as it interfaces directly with the stored mail on the server. Usually this is good, except now when it breaks. "Its possible the issue is SSL related but its difficult to say. There have been a number of breaks with SSL encryption in recent years which is why the cipher list has been adjusted," SSL is definitely my weakest point of knowledge. I know I had it working smoothly but it was basically following how to stuff. The reason we're using it even for mail has to do with someone in my husband's life who would get in and mess things up if he had the chance, so I'm trying to make sure he doesn't have that chance by locking things down tight. "google can also be out of date I'd recommend using a date filter when using it for checking configurations and limit it only to the last 1-2 years as you will get more relevant information typically." Oh my gosh, that alone would be extremely helpful. The number of seriously outdated articles I had to filter through when I set this up in the first place is just unreal. Mind telling me how to do a date filter? Otherwise, I'll google how to use google, hahaha. "The configuration parameter for the cipher list uses HIGH as a default profile and if I recall correctly that disables lower TLS versions that are susceptible to certain types of attacks. (SSL3,TLS1,TLS1.1,1.2 I think) The dovecot documentation explains what the defaults are for HIGH. The ! prevents using specifica protocols and configuration is usually a chain (processed from left to right until a match is found). DH is the diffie-helman exchange. Usually this file is recalculated on a per server basis to prevent pre-calculation attacks on SSL and usually it must meet a certain key length. DH Groups 1 and 2 are known to be insecure." Okay, that went way over my head, but it sounds like good information to have and study up more on, hopefully after I get the immediate issue solved. If I'm following you correctly at all though I could see that potentially being my issue, hmm. I will for sure see if I can get my hands on that book. "Some quick thoughts here — if the changes you mentioned did not solve the issue, I would definitely comment those back out so you are only troubleshooting one thing at a time." Fair point. I commented out the one about dh high, then hopefully reloaded the configuration, dovecot reload? That done, I tried sending an email from the domain to my gmail using thunderbird. I got the same message, but it did actually send this time. However, when I replied to the test message with my gmail account, it wasn't received by thunderbird. I do see it using the mail app on the server, though. "Next, are you able to send email using any other client?" I can send mail locally on the server from one account to another for sure. I managed it once, at least. Those mail clients seem clunky though so I may not be doing things correctly to test. "Third, try disabling all SSL and see if you are able to send via Thunderbird or really, any client at all…" Is there an easy way to disable ssl for now and then reenable it? That would definitely help narrow this down. "Your DH parameters are too weak. You should generate at least 2048 byte parameters." To be honest, I don't even recall setting up DH parameters. I would guess that probably happened when I was setting up ssl? Again, thank you to each of you for helping with this. I really try not to send stuff like this to mailing lists that are technical in nature, but this is important business mail he's potentially missing and I'm a bit out of my league. First project once this gets fixed will be learning how to back up the server. Christy

3 2

New Version
by Jorge Bastos 02 Dec '20

02 Dec '20

Howdy, Sorry the question, I havent seen any update on github since July, is everything ok with Dovecot? Thanks,

2 3

Re: Re: Dovecot+Solr+Tika segmentation fault
by Manuel Josupeit-Walter 02 Dec '20

02 Dec '20

>> A fix was posted on this list which seems to avoid segfaults when >> using solr+tika. I reposted it not long ago in response to another >> question. Before exploring the other segfaults it would be worthwhile >> getting that fix. If other segfaults continue then I suggest posting >> the logging from Dovecot >> > > https://github.com/dovecot/core/commit/043d111cea78533cf22d3bfe0872c8376d84… > (DOV-4090) > > Scheduled for 2.3.13. Meanwhile, I also found the fix from October. I can confirm that the issue occurred on 2.3.11.3 only. I switched to the previous version (which is pretty easy thanks to Docker) and the issue disappeared. All messages were indexed properly afterwards. I applied the fix and will observe it for the next couple of days. The assertions have been misplaced anyway from my viewpoint because the asserted fields (or properties or whatever, I don't know how they're called in C) are not accessed in every case. I am happy that this is scheduled for 2.3.13. Although, I hope that the payload_input or payload_output won't be NULL when payload_finished is true. :) Thanks a lot to you guys for all the effort. I would not be able to resolve this on my own without your help. Best, Manuel

1 0

Error: Mailbox INBOX: Sync failed
by Jos Chrispijn 02 Dec '20

02 Dec '20

Just saw in my logfile this line appeared: imap(xxxxxx)<56614><CMMJw9i0aoBSsH9H>: Error: Mailbox INBOX: Sync failed for mbox: Expunged message reappeared to mailbox (UID 174550 < 174552, seq=11, idx_msgs=9) Looks like messages' UIDs are mixed up? I thought that if Dovecot sees broken X-UID headers for "new" messages, they're silently fixed? Can you tell how I can correct this error/warning? Thanks! BR, Jos

1 1

Dovecot+Solr+Tika segmentation fault
by Manuel Josupeit-Walter 02 Dec '20

02 Dec '20

3 2

Bug report: invalid \HasNoChildren in response when asterisk is used in the name
by Merijn Schering 01 Dec '20

01 Dec '20

Hi, I'm using dovecot 2.3.4.1 (f79e8e7e4). I have created a folder called "Test with *". Now when I do a command: A3 LIST "" "%" RETURN (CHILDREN STATUS (MESSAGES UNSEEN)) It responds: * STATUS Test with * ( MESSAGES 0 UNSEEN 0 ) * LIST ( \HasNoChildren \UnMarked ) / Spam But it actually has children! This only happens when I use a * character in the name. I hope you can fix this some day. Thanks! Best regards, Merijn Schering Intermesh https://www.group-office.com https://twitter.com/GroupOffice

2 1

doveadm crash on import
by Ralf Hildebrandt 01 Dec '20

01 Dec '20

% doveadm import -u restore(a)backup.invalid mdbox:/home/copymail2/mdbox '' mailbox INBOX header Reply-To gmail.com results in: doveadm(restore(a)backup.invalid): Panic: fcntl(/home/copymail2/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log, write-lock, F_SETLKW) locking failed: Resource deadlock avoided (WRITE lock held by pid 37462) doveadm(restore(a)backup.invalid): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7fb131662d22] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7fb131662e3e] -> /usr/lib/dovecot/libdovecot.so.0(+0xed75e) [0x7fb13166d75e] -> /usr/lib/dovecot/libdovecot.so.0(+0xed7a1) [0x7fb13166d7a1] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fb1315c2beb] -> /usr/lib/dovecot/libdovecot.so.0(+0xf2932) [0x7fb131672932] -> /usr/lib/dovecot/libdovecot.so.0(file_wait_lock_error+0x26) [0x7fb131672966] -> /usr/lib/dovecot/libdovecot.so.0(file_wait_lock+0x20) [0x7fb131672a00] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_transaction_log_file_lock+0x90)[0x7fb131a28050] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_transaction_log_lock_head+0x3d)[0x7fb131a2d4dd] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_transaction_log_append_begin+0x3e)[0x7fb131a26a8e] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xf52f2) [0x7fb131a202f2] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_transaction_commit_full+0x76)[0x7fb131a20826] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_transaction_commit+0x1c)[0x7fb131a208bc] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xdd619) [0x7fb131a08619] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_cache_add+0x214) [0x7fb131a08f24] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header+0x5a5) [0x7fb1319eaef5] -> /usr/lib/dovecot/libdovecot.so.0(+0xc5b13) [0x7fb131645b13] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_memarea+0x74) [0x7fb13167ab14] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x29) [0x7fb13167ae49] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7fb13167b7bd] -> /usr/lib/dovecot/libdovecot.so.0(message_parse_header_next+0x68) [0x7fb13164c288] -> /usr/lib/dovecot/libdovecot.so.0(message_parse_header+0x4f) [0x7fb13164cc5f] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xc9f61) [0x7fb1319f4f61] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcb214) [0x7fb1319f6214] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblock+0x10d)[0x7fb1319f69ed] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonblock+0x22)[0x7fb131978df2] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next+0x3d)[0x7fb131978e5d] -> doveadm(+0x31c14) [0x557ac63f1c14] -> doveadm(+0x2c518) [0x557ac63ec518] -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | https://www.charite.de

2 2