October 2020 - dovecot

Newbie question about replication
by Piotr Auksztulewicz 27 Oct '20

27 Oct '20

Hi, I am a long-time dovecot user/admin for my family mail server, but I had ventured into configuring replication only recently and I am not sure about how it should work and what are the good practices. My intent for replication was to have another instance to switch to in a short time if something fails. I have daily backups, but it will take much more time to rebuild and restore than just to switch to a good instance while taking time to restore the bad one. The setup seems to work just fine from IMAP client point of view (minus sieve scripts - but I have worked it around as I am not yet ready to upgrade dovecot beyond what's in stable Debian at the moment). I have my clients configured to access instance A exclusively; instance B is just sitting around with no clients routinely connected. Also only instance A is receiving deliveries from both my primary and secondary MTAs. I have some doubt however if this is the right thing to do and if it is correct from the dovecot server point of view. I have observed my mail storage directories on both instances and have noticed that they diverge a little over time - the instance B not having all the updates. However the moment I connect another client to instance B and log in as a particular user, the state on instance B springs to the correct state and I see the same content on both clients, and the storage directories become identical as well. I am afraid that, if instance A fails, I would not have the current state on instance B. Is there anything I should have done differently? Here is dovecot -n output of instance A. Instance B config is symmetrical with imap1/2 hostnames switched. FYI: actual domain is substituted for privacy. mail.mydomain.net is an internal subdomain with private IP addresses. Instances of dovecot are containerized with access to internal virtual bridge on which MTAs and supporting processes are present as well. The access from outside is provided by port forwarding/NAT, only for imaps and pop3s ports. Other ports are only internally accessible for local clients (postfix, mutt, doveadm command line tool etc.) or debug purposes. # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.4 () # OS: Linux 4.19.0-9-cloud-amd64 x86_64 # Hostname: imap1.mail.mydomain.net auth_failure_delay = 3 secs auth_verbose = yes auth_worker_max_count = 5 deliver_log_format = msgid=%m; from=%f: %$ disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it doveadm_port = 4551 first_valid_uid = 100 login_greeting = # <redacted> mail_attribute_dict = file:mail/dovecot-attributes mail_gid = 122 mail_home = /var/vmail/%u mail_location = maildir:~/mail:LAYOUT=fs:DIRNAME=.MailDir:UTF-8 mail_plugins = " notify replication" mail_privileged_group = vmail mail_uid = 113 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext editheader vnd.dovecot.debug namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext.local driver = sql } plugin { mail_replica = tcp:imap2.mail.mydomain.net sieve = ~/.sieve sieve_dir = ~/sieve sieve_extensions = +vnd.dovecot.debug +editheader sieve_user_log = ~/log/sieve.log } pop3_fast_size_lookups = yes pop3_lock_session = yes pop3_no_flag_updates = yes pop3_save_uidl = yes protocols = " imap lmtp sieve pop3" replication_max_conns = 5 service aggregator { fifo_listener replication-notify-fifo { group = root mode = 0660 user = vmail } unix_listener replication-notify { group = root mode = 0660 user = vmail } } service auth-worker { chroot = /etc/dovecot/db process_limit = 10 unix_listener auth-worker { mode = 0600 user = doveauth } user = doveauth } service auth { client_limit = 64 inet_listener sasl { address = imap1.mail.mydomain.net port = 9192 } unix_listener auth-userdb { group = vmail mode = 0660 user = doveauth } user = doveauth } service dict-async { protocol = none } service dict { protocol = none } service director { protocol = none } service doveadm { inet_listener doveadm { address = imap1.mail.mydomain.net port = 4551 } } service imap-hibernate { protocol = none } service imap-login { inet_listener imap { address = imap1.mail.mydomain.net port = 143 } process_limit = 30 service_count = 1 } service imap-urlauth-login { protocol = none } service imap-urlauth-worker { protocol = none } service imap-urlauth { protocol = none } service imap { chroot = /var/vmail client_limit = 15 process_limit = 30 service_count = 32 user = vmail } service indexer-worker { process_limit = 3 user = vmail } service indexer { chroot = $base_dir } service lmtp { client_limit = 1 extra_groups = group = vmail inet_listener lmtp { address = imap1.mail.mydomain.net port = 2306 } process_limit = 6 user = vmail } service log { user = $default_internal_user } service managesieve-login { inet_listener sieve { address = imap1.mail.mydomain.net port = 4190 } process_limit = 3 service_count = 1 } service managesieve { chroot = /var/vmail process_limit = 3 service_count = 1 user = vmail } service pop3-login { inet_listener pop3 { port = 0 } process_limit = 4 service_count = 1 } service pop3 { chroot = /var/vmail process_limit = 4 service_count = 1 user = vmail } service replicator { unix_listener replicator-doveadm { group = root mode = 0600 user = vmail } } service submission-login { protocol = none } service tcpwrap { protocol = none } ssl_cert = </etc/dovecot/private/dovecot.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-sql.conf.ext.local driver = sql } valid_chroot_dirs = /var/vmail /etc/dovecot/db verbose_proctitle = yes protocol lmtp { mail_plugins = " notify replication sieve" } protocol imap { imap_metadata = yes } -- Piotr "Malgond" Auksztulewicz firstname(a)lastname.net

2 5

Re: Dovecot replication not picking up new mail in maildir
by Justin Ridge 27 Oct '20

27 Oct '20

On 26/10/2020 19:25, John Fawcett wrote: > On 25/10/2020 17:19, jayare~dcml at videocoding.org wrote: >> I've got a basic dovecot master-master replication setup between two >> servers (imap1 and imap2). Actions such as deleting messages or moving >> them between folders are replicated correctly and quickly. >> >> However, when new mail is delivered by postfix, replication does not >> occur until something else triggers it, such as one of the delete/move >> actions mentioned earlier. A search suggested this may be due to the >> notify plugin, but I don't see any issue with the settings. Output of >> doveconf -n: > > Hi > > is there any error in the log? > > Can you show some evidence or an example? ie some email delivery in the > master but not on the slave? > > Can you be sure to check both new and cur subdirectories on master and > slave? > > John There's precisely nothing in the log. The new mail arrives on the master, and it is as though dovecot completely ignores it for the purposes of replication. However, I think I have solved the problem by enabling LMTP and forcing postfix to go through that rather than writing to the maildir directly. Perhaps LMTP is required for replication and I just didn't realize it. End result is (I think) it is working properly. J

3 3

imapsieve: setting imapsieve_url disables admin scripts
by Gedalya 27 Oct '20

27 Oct '20

Hello, The documentation says imapsieve_url "has no effect on the administrator-controlled Sieve scripts". However, when setting this item, I get lines such as: Error: imapsieve: mailbox INBOX: Failed to read /shared/imapsieve/script mailbox attribute: Mailbox attributes not enabled and that's it. imapsieve_mailboxXXX_* settings are completely ignored. Unsetting imapsieve_url makes it all work again. # 2.3.11.3 (502c39af9): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.11 (6c69c917) # OS: Linux 4.19.0-5-amd64 x86_64 Debian bullseye/sid xfs # Hostname: ------ auth_master_user_separator = * auth_mechanisms = plain login auth_verbose = yes imapc_features = rfc822.size fetch-headers imapc_host = imap.gmail.com imapc_port = 993 imapc_ssl = imaps log_timestamp = "%Y-%m-%d %H:%M:%S " mail_gid = vmail mail_home = /srv/mail/domains/%d/%n mail_location = mdbox:/srv/mail/domains/%d/%n/mdbox mail_plugins = quota fts fts_solr mail_prefetch_count = 20 mail_privileged_group = mail mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve mdbox_preallocate_space = yes namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe autoexpunge = 30 days special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe autoexpunge = 30 days special_use = \Trash } prefix = separator = / } passdb { args = /etc/dovecot/master-users driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { fts = solr fts_autoindex = yes fts_solr = url=http://127.0.0.1:8983/solr/dovecot/ fts_tika = http://127.0.0.1:9998/tika/ imapsieve_mailbox1_before = file:/usr/local/lib/imapsieve/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Junk imapsieve_mailbox2_before = file:/usr/local/lib/imapsieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Junk imapsieve_mailbox2_name = * imapsieve_mailbox3_before = file:/usr/local/lib/imapsieve/report-spam.sieve imapsieve_mailbox3_causes = COPY imapsieve_mailbox3_name = Junk2 imapsieve_mailbox4_before = file:~/sieve/IMAP-Sent.sieve imapsieve_mailbox4_causes = APPEND COPY imapsieve_mailbox4_name = Sent quota = count:User quota:noenforcing quota_rule = *:storage=5120M quota_vsizes = yes sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = /etc/dovecot/sieve-global/fileinto-spam.sieve sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment sieve_pipe_bin_dir = /usr/local/lib/imapsieve sieve_plugins = sieve_imapsieve sieve_extprograms } pop3_fast_size_lookups = yes pop3_no_flag_updates = yes protocols = " imap lmtp sieve pop3" quota_full_tempfail = yes service auth-worker { user = $default_internal_user } service auth { unix_listener auth-client { group = Debian-exim mode = 0660 user = Debian-exim } unix_listener auth-userdb { group = vmail mode = 0660 user = vmail } user = $default_internal_user } service imap-login { inet_listener imap { port = 143 } service_count = 0 } service imap { vsz_limit = 512 M } service lmtp { inet_listener lmtp { address = ------- port = 2525 } unix_listener lmtp { group = Debian-exim mode = 0660 user = root } } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 0 } service pop3-login { inet_listener pop3 { port = 110 } service_count = 0 } ssl = required ssl_cert = </etc/letsencrypt/live/--------/fullchain.pem ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } verbose_proctitle = yes protocol lmtp { mail_plugins = quota fts fts_solr sieve ssl = no } protocol !indexer-worker { mail_vsize_bg_after_count = 100 } protocol lda { mail_plugins = quota fts fts_solr sieve } protocol imap { mail_max_userip_connections = 30 mail_plugins = quota fts fts_solr imap_quota imap_sieve } protocol sieve { mail_max_userip_connections = 10 }

2 2

imapc_port not working
by David Tildesley 27 Oct '20

27 Oct '20

Hi, Dovecot v2.3.7.2 on Ubuntu 18.04LTS I have configured for imap proxy to imap.gmail.com However whilst the imapc_host setting is working, imapc_port is not - it ignores the value I set and dovecot always tries to connect to gmail on port 143 (as per error log output). I have the imapc settings in the 10-mail.conf which gets loaded by dovecoat.conf Looks like a bug. Any assistance would appreciated. Regards,David.

2 9

forwarding email with sieve of spf domains
by Marc Roos 27 Oct '20

27 Oct '20

Say someone has setup spf for his domain and sends an email to a user that has in roundcube enabled the sieve forward. If the message is forwarded without altering the message headers, this could result in a message being blocked or not relayed, because sending hosts ip, is not in the spf of the from: domain. Possible solutions are: - add option if enabled, it replaces the From: with that of the email address of the sieve user. (Maybe move the original sender to the Reply-To header? Maybe exception for 'internal' forward?) - Upon processing the message, check the spf records, if they are enforced, do the above, otherwise do nothing. https://tools.ietf.org/html/rfc5228#section-4.2

6 7

Roundcube can't receive emails
by direktor 26 Oct '20

26 Oct '20

Hello, Can someone guide me how to test the mail software? I can send emails but can't receive. I have Dovecot - telnet to 143 port says Dovecot OK. Same with port 25. I get back mail delivery problem. 550. No mail shows in Roundcube. Log files maillog shows nothing, seems empty. It's clean install. On testing Exim log I get something like this: pop3-login auth Error: passrd-file: stat (/etc/exim/domains/myserver/passwd) failed: No such file or directory So what is wrong, how to check and how to fix it? -- Sent from: http://dovecot.2317879.n4.nabble.com/

1 0

Dovecot replication not picking up new mail in maildir
by jayare~dcml＠videocoding.org 26 Oct '20

26 Oct '20

I've got a basic dovecot master-master replication setup between two servers (imap1 and imap2). Actions such as deleting messages or moving them between folders are replicated correctly and quickly. However, when new mail is delivered by postfix, replication does not occur until something else triggers it, such as one of the delete/move actions mentioned earlier. A search suggested this may be due to the notify plugin, but I don't see any issue with the settings. Output of doveconf -n: # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf # OS: Linux 4.19.118-0-vanilla x86_64 CentOS Linux release 7.8.2003 (Core) # Hostname: imap1 disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it first_valid_uid = 1000 haproxy_trusted_networks = 172.16.0.0/24 mail_location = maildir:~/Mail mail_plugins = zlib notify replication mailbox_list_index = yes mailbox_list_index_include_inbox = yes maildir_empty_new = yes mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { mail_replica = tcp:imap2:9943 replication_sync_timeout = 2 } protocols = imap service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = dovecot } unix_listener replication-notify { mode = 0666 user = dovecot } } service config { unix_listener config { user = dovecot } } service doveadm { inet_listener { port = 9943 } } service imap-login { inet_listener imap_hpx { haproxy = yes port = 943 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } ssl = no userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lda { mail_plugins = zlib notify replication } protocol imap { mail_plugins = zlib notify replication imap_zlib } I specifically tried setting the following as no and yes as they can affect notifications: mailbox_list_index = yes mailbox_list_index_include_inbox = yes maildir_empty_new = yes The other server has the same settings except obviously mail_replica = tcp:imap1:9943. Any suggestions welcome, thanks!

2 1

dovecot-uidlist invalid data
by Maciej Milaszewski 26 Oct '20

26 Oct '20

Hello I have a problem with Invalid data System debian10 dovecot-2.2.36.4 # 2.2.36.4 (baf9232c1): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.24.rc1 (debaa297) # OS: Linux 4.19.0-12-amd64 x86_64 Debian 10 Oct 23 15:57:52 dovecot6 dovecot: lmtp(33973,media4_js,2KEXD2Dhkl+1hAAAe3x6RQ): Error: Broken file /vmail/me/media4_js/Maildir/dovecot-uidlist line 6875: Invalid data: In debian9 - kernel-4.9.0-13 - problem exists In debian10 - kernel-4.19.0-12 - problem exist In debian8 - kernel 3.16.0-11-amd64 - problem not exists In debian9 - kernel 3.16.0-11-amd64 - problem not exists storage mount from storage NetApp storage:/vmail on /vmail type nfs (rw,noexec,noatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=120,acregmax=120,acdirmin=120,acdirmax=120,hard,nocto,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.19.19.19,mountvers=3,mountport=635,mountproto=tcp,local_lock=none,addr=10.19.19.19) cat /etc/fstab storage:/vmail /vmail nfs rw,sec=sys,noexec,noatime,tcp,hard,rsize=65536,wsize=65536,intr,nordirplus,nfsvers=3,tcp,actimeo=120,nocto 0 0 Probably somthing in kernel or mount options. Any idea ?

1 1

Re: Looking for a guide to collect all e-mail from the ISP mail server
by R. Diez 26 Oct '20

26 Oct '20

> What you are looking for would be a very advanced setup > [...] I don't think so. But we'll see! I would be happy to take a pre-packaged mail server solution like iRedMail which includes RoundCube or whatever. I just need a "easy", practical guide to reconfigure it to 1) download e-mails from a multidrop, and 2) relay external e-mail to the ISP's SMTP server. Well, at least saying it like that does not sound hard. 8-) > More importantly, most ISPs are very limited in the way of support. > [...] While that assertion is generally true, most ISPs I have seen do get the basic e-mail service right. All have a kind of "catch all" e-mail address. Level of service is normally fine for a small business. If it is not, you can always change ISP. Hardware costs are not a problem. Most small companies I have seen have a file server sitting idle most of the time. Creating a VM with VirtualBox or KVM is relatively easy. Backing the mail server up is easy too: you just copy the VM as a big file. In a small business you may need a few Gigabytes per year if you want to keep all e-mails. I have even written a script that stops a KVM virtual machine every day at 4 am, backs the big file up to a NAS, and the restarts the VM. OK, I am not actually using it, and there is room for improvement, but the basic idea would work. If the server breaks, you have some downtime and you lose some data, but not too much. Most small businesses have more downtime and more data loss for other reasons. Retention is problematic anyway even if you use an external provider. With an external provider you also have to consider data protection issues. Small businesses are often exempt from difficult legal requirements (but I am not a lawyer either). > The server would still need to be accessible publicly for email to be routed to it. That's not the case. It hasn't been for an Exchange Server with "POP3 Connector" that I have seen. And it will not be the case with a Dovecote that fetches e-mail from the ISP over multidrop. > [...] > things unexpectedly and you still have maintenance regarding deliverability > (i.e. reputation, dkim/spf/dmarc) and will still be paying additional Not applicable. Reputation, SPF, etc. is handled by the ISP. Those are standard things, a commodity nowadays. > [...] > Webform attacks are rising so there would Not applicable. No web interface exposed. Only accessible over a VPN connection. > [...] > This includes the workaround.org guide which you'll inevitably run across. > That guide was designed for a personal server, its a good step forward > but there are many more requirements needed for business. That is true. On the other hand, you may be overestimating the needs of a small business. Some of them still use Hotmail addresses! > [...] > Stakeholders will need to include all emails related to it at the final signing. > [...] I am thinking of a small business here, where most things are rather more informal. Something along the lines of "I would do it this way, but if you need something professional, I am out of my depth, because I am not actually an admin at all, you know". That's how real life often works. 8-) In other words, I am looking for a workaround.org guide for multidrop. Just for fun! Regards, rdiez

3 4

Panic mbox_sync_handle_header
by Igor Russkih 25 Oct '20

25 Oct '20

Hello all, I'm getting a very often panic on my local dovecot setup when accessing some of the local folders. Error message is always Oct 25 14:34:50 imap(irusskikh): Panic: file mbox-sync.c: line 665 (mbox_sync_handle_header): assertion failed: (mail_ctx->mail.from_offset + move_diff != 1 && mail_ctx->mail.from_offset + move_diff != 2) Oct 25 14:34:50 imap(irusskikh): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xa05a2) [0x7f05e40005a2] -> /usr/lib/dovecot/libdovecot.so.0(+0xa069a) [0x7f05e400069a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f05e3f901d8] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x8b1d7) [0x7f05e432b1d7] -> /usr/lib/dovecot/libdovecot-storage.so.0(mbox_sync+0x48) [0x7f05e432bbf8] -> /usr/lib/dovecot/libdovecot-storage.so.0(mbox_storage_sync_init+0x3b) [0x7f05e432bd3b] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44) [0x7f05e42eb704] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37) [0x7f05e42eb7a7] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_get_status+0x31) [0x7f05e43663c1] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_status+0x31) [0x7f05e42ebb51] -> dovecot/imap(imap_status_get+0x77) [0x7f05e4c25017] -> dovecot/imap(cmd_status+0x19e) [0x7f05e4c171de] -> dovecot/imap(command_exec+0x5c) [0x7f05e4c1ce0c] -> dovecot/imap(+0x1b382) [0x7f05e4c1b382] -> dovecot/imap(+0x1b41c) [0x7f05e4c1b41c] -> dovecot/imap(client_handle_input+0x18d) [0x7f05e4c1b7dd] -> dovecot/imap(client_input+0xac) [0x7f05e4c1bd2c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f05e4016642] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) [0x7f05e4017cee] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) [0x7f05e40166d6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f05e4016888] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f05e3f9aa83] -> dovecot/imap(main+0x329) [0x7f05e4c0e139] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f05e3b81b97] -> dovecot/imap(_start+0x2a) [0x7f05e4c0e2ca] Oct 25 14:34:50 imap(irusskikh): Fatal: master: service(imap): child 12785 killed with signal 6 (core dumped) # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS: Linux 4.4.0-17763-Microsoft x86_64 Ubuntu 18.04.4 LTS Running now on WSL, but have the same badness on generic ubuntu setup as well. Could somebody guide me if thats something known, or me doing bad things? I'm not on the list, so please CC me. Igor

1 0