- dovecot - dovecot.org

Dovecot replication not picking up new mail in maildir
by jayare~dcml＠videocoding.org 26 Oct '20

26 Oct '20

I've got a basic dovecot master-master replication setup between two servers (imap1 and imap2). Actions such as deleting messages or moving them between folders are replicated correctly and quickly. However, when new mail is delivered by postfix, replication does not occur until something else triggers it, such as one of the delete/move actions mentioned earlier. A search suggested this may be due to the notify plugin, but I don't see any issue with the settings. Output of doveconf -n: # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf # OS: Linux 4.19.118-0-vanilla x86_64 CentOS Linux release 7.8.2003 (Core) # Hostname: imap1 disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it first_valid_uid = 1000 haproxy_trusted_networks = 172.16.0.0/24 mail_location = maildir:~/Mail mail_plugins = zlib notify replication mailbox_list_index = yes mailbox_list_index_include_inbox = yes maildir_empty_new = yes mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { mail_replica = tcp:imap2:9943 replication_sync_timeout = 2 } protocols = imap service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = dovecot } unix_listener replication-notify { mode = 0666 user = dovecot } } service config { unix_listener config { user = dovecot } } service doveadm { inet_listener { port = 9943 } } service imap-login { inet_listener imap_hpx { haproxy = yes port = 943 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } ssl = no userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lda { mail_plugins = zlib notify replication } protocol imap { mail_plugins = zlib notify replication imap_zlib } I specifically tried setting the following as no and yes as they can affect notifications: mailbox_list_index = yes mailbox_list_index_include_inbox = yes maildir_empty_new = yes The other server has the same settings except obviously mail_replica = tcp:imap1:9943. Any suggestions welcome, thanks!

2 1

dovecot-uidlist invalid data
by Maciej Milaszewski 26 Oct '20

26 Oct '20

Hello I have a problem with Invalid data System debian10 dovecot-2.2.36.4 # 2.2.36.4 (baf9232c1): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.24.rc1 (debaa297) # OS: Linux 4.19.0-12-amd64 x86_64 Debian 10 Oct 23 15:57:52 dovecot6 dovecot: lmtp(33973,media4_js,2KEXD2Dhkl+1hAAAe3x6RQ): Error: Broken file /vmail/me/media4_js/Maildir/dovecot-uidlist line 6875: Invalid data: In debian9 - kernel-4.9.0-13 - problem exists In debian10 - kernel-4.19.0-12 - problem exist In debian8 - kernel 3.16.0-11-amd64 - problem not exists In debian9 - kernel 3.16.0-11-amd64 - problem not exists storage mount from storage NetApp storage:/vmail on /vmail type nfs (rw,noexec,noatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=120,acregmax=120,acdirmin=120,acdirmax=120,hard,nocto,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.19.19.19,mountvers=3,mountport=635,mountproto=tcp,local_lock=none,addr=10.19.19.19) cat /etc/fstab storage:/vmail /vmail nfs rw,sec=sys,noexec,noatime,tcp,hard,rsize=65536,wsize=65536,intr,nordirplus,nfsvers=3,tcp,actimeo=120,nocto 0 0 Probably somthing in kernel or mount options. Any idea ?

1 1

Re: Looking for a guide to collect all e-mail from the ISP mail server
by R. Diez 26 Oct '20

26 Oct '20

> What you are looking for would be a very advanced setup > [...] I don't think so. But we'll see! I would be happy to take a pre-packaged mail server solution like iRedMail which includes RoundCube or whatever. I just need a "easy", practical guide to reconfigure it to 1) download e-mails from a multidrop, and 2) relay external e-mail to the ISP's SMTP server. Well, at least saying it like that does not sound hard. 8-) > More importantly, most ISPs are very limited in the way of support. > [...] While that assertion is generally true, most ISPs I have seen do get the basic e-mail service right. All have a kind of "catch all" e-mail address. Level of service is normally fine for a small business. If it is not, you can always change ISP. Hardware costs are not a problem. Most small companies I have seen have a file server sitting idle most of the time. Creating a VM with VirtualBox or KVM is relatively easy. Backing the mail server up is easy too: you just copy the VM as a big file. In a small business you may need a few Gigabytes per year if you want to keep all e-mails. I have even written a script that stops a KVM virtual machine every day at 4 am, backs the big file up to a NAS, and the restarts the VM. OK, I am not actually using it, and there is room for improvement, but the basic idea would work. If the server breaks, you have some downtime and you lose some data, but not too much. Most small businesses have more downtime and more data loss for other reasons. Retention is problematic anyway even if you use an external provider. With an external provider you also have to consider data protection issues. Small businesses are often exempt from difficult legal requirements (but I am not a lawyer either). > The server would still need to be accessible publicly for email to be routed to it. That's not the case. It hasn't been for an Exchange Server with "POP3 Connector" that I have seen. And it will not be the case with a Dovecote that fetches e-mail from the ISP over multidrop. > [...] > things unexpectedly and you still have maintenance regarding deliverability > (i.e. reputation, dkim/spf/dmarc) and will still be paying additional Not applicable. Reputation, SPF, etc. is handled by the ISP. Those are standard things, a commodity nowadays. > [...] > Webform attacks are rising so there would Not applicable. No web interface exposed. Only accessible over a VPN connection. > [...] > This includes the workaround.org guide which you'll inevitably run across. > That guide was designed for a personal server, its a good step forward > but there are many more requirements needed for business. That is true. On the other hand, you may be overestimating the needs of a small business. Some of them still use Hotmail addresses! > [...] > Stakeholders will need to include all emails related to it at the final signing. > [...] I am thinking of a small business here, where most things are rather more informal. Something along the lines of "I would do it this way, but if you need something professional, I am out of my depth, because I am not actually an admin at all, you know". That's how real life often works. 8-) In other words, I am looking for a workaround.org guide for multidrop. Just for fun! Regards, rdiez

3 4

Panic mbox_sync_handle_header
by Igor Russkih 25 Oct '20

25 Oct '20

Hello all, I'm getting a very often panic on my local dovecot setup when accessing some of the local folders. Error message is always Oct 25 14:34:50 imap(irusskikh): Panic: file mbox-sync.c: line 665 (mbox_sync_handle_header): assertion failed: (mail_ctx->mail.from_offset + move_diff != 1 && mail_ctx->mail.from_offset + move_diff != 2) Oct 25 14:34:50 imap(irusskikh): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xa05a2) [0x7f05e40005a2] -> /usr/lib/dovecot/libdovecot.so.0(+0xa069a) [0x7f05e400069a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f05e3f901d8] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x8b1d7) [0x7f05e432b1d7] -> /usr/lib/dovecot/libdovecot-storage.so.0(mbox_sync+0x48) [0x7f05e432bbf8] -> /usr/lib/dovecot/libdovecot-storage.so.0(mbox_storage_sync_init+0x3b) [0x7f05e432bd3b] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44) [0x7f05e42eb704] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37) [0x7f05e42eb7a7] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_get_status+0x31) [0x7f05e43663c1] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_status+0x31) [0x7f05e42ebb51] -> dovecot/imap(imap_status_get+0x77) [0x7f05e4c25017] -> dovecot/imap(cmd_status+0x19e) [0x7f05e4c171de] -> dovecot/imap(command_exec+0x5c) [0x7f05e4c1ce0c] -> dovecot/imap(+0x1b382) [0x7f05e4c1b382] -> dovecot/imap(+0x1b41c) [0x7f05e4c1b41c] -> dovecot/imap(client_handle_input+0x18d) [0x7f05e4c1b7dd] -> dovecot/imap(client_input+0xac) [0x7f05e4c1bd2c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f05e4016642] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) [0x7f05e4017cee] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) [0x7f05e40166d6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f05e4016888] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f05e3f9aa83] -> dovecot/imap(main+0x329) [0x7f05e4c0e139] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f05e3b81b97] -> dovecot/imap(_start+0x2a) [0x7f05e4c0e2ca] Oct 25 14:34:50 imap(irusskikh): Fatal: master: service(imap): child 12785 killed with signal 6 (core dumped) # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS: Linux 4.4.0-17763-Microsoft x86_64 Ubuntu 18.04.4 LTS Running now on WSL, but have the same badness on generic ubuntu setup as well. Could somebody guide me if thats something known, or me doing bad things? I'm not on the list, so please CC me. Igor

1 0

dovecot.lda-dupes
by ＠lbutlr 24 Oct '20

24 Oct '20

What is the purpose and/or use of the dovecot.lda-dupes file that (sometimes) exists in the home folder for each user? I've seen some posts about issues with the file or losing track of the location, etc, but nothing on what the file actually does or how it could be useful. I assume it is just a dovecot file that helps not keep track of … I guess duplicate mails? It looks like it contains the message-ids and email destinations for certain, but not all, duplicate emails, but in a binary format. -- "The good news: Hadron Collider went live and did not destroy ALL reality. Bad: I'm the only one who remembers President Gore's 2 terms." --Andy Ihnatko

2 1

net_connect_unix(/var/run/dovecot/stats-writer) failed: Permission denied
by mj 24 Oct '20

24 Oct '20

Hi, We are getting very occasional messags from dovecot: > net_connect_unix(/var/run/dovecot/stats-writer) failed: Permission denied Over the last week, the message appeared five times. (on a mail server with over 100 users, to that's basically almost never) doveconf -n below > # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf > # Pigeonhole version 0.5.4 () > # OS: Linux 4.19.0-10-amd64 x86_64 Debian 10.6 xfs snip... > service stats { > unix_listener stats-reader { > group = vmail > mode = 0660 > user = vmail > } > unix_listener stats-writer { > group = vmail > mode = 0660 > user = vmail > } > } and the on-disk permissions are: > root@dovecot:~# ls -l /var/run/dovecot/*stat* > srw------- 1 root root 0 Oct 6 00:25 /var/run/dovecot/old-stats > prw------- 1 root root 0 Oct 6 00:25 /var/run/dovecot/old-stats-mail > prw------- 1 root root 0 Oct 6 00:25 /var/run/dovecot/old-stats-user > srw-rw---- 1 vmail vmail 0 Oct 6 00:25 /var/run/dovecot/stats-reader > srw-rw---- 1 vmail vmail 0 Oct 6 00:25 /var/run/dovecot/stats-writer We're not sure what makes the Permission denied error happen... Anyone with an idea? MJ

3 3

Sieve_before
by ＠lbutlr 24 Oct '20

24 Oct '20

So, this is what I did instead of default_sieve: sieve_before = file:/usr/lib/dovecot/sieve/bcc.sieve sieve_before2 = file:/usr/lib/dovecot/sieve/filespam.sieve # cat /usr/lib/dovecot/sieve/bcc.sieve require ["variables", "copy"]; if address :matches ["To", "Cc"] ["*@*."] { redirect :copy "backup+295.${1}.${2}@. . ."; } # cat /usr/lib/dovecot/sieve/filespam.sieve require ["fileinto", "imap4flags"]; if header :contains "X-spam-flag" "YES" { fileinto :flags "\Seen" "Junk"; } The bcc.sieve file is generated each day with a little script to put in the day of the year, so that for the backup user the folder structure for backups for my list mail is, for example: 295 kremels kreme <messages> otheruser kreme <messages> And tomorrow new folders for 296 will be created. So far, I think it's working. 🤞 -- C code. C code run. Run, code, run.

4 17

antispam plugin again
by Maciej Milaszewski 23 Oct '20

23 Oct '20

Hello I have a problem with migrating dovecot from 2.2.36 to 2.3.8 - everything works fine, but a problem with migrating anti-spam plugins New dovecot 2.3.x has implemented own antispam-plugin like: ---- new from dovecot 2.3.8 ----- # From elsewhere to Spam folder imapsieve_mailbox1_name = Spam imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_before = file:/usr/lib64/dovecot/sieve/report-spam.sieve # From Spam folder to elsewhere imapsieve_mailbox2_name = * imapsieve_mailbox2_from = Spam imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_before = file:/usr/lib64/dovecot/sieve/report-ham.sieve sieve_pipe_bin_dir = /usr/lib64/dovecot/sieve sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment #setting_name = value sieve_global_dir = /etc/sieve_global sieve_max_redirects = 20 sieve_vacation_use_original_recipient = yes expire = SPAM #expire_dict = proxy::expire expire_dict = redis:host=127.0.0.1:prefix=expire/ --------------------------------------------------------------------------------------- In older version dovecot I use antispam_plugin + simple script: ---------------- old from 2.2.36.4 + antispam plugin ---------- ... antispam_backend = MAILTRAIN antispam_mail_spam = --spam antispam_mail_notspam = --ham antispam_mail_sendmail = /usr/local/bin/spam-learn.sh antispam_pipe_tmpdir = /tmp antispam_spam_pattern_ignorecase = spam;inbox.spam;Unwanted antispam_trash_pattern_ignorecase = trash;Deleted *;Junk*;kosz antispam_debug_target = syslog antispam_verbose_debug = 1 ------------------------- How do I change the bash file to make it work like on the old system (I use pyzor) like: cat /usr/local/bin/spam-learn.sh #!/bin/sh date >> /tmp/spam.txt echo $@ >> /tmp/spam.txt if [ "x$1" = "x--spam" ]; then /usr/bin/pyzor report >> /tmp/spam.txt 2>&1 fi if [ "x$1" = "x--ham" ]; then /usr/bin/pyzor whitelist >> /tmp/ham.txt 2>&1 fi

1 0

Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: transaction context
by Graham Leggett 22 Oct '20

22 Oct '20

Hi all, While trying to reindex a large mailbox to take advantage of Apache Tika, dovecot logs the following message: Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: transaction context The message is vague in that a transaction has failed, but nowhere does it say what the transaction boundaries are. Did the transaction affect the indexing of one message, or many messages? Did the indexing of the whole mailbox get aborted and rolled back, or just a single message? A wide question. How do I ensure that my shiny new dovecot index gets completely reindexed? How do I know if dovecot is indexing just the new messages, and ignoring the existing messages? Regards, Graham —

1 0

Best prometheus exporter for Dovecot 2.3 with the new metrics/stats module
by Sidsel Jensen 22 Oct '20

22 Oct '20

Hi ppl. I am trying to locate the best prometheus exporter for use with Dovecot 2.3 in connection with the new metrics/stats module. Any pointers in the right direction will be appreciated - seems like there is a bunch out there which all works with the old stats module. Anybody have something “in the works” for 2.3 ? Kind Regards, Sidsel Jensen

2 1