January 2019 - dovecot

doveadm + HA
by Maciej Milaszewski IQ PL 07 Jan '19

07 Jan '19

Hi I have two server directors in ring and 5 dovecot servers (2.2.36) IP for IMAP and POP3 is a VIP (keepalived) What is the best solutions to get realy HA for 5 dovecot servers ? Maby corosync+pacemeker ? But this solution is too problematic and hardcore Why I need HA ? Doveadmin is too lazy and doveadm director does not know that one machine broke down and still sends traffic

2 1

Dovecot no logs by sieve user script
by Christian Wolf 07 Jan '19

07 Jan '19

Dear mailinglist, I have the problem that I had to migrate a setup and since then my user's sieve script does no longer log anything. I have sieve_user_log unset and sieve points to a folder (ready for ManageSievev service). In the old config I had sieve point to the symlink and sieve_dir point to the folder of all scripts. As sieve_dir is deprecated I use the sieve directive only (see below). According to my understanding of https://wiki2.dovecot.org/Pigeonhole/Sieve/ Configuration this should result in user logging in ~/.dovecot.sieve.log. This file is never generated by dovecot and if I create it manually, it is not filled. Can you tell me, what I am missing here in order to get the per-user logs running? Thank you very much Christian # dovecot --version # dovecot -n # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.6 auth_mechanisms = plain login disable_plaintext_auth = no mail_location = maildir:~/Mailbox managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { auto = create special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } passdb { args = scheme=CRYPT username_format=%u /etc/dovecot/users driver = passwd-file } plugin { sieve = file:~/sieve;active=~/.dovecot.sieve } postmaster_address = postmaster@wolf- stuttgart.net protocols = " imap sieve sieve" service auth { unix_listener /var/spool/postfix/ private/auth { mode = 0666 } } service managesieve-login { inet_listener sieve { port = 4190 } } ssl = required ssl_cert = </etc/letsencrypt/live/imap/fullchain.pem ssl_key = # hidden, use -P to show it userdb { driver = passwd } userdb { args = username_format=%u /etc/dovecot/users driver = passwd-file } protocol lda { mail_plugins = " sieve" }

2 3

special_use folders still created
by Marc Roos 07 Jan '19

07 Jan '19

dovecot-2.2.36-3.el7.x86_64 The below configuration seems to work for Thunderbird and emclient, when moving a message to Junk. Emclient uses "Junk E-mail" and Thunderbird uses "Junk". On the server is only the file .Junk created as expected. When I move a message to Archive in Thunderbird. On the server the files .Archives and .Archives.2018 are created (Thunderbird creates a subfolder in Archives) How is this possible? On the server there is already an .Archive folder so it should create .Archive.2018 not? imap_capability = +SPECIAL-USE # drafts folders merging mailbox Drafts { special_use = \Drafts auto = create } # archive folders merging mailbox Archive { special_use = \Archive auto = create } mailbox Archives { special_use = \Archive auto = no } # spam folders merging mailbox Junk { special_use = \Junk auto = create } mailbox Spam { special_use = \Junk auto = no } mailbox "Junk E-mail" { special_use = \Junk auto = no }

4 3

reprocessing mail
by Marc Lucke 06 Jan '19

06 Jan '19

Hey guys, I had some corruptions whereby dovecot would bomb out on mail files in my maildir. I found that if piped the files to dovecot-lda the all email was clean &lost. I even retrieved email I thought might have been lost for good. I have some better ideas now, but I still processed a few thousand emails like that now & would like to do something like this: * stop dovecot * iterate over each file ** get date ** touch date (linux - to set the date/time) ** rename the file with the correct epoch ** update dovecot-uidlist with correct epoch * start dovecot Will that work? Or is there something I've missed or should do? - Marc

1 0

Sieve "OOO" configuration
by Jerry 05 Jan '19

05 Jan '19

I am able to get sieve issuing an "out of office"message correctly. However, I want to configure it to send an "OOO" message only during certain dates, say on weekends, or only between certain hours. I can do that manually; however, was wondering if there is any automatic method available that could handle this chore. Thanks! -- Jerry

2 2

zlib - some emails maybe double or partial gzipped
by admin 05 Jan '19

05 Jan '19

Hi, I found these in my logs: Jan 5 15:58:56 mail dovecot: imap-login: Login: user=<xy(a)z.com>, method=PLAIN, rip=1.2.3.4, lip=2.3.4.5, mpid=5852, TLS, session=<K2oaPLd+885fWs6U> Jan 5 15:58:56 mail dovecot: imap-login: Login: user=<xy(a)z.com>, method=PLAIN, rip=1.2.3.4, lip=2.3.4.5, mpid=5854, TLS, session=<uxAjPLd+ys5fWs6U> Jan 5 15:58:56 mail dovecot: imap(<xy(a)z.com>): Error: Maildir filename has wrong S value, renamed the file from /var/vmail/mailboxes/<xy(a)z.com>/mail/Sent/cur/1537260554.M80670P8164.mail,S=17247,W=17549:2,Sab,Z to /var/vmail/mailboxes/<xy(a)z.com>/mail/Sent/cur/1537260554.M80670P8164.mail,S=4172:2,Sab,Z Jan 5 15:58:56 mail dovecot: imap(<xy(a)z.com>): Error: Corrupted index cache file /var/vmail/mailboxes/<xy(a)z.com>/mail/Sent/dovecot.index.cache: Broken physical size for mail UID 99 in mailbox Sent: read(zlib(/var/vmail/mailboxes/<xy(a)z.com>/mail/Sent/cur/1537260554.M80670P8164.mail,S=17247,W=17549:2,Sab,Z)) failed: Cached message size larger than expected (17247 > 4268, box=Sent, UID=99) Jan 5 15:58:56 mail dovecot: imap(<xy(a)z.com>): Panic: file istream.c: line 175 (i_stream_read): assertion failed: (old_size <= _stream->pos - _stream->skip) Jan 5 15:58:56 mail dovecot: imap(<xy(a)z.com>): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7f5068d3ee92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7f5068d3ef8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f5068cd4a91] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x290) [0x7f5068d49f20] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f5068d4a72d] -> /usr/lib/dovecot/libdovecot.so.0(message_parse_header_next+0x72) [0x7f5068d25d12] -> /usr/lib/dovecot/libdovecot.so.0(+0x76fd1) [0x7f5068d1ffd1] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x53) [0x7f5068d49ce3] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f5068d4a72d] -> /usr/lib/dovecot/libdovecot.so.0(message_parse_header_next+0x72) [0x7f5068d25d12] -> /usr/lib/dovecot/libdovecot.so.0(+0x76fd1) [0x7f5068d1ffd1] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x53) [0x7f5068d49ce3] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f5068d4a72d] -> /usr/lib/dovecot/libdovecot.so.0(message_get_header_size+0x74) [0x7f5068d29904] -> /usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x32f) [0x7f50690b52ef] -> dovecot/imap(+0x1ebbc) [0x556dd2cd7bbc] -> dovecot/imap(+0x1cfb6) [0x556dd2cd5fb6] -> dovecot/imap(imap_fetch_more+0x39) [0x556dd2cd70e9] -> dovecot/imap(cmd_fetch+0x33b) [0x556dd2cc8d9b] -> dovecot/imap(command_exec+0xa5) [0x556dd2cd4735] -> dovecot/imap(+0x199c2) [0x556dd2cd29c2] -> dovecot/imap(+0x19a4c) [0x556dd2cd2a4c] -> dovecot/imap(client_handle_input+0x1b5) [0x556dd2cd2e55] -> dovecot/imap(client_input+0x86) [0x556dd2cd33c6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f5068d539f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7f5068d55029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f5068d53a8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f5068d53c38] Jan 5 15:58:56 mail dovecot: imap(<xy(a)z.com>): Fatal: master: service(imap): child 5852 killed with signal 6 (core dumps disabled) # and again: Jan 5 15:58:56 mail dovecot: imap(<xy(a)z.com>): Error: Maildir filename has wrong S value, renamed the file from /var/vmail/mailboxes/<xy(a)z.com>/mail/Sent/cur/1537260554.M80670P8164.mail,S=4172:2,Sab,Z to /var/vmail/mailboxes/<xy(a)z.com>/mail/Sent/cur/1537260554.M80670P8164.mail,S=4268:2,Sab,Z Jan 5 15:58:56 mail dovecot: imap(<xy(a)z.com>): Error: Corrupted index cache file /var/vmail/mailboxes/<xy(a)z.com>/mail/Sent/dovecot.index.cache: Broken physical size for mail UID 254 in mailbox Sent: read(zlib(/var/vmail/mailboxes/<xy(a)z.com>/mail/Sent/cur/1537260554.M80670P8164.mail,S=4172:2,Sab,Z)) failed: Cached message size smaller than expected (4172 < 4268, box=Sent, UID=254) Jan 5 15:58:56 mail dovecot: imap(<xy(a)z.com>): Panic: file istream.c: line 212 (i_stream_read): assertion failed: (!i_stream_is_buffer_invalid(_stream)) Jan 5 15:58:56 mail dovecot: imap(<xy(a)z.com>): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7f1dd9f6be92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7f1dd9f6bf8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f1dd9f01a91] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x190) [0x7f1dd9f76e20] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f1dd9f7772d] -> /usr/lib/dovecot/libdovecot.so.0(message_get_header_size+0x74) [0x7f1dd9f56904] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x388) [0x7f1dda2a7538] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x66fc0) [0x7f1dda25efc0] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream_because+0x5d) [0x7f1dda22ee1d] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x66d87) [0x7f1dda25ed87] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_virtual_size+0x33) [0x7f1dda22ea83] -> dovecot/imap(+0x1eae3) [0x5595eb2dfae3] -> dovecot/imap(+0x1cfb6) [0x5595eb2ddfb6] -> dovecot/imap(imap_fetch_more+0x39) [0x5595eb2df0e9] -> dovecot/imap(cmd_fetch+0x33b) [0x5595eb2d0d9b] -> dovecot/imap(command_exec+0xa5) [0x5595eb2dc735] -> dovecot/imap(+0x199c2) [0x5595eb2da9c2] -> dovecot/imap(+0x19a4c) [0x5595eb2daa4c] -> dovecot/imap(client_handle_input+0x1b5) [0x5595eb2dae55] -> dovecot/imap(client_input+0x86) [0x5595eb2db3c6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f1dd9f809f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7f1dd9f82029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f1dd9f80a8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f1dd9f80c38] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f1dd9f07fd3] -> dovecot/imap(main+0x328) [0x5595eb2cde68] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f1dd9b572e1] -> dovecot/imap(_start+0x2a) [0x5595eb2cdfea] Jan 5 15:58:56 mail dovecot: imap(<xy(a)z.com>): Fatal: master: service(imap): child 5854 killed with signal 6 (core dumps disabled) Trying to read them with zcat they turned out to be binary. I figured out they might accidentally be compressed twice - they are. How could this happen? I used this script: dovecot-maildir-compress.sh (https://gist.github.com/cviebrock/e37f80a3223dfee19051c936c5ebe509) which appends an additional flag 'Z' after processing/conmpressong the mail and only compresses those without the flag: > find=$(find "$maildir" -type f -name "*,S=*" -mtime +30 ! -name "*,*:2,*,*Z*" -printf "%f\n") ... > if echo $flags | grep ','; > then > newname=$filename"Z" > else > newname=$filename",Z" which is actually neat. Now two possible things: (1) I am not quite sure yet but my best guess is that moving messages from one folder to another (via mail client) the message looses it's 'Z' flag. (2) compressing might have failed to rename but successfully compressed the file and the another consecutive run of the script compressed the file again. How could I recursively (a) check each maildir file if e.g. the output of(zcat $file | zcat) is ASCII or !binary (b) find all mails which are compressed - i.e. valid gzip - but miss the 'Z' flag ? (c) find all mails that are valid gzip but dont contain ASCII if / dont uncompress to mail ? There are backups in place for all files I get a error for, but I want to recursively fix the problem. Any hint? Thanks in advance! -M -- info(a)awib.it

1 0

Searching small(ish) sites on ubuntu
by Dirk Koopman 05 Jan '19

05 Jan '19

I run a small dovecot site with ~5 users, some of whom have many large mailboxes. I used to use lucene until the packagers decided that it broke some licencing rules and stopped including it. I have dismally failed to configure Solr, plus it is huge and potentially exposes too many services as well. Squash is deprecated (but still used?) in 2.2+. I am currently on 2.2.33. So what can I use? Or do I have to bite the bullet and install from source, thus losing the advantages of packaging and automatic updates? Dirk

1 0

Compiling Dovecot on Solaris 11 fails
by Andrew Watkins 04 Jan '19

04 Jan '19

Hi, Tried to build dovecot-2.3.4 on Solaris 11 x86 and it fails at configure part. I just went checked and it last version it works on is v2.3.2.1 Version 2.3.2.1 (OK) # ./configure .... checking Linux compatible mremap()... no checking whether shared mmaps get updated by write()s... yes checking whether fd passing works... yes ... Version 2.3.4 (FAIL) .... checking Linux compatible mremap()... no checking whether shared mmaps get updated by write()s... no checking whether fd passing works... no configure: error: fd passing is required for Dovecot to work .... config.log shows: configure:22538: checking whether shared mmaps get updated by write()s configure:22585: gcc -o conftest -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -mfunction-return=thunk -mindirect-branch=thunk -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 conftest.c >&5 configure:22585: $? = 0 configure:22585: ./conftest ./configure[2026]: eval: line 1: 29183: Memory fault(coredump) .... onfigure:22610: checking whether fd passing works configure:22685: gcc -o conftest -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -mfunction-return=thunk -mindirect-branch=thunk -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I./src/lib ./src/lib/fdpass.c conftest.c >&5 configure:22685: $? = 0 configure:22685: ./conftest ./configure[2026]: eval: line 1: 29206: Memory fault(coredump) ... $ ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files (-n) 512 pipe size (512 bytes, -p) 10 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 29995 virtual memory (kbytes, -v) unlimited Any ideas and thanks. Andrew -- Andrew Watkins * Birkbeck, University of London * Computer Science * * http://notallmicrosoft.blogspot.com * * UKOUG Systems SIG Chair * * UKOUG Tech committee * * tel: 020 7631 6720 * ::e-mails are only read & actioned between 0900 and 1800 hours (Monday - Friday)

4 6

doveadm_allowed_commands doesn't work as expected
by Ronald Poon 04 Jan '19

04 Jan '19

Trying to limit the API calls to doveadm-http-api by configure allowed commands, but once the commands added to the list, the RestAPI no longer work. 1) Return correct reply when doveadm_allowed_commands is empty # curl -k -H "Content-Type: application/json" -H "Authorization: X-Dovecot-API <base64 api key>" https://localhost:9088/doveadm/v1 -d'[["quotaGet",{"user":"user1(a)mydomain.com"},"c01"]]' [["doveadmResponse",[{"root":"User quota","type":"STORAGE","value":"0","limit":1024","percent":"0"},{"root":"Userquota","type":"MESSAGE","value":"0","limit":"-","percent":"0"}],"c01"]] 2) Return unAuthorized when doveadm_allowed_commands = quotaGet,quotaRecalc,expunge # curl -k -H "Content-Type: application/json" -H "Authorization: X-Dovecot-API <base64 api key>" https://localhost:9088/doveadm/v1 -d'[["quotaGet",{"user":"user1(a)mydomain.com"},"c01"]]' [["error",{"type":"unAuthorized", "exitCode":0},"c01"]] Here is my config, # uname -a Linux ad92422d8e94 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 Linux # free -m total used free shared buffers cached Mem: 15885 7133 8751 0 1 4374 -/+ buffers/cache: 2758 13126 Swap: 0 0 0 / # dovecot -n # 2.3.2.1 (0719df592): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.2 (7704de5e) # OS: Linux 3.10.0-862.2.3.el7.x86_64 x86_64 xfs # Hostname: ad92422d8e94 auth_mechanisms = plain login doveadm_allowed_commands = quotaGet,quotaRecalc,expunge doveadm_api_key = # hidden, use -P to show it hostname = mailhost.mydomain.com info_log_path = /dev/stdout lda_mailbox_autosubscribe = yes log_path = /dev/stderr login_greeting = Dovecot ready. mail_gid = vmail mail_home = /var/vmail/%d/%n mail_location = maildir:/var/vmail/%d/%n/Maildir mail_plugins = " quota zlib" mail_privileged_group = vmail mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { imapsieve_mailbox1_before = file:/etc/dovecot/sieve/global/learn-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Junk imapsieve_mailbox2_before = file:/etc/dovecot/sieve/global/learn-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Junk imapsieve_mailbox2_name = * quota = maildir:User quota quota_exceeded_message = User %u has exhausted allowed storage space. recipient_delimiter = - sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = /etc/dovecot/sieve/global/spam-to-folder.sieve sieve_global_extensions = +vnd.dovecot.pipe sieve_pipe_bin_dir = /usr/lib/dovecot/sieve sieve_pipe_exec_timeout = 60s sieve_plugins = sieve_imapsieve sieve_extprograms zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster(a)mydomain.com protocols = lmtp imap pop3 sieve recipient_delimiter = - service auth { inet_listener { port = 9000 } } service doveadm { client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = $default_internal_group inet_listener http { port = 9088 ssl = yes } service_count = 1 } service lmtp { inet_listener lmtp { port = 24 } } service managesieve-login { inet_listener sieve { port = 4190 } } ssl_cert = </etc/tls/mailserver.crt ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it submission_host = mta-host.mydomain.com userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_plugins = " quota zlib sieve" } protocol imap { mail_plugins = " quota zlib imap_sieve imap_quota imap_zlib" }

2 2

gcc -> clang
by Rupert Gallagher 03 Jan '19

03 Jan '19

Please, use clang instead of gcc. Code quality can only profit from it. I just compiled 2.3.4 and compiler stderr is full of interesting problems.

5 4