April 2016 - dovecot - dovecot.org

sis deduplication broken from 2.2.16 upwards
by Pavel Stano 12 Apr '16

12 Apr '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, sis attachment deduplication is broken in 2.2.16 upwards. It is caused by this commit. https://github.com/dovecot/core/commit/664bf3e236c214aee86294483c379e4fa66c… in src/lib-fs/fs-sis.c function fs_sis_try_link() is comparation of inodes of hash files. Because fs_stat() after that commit use fstat() on open fd of temporary file instead of stat on filename. But that temporary file has differnt inode. It not cause any corruption but it will not save any space. Because every duplicate attachment will be in separate file. - -- [ Ohodnotte kvalitu mailu: https://www.nicereply.com/websupport/4afafd34 ] Pavel Stano | Troubleshooter http://WebSupport.sk *** BERTE A VYCHUTNAVAJTE *** -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJW3XkgAAoJEJDF0QA0DLajmJ0QAIZTFTkN0KMJDT3I/rft5Dpp QYM6kyTVdr/FNeR200LLPjXsHXHF8hhoxkQbpfq1D9ceQO0ERDVaL6knKK7/j5Zj EffO+ercXUkcJbE8iYRNcbdpXltUjgYYzgOha7ULiCk0VxROxcujThNTGCDHY8g/ zHXre2aW7hQ8o1yjjhITIOU5WsENSI8PbFktvvWF6OEMVwv8EnaEckJTRGwEl2fu FEPSV+eQg6jHTs/fgiMxAbd4DSzRa2tkeOrw4l4oprtIuthU5hAi9G42Dk+IFvXk k6imnsI6QMPhlrHCm+8Ym+8FwCv7S75JJ/iRp/sS2zHuEwvfwFcVi8pBSjeV50kZ Z0Xglm35tFrHNLAFFb1CVnEw6ftQpuMEE7o/IYrLMVVdPB0C/1IP0s9ucKK0ccQ0 Ld9RULBo6jLgxsVodEntgtMABkfvxybut/tF3O4mm/iN0rDzlOZHMeQ+lsgcdrZd J7h3ow/J0OYosR7CfRFLA875ue2g4GsF+FFP9/oKx/1TkHXJNXpKAhhjOCnSgxHe h44bDrpJyYKlrCaLcOjQM4iUO0upBPpepOnJu50Fde1hNmyr4NaiHLTo23OGX/v3 kEA6xEgmTsXyr3kjhg51/0tdradqKvih2yIptmhpw8P+RWp+ceMoDX2hF95EGB8q FGklrxdxCIoGPAZTOVyf =imfX -----END PGP SIGNATURE-----

5 7

doveadm sync ignores -u option
by Max 12 Apr '16

12 Apr '16

Hi, Since I upgraded from 2.2.21 to 2.2.23, the user option -u seems to be ignored by doveadm sync, or at least it doesn't handle it correctly. If I run the following in the shell as user 'max': % doveadm sync -u foo(a)example.com -d doveadm(max): Error: User doesn't exist I get it to work if I set the USER environment variable to foo(a)example.com: % USER=foo(a)example.com doveadm sync -u foo(a)example.com -d # sync runs as expected. Kind Regards, Max

3 2

[Dovecot] Replication and public namespaces
by Markus Weippert 11 Apr '16

11 Apr '16

Hi, I'm having some issues with replicating public namespaces. Everything seems to work fine for private namespaces, but while importing some huge mailboxes (many small mails) into a public namespace via imapsync, something goes wrong. The expected mail flow is: old-server (imapsync)> new-server1 (replication)> new-server2 But then, dovecot seems to run into race conditions when the replications process tries to sync the same public mailbox under two or more different users at the same time. As a result, messages get duplicated, new-server2 sends those back to new-server1 which then starts to produce duplicates too. If I don't kill the processes in time and delete the faulty mailbox, they start to produce thousands of mails. In fact, server2 should not export messages at all, since it's not productive yet and does not get any mail except from the replication. The only thing getting logged (only few compared to the huge amount of duplicates produced): "dsync-server(user(a)example.com): Warning: Maildir /...: Expunged message reappeared, giving a new UID" Is there any way to fix this? Regards, Markus doveadm_password = *********** doveadm_port = ***** mail_home = /var/vmail/private/%d/%n mail_location = maildir:~/mail mail_plugins = acl virtual listescape notify replication namespace inbox { inbox = yes location = prefix = separator = / type = private } namespace legacy { alias_for = # ... } namespace ns_public { list = children location = maildir:/var/vmail/public/%d prefix = Public/ separator = / subscriptions = no type = public } namespace virtual { # ... } plugin { mail_replica = tcps:************** } protocols = imap lmtp sieve replication_dsync_parameters = -d -l 60 -N -x virtual -x ns_public -U replication_full_sync_interval = 4 hours replication_max_conns = 20 service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service doveadm { inet_listener { port = 8143 ssl = yes } process_min_avail = 5 user = vmail vsz_limit = 4 G } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = vmail } }

3 7

enable/disable shared namespace in passdb
by abi＠oleane.net 11 Apr '16

11 Apr '16

hello all i'm currently using passdb to enable and disable various namespaces conditionally : for example the following work namespace/inbox/inbox=no namespace/legacy_pfso/disabled=no namespace/legacy_pfso/inbox=no namespace/local_storage/disabled=no namespace/comboINBOX_local_pfso/disabled=no namespace/comboINBOX_local_pfso/inbox=yes but i can't get things like the following to work namespace/shared/disabled=yes -- the shared namespace is configured as shown below namespace shared { type = shared list = yes # children disabled = yes separator = / prefix = shared/%%u/ location = imapc:~/dovemail/shared/%%u/ } imapc_host = localhost imapc_port = 143 imapc_password = superpassword # either super or a dummy imapc_master_user = %u -- it seems the shared namespaces do not have names any more so they can't be modified dynamically is there a way to circumvent ? something i missed ? i'm running 2.2.22 but i can switch easily to 2.2.9 or any other version providing master_user or one of the variables that can let me know if a master login was performed is available in passdb the reason i'm doing this is so i can prevent long chains/loops from occuring : then accessed by imapc: , the passdb should disable the shared namespace so it does not produce (an)other imapc connection(s) thanks for your help alexis

2 5

Segfaults in indexer when after Maildir filename S value is updated
by Daniel Demus 10 Apr '16

10 Apr '16

Every now and then the S value in the Maildir filname is wrong, and is automaticcaly fixed by Dovecot while updating the index. It seems the indexer retains a handle pointing to the old filename and crashes immediately afterwards. The index is updated the next time the indexer is run, is this is only an annoyance and somethign that fills up the log. dovecot: indexer-worker(demus): Error: Maildir filename has wrong S value, renamed the file from /home/demus/Maildir/.INBOX.NYTimes/cur/1100506665.M177356P6525V0000000000000302I010100CD_0.server.demus.dk,S=12206:2,S to /home/demus/Maildir/.INBOX.NYTimes/cur/1100506665.M177356P6525V0000000000000302I010100CD_0.server.demus.dk,S=12658:2,S: 1 Time(s) dovecot: indexer-worker(demus): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x79988) [0xb6dc7988] -> /usr/lib/dovecot/libdovecot.so.0(i_warning+0) [0xb6dc7aa8] -> /usr/lib/dovecot/libdovecot.so.0(i_panic+0x64) [0xb6d6a948] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9d27c) [0xb6ed627c] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x1c8) [0xb6ed67c0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x568a0) [0xb6e8f8a0] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream+0x68) [0xb6e5f3ac] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x56678) [0xb6e8f678] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_virtual_size+0x48) [0xb6e5efd4] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_precache+0x124) [0xb6ed7ee8] -> /usr/lib/dovecot/lib20_fts_plugin.so(+0xd5d4) [0xb6b6f5d4] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x40) [0xb6e5f804] -> dovecot/indexer-worker(+0x1ee8) [0x7f570ee8] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x68) [0xb6ddc5cc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x120) [0xb6dddd10] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x34) [0xb6ddc684] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x50) [0xb6ddc8b4] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x14) [0xb6d70c84] -> dovecot/indexer-worker(main+0xe8) [0x7f570880] -> /lib/libc.so.6(__libc_start_main+0x114) [0xb6bd2b54]: 1 Time(s) dovecot: indexer-worker(demus): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x79988) [0xb6e43988] -> /usr/lib/dovecot/libdovecot.so.0(i_warning+0) [0xb6e43aa8] -> /usr/lib/dovecot/libdovecot.so.0(i_panic+0x64) [0xb6de6948] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9d27c) [0xb6f5227c] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x1c8) [0xb6f527c0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x568a0) [0xb6f0b8a0] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream+0x68) [0xb6edb3ac] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x56678) [0xb6f0b678] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_virtual_size+0x48) [0xb6edafd4] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_precache+0x124) [0xb6f53ee8] -> /usr/lib/dovecot/lib20_fts_plugin.so(+0xd5d4) [0xb6beb5d4] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x40) [0xb6edb804] -> dovecot/indexer-worker(+0x1ee8) [0x7f5eeee8] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x68) [0xb6e585cc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x120) [0xb6e59d10] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x34) [0xb6e58684] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x50) [0xb6e588b4] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x14) [0xb6decc84] -> dovecot/indexer-worker(main+0xe8) [0x7f5ee880] -> /lib/libc.so.6(__libc_start_main+0x114) [0xb6c4eb54]: 1 Time(s) Regards, Daniel Demus

2 1

mdbox and shared \Seen flag
by Dmitry Nezhevenko 09 Apr '16

09 Apr '16

Hi, I've just switched mail storage from maildir to mdbox and found it's very amazing. The only question I've is related to shared mailboxes. In the past I've had a few mailboxes that were shared between two users by just symlinking maildir. So everything was shared including flags and \Seen status. Now I want to get something like this. As far as I understand, "shared" namespace will keep \Seen flag private. So any suggestions how to make \Seen shared? My idea is to create some 'private' mdbox-based namespace that points to fixed location on server, that doesn't depends on username? Is it safe (in terms of locking, accessing indexes, etc)? I'm 100% sure that nobody except dovecot will access it. PS. It's latest version of dovecot. Thanks. -- WBR, Dmitry

2 2

Dovecot Proxy LTMP client connect to TCP port 0
by Wido den Hollander 09 Apr '16

09 Apr '16

Hi, I am trying to set up a Dovecot proxy which proxies through POP3, IMAP and LTMP towards a different Dovecot machine. On the proxy machine I use a MySQL database as a userdb and passwdb backend and it returns the proper information as described here: http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy IMAP and POP3 works just fine, but with LMTP I run into a problem. On the 'proxy' machine Postfix is also running and it deliver locally to LTMP via Socket: virtual_transport = lmtp:unix:private/dovecot-lmtp Dovecot there is also configured to proxy LTMP: lmtp_proxy = yes service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } So far so good. A telnet to localhost 110 or 143 allows me to log in to the POP3/IMAP. When Postfix delivers the message locally to Dovecot through LTMP it tries to proxy it though. When doing so it tries to connect to TCP port 0 (zero). dovecot: lmtp(22580): Error: lmtp client: connect(mbox01.XXXX.nl, 0) failed: Connection refused I know I can return the 'port' field in the userdb query, but the same query is used for POP3, IMAP and LMTP. So that can't be hardcoded. I fixed it for now with a CASE statement in SQL: password_query = SELECT b.hostname AS host, NULL AS password, \ 'Y' AS nopassword, u.email AS destuser, 'Y' AS proxy, \ CASE '%s' WHEN 'lmtp' THEN 2525 WHEN 'pop3' THEN 110 WHEN 'imap' THEN 143 END AS port \ FROM User u, Backend b, Domain d \ WHERE u.domainID = d.domainID \ AND b.backendID = d.backendID \ AND u.email = '%u' %s is a variable containing the service Dovecot is trying to look up. This is however rather hacky. On my destination machine LMTP is listening on port 2525. Is there any way to tell the Dovecot LTMP client to connect to port 2525 by default? Thanks, Wido

1 1

fts_solr: Lookup failed: 400 Bad Request / GET null null
by Chris Laif 08 Apr '16

08 Apr '16

Hi, the log file shows suspicious entries as follows: Apr 3 09:14:49 xxx dovecot: imap(user@domain): Error: fts_solr: Lookup failed: 400 Bad Request The tomcat7 log shows for this request: 10.0.0.234 - - [03/Apr/2016:09:14:49 +0200] "GET null null" 400 - Dovecot is latest 2.2.23 with a pretty standard FTS config copied from the wiki: mail_plugins = " acl fts fts_solr" fts = solr fts_solr = break-imap-search url=http://10.0.0.123:8080/solr/ Unfortunately I do not know (yet) which client action produces these log lines. "GET null null" seems not to be a sensible request in any case. Anyone seeing the same effect? Chris

3 5

Intermittent IMAP Login failures - about 25% fail
by Mobile Phone 08 Apr '16

08 Apr '16

New server & just added three domains: now I see "dovecot: imap-login: Aborted login (auth failed, 1 attempts in 5 secs): user=<username.companyname>, method=PLAIN, rip=1.1.1.1, lip=99.99.99.99, TLS, session=<kw6Y2NYviQBex5Un>" Plus all Outlook users keep being bounced and finally get in. I am going to kill all the users soon... Not too many files open, nor bandwidth, nor load. Any light/help appreciated. root@brazil:/var/log# dovecot --version 2.2.9 root@brazil:/var/log# dovecot -n # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.19.0-58-generic x86_64 Ubuntu 14.04.4 LTS auth_debug_passwords = yes auth_mechanisms = plain login mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } protocols = imap pop3 pop3 ssl_cert = </etc/dovecot/dovecot.pem ssl_key = </etc/dovecot/private/dovecot.pem userdb { driver = passwd } Thanks

4 7

Permissions for new folders
by David Cigánek 08 Apr '16

08 Apr '16

Hi, I have a problem with permissions for newly created folders in Dovecot. Everything worked fine until few days ago, when i did a little change in userdb lookup - but i don't know why it is causing problems. Server uses following directory structure: index - /data/mail/index mail - /data/mail/mail alt - /data/mail/archive all 3 on separate volumes, for performance reasons. Up until few days, i used following: # doveadm user dave(a)example.com field value user dave(a)example.com uid 13726 gid 1004 home /data/mail/mail/003/3726/da/dave mail mdbox:~/mbox/:DIRNAME=.00f-dbox-Mails:INDEX=/data/mail/stub/index/data/mail/mail/003/3726/da/dave:ALT=/data/mail/stub/archive/data/mail/mail/003/3726/da/dave config is quite simple: mail_location = mdbox:~/mbox/:DIRNAME=.00f-dbox-Mails:INDEX=/data/mail/stub/index%h:ALT=/data/mail/stub/archive%h passdb { driver = sql args = /data/mail/sql/dovecot-sql.conf.ext } userdb { driver = static args = gid=vmail } from SQL i get those fileds username domain password userdb_home userdb_uid userdb_quota_rule userdb_quota_over_flag For nicer paths, i created a symlinks /data/mail/stub/index/data/mail -> /data/mail/index /data/mail/stub/archive/data/mail -> /data/mail/archive so path /data/mail/stub/index/data/mail/mail/003/3726/da/dave is in fact in /data/mail/index/003/3726/da/dave In that setup, everything worked fine, but i didn't like those simlinks, so i altered the SQL query to return all paths in userdb_mail filed: # doveadm user dave(a)example.com field value user dave(a)example.com uid 13726 gid 1004 home /data/mail/mail/003/3726/da/dave mail mdbox:~/mbox/:DIRNAME=.00f-dbox-Mails:INDEX=/data/mail/index/003/3726/da/dave:ALT=/data/mail/archive/003/3726/da/dave So everything is in fact the same but little nicer. But since then, newly created directories doesn't get the same permissions like parent folder as day should (2770), but instead they are created with 2700: # stat -c "%a (%A)" /data/mail/mail/ 2770 (drwxrws---) # stat -c "%a (%A)" /data/mail/mail/003 2700 (drwx--S---) I need to have there 2770 perms, especialy because when other user like this is created: user dave(a)example.com uid 13999 gid 1004 home /data/mail/mail/003/3999/da/dave Dovecot is unable to create directory /data/mail/mail/003/3999, because it doesn't have write permissions for group on /data/mail/mail/003. I can't find any reason for this behavior - is it something in Dovecot or am I doing something wrong? Thank you, David Dovecot version: 2.2.21 # doveconf -n # 2.2.21 (5345f22): /usr/local/dovecot/2.2.21/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.12 (c1c0a23) # OS: Linux 3.10.0-327.4.5.el7.x86_64 x86_64 CentOS Linux release 7.2.1511 (Core) auth_master_user_separator = xxxxxx dict { sieve_movespam = mysql:/data/mail/sql/dovecot-movespam-sql.conf.ext sieve_redirect = mysql:/data/mail/sql/dovecot-redirects-sql.conf.ext } mail_location = mdbox:~/mbox/:DIRNAME=.00f-dbox-Mails:INDEX=/data/mail/stub/index%h:ALT=/data/mail/stub/archive%h mail_plugins = " quota notify replication" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mdbox_rotate_size = 15 M passdb { args = xxxxxx driver = passwd-file master = yes pass = yes skip = authenticated } passdb { args = /data/mail/sql/dovecot-sql.conf.ext driver = sql } passdb { args = /data/mail/sql/dovecot-catchall-sql.conf.ext driver = sql } plugin { mail_replica = tcp:xxx:xxx quota = dict:user-quota::file:~/user_quota.dat quota2 = dict:domain-quota:%d:file:~/../../domain_quota.dat quota2_over_flag_value = 1 quota2_over_script = quota-warning domain %d %n@%d %i missmatch2 quota2_rule = *:bytes=500M quota2_rule2 = Trash:bytes=+5%% quota2_warning = bytes=90%% quota-warning domain %d %n@%d %i warn 90 quota2_warning2 = bytes=100%% quota-warning domain %d %n@%d %i over 100 quota2_warning3 = -bytes=100%% quota-warning domain %d %n@%d %i under 100 quota_grace = 5%% quota_over_flag_value = 1 quota_over_script = quota-warning user %n %n@%d %i missmatch1 quota_rule = *:bytes=0 quota_rule2 = Trash:bytes=+5%% quota_warning = bytes=90%% quota-warning user %n %n@%d %i warn 90 quota_warning2 = bytes=100%% quota-warning user %n %n@%d %i over 100 quota_warning3 = -bytes=100%% quota-warning user %n %n@%d %i under 100 sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = dict:proxy::sieve_movespam;name=sieve_movespam;bindir=~ sieve_before2 = dict:proxy::sieve_redirect;name=sieve_before;bindir=~ sieve_max_actions = 500 sieve_max_redirects = 500 sieve_redirect_envelope_from = recipient } pop3_no_flag_updates = yes protocols = imap pop3 lmtp sieve replication_dsync_parameters = -1 -d -N -l 30 -U replication_max_conns = 5 service aggregator { fifo_listener replication-notify-fifo { group = vmail mode = 0660 user = vmail } unix_listener replication-notify { group = vmail mode = 0660 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0660 user = vmail } } service doveadm { inet_listener { port = xxxx } } service imap-login { client_limit = 10000 inet_listener imap { port = 143 } process_min_avail = 50 service_count = 0 vsz_limit = 256 M } service imap { client_limit = 1 process_limit = 20000 vsz_limit = 1 G } service lmtp { inet_listener lmtp { port = 2525 } } service pop3-login { inet_listener pop3 { port = 110 } } service quota-warning { executable = script /data/mail/scripts/quota-actions.sh unix_listener quota-warning { mode = 0666 } user = root } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } ssl = no ssl_client_ca_file = /etc/pki/tls/cert.pem syslog_facility = local5 userdb { args = gid=vmail driver = static } userdb { args = /data/mail/sql/dovecot-sql.conf.ext driver = sql override_fields = gid=vmail } protocol imap { mail_max_userip_connections = 1000 mail_plugins = " quota notify replication imap_quota" } protocol lmtp { mail_plugins = " quota notify replication sieve" } protocol lda { mail_plugins = " quota notify replication sieve" }

1 0