June 2015 - dovecot - dovecot.org

Large jumps in dovecot-uidlist
by Mark 07 Jul '15

07 Jul '15

Since upgrading to Debian Wheezy I have been observing large jumps in the value stored in dovecot-uidlist. The effect of these jumps is to confuse some mail clients (Thunderbird, Opera Mail) causing them not to display messages from a (random?) point in time onwards in the affected folder. I have been unable to determine the root cause of these jumps. What I do know: 1. I never observed the problem while running Debian Squeeze. 2. The only way I have found to fix the issue is to remove all the dovecot* files from the affected folder and restart dovecot. I have tried removing various subsets of the dovecot* files but haven't yet found a subset that works. 3. I first tried upgrading from Squeeze to Wheezy within a few weeks of the Wheezy release. At that point the problem occurred so frequently that I couldn't sync my mail client with my mail box. Multiple folders exhibited this problem and while I was fixing them the problem appeared in other folders. I was unable to reach a point where the system was stable so I reverted back to Squeeze. 4. Some time later (I think a year but I am not sure) I tried the upgrade to Wheezy again. This time the system was stable. After a few days I noticed that the problem was still occurring but less frequently. Since then I have been fixing the problem as I notice it and experimenting with various settings (mainly locking related) without success. I now have a cron job that looks for problematic folders and fixes them when found. 5. I have updated Dovecot packages to those from Jessie via wheezy-backports and the problem still occurs. 6. Dovecot is installed along with postfix, amavisd-new, clamav-daemon, procmail and fetchmail. 7. The mail server is running on a eSATA SheevaPlug that boots of an external USB drive. All user home dirs (and associated mailboxes) are located on a LUKS/ext3 encrypted partition. 8. The output of dovecot -n may be found at the end of this mail. 9. Uid values for the three most recent instances of this problem are: 3801596527, 3371927249, 3443181615 10. Some folders seem more susceptible to this problem but I have not been able to identify any common factors/triggers. 11. The problem occurs - on average - once a day. I am happy to provide any further information, configure any debug logging and/or test suggested configuration settings that may help track down the root cause of this problem. Any help gratefully appreciated. Mark # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-kirkwood armv5tel Debian 7.8 mail_debug = yes mail_location = maildir:~/Maildir mail_max_userip_connections = 100 namespace { inbox = yes location = mailbox { special_use = \Drafts name = Drafts } mailbox { special_use = \Junk name = Junk } mailbox { special_use = \Sent name = Sent } mailbox { special_use = \Sent name = Sent Messages } mailbox { special_use = \Trash name = Trash } prefix = name = inbox } passdb { driver = pam } protocols = " imap" service replication-notify-fifo { name = aggregator } service anvil-auth-penalty { name = anvil } service auth-worker { name = auth-worker } service { unix_listener { group = postfix mode = 0666 user = postfix path = /var/spool/postfix/private/auth-client } name = auth } service config { name = config } service dict { name = dict } service login/proxy-notify { name = director } service dns-client { name = dns_client } service doveadm-server { name = doveadm } service { inet_listener { port = 0 name = imap } inet_listener { port = 993 ssl = yes name = imaps } process_min_avail = 5 name = imap-login } service imap-urlauth { name = imap-urlauth-login } service imap-urlauth-worker { name = imap-urlauth-worker } service token-login/imap-urlauth { name = imap-urlauth } service { executable = imap postlogin name = imap } service indexer-worker { name = indexer-worker } service indexer { name = indexer } service ipc { name = ipc } service lmtp { name = lmtp } service log-errors { name = log } service pop3 { name = pop3-login } service login/pop3 { name = pop3 } service { executable = script-login -d rawlog name = postlogin } service replicator-doveadm { name = replicator } service login/ssl-params { name = ssl-params } service stats-mail { name = stats } ssl_cert = </etc/ssl/certs/smtpd.crt ssl_cipher_list = HIGH:MEDIUM:!aNULL:!MD5:!RC4 ssl_key = </etc/ssl/private/smtpd.key userdb { driver = passwd } protocol imap { service replication-notify-fifo { name = aggregator } service anvil-auth-penalty { name = anvil } service auth-worker { name = auth-worker } service auth-client { name = auth } service config { name = config } service dict { name = dict } service login/proxy-notify { name = director } service dns-client { name = dns_client } service doveadm-server { name = doveadm } service imap { name = imap-login } service imap-urlauth { name = imap-urlauth-login } service imap-urlauth-worker { name = imap-urlauth-worker } service token-login/imap-urlauth { name = imap-urlauth } service login/imap { name = imap } service indexer-worker { name = indexer-worker } service indexer { name = indexer } service ipc { name = ipc } service lmtp { name = lmtp } service log-errors { name = log } service pop3 { name = pop3-login } service login/pop3 { name = pop3 } service replicator-doveadm { name = replicator } service login/ssl-params { name = ssl-params } service stats-mail { name = stats } }

2 4

Questions about hardlinks, alternate storage and compression
by Gionatan Danti 06 Jul '15

06 Jul '15

Hi all, I have some questions about hardlinks, alternate storage and compression. I already scanned the list for related information and I have an idea of how things works, but I would like to have a definite answer. System spec: - CentOS 6.6 x64 - dovecot-2.0.9-8.el6_6.4.x86_64 RPM package/version - sdbox mail store 1) About hardlinks: when sending the same message to two different recipients, I see that the two u.x files are created as two different files. Diffing them, I see that the only difference is a single char (see [1] for an example). My questions are: a) it is possible to tell dovecot to create a single file + a single hardlink (linkref=2)? As other IMAP servers support that features (eg: Cyrus, CommunigatePro, etc) I am wondering if I missed something in configuring dovecot... b) If it is not possible, can I run a script that compare the various files and substitute equal ones (minus the changing line) with hardlinks, or it will confuse dovecot? As a side note, why the changing line ever exists? 2) About alternate storage and compression: actually, I use a single mail_location without compression. I would like to have an alternate storage and to enable compression on it only, leaving the main location without compression. I if understand it correctly, it _should_ be done using a command similar to "doveadm -Dv -o "plugin/zlib_save=gz" altmove -uu testuser sentbefore 8d". I'm right thinking that it should work? I will really end with a primary uncompressed mail store and an alternate, zlib-compressed one? Thank you all and sorry if I did some naive questions. [1] 63c63 < G2fd0811c64be8e553d970000eaa8309f --- > G2ed0811c64be8e553d970000eaa8309f -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti(a)assyoma.it - info(a)assyoma.it GPG public key ID: FF5F32A8

1 2

Re: dovecot auth using 100% CPU
by Edward Betts 03 Jul '15

03 Jul '15

Jorge Bastos <mysql.jorge(a)decimal.pt> wrote: > What do you see in the logs? > My guess is that someone is trying a brute force auth against you, Thanks Jorge, I think this is the answer. I'm using dovecot for exim4 SMTP authentication. The exim4 logs show brute force attacks. -- Edward.

3 4

Outlook 2013 not fetching new mail/synchronization issues
by Dragos Pacher 02 Jul '15

02 Jul '15

Hi all, Dovecot 2.2.10 CentOS 6 64 bit. We are having terrible synchronization issues with Outlook 2013 (all updates) on Windows 7 SP1 and 8.1 hosts (all updates). Scenario: 1. Emails were sent since user closed Outlook 2. Next day: start computer, open outlook 3. No new email (even though email were sent overnight) 4. half a day later Outlook fetches emails like nothing happened Second issue: folders created or email send in Outlook 2013 does not synchronize on IMAP, they remain only local .. But I am mostly interested into finding a fix for the first synchronization issue described in scenario. Second is also important but I have a feeling its connected to the first. Any help will be highly appreciated! dovecot.conf listen = * #base_dir = /var/run/dovecot # Enabled mail protocols. #protocols = pop3 imap sieve protocols = pop3 imap # User/group who owns the message files: mail_uid = 89 mail_gid = 89 # Assign uid to virtual users. first_valid_uid = 89 last_valid_uid = 89 mail_max_userip_connections = 200 default_client_limit = 3000 namespace inbox { separator = . prefix = INBOX. inbox = yes # These mailboxes are widely used and could perhaps be created automatically: mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Trash { special_use = \Trash } # For \Sent mailboxes there are two widely used names. We'll mark both of # them as \Sent. User typically deletes one of them if duplicates are created. mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } } # Logging. Reference: http://wiki2.dovecot.org/Logging log_path = /var/log/dovecot.log mail_debug = no auth_verbose = no auth_debug = no auth_debug_passwords = no auth_verbose_passwords = no auth_worker_max_count = 2048 # SSL: Global settings. # Refer to wiki site for per protocol, ip, server name SSL settings: # http://wiki2.dovecot.org/SSL/DovecotConfiguration ssl = yes verbose_ssl = no ssl_ca = </etc/ssl/savvy.cz-CA.pem ssl_cert = </etc/ssl/savvy.cz.crt ssl_key = </etc/ssl/savvy.cz.key disable_plaintext_auth = no # Mail location and mailbox format. #mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/ mail_location = maildir:~/Maildir:INDEX=/home/dovecotindex/%Lh/Maildir/ # Authentication related settings. # Append this domain name if client gives empty realm. auth_default_realm = # Authentication mechanisms. auth_mechanisms = PLAIN LOGIN service imap-login { client_limit = 3024 process_limit = 2048 process_min_avail = 64 vsz_limit = 64 M } service auth { client_limit = 3072 unix_listener auth-userdb { } inet_listener { port = 12345 } } service imap { vsz_limit = 256M # Max. number of IMAP processes (connections) process_limit = 2048 #imap_idle_notify_interval = 2 mins } service pop3 { # Max. number of POP3 processes (connections) process_limit = 1024 } # Virtual mail accounts. userdb { args = /etc/dovecot/dovecot-mysql.conf driver = sql } passdb { args = /etc/dovecot/dovecot-mysql.conf driver = sql } plugin { auth_socket_path = /var/run/dovecot/auth-master # Plugin: autocreate. Create and subscribe to default IMAP folders. autocreate = INBOX autocreate2 = Sent autocreate3 = Trash autocreate4 = Drafts autocreate5 = Junk autosubscribe = INBOX autosubscribe2 = Sent autosubscribe3 = Trash autosubscribe4 = Drafts autosubscribe5 = Junk # Plugin: expire. expire = Trash 7 Trash/* 7 Junk 30 expire_dict = proxy::expire # Pigeonhole managesieve service. # Reference: http://wiki2.dovecot.org/Pigeonhole/Sieve/Configuration # Per-user sieve settings. sieve_dir = /%Lh/sieve sieve = /%Lh/sieve/dovecot.sieve # Global sieve settings. sieve_global_dir = /home/vpopmail/sieve sieve_global_path = /home/vpopmail/sieve/dovecot.sieve #sieve_before = #sieve_after = } service dict { unix_listener dict { # mode = 0660 # user = vmail # group = vmail } } protocol lda { # Reference: http://wiki2.dovecot.org/LDA #mail_plugins = quota sieve autocreate #auth_socket_path = /var/run/dovecot/auth-master #log_path = /var/log/sieve.log #lda_mailbox_autocreate = yes #postmaster_address = root } protocol imap { mail_max_userip_connections = 2048 imap_client_workarounds = tb-extra-mailbox-sep #mail_plugins = quota imap_quota autocreate #mail_plugins = autocreate } protocol pop3 { #mail_plugins = quota pop3_uidl_format = UID%u-%v pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } Regards, Pacher Dragos

3 12

LDAP default value variable interpolation
by Leon Kyneur 01 Jul '15

01 Jul '15

I'm wondering if there is a way I can dynamically generate an LDAP default value by using variables. I have a configuration that works something like this on my front end proxy: pass_attrs = mail=user,\ =nopassword=y,\ =proxy=y,\ =host=mail.%d The above works perfectly well even though it seems a bit hack. I want to fetch the host field from LDAP and default it to "mail.%d" if the attribute isn't populated for the user. I have tried the following : pass_attrs = mail=user,\ =nopassword=y,\ =proxy=y,\ =host=%{ldap:mailHost:mail.%d} however the interpolation doesn't occur on the default. Just wondering if there is a way to achieve this? thanks

2 2

Getting regularly mailbox desyncs while syncing with imapc
by Michael 30 Jun '15

30 Jun '15

Hello list! I sync approximately ~1000 Mailboxes actually without any problems, except my very own mailbox - Dovecot 2.2.18 (mdbox format) running here. The server from where I sync my mails from runs Dovecot 1.2.4 (with mbox format). The last log line that I receive is: dsync(ms(a)ddnetservice.de): Warning: Mailbox changes caused a desync. You may want to run dsync again. dovecot error log: 2015-06-30 13:09:50 imap(ms(a)ddnetservice.de): Error: Corrupted transaction log file /var/vmail/ddnetservice.de/ms/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log seq 2: indexid changed 1435401731 -> 1435662589 (sync_offset=0) This is the command for running my syncs: doveadm -o imapc_user=<remote_user> -o imapc_password=<remote_pw> -o imapc_host=<remote_host> -o imapc_features="rfc822.size fetch-headers" -o mail_prefetch_count=20 -v -D sync -R -u<local_user> imapc: &> $LOGS/$BOX.log A re-run of this command for my mailbox doesn't fix the problem. The "Inbox" remains empty. Only a "doveadm force-resync -ums(a)ddnetservice.de" fixes this problem with the exception that the Inbox still remains empty. So after force-resyncing I need to run the sync again and my Inbox remains consistent until the next imapc sync run. Actually the desync message is self-explanatory, but I wonder why this behaviour only occurs for my own Inbox (only Inbox, other directories aren't affected) and not anywhere else. Is there any possiblity to debug this? Could also the amount of new incoming mails cause this or do I have maybe a bugged mail in my original mailbox that is causing my trouble? Any hint is appreciated. Thanks, Michael

1 0

Same-server mail transfer
by Andrew Obertas 30 Jun '15

30 Jun '15

Hello, What would be the best way to perform a same-server Maildir transfer, while retaining the IMAP structure already in place, between users with the same GID but different UIDs? I've tried doveadm import but I presume it is encountering a permissions error due to the different UIDs. Worst case scenario, I could move everything over manually and then update the UIDs. I feel like there should be a more elegant solution though that I'm not immediately grasping however. -- Best regards, Andrew Obertas Junior Systems Administrator Infinet Communications Group

1 0

replication - remove/delete in storage
by lejeczek 29 Jun '15

29 Jun '15

hi, novice question - is replication supposed to replicate maildir in it's entirety? I get new mails replicated but not operations of removal/deletion, is this normal? many thanks

2 3

authenticate LDAP to email server
by robert k Wild 29 Jun '15

29 Jun '15

hi all, first things first my version and config - # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-504.el6.x86_64 x86_64 CentOS release 6.6 (Final) auth_mechanisms = plain login mail_location = maildir:~/Maildir mbox_write_locks = fcntl passdb { driver = pam } protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { driver = passwd } - i have installed dovecot+postfix email server and a LDAP server, if i create users on the LDAP server can dovecot pick up the usernames and create a mailbox for the individual users as atm if i want to create an email address for users i add the user to the email server and make a directory for the users mailbox and have to chmod and chown also just another question, can i set up mail groups on dovecot? cheers rob -- Regards, Robert K Wild.

4 12

cram-md5 cache problem
by d tbsky 28 Jun '15

28 Jun '15

hi: I was using scientific linux 6 and had cache problem. I upgrade to scientific linux 7 but the problem is the same, so I try to find out why. dovecot version: 2.2.10. postfix + postfixadmin environment. use dovecot-lda to deliver local mail. if I enable cache with "auth_cache_size = 102400" and enable cram-md5 with "auth_mechanisms = plain login cram-md5". I have cache problem: if I change password at postfixadmin, I can not use the new password to login imap when old password is cached. dovecot log below: Jun 28 11:50:32 auth: Debug: client in: AUTH 6 CRAM-MD5 service=imap securedsession=xjn534sZ9gB/AAAB lip=127.0.0.1 rip=127.0.0.1 lport=143 rport=36342 Jun 28 11:50:32 auth: Debug: client passdb out: CONT 6 PDQzNTU5NTA4MTQxODg5NzUuMTQzNTQ2MzQzMkBtYWlsLmxoeS5jb20udHc+ Jun 28 11:50:32 auth: Debug: client in: CONT<hidden> Jun 28 11:50:32 auth: Debug: cache(test(a)test.example.com,127.0.0.1,<xjn534sZ9gB/AAAB>): hit: <hidden> user=test(a)test.example.com userdb_home=/share/postfix/domains/test.example.com/test/home userdb_mail=maildir:/share/postfix/domains/test.example.com/test userdb_uid=143 userdb_gid=143 Jun 28 11:50:32 auth: Info: cram-md5(test(a)test.example.com,127.0.0.1,<xjn534sZ9gB/AAAB>): password mismatch Jun 28 11:50:34 auth: Debug: client passdb out: FAIL 6 user=test(a)test.example.com Jun 28 11:50:34 imap-login: Debug: Ignoring unknown passdb extra field: Jun 28 11:50:34 imap-login: Info: Disconnected (auth failed, 1 attempts in 2 secs): user=<test(a)test.example.com>, method=CRAM-MD5, rip=127.0.0.1, lip=127.0.0.1, secured, session=<xjn534sZ9gB/AAAB> if I disable "cram-md5" with "auth_mechanisms = plain login". everything is fine. dovecot log below: Jun 28 11:32:24 auth: Debug: cache(test(a)test.example.com,127.0.0.1,<WNsin4sZ6gB/AAAB>): hit: <hidden> user=test(a)test.example.com userdb_home=/share/postfix/domains/test.example.com/test/home userdb_mail=maildir:/share/postfix/domains/test.example.com/test userdb_uid=143 userdb_gid=143 Jun 28 11:32:24 auth: Info: cache(test(a)test.example.com,127.0.0.1,<WNsin4sZ6gB/AAAB>): Password mismatch Jun 28 11:32:24 auth-worker(3711): Debug: sql(test(a)test.example.com,127.0.0.1): query: SELECT username as user, password, '/share/postfix/domains/test.example.com/test/home' as userdb_home, 'maildir:/share/postfix/domains/test.example.com/test' as userdb_mail, 143 as userdb_uid, 143 as userdb_gid FROM mailbox WHERE username = 'test(a)test.example.com' AND active = '1' Jun 28 11:32:24 auth: Debug: client passdb out: OK 15 user=test(a)test.example.com Jun 28 11:32:24 auth: Debug: master in: REQUEST 1951268865 3672 15 0b198494a7e732324f6d6c507b25c13e session_pid=3956 request_auth_token Jun 28 11:32:24 auth: Debug: prefetch(test(a)test.example.com,127.0.0.1,<WNsin4sZ6gB/AAAB>): success Regards, tbskyd

1 0