May 2014 - dovecot - dovecot.org

[Dovecot] Odd ownership of the dovecot-uidlist file
by dclist.hook＠hook.net.nz 27 May '14

27 May '14

Hi, We have just recently switched from Courier to Dovecot for both IMAAP and POP access for our shared hosting platform, and while the issues we have been seeing with Courier have gone away a new one has popped up. Unfortunately googling for an answer is difficult as it falls into the ownership and permissions category which people seem to have all the time. But in those cases it always happens, whereas what we are seeing is a random ownership change on the dovecot-uidlist file. Basically what we see is the user is able to login fine for days and then all of a sudden the file has changed ownership to a user under a different domain, what's worse is the other user has not even logged in during that time. Here's a snippet from a case from a month ago which was escalated to us, there are plenty of other examples but its so random we only know its happened when the customer complains or we happen to watch the logs, the current help desk procedure when the customer calls is to just fix the ownership of the file. -- 8<-- Mar 25 11:18:05 brio dovecot: pop3-login: Login: user=<accounts@DOMAIN1>, method=PLAIN, rip=IP, lip=IP, mpid=21739, TLS, session=<mwQLnGH1ZAB2XalI> Mar 25 11:18:08 brio dovecot: pop3(accounts@DOMAIN1): Disconnected: Logged out top=0 (0 b), retr=4 (274142 b), messages=27 (10797760 b), del=0 Mar 25 11:19:24 brio dovecot: pop3(accounts@DOMAIN1): Error: open(/mnt/spool/keepers/t/DOMAIN1/accounts/Maildir/dovecot-uidlist) failed: Permission denied Mar 25 11:19:24 brio dovecot: pop3(accounts@DOAMIN1): Error: open(/mnt/spool/keepers/t/DOMAIN1/accounts/Maildir/dovecot-uidlist) failed: Permission denied -- 8<-- In this case the ownership of the file was: -rw------- 1 jacqui-30747 DOMAIN2 48839 Mar 25 10:31 dovecot-uidlist when it should be: -rw------- 1 accounts-25105 DOMAIN1 48839 Mar 25 10:32 dovecot-uidlist As you can see both the user and the group have changed to a different user as if that user had recreated the file. Its worth noting that the parent directory does not change: drwx------ 8 accounts-25105 DOMAIN1 4096 Mar 25 10:31 . so this user should not even be able to access this file. It would be great if someone can give us some hints where the problem maybe as this has us stumped. Some more detail if it helps: root@brio:~# dovecot -n # 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.2 auth_mechanisms = plain login cram-md5 apop auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@& default_process_limit = 3000 disable_plaintext_auth = no lock_method = dotlock mail_fsync = always mail_location = maildir:~/Maildir:INDEX=/home/indexes/%1d/%n@%d mail_nfs_index = yes mail_nfs_storage = yes mail_plugins = " quota" maildir_stat_dirs = yes mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = INBOX. separator = . subscriptions = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } passdb { args = /etc/dovecot/dovecot-sql-cram.conf.ext driver = sql } plugin { quota = maildir:User quota } protocols = " imap pop3" service imap-login { inet_listener imap25 { port = 25143 } } service imap-postlogin { executable = script-login -d /usr/local/bin/postlogin.sh } service imap { executable = imap imap-postlogin } service pop3-login { inet_listener pop325 { port = 25110 } } service pop3 { executable = pop3 imap-postlogin } ssl_cert = </var/www/mailadmin/ssl/cert.pem ssl_crypto_device = dynamic ssl_key = </var/www/mailadmin/ssl/mailx.hosts.net.nz.key userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } userdb { args = /etc/dovecot/dovecot-sql-cram.conf.ext driver = sql } verbose_proctitle = yes protocol imap { imap_capability = +XHOST_brio imap_logout_format = bytes=%i/%o mail_max_userip_connections = 50 mail_plugins = " quota imap_quota" } protocol pop3 { mail_max_userip_connections = 10 pop3_lock_session = no pop3_logout_format = top=%t (%p b), retr=%r (%b b), messages=%m (%s b), del=%d pop3_save_uidl = yes pop3_uidl_format = %f } root@brio:~# dpkg -l|grep dovecot ii dovecot-core 1:2.1.7-7 amd64 secure mail server that supports mbox, maildir, dbox and mdbox mailboxes ii dovecot-imapd 1:2.1.7-7 amd64 secure IMAP server that supports mbox, maildir, dbox and mdbox mailboxes ii dovecot-mysql 1:2.1.7-7 amd64 MySQL support for Dovecot ii dovecot-pop3d 1:2.1.7-7 amd64 secure POP3 server that supports mbox, maildir, dbox and mdbox mailboxes Mailbox storage is done over various NFS stores, however each user has a specific NFS store Connections are load balanced using LVS and keepalived over approx 10 real servers, persistant connections are setup so that a single user should only hit a single dovecot server (this is different from our courier setup due to the known NFS limitations with dovecot) We also have IMAP indexes stored on the dovecot servers ext3 file system itself rather than NFS. Each server is setup and maintained by puppet, and any one of the servers may see the issue. All servers are x86 with at least 2Gb of RAM User information comes from a MySQL database, and testing the SQL statements only ever return the correct record. Exim is writing to the Maildir with the correct permissions, and after fixing the ownership of the file dovecot can read and change the mailboxes. We do have a postlogin.sh script which runs after authentication to convert the old courier files over to the dovecot ones, this only happens however when the process has not run before (it appears dovecot deletes the files and recreates them, so we use a state file). Adding debugging to the script shows its running as the correct user at the last successful login. It does not run at all after the ownership has changed. There are some users who this is happening regularly, and other users who it has happened to only once. It is also not just limited to POP3, although this is where we commonly see it as dovecot accepts the IMAP connection if the ownership on a sub folders uidlist is different. If you need any further detail, I can provide it. Debugging could be much more difficult because of how random it is, while it can happen daily there have been periods of up to one week before its seen. Any help in finding this would be greatly appreciated, Cheers, Bruce

2 1

[Dovecot] Incorrect IMAP search results when FTS/Squat indexes are present with 2.1.7
by Robert L Mathews 27 May '14

27 May '14

We recently upgraded from Dovecot 1.2.15 (on Debian squeeze) to Dovecot 2.1.7 (on Debian wheezy/stable). For now, we continue to use FTS with Squat indexes. We'd like to eventually switch to Lucene but it's not yet supported in Debian stable. Since upgrading to 2.1.7, we've seen some problems with IMAP search results when an FTS Squat index is present (i.e., when the "dovecot.index.search" and "dovecot.index.search.uids" files are present). If it's present, even header searches sometimes give incorrect results (such as searches on FROM or SUBJECT headers). I'm wondering if this is a known issue with 2.1.7 and whether upgrading to a later version of Dovecot with Squat is likely to fix this (due to, say, <http://www.dovecot.org/list/dovecot-cvs/2012-June/020881.html>), or if Squat in all versions is a lost cause and I need to switch to Lucene. I can demonstrate the problem by showing the results of three searches on my mailbox. First, here are the correct results for header FTS searches when no "dovecot.index.search[.uids]" file is present: ---- . UID SEARCH FROM "Nick" * SEARCH 9 66 100 116 179 212 274 338 347 386 494 519 579 603 643 766 781 788 827 1107 1280 1626 1641 1667 1688 1702 1703 2184 2187 3202 3519 3716 3798 3885 3886 4024 4166 4345 4400 4401 4436 4491 4561 4595 4618 4676 4711 4712 4746 4788 4883 5217 5253 5577 5602 5722 . OK Search completed (0.059 secs). . UID SEARCH ALL FROM "Nick" * SEARCH 9 66 100 116 179 212 274 338 347 386 494 519 579 603 643 766 781 788 827 1107 1280 1626 1641 1667 1688 1702 1703 2184 2187 3202 3519 3716 3798 3885 3886 4024 4166 4345 4400 4401 4436 4491 4561 4595 4618 4676 4711 4712 4746 4788 4883 5217 5253 5577 5602 5722 . OK Search completed (0.058 secs). . UID SEARCH SUBJECT "Nick" * SEARCH 3466 3467 3507 . OK Search completed (0.054 secs). ---- These results are consistent (and correct) whether "mail_plugins = fts fts_squat" is enabled or not, as long as no "dovecot.index.search[.uids]" file is present. Now I generate an FTS index: ---- . UID SEARCH TEXT "Nick" * OK Indexed 98% of the mailbox, ETA 0:00 * OK Mailbox indexing finished * SEARCH 9 32 66 100 111 112 113 116 179 199 212 228 229 240 274 303 338 347 386 393 448 478 491 494 519 579 591 603 614 618 624 643 649 742 766 781 788 827 837 839 841 1095 1106 1107 1174 1188 1225 1227 1228 1240 1254 1261 1278 1280 1452 1482 1519 1520 1595 1626 1641 1656 1667 1688 1702 1703 1724 1758 2184 2187 2798 2883 2932 3034 3198 3202 3206 3230 3371 3432 3454 3457 3466 3467 3507 3519 3631 3637 3685 3702 3703 3716 3765 3798 3799 3803 3853 3854 3855 3857 3873 3885 3886 3962 3969 4024 4033 4050 4051 4058 4161 4166 4173 4176 4224 4273 4345 4400 4401 4436 4461 4490 4491 4498 4561 4595 4618 4651 4676 4711 4712 4713 4729 4731 4736 4737 4738 4739 4746 4786 4787 4788 4813 4816 4822 4847 4883 4885 4933 4941 4942 4957 4958 5029 5052 5097 5135 5136 5182 5217 5253 5271 5357 5543 5551 5565 5577 5589 5602 5612 5643 5692 5722 . OK Search completed (11.078 secs). ---- Now I run the same original three searches, but they return completely different results(!): ---- . UID SEARCH FROM "Nick" * SEARCH 9 66 100 116 179 212 228 240 274 338 347 386 448 494 519 579 603 643 766 781 788 827 1095 1107 1174 1240 1254 1261 1280 1595 1626 1641 1667 1688 1702 1703 1724 2184 2187 2883 2932 3202 3371 3432 3457 3466 3467 3507 3519 3631 3637 3685 3716 3765 3798 3803 3853 3854 3855 3885 3886 3969 4024 4050 4166 4176 4224 4345 4400 4401 4436 4461 4490 4491 4498 4561 4595 4618 4676 4711 4712 4729 4731 4746 4788 4883 5029 5135 5136 5182 5217 5253 5577 5602 5722 . OK Search completed (0.001 secs). . UID SEARCH ALL FROM "Nick" * SEARCH . OK Search completed (0.001 secs). . UID SEARCH SUBJECT "Nick" * SEARCH 9 66 100 116 179 212 228 240 274 338 347 386 448 494 519 579 603 643 766 781 788 827 1095 1107 1174 1240 1254 1261 1280 1595 1626 1641 1667 1688 1702 1703 1724 2184 2187 2883 2932 3202 3371 3432 3457 3466 3467 3507 3519 3631 3637 3685 3716 3765 3798 3803 3853 3854 3855 3885 3886 3969 4024 4050 4166 4176 4224 4345 4400 4401 4436 4461 4490 4491 4498 4561 4595 4618 4676 4711 4712 4729 4731 4746 4788 4883 5029 5135 5136 5182 5217 5253 5577 5602 5722 . OK Search completed (0.001 secs). ---- The "ALL" one returns nothing at all, even though an "AND" search on "ALL" should make no difference, and the "FROM" and "SUBJECT" ones now return identical incorrect results, many of which do not actually match the header requested when examined -- it's returning messages with "Nick" in *any* header. For example, message 9 does not contain "Nick" in the subject, despite the results shown above: ---- . FETCH 9 UID * 9 FETCH (UID 9) . OK Fetch completed. FETCH 9 BODY.PEEK[HEADER.FIELDS (SUBJECT)] * 9 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {47} Subject: Re: Finishing Gallery support page ) . OK Fetch completed. ---- If I delete the "dovecot.index.search" and "dovecot.index.search.uids" files, or if I disable fts in the configuration entirely, the results immediately become correct again. As I mentioned above, I'm wondering if anyone knows the cause of (and fix for) this. Here's my doveconf -n: # 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.4 auth_master_user_separator = * auth_mechanisms = plain login auth_username_format = default_vsz_limit = 1 G disable_plaintext_auth = no lda_mailbox_autosubscribe = yes listen = * log_timestamp = "%Y-%m-%d %H:%M:%S " login_log_format_elements = pid=[%p] user=<%u> method=%m rip=%r lip=%l %c mail_fsync = never mail_location = maildir:~/ mail_max_userip_connections = 100 mail_plugins = fts fts_squat mail_log notify managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave spamtest spamtestplus namespace { inbox = yes location = prefix = INBOX. separator = . type = private } namespace { hidden = yes inbox = no list = no location = prefix = separator = . type = private } passdb { args = /etc/dovecot/dovecot.users driver = passwd-file } passdb { args = /etc/dovecot/master-passwords driver = passwd-file master = yes pass = yes } plugin { fts = squat mail_log_events = copy expunge mail_log_fields = box msgid size from sieve = %h/mailbox.sieve sieve_after = %h/../../domain-after.sieve sieve_before = %h/../../domain-before.sieve sieve_dir = %h/sieve sieve_extensions = -reject +spamtest +spamtestplus sieve_global_dir = %h/../../sieve-global-include-scripts sieve_spamtest_max_value = 7 sieve_spamtest_status_header = X-Spam-Level sieve_spamtest_status_type = strlen } protocols = pop3 imap lmtp sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } service imap-login { process_min_avail = 8 service_count = 0 } service imap { process_limit = 2000 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service pop3-login { process_min_avail = 8 service_count = 0 } service pop3 { process_limit = 2000 } ssl = no userdb { args = /etc/dovecot/dovecot.users driver = passwd-file } verbose_proctitle = yes protocol imap { imap_client_workarounds = delay-newmail imap_logout_format = %i bytes in, %o bytes out } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_logout_format = ; %i bytes in, %o bytes out, %t top (%p bytes), %r retr (%b bytes), %d of %m deleted, mailbox size %s bytes pop3_uidl_format = UID%u-%v } protocol lda { mail_fsync = optimized mail_plugins = sieve postmaster_address = postmaster(a)tigertech.net sendmail_path = /usr/local/bin/dovecot-sendmail-wrapper } protocol lmtp { mail_fsync = optimized mail_plugins = sieve postmaster_address = postmaster(a)tigertech.net sendmail_path = /usr/local/bin/dovecot-sendmail-wrapper } -- Robert L Mathews, Tiger Technologies, http://www.tigertech.net/

1 0

[Dovecot] LDAP: allow pop3, restrict imap
by Andreas Schulze 26 May '14

26 May '14

Hello, I have all userdata in a ldapserver. Every user has the right to use pop3. There is no explicit attribute allowing that. It's simply possible. Now I like to add imap. For a starting period I like to restrict, who may use imap. http://wiki2.dovecot.org/Authentication/RestrictAccess mention a solution where I could modify ldap pass_filter. But that require an attribute "service=pop3" to be present in the ldap servers. In my case I have to add it for every existing user :-/ No way... Are there other options? Thanks, Andreas

3 3

[Dovecot] socket /var/spool/postfix/private/auth not created
by lister171254 26 May '14

26 May '14

I'm on Ubuntu 13.10 and followed these instructions http://wiki2.dovecot.org/HowTo/PostfixAndDovecotSASL to configure Dovecot for Postfix. No matter what I try, the listener is not created. <http://dovecot.2317879.n4.nabble.com/file/n48182/32.png> dovecot -n does not show the listener (I guess that's why its not created) This is the extract from the /etc/dovecot/conf.d/10-master.conf --------------- service auth { # auth_socket_path points to this userdb socket by default. It's typically # used by dovecot-lda, doveadm, possibly imap process, etc. Users that have # full permissions to this socket are able to get a list of all usernames and # get the results of everyone's userdb lookups. # # The default 0666 mode allows anyone to connect to the socket, but the # userdb lookups will succeed only if the userdb returns an "uid" field that # matches the caller process's UID. Also if caller's uid or gid matches the # socket's uid or gid the lookup succeeds. Anything else causes a failure. # # To give the caller full permissions to lookup all users, set the mode to # something else than 0666 and Dovecot lets the kernel enforce the # permissions (e.g. 0777 allows everyone full permissions). unix_listener auth-userdb { #mode = 0666 #user = #group = } # Postfix smtp-auth unix_listener /var/spool/postfix/private/auth { mode = 0600 user = postfix group = postfix } # Auth process is run as this user. #user = $default_internal_user } ---------------- Postfix is not involved in this right now as I expect the socket to be created when I restart dovecot. Can't find any meaningfull errors in the logs either. I hope I'm missing something obvious Thanks, Leo -- View this message in context: http://dovecot.2317879.n4.nabble.com/socket-var-spool-postfix-private-auth-… Sent from the Dovecot mailing list archive at Nabble.com.

6 14

[Dovecot] Copies of outgoing emails in the Sent folder
by Dmitry Podkovyrkin 26 May '14

26 May '14

Hi! My mail server works in tandem: Dovecot (IMAP) + Postfix. How can I configure in Gmail: Outgoing email copied to the Sent folder using postfix without mail client? This question is for the mailing list for Postfix? Mail client itself copies outgoing messages in the folder Sent, but the messages are large, and the server is far away. And to transmit email twice for sending and copying is not effective. Sorry for my English ). -- Regards Dmitry Podkovyrkin mobile: +7 922 20 56 756 email: dmirty(a)rutelecom.company skype: dmitryrw

6 15

[Dovecot] Multilayer mail handling
by Jimmy Thrasibule 26 May '14

26 May '14

Hi, I'm wondering what is the correct architecture and configuration to setup a multilayer mail handling? What I have in mind is something more like what we can found in Web platform architectures. 1. First layer does first checks: user authentication and SSL offloading. If all checks are OK request is proxied to layer 2. 2. Second layer, will actually handle the IMAP/POP3 connection proxied by layer 1, no major checks should be done as already made by layer 1. 3. Finally, layer 3 to do some virus checks on the emails and maybe some other things I can't think about. Do you have some pointers on how to set up such a mail stack where each layer can be on multiple other hosts? Best regards, Jimmy

1 0

[Dovecot] Index cache errors worse with 2.2.x
by Alexander 'Leo' Bergolth 25 May '14

25 May '14

Hi! I am getting "Cached message size smaller than expected" errors since dovecot-2.1.x. Until now, I thought that those errors don't do any harm, I assumed that the cache will just be rebuilt after the error had been detected. However, since dovecot 2.2.x (I tried 2.2.7 and 2.2.10), the errors sometimes seem to actually cause mail access to fail. With dovecot-2.1.1, only the first two messages were logged, since using 2.2.7 and 2.2.10, the "read ... failed: Invalid argument" sometimes appear: -------------------- 8< -------------------- May 7 17:03:52 samba dovecot: imap(xyz): Error: Cached message size smaller than expected (2246 < 2247) May 7 17:03:52 samba dovecot: imap(xyz): Error: Corrupted index cache file /home/xyz/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 11277 May 7 17:03:52 samba dovecot: imap(xyz): Error: read(/var/mail/xyz) failed: Invalid argument May 7 17:03:52 samba dovecot: imap(xyz): Error: read(/var/mail/xyz) failed: Invalid argument (uid=11277) -------------------- 8< -------------------- I am using dovecot on RHEL6 with mbox mailboxes. Message size differences are always 1 byte. Any hints would be appreciated! Cheers, --leo P.S.: Those errors occur some times a day and affect different users.. (Upgrade 2.1.1 -> 2.2.7 was on 2013-11-22, 2.2.10 was installed on 2014-01-20.) # for f in maillog* ; do echo -n "$f: "; grep "dovecot:.*Error: read.*failed.*Invalid arg" $f | wc -l; done maillog: 31 maillog-20130801: 0 maillog-20130901: 1 maillog-20131001: 0 maillog-20131101: 0 maillog-20131201: 30 maillog-20140101: 92 maillog-20140201: 110 maillog-20140301: 99 maillog-20140401: 133 maillog-20140501: 100 # for f in maillog* ; do echo -n "$f: "; grep "dovecot:.*Error: Cached message size smaller than expected" $f | wc -l; done maillog: 99 maillog-20130801: 465 maillog-20130901: 484 maillog-20131001: 512 maillog-20131101: 460 maillog-20131201: 520 maillog-20140101: 402 maillog-20140201: 400 maillog-20140301: 355 maillog-20140401: 481 maillog-20140501: 434 -- e-mail ::: Leo.Bergolth (at) wu.ac.at fax ::: +43-1-31336-906050 location ::: IT-Services | Vienna University of Economics | Austria

4 13

[Dovecot] Conditional jump or move depends on uninitialised value
by Daminto Lie 23 May '14

23 May '14

Hi, My Server runs on Ubuntu Server 12.04 LTS 32 bits. I'm getting the following error messages when I run "make check" during the compilation of dovecot-2.2.13. <snip> ==2058== Conditional jump or move depends on uninitialised value(s) ==2058== at 0x4049DD8: inflateReset2 (in /lib/i386-linux-gnu/libz.so.1.2.3.4) ==2058== by 0x4049EC7: inflateInit2_ (in /lib/i386-linux-gnu/libz.so.1.2.3.4) ==2058== by 0x804AFEF: i_stream_zlib_init (istream-zlib.c:320) ==2058== by 0x804B122: i_stream_create_zlib (istream-zlib.c:475) ==2058== by 0x804AA18: test_compression_handler (test-compression.c:72) ==2058== by 0xEFCDAB88: ??? ==2058== make[2]: *** [check-test] Error 1 make[2]: Leaving directory `/usr/src/dovecot-2.2.13/src/lib-compression' make[1]: *** [check-recursive] Error 1 make[1]: Leaving directory `/usr/src/dovecot-2.2.13/src' make: *** [check-recursive] Error 1 Any help would be greatly appreciated. Thank you

2 1

[Dovecot] dsync incredibly slow
by Alan McGinlay 23 May '14

23 May '14

Hi! While performing a dsync from cyrus imap to dovecot 2.2.12, dsync seems to stop for perhaps a minute without disk / cpu / memory activity (that I can see). This happens several times per sync per user so it takes an enormous amount of time to sync just a couple of gigs of mail. dsync -D -v -o mail_fsync=never mirror -f -R -u user(a)example.com imapc: What could possibly be the cause of this slowness? Is it protocol dependent or just dsync it's self? Thanks!

2 1

[Dovecot] Mail logger plugin, improvement request
by Gilles Chauvin 23 May '14

23 May '14

Hi Timo, Would it be possible for the mail logger plugin to also log the imap MOVE events (RFC6851) ? Thanks. Regards, Gilles.

1 0