January 2012 - dovecot

[Dovecot] Mountpoints
by Timo Sirainen 30 Jan '12

30 Jan '12

I've been thinking about mountpoints recently. There have been a few problems related to them: - If dbox mails and indexes are in different filesystems, and index fs isn't mounted and mailbox is accessed -> Dovecot rebuilds indexes from scratch, which changes UIDVALIDITY, which causes client to redownload mails. All mails will also show up as unread. Once index fs gets mounted again, the UIDVALIDITY changes again and client again redownloads mails. What should happen instead is that Dovecot simply refuses to rebuild indexes when the index fs isn't mounted. This isn't as critical for mbox/maildir, but probably a good idea to do there as well. - If dbox's alternative storage isn't mounted and a mail from there is tried to be accessed -> Dovecot rebuilds indexes and sees that all mails in alt path are gone, so Dovecot also deletes them from indexes as well. Once alt fs is mounted again, the mails in there won't come back without manual index rebuild and then they have also lost flags and have updated UIDs causing clients to redownload them. So again what should happen is that Dovecot won't rebuild indexes while alt fs isn't mounted. - For dsync-based replication I need to keep a state of each mountpoint (online, offline, failover) to determine how to access user's mails. So in the first two cases the main problem is: How does Dovecot know where a mountpoint begins? If the mountpoint is actually mounted there is no problem, because there are functions to find it (e.g. from /etc/mtab). So how to find a mountpoint that should exist, but doesn't? In some OSes Dovecot could maybe read and parse /etc/fstab, but that doesn't exist in all OSes, and do all installations even have all of the filesystems listed there anyway? (They could be in some startup script.) So, I was thinking about adding doveadm commands to explicitly tell Dovecot about the mountpoints that it needs to care about. When no mountpoints are defined Dovecot would behave as it does now. doveadm mount add|remove <path> - add/remove mountpoint doveadm mount state [<path> [<state>]] - get/set state of mountpoint (used by replication) - if path isn't given list states of all mountpoints List of mountpoints is kept in /var/lib/dovecot/mounts. But because the dovecot directory is only accessible to root (and probably too much trouble to change that), there's another list in /var/run/dovecot/mounts. This one also contains the states of the mounts. When Dovecot starts up and can't find the mounts from rundir, it creates it from vardir's mounts. When mail processes notice that a directory is missing, it usually autocreates it. With mountpoints enabled, Dovecot first finds the root mountpoint for the directory. The mount root is stat()ed and its parent is stat()ed. If their device numbers equal, the filesystem is unmounted currently, and Dovecot fails instead of creating a new directory. Similar logic is used to avoid doing a dbox rebuild if its alt dir is currently in unmounted filesystem. The main problem I see with all this is how to make sysadmins remember to use these commands when they add/remove mountpoints?.. Perhaps the additions could be automatic at startup. Whenever Dovecot sees a new mountpoint, it's added. If an old mountpoint doesn't exist at startup a warning is logged about it. Of course many of the mountpoints aren't intended for mail storage. They could be hidden from the "mount state" list by setting their state to "ignore". Dovecot could also skip some of the common known mountpoints, such as where type is proc/tmpfs/sysfs. Thoughts?

1 1

[Dovecot] A namespace error on 2.1rc5
by Bill Ryder 30 Jan '12

30 Jan '12

Hello all, I'm not sure if this is a bug. It's probably just an upgrade note. In summary I had no namespace section in my 2.0.17 config. When trying out 2.1rc5 no user could login because of a namespace error. 2.1rc5 adds a default namespace clause which broke my logins (It was noted in the changelog) I seemed to fix it by just putting this in the config file: namespace inbox { inbox = yes } Long story: I've been recently testing dovecot against cyrus to decide where we should go for our next mail server(s) I loaded up the mail server with mail delivered via postfix all on dovecot 2.0.15 (I've since moved to 2.0.17) I have three dovecot directors, two backends on the same NFS mail store. With dovecot 2.0.xx the tester works fine (it't just a script which logins in and emulates thunderbird when a user is idle - without using IDLE so the client asks for mail every few minutes). When I moved to 2.1rc5 I got namespace errors and the user can not login. The server said: dovecot-error.log-20120128:Jan 27 13:37:59 imap(ethab01): Error: user ethab01: Initialization failed: namespace configuration error: inbox=yes namespace missing The client says: * BYE Internal error occurred. Refer to server log for more information. The session looks like 0.000000 192.168.121.37 -> 192.168.121.2 TCP 33213 > imap [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=1056649457 TSER=0 WS=5 0.000036 192.168.121.2 -> 192.168.121.37 TCP imap > 33213 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=3264407631 TSER=1056649457 WS=7 0.000187 192.168.121.37 -> 192.168.121.2 TCP 33213 > imap [ACK] Seq=1 Ack=1 Win=5856 Len=0 TSV=1056649458 TSER=3264407631 0.006338 192.168.121.2 -> 192.168.121.37 IMAP Response: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. 0.006889 192.168.121.37 -> 192.168.121.2 TCP 33213 > imap [ACK] Seq=1 Ack=124 Win=5856 Len=0 TSV=1056649465 TSER=3264407637 0.006973 192.168.121.37 -> 192.168.121.2 IMAP Request: I ID ("x-originating-ip" "192.168.114.249" "x-originating-port" "49403" "x-connected-ip" "192.168.121.37" "x-connected-port" "143") 0.006980 192.168.121.2 -> 192.168.121.37 TCP imap > 33213 [ACK] Seq=124 Ack=178 Win=6912 Len=0 TSV=3264407638 TSER=1056649465 0.007086 192.168.121.2 -> 192.168.121.37 IMAP Response: * ID NIL 0.018471 192.168.121.2 -> 192.168.121.37 IMAP Response: * BYE Internal error occurred. Refer to server log for more information. (interestingly the tshark output strips out the user name and password which is convenient but which may mean there's not enough information?) I rolled back to 2.0.17 and it was fine again. It's the same config files for both, same maildirs etc etc. All I did was change the dovecot version from 2.0.17 to 2.1rc5 However I see from the changelog that 2.1rc5 added a default namespace inbox: diff doveconf-n.2.0.17 doveconf-n.2.1-rc5 1c1 < # 2.0.17 (684381041dc4+): /etc/dovecot/dovecot.conf --- > # 2.1.rc5: /etc/dovecot/dovecot.conf 20a21,39 > namespace inbox { > location = > mailbox Drafts { > special_use = \Drafts > } > mailbox Junk { > special_use = \Junk > } > mailbox Sent { > special_use = \Sent > } > mailbox "Sent Messages" { > special_use = \Sent > } > mailbox Trash { > special_use = \Trash > } > prefix = > } We had this section commented out in 2.0.x so there was no namespace inbox anywhere. ============== doveconf -n for 2.0.17 (for the backends) # 2.0.17 (684381041dc4+): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-131.6.1.el6.x86_64 x86_64 Scientific Linux release 6.1 (Carbon) nfs auth_mechanisms = plain login auth_username_format = %n auth_verbose = yes debug_log_path = /var/log/dovecot/dovecot-debug.log disable_plaintext_auth = no first_valid_uid = 200 info_log_path = /var/log/dovecot/dovecot-info.log log_path = /var/log/dovecot/dovecot-error.log mail_debug = yes mail_fsync = always mail_gid = vmail mail_location = maildir:/vol/dt_mailstore1/spool/%n:INDEX=/var/indexes/%n mail_nfs_storage = yes mail_plugins = " fts fts_solr mail_log notify quota" mail_uid = vmail maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { driver = pam } plugin { autocreate = Trash autocreate2 = Drafts autocreate3 = Sent autocreate4 = Templates autosubscribe = Trash autosubscribe2 = Drafts autosubscribe3 = Sent autosubscribe4 = Templates fts = solr fts_solr = break-imap-search debug url=http://dovecot-solr1.wetafx.co.nz:8080/solr/ mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = imap pop3 lmtp sieve service auth { unix_listener auth-userdb { group = vmail user = vmail } } service lmtp { inet_listener lmtp { address = 192.168.121.2 127.0.0.1 port = 24 } process_min_avail = 20 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } } ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { args = uid=vmail gid=vmail home=/vol/dt_mailstore1/%n/%u driver = passwd } protocol lmtp { mail_plugins = sieve } protocol lda { mail_plugins = " fts fts_solr mail_log notify quota sieve" } protocol imap { mail_plugins = " fts fts_solr mail_log notify quota imap_quota" } protocol sieve { managesieve_notify_capability = managesieve_sieve_capability = } ===== doveconf for 2.1rc5 # 2.1.rc5: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-131.6.1.el6.x86_64 x86_64 Scientific Linux release 6.1 (Carbon) nfs auth_mechanisms = plain login auth_username_format = %n auth_verbose = yes debug_log_path = /var/log/dovecot/dovecot-debug.log disable_plaintext_auth = no first_valid_uid = 200 info_log_path = /var/log/dovecot/dovecot-info.log log_path = /var/log/dovecot/dovecot-error.log mail_debug = yes mail_fsync = always mail_gid = vmail mail_location = maildir:/vol/dt_mailstore1/spool/%n:INDEX=/var/indexes/%n mail_nfs_storage = yes mail_plugins = " fts fts_solr mail_log notify quota" mail_uid = vmail maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { autocreate = Trash autocreate2 = Drafts autocreate3 = Sent autocreate4 = Templates autosubscribe = Trash autosubscribe2 = Drafts autosubscribe3 = Sent autosubscribe4 = Templates fts = solr fts_solr = break-imap-search debug url=http://dovecot-solr1.wetafx.co.nz:8080/solr/ mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = imap pop3 lmtp sieve service auth { unix_listener auth-userdb { group = vmail user = vmail } } service lmtp { inet_listener lmtp { address = 192.168.121.2 127.0.0.1 port = 24 } process_min_avail = 20 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } } ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { args = uid=vmail gid=vmail home=/vol/dt_mailstore1/%n/%u driver = passwd } protocol lmtp { mail_plugins = sieve } protocol lda { mail_plugins = " fts fts_solr mail_log notify quota sieve" } protocol imap { mail_plugins = " fts fts_solr mail_log notify quota imap_quota" } protocol sieve { managesieve_notify_capability = managesieve_sieve_capability = }

1 0

[Dovecot] 2.1.rc3 (1a722c7676bb): Panic: file ostream.c: line 173 (o_stream_sendv): assertion failed: (stream->stream_errno != 0)
by Pascal Volk 30 Jan '12

30 Jan '12

Hi Timo, today some imap processes are crashed. Regards, Pascal -- The trapper recommends today: f007ba11.1201322(a)localdomain.org

1 1

[Dovecot] imap-login process_limit reached
by Don Buchholz 29 Jan '12

29 Jan '12

I've been having some problems with IMAP user connections to the Dovecot (v2.0.8) server. The following message is being logged. Jan 16 10:51:36 postal dovecot: master: Warning: service(imap-login): process_limit reached, client connections are being dropped The server is running Red Hat Enterprise Linux release 4 (update 6). Dovecot is v2.0.8. We have only 29 user accounts in /etc/dovecot/users. There were 196 "dovecot/imap" processes and 6 other dovecot processes, for a total of 202 "dovecot" processes, listed in the 'ps aux' output when problems were being experienced. Stopping and restarting the Dovecot system fixes the problem -- for a while. The 'doveconf -n' output is attached. I have not set any "process_limit" values, and I don't think I'm getting anywhere close to the 1024 default, so I'm pretty confused as to what might be wrong. Any suggestions on what to do next are appreciated. Thanks, - Don ------------------------------------------------------------------------ # 2.0.8: /etc/dovecot/dovecot.conf # OS: Linux 2.6.9-67.0.1.ELsmp i686 Red Hat Enterprise Linux WS release 4 (Nahant Update 6) ext3 auth_username_format = %Lu auth_verbose = yes auth_verbose_passwords = sha1 login_log_format_elements = user=<%u> service=%Us method=%m rip=%r lip=%l mpid=%e %c login_trusted_networks = xx.xx.xx.0/24 mail_location = maildir:/Virtual_Users/%u/Mail passdb { args = scheme=CRYPT username_format=%u /etc/dovecot/users driver = passwd-file } passdb { driver = pam } ssl_ca = </etc/dovecot/ssl/iMove_2011_CAcert.pem ssl_cert = </etc/dovecot/ssl/postal_cert.pem ssl_key = </etc/dovecot/ssl/postal_key+req.pem ssl_key_password = ********** ssl_parameters_regenerate = 12 userdb { args = username_format=%u /etc/dovecot/users driver = passwd-file } userdb { driver = passwd } verbose_proctitle = yes protocol imap { ssl_cert = </etc/dovecot/ssl/imap_cert.pem ssl_key = </etc/dovecot/ssl/imap_key+req.pem ssl_key_password = </etc/dovecot/ssl/imap.pw } protocol pop3 { ssl_cert = </etc/dovecot/ssl/pop3_cert.pem ssl_key = </etc/dovecot/ssl/pop3_key+req.pem ssl_key_password = </etc/dovecot/ssl/pop3.pw }

5 16

[Dovecot] XFS Developer Takes Shots At Btrfs, EXT4
by Adrian Minta 29 Jan '12

29 Jan '12

Nice article about XFS improvements: http://tinyurl.com/7pvr9ju

2 2

[Dovecot] 2.1.rc1 (056934abd2ef): virtual plugin mailbox search pattern
by e-frog 29 Jan '12

29 Jan '12

Hello Timo, With dovecot 2.1.rc1 (056934abd2ef) there seems to be something wrong with virtual plugin mailbox search patterns. I'm using a virtual mailbox 'unread' with the following dovecot-virtual file $ cat dovecot-virtual * unseen For testing propose I created the following folders with each containing one unread message INBOX, INBOX/level1 and INBOX/level1/level2 2.1.rc1 (056934abd2ef) 1 LIST "" "*" * LIST (\HasChildren) "/" "INBOX" * LIST (\HasChildren) "/" "INBOX/level1" * LIST (\HasNoChildren) "/" "INBOX/level1/level2" * LIST (\HasChildren) "/" "virtual" * LIST (\HasNoChildren) "/" "virtual/unread" 1 OK List completed. 2 STATUS "INBOX" (UNSEEN) * STATUS "INBOX" (UNSEEN 1) 2 OK Status completed. 3 STATUS "INBOX/level1" (UNSEEN) * STATUS "INBOX/level1" (UNSEEN 1) 3 OK Status completed. 4 STATUS "INBOX/level1/level2" (UNSEEN) * STATUS "INBOX/level1/level2" (UNSEEN 1) 4 OK Status completed. 5 STATUS "virtual/unread" (UNSEEN) * STATUS "virtual/unread" (UNSEEN 1) 5 OK Status completed. Result: virtual/unread shows only 1 unseen message. Further tests showed it's the one from INBOX. The mails from the deeper levels are not found. Downgrading to 2.0.16 restores the correct behavior: 1 LIST "" "*" * LIST (\HasChildren) "/" "INBOX" * LIST (\HasChildren) "/" "INBOX/level1" * LIST (\HasNoChildren) "/" "INBOX/level1/level2" * LIST (\HasChildren) "/" "virtual" * LIST (\HasNoChildren) "/" "virtual/unread" 1 OK List completed. 2 STATUS "INBOX" (UNSEEN) * STATUS "INBOX" (UNSEEN 1) 2 OK Status completed. 3 STATUS "INBOX/level1" (UNSEEN) * STATUS "INBOX/level1" (UNSEEN 1) 3 OK Status completed. 4 STATUS "INBOX/level1/level2" (UNSEEN) * STATUS "INBOX/level1/level2" (UNSEEN 1) 4 OK Status completed. 5 STATUS "virtual/unread" (UNSEEN) * STATUS "virtual/unread" (UNSEEN 3) 5 OK Status completed. Result: virtual/unread shows 3 unseen messages as it should The namespace configuration is as following namespace { hidden = no inbox = yes list = yes location = prefix = separator = / subscriptions = yes type = private } namespace { location = virtual:~/virtual prefix = virtual/ separator = / subscriptions = no type = private } I've also tried this with location = virtual:~/virtual:LAYOUT=maildir++ leading to the same result. Thanks, e-frog

2 3

[Dovecot] Migration to multi-dbox and SiS
by Ronald MacDonald 29 Jan '12

29 Jan '12

Dear list, A huge thank-you first of all for all the work that's gone into Dovecot itself. I'm rebuilding a mail server next week and so, taking the rare opportunity to re-consider all the options I've had running over the past couple of years. Around the time of the last re-build (2010), there had been some discussion on single instance storage, which was quite new on Dovecot around then. I chickened out of setting it up though. Now with it having been in the wild for a couple of years, I wonder, how have people found SiS to behave? Additionally, though there was talk of the prospect of it being merged with 2.x am I right in thinking it's not yet in the main project? Couldn't find any 2.x changelogs that could confirm this. With best wishes, Ronald.

2 1

[Dovecot] Attribute Cache flush errors on FreeBSD 8.2
by Phil Kernick 29 Jan '12

29 Jan '12

I'm running dovecot 2.0.16 on FreeBSD 8.2 with the mail spool and indexes on an NFS server. Lines like the following keep appearing in syslog for access to each mailbox: Error: nfs_flush_attr_cache_fd_locked: fchown(/home/philk/Mail/Deleted) failed: Bad file descriptor This is coming from nfs-workarounds.c line 210, which tracing back seems to be coming from the call to mbox_lock on lib-storage/index/mbox/mbox-lock.c line 774. I have /home mounted with options acregmin=0,acregmax=0,acdirmin=0,acdirmax=0 (as FreeBSD doesn't have a noac option), but it throws the same error either way. The output of dovecot -n is below. Phil. # 2.0.16: /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 8.2-RELEASE-p3 i386 auth_mechanisms = plain login auth_username_format = %Lu disable_plaintext_auth = no first_valid_gid = 1000 first_valid_uid = 1000 listen = *, [::] mail_fsync = always mail_location = mbox:~/Mail/:INBOX=/var/mail/%u mail_nfs_index = yes mail_nfs_storage = yes mail_privileged_group = mail mmap_disable = yes passdb { args = session=yes dovecot driver = pam } protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = root } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { args = blocking=yes driver = passwd } verbose_proctitle = yes protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv } protocol lda { postmaster_address = postmaster(a)example.com sendmail_path = /usr/sbin/sendmail }

2 1

[Dovecot] 2.0.17: Index lost -> SAVEDON lost as well?
by Ralf Hildebrandt 29 Jan '12

29 Jan '12

Today I encoundered these errors: Jan 9 08:30:06 mail dovecot: lmtp(31174, backup(a)backup.invalid) Error: Log synchronization error at seq=858,offset=44672 for /home/mailboxname/mdbox/storage/dovecot.map.index: Append with UID 282388, but next_uid = 282389 Jan 9 08:30:06 mail dovecot: lmtp(31819, backup(a)backup.invalid) Error: Log synchronization error at seq=858,offset=44672 for /home/mailboxname/mdbox/storage/dovecot.map.index: Append with UID 282388, but next_uid = 282389 Jan 9 08:30:06 mail dovecot: lmtp(32148, backup(a)backup.invalid) Error: Log synchronization error at seq=858,offset=44672 for /home/mailboxname/mdbox/storage/dovecot.map.index: Append with UID 282388, but next_uid = 282389 After that, the SAVEDON date for all mails was reset to today: mail:~# doveadm search -u backup(a)backup.invalid mailbox INBOX SAVEDON 2012-01-09 | wc -l 75650 mail:~# doveadm search -u backup(a)backup.invalid mailbox INBOX SAVEDON 2012-01-08 | wc -l 0 mail:~# doveadm search -u backup(a)backup.invalid mailbox INBOX SAVEDON 2012-01-07 | wc -l 0 Before, I was running this: vorgestern=`date -d "-2 day" +"%Y-%m-%d"` doveadm expunge -u backup(a)backup.invalid mailbox INBOX SAVEDBEFORE $vorgestern doveadm purge -u backup(a)backup.invalid Is there a way of restoring the SAVEDON info? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

2 4

[Dovecot] compressed mboxes very slow
by kjonca＠o2.pl 29 Jan '12

29 Jan '12

I have some archive mails in gzipped mboxes. I could use them with dovecot 1.x without problems. But recently I have installed dovecot 2.0.12, and they are slow. very slow. Creating index files takes about 10 minutes for ~20M file with 560 messages for bzipped mbox, for gzipped is little better but still unusable :( Stracing dovecot process shows that every ~ 20 messages it rereads complete mbox file. Am I doing something wrong? KJ -- http://modnebzdury.wordpress.com/2009/10/01/niewiarygodny-list-prof-majewsk…

4 13