June 2012 - dovecot - dovecot.org

[Dovecot] dovecot stats: useful data to gather
by Patrick Ben Koetter 06 Sep '12

06 Sep '12

Timo, following our discussion on dovecot stats at the LinuxTag 2012 my team and I sat down and put together a list of stat items we think to be useful in daily dovecot usage. Besides pulling together all the data we also think it would be useful to have an SNMP interface to access the stats. Our offer to create and contribute a standalone web interface for dovecot stats stands. Here are the stats we believe to be useful: Login/Logout - total number login success/time - total number login failure/time - total number per authentication mechanism - total number plain sessions - total number STARTTLS sessions - total number of currently connected users (pop3/pop3s/imap/imaps/managesieve) - login names of connected users (not really stats, but great for actions regarding those uses e.g. force logout) - total number logout commands/time - total number BYE responses (autologout) Mailbox state - Inflow rate (number incoming messages/time) - Deleted rate (number \Deleted flagged messages/time) - Expunge rate (number Expunge operations/time) - total number current messages mailboxes normal storage - total number current messages mailboxes alt storage - total number read messages mailboxes normal storage - total number read messages mailboxes alt storage - per user number current messages mailboxes normal storage - per user number current messages mailboxes alt storage - per user number read messages mailboxes normal storage - per user number read messages mailboxes alt storage Mailbox Quota - total number persons under soft-quota per quota - total number persons above or equal soft-quota per quota - total number persons above or equal hard-quota per quota Performance - minimum time to write a message - maximum time to write a message - average time to write a message - minimum time to modify a message - maximum time to modify a message - average time to modify a message - minimum time to delete a message - maximum time to delete a message - average time to delete a message - minimum time search operations - maximum time search operations - average time search operations Regards, p@rick -- state of mind () http://www.state-of-mind.de Franziskanerstraße 15 Telefon +49 89 3090 4664 81669 München Telefax +49 89 3090 4666 Amtsgericht München Partnerschaftsregister PR 563

5 5

[Dovecot] dovecot stats error
by Robert Schetterer 25 Aug '12

25 Aug '12

Hi Timo, any idea whats this related too ? dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mrbytes 21703727 < 25193928 -- Best Regards MfG Robert Schetterer

3 3

[Dovecot] dsync replication available for testing
by Timo Sirainen 25 Aug '12

25 Aug '12

In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: - public namespace isn't replicated at all - shared namespace is replicated, but not private mail flags - I've only tested SSH replication setup now, not director replication setup (and director setup is still missing many things) - SSH replication setup uses aggregator process, which isn't really necessary and can probably be avoided in future Below is a configuration for virtual user setup. System user configuration works pretty much the same, except doveadm/ssh is run as root. Try first that dsync works successfully with ssh in host1: doveadm sync -u user@domain remote:vmail@host2.example.com and also in host2: doveadm sync -u user@domain remote:vmail@host1.example.com ------ mail_plugins = $mail_plugins notify replication service aggregator { # give enough permissions for mail processes fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup process_min_avail = 1 } plugin { # host1 replicates to host2 mail_replica = remote:vmail@host2.example.com # host2 replicates to host1 #mail_replica = remote:vmail@host1.example.com } #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service doveadm { # if you're using a single virtual user, set this to # start ssh as vmail (not root) user = vmail }

4 20

Re: [Dovecot] problem with virtual plugin/index files?
by Lutz Preßler 24 Aug '12

24 Aug '12

Hello, Timo: on Oct 29th I wrote ---- with latest 2.0.6 (601065674f74), the following virtual mailbox is not behaving correctly. I'm quite sure that the problem has not existed in some earlier 2.0.x (maybe only INBOX as mailbox then though): ~user/Maildir/virtual/.week/dovecot-virtual contains INBOX INBOX.* -INBOX.Trash inthread refs younger 604800 This works as expected for the first time. But later on, also older messages/ threads are included until I delete the virtual/.week/dovecot.index*. ---- Just want to state that this problem still persists with latest version. Any idea? Thanks and greetings, Lutz -- Lutz Preßler <Lutz.Pressler(a)SerNet.DE> http://www.SerNet.DE/ SerNet Service Network GmbH, Bahnhofsallee 1b, D-37081 Göttingen Tel.: +49-551-370000-2, FAX: +49-551-370000-9 AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen

2 4

[Dovecot] namespace from snapshots
by Karl Oulmi 21 Aug '12

21 Aug '12

hi all, I'm trying to give access to snapshots taken from by dell iscsi MD3200i to my maildirs users. snapshot are mounted in read only mode from my FreeBSD box. In my /usr/local/etc/dovecot/conf.d/10-mail.conf, I have : namespace inbox { inbox = yes } namespace da1 { prefix = INBOX.backup.da1. hidden = no list = yes inbox = no location = maildir:/da1/%u/Maildir type = private } I have two problems : - In my log, dovecot complain about the fact that my snapshot filsystem is read only Error: open(/da1/test2(a)toto.com/Maildir/.Trash/dovecot-uidlist) failed: Read-only file system The second problem is that through my thunderbird I can't see the Inbox saved in the snapshot (whereas I see Sent and Trash folders) If anyone could help me, It would be nice. Regard, Karl.

3 7

[Dovecot] Statistics scripts
by Timo Sirainen 08 Aug '12

08 Aug '12

I wrote two scripts, which you can use with the stats plugin enabled to get a better "doveadm stats top" than it currently is: http://dovecot.org/tools/stats-top.pl http://dovecot.org/tools/stats.pl

2 1

[Dovecot] mysql auth failover failing
by Paul B. Henson 29 Jul '12

29 Jul '12

We are running dovecot to provide authentication for postfix, using two mysql servers in a multi-master replication set as the password source: ---------------------------------------- # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.37-gentoo-r4 x86_64 Gentoo Base System release 2.0.2 auth_mechanisms = plain login digest-md5 cram-md5 auth_verbose = yes passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocols = none service auth-worker { unix_listener auth-worker { user = postfix } user = $default_internal_user } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = postfix } ssl = no userdb { driver = passwd } --------------------------------------- With an sql config of: ------------------------- driver = mysql connect = host=mysql-1.unx.csupomona.edu host=mysql-2.unx.csupomona.edu dbname=idmgmt user=postfix password=XXXXXXX default_pass_scheme = PLAIN password_query = XXXXXXXXX ------------------------- According to the sample SQL configuration file "HA / round-robin load-balancing is supported by giving multiple host settings, like: host=sql1.host.org host=sql2.host.org". However, as far as I can tell dovecot only connects to the first listed host, and processes all queries through it, there does not appear to be any load-balancing going on. That's not necessarily a dealbreaker; however, high-availability does not appear to be working either. If I shutdown the first mysql server, dovecot starts to log connection failures: Sep 9 15:47:34 tweak dovecot: auth: Error: mysql(mysql-1.unx.csupomona.edu): Connect failed to database (idmgmt): Can't connect to MySQL server on 'mysql-1.unx.csupomona.edu' (111) - waiting for 1 seconds before retry Sep 9 15:47:39 tweak dovecot: auth: Error: mysql(mysql-1.unx.csupomona.edu): Connect failed to database (idmgmt): Can't connect to MySQL server on 'mysql-1.unx.csupomona.edu' (111) - waiting for 25 seconds before retry And postfix starts to fail authentications: Sep 9 15:47:35 tweak postfix/smtpd[5119]: warning: bender.iitsys.csupomona.edu[134.71.250.134]: SASL DIGEST-MD5 authentication failed: Connection lost to authentication server Now and again the authentication process dies: Sep 9 15:47:39 tweak dovecot: auth: Panic: file auth-request-handler.c: line 697 (auth_request_handler_flush_failures): assertion failed: (auth_request->state == AUTH_REQUEST_STATE_FINISHED) Sep 9 15:47:39 tweak dovecot: auth: Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x3f71a) [0x7f25822ca71a] -> /usr/lib64/dovecot/libdovecot.so.0(+0x3f766) [0x7f25822ca766] -> /usr/lib64/dovecot/libdovecot.so.0(+0x198ca) [0x7f25822a48ca] -> dovecot/auth() [0x4137f4] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xd4) [0x7f25822d5fe4] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5b) [0x7f25822d6bcb] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f25822d5c48] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f25822c3de3] -> dovecot/auth(main+0x2be) [0x4179de] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f2581898bbd] -> dovecot/auth() [0x40bdc9] Sep 9 15:47:39 tweak dovecot: master: Error: service(auth): child 4154 killed with signal 6 (core dumps disabled) Requests start to pile up: Sep 9 15:51:46 tweak dovecot: auth: Warning: auth workers: Auth request was queued for 25 seconds, 45 left in queue Lookups time out: Sep 9 15:57:22 tweak dovecot: auth: Error: auth worker: Aborted request: Lookup timed out This occasionally pops up: Sep 9 15:58:38 tweak dovecot: auth: Fatal: net_connect_unix(auth-worker) failed: Resource temporarily unavailable And sometimes the auth process gets temporarily disabled: Sep 9 15:58:57 tweak dovecot: master: Error: service(auth): command startup failed, throttling Resulting in more postfix authentication failures: Sep 9 15:58:57 tweak postfix/smtpd[6531]: warning: bender.iitsys.csupomona.edu[134.71.250.134]: SASL DIGEST-MD5 authentication failed: Sep 9 15:59:08 tweak postfix/smtpd[6551]: fatal: no SASL authentication mechanisms To the point where postfix also temporarily throttles smtpd: Sep 9 15:59:21 tweak postfix/master[6526]: warning: /usr/lib64/postfix/smtpd: bad command startup -- throttling Resulting in a complete unavailability of smtp service, not just unavailability of authenticated services. I don't think all authentications fail during the scenario, but I think the majority do. Based on the network traffic, dovecot is almost continuously trying to connect to the first listed server. It sometimes connects to the second listed server, but when it does, the connection does not persist, it goes away almost immediately. Ideally, I would like no authentications to fail if one of the MySQL servers is unavailable. If a few fail just when the server dies, that would be undesirable but acceptable as long as they do not continuously fail while the server is down. Am I doing something wrong? Does the example sql config have incorrect information? We were previously running dovecot 1.2.11, we just recently upgraded to 2. In the previous version, we actually had two different passdb's configured, each one listing only one of the mysql servers. I seem to recall that was the recommendation at the time for high-availability. When that configuration did not seem to work under version 2, I found an updated recommendation to list both servers in the same passdb, which also does not appear to work correctly. I actually went back and tested the older version, and determined it seemed to work okay in the case where the server was up but the service was down, and connections were refused, but also failed a large number of authentication attempts when the server was completely down and connections were timing out. Thanks much... -- Paul B. Henson | (909) 979-6361 | http://www.csupomona.edu/~henson/ Operating Systems and Network Analyst | henson(a)csupomona.edu California State Polytechnic University | Pomona CA 91768

5 19

[Dovecot] SQLite dovecot query caching
by Dominic Malolepszy 19 Jul '12

19 Jul '12

Hi, I am wondering if Dovecot caches SQLite queries, and how well it works in high performance setups. I am particularly interested because in the below thread SQLite has been suggested as a means of Dovecot proxying connections to different ports. http://old.nabble.com/director%3A-non-standart-ports-at-backends-td33991991… Cheers, Dominic.

2 7

[Dovecot] director: non standart ports at backends
by Костырев Александр Алексеевич 16 Jul '12

16 Jul '12

hello, I'm trying to figure out how to proxy pop3 and pop3s that listens on non-standart ports at backends. For example, pop3 is at 1110 and pop3s at 1995 (on backend side). is it possible? how should I separate this ports in director's config? it's easy for one port: for example lmtp - you just use passdb in protocol lmtp {}

3 5

[Dovecot] dict Panic after upgrade to 2.1.7
by ckubu 03 Jul '12

03 Jul '12

Hello, after upgrade my mailsystem to dovecot version 2.1.7, dovecot doesn't work properly. something went wrong in dict service connecting the postgres backend. that happens not on every connection. the db connection data are correct, no difference connecting via tcp or linux socket. dovecot log entries: Jun 23 23:19:10 mx dovecot: dict: Panic: file driver-pgsql.c: line 84 (driver_pgsql_set_state): assertion failed: (state == SQL_DB_STATE_BUSY || db- >cur_result == NULL) Jun 23 23:19:10 mx dovecot: dict: Error: Raw backtrace: /usr/local/dovecot-2.1.7/lib/dovecot/libdovecot.so.0(+0x4478a) [0x7ffc7d8e578a] -> /usr/local/dovecot-2.1.7/lib/dovecot/libdovecot.so.0(+0x447d6) [0x7ffc7d8e57d6] -> /usr/local/dovecot-2.1.7/lib/dovecot/libdovecot.so.0(i_error+0) [0x7ffc7d8bc5ef] -> dovecot/dict() [0x40a9a6] -> dovecot/dict() [0x40aa01] -> dovecot/dict() [0x40be43] -> dovecot/dict() [0x409474] -> dovecot/dict(sql_db_cache_deinit+0x20) [0x4089d0] -> dovecot/dict(main+0x169) [0x4059f9] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7ffc7d335c8d] -> dovecot/dict() [0x404b59] Jun 23 23:19:10 mx dovecot: dict: Fatal: master: service(dict): child 13812 killed with signal 6 (core dumps disabled) Jun 23 23:23:09 mx dovecot: dict: Error: dict sql iterate failed: Not connected to database Jun 23 23:23:09 mx dovecot: pop3(xxx(a)yyy.zz): Error: acl: dict iteration failed, can't update dict Jun 23 23:23:09 mx dovecot: dict: Error: dict sql iterate failed: Not connected to database Jun 23 23:23:09 mx dovecot: pop3(xxx(a)yyy.zz): Error: acl: dict iteration failed, can't update dict Jun 23 23:23:17 mx dovecot: dict: Error: dict sql lookup failed: Not connected to database Jun 23 23:23:17 mx dovecot: imap(xxx(a)yyy.zz): Error: Internal quota calculation error Jun 23 23:23:19 mx dovecot: dict: Error: dict sql lookup failed: Not connected to database Jun 23 23:23:40 mx dovecot: dict: Error: dict sql lookup failed: Not connected to database maybe i have missconfigured the dovecot system, but i don't find the mistake. can anybody give me a hint ? best wiches christoph ----- doveconf -n # 2.1.7: /usr/local/dovecot-2.1.7/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-vserver-amd64 x86_64 Debian 6.0.5 ext3 auth_mechanisms = plain login digest-md5 cram-md5 apop auth_socket_path = /var/run/dovecot/auth-userdb auth_username_translation = %@ auth_verbose = yes auth_verbose_passwords = plain base_dir = /var/run/dovecot/ dict { acl = pgsql:/usr/local/dovecot/etc/dovecot/sql-dict.conf.ext expire = pgsql:/usr/local/dovecot/etc/dovecot/sql-dict.conf.ext quota = pgsql:/usr/local/dovecot/etc/dovecot/sql-dict.conf.ext } disable_plaintext_auth = no first_valid_gid = 5000 first_valid_uid = 5000 hostname = mx.warenform.de last_valid_gid = 5000 last_valid_uid = 5000 listen = 178.63.63.151 2a01:4f8:121:c5::2 mail_gid = vmail mail_location = maildir:/var/vmail/%d/%n/Maildir mail_plugins = autocreate quota expire acl mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { list = children location = maildir:/var/vmail/%%d/%%n/Maildir:INDEX=~/Maildir/shared/%%u prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { special_use = \Junk } mailbox Trash { special_use = \Trash } prefix = separator = / type = private } passdb { args = /usr/local/dovecot/etc/dovecot/sql-connect.conf.ext driver = sql } plugin { acl = vfile acl_shared_dict = proxy::acl autocreate = Spam autocreate2 = Sent autocreate3 = Trash autocreate4 = Drafts autosubscribe = Spam autosubscribe2 = Sent autosubscribe3 = Trash autosubscribe4 = Drafts expire = Trash expire2 = Trash.* expire3 = Spam expire_dict = proxy::expire quota = dict:User quota::proxy::quota quota_rule = *:storage=1G quota_rule2 = Trash:storage=+200M quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u recipient_delimiter = sieve = ~/.dovecot.sieve sieve_before = /usr/local/dovecot/etc/dovecot/sieve/move-spam.sieve sieve_dir = ~/sieve sieve_global_dir = /usr/local/dovecot/etc/dovecot/sieve/global/ } postmaster_address = admin(a)warenform.de protocols = imap pop3 sieve lmtp service auth { unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { inet_listener imap { address = 127.0.0.1 178.63.63.151 2a01:4f8:121:c5::2 } inet_listener imaps { address = 178.63.63.151 2a01:4f8:121:c5::2 } process_min_avail = 16 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service managesieve-login { inet_listener sieve { address = 127.0.0.1 port = 4190 } } service pop3-login { inet_listener pop3 { address = 178.63.63.151 2a01:4f8:121:c5::2 } inet_listener pop3s { address = 178.63.63.151 2a01:4f8:121:c5::2 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } shutdown_clients = no ssl_cert = </etc/postfix/ssl/servercert.pem ssl_key = </etc/postfix/ssl/servercert.pem syslog_facility = local1 userdb { args = /usr/local/dovecot/etc/dovecot/sql-connect.conf.ext driver = sql } protocol lmtp { mail_plugins = autocreate quota expire acl sieve } protocol lda { mail_plugins = autocreate quota expire acl sieve } protocol imap { imap_client_workarounds = delay-newmail mail_max_userip_connections = 24 mail_plugins = autocreate quota expire acl imap_quota imap_acl ssl_cert = </etc/postfix/ssl/imap_warenform_de.crt ssl_key = </etc/postfix/ssl/imap_warenform_de.key } protocol pop3 { mail_max_userip_connections = 10 pop3_client_workarounds = outlook-no-nuls oe-ns-eoh ssl_cert = </etc/postfix/ssl/pop_warenform_de.crt ssl_key = </etc/postfix/ssl/pop_warenform_de.key } ----- sql-dict.conf.ext: connect = host=/var/run/postgresql user=db_user password=db_passwd dbname=db_name # quota map { pattern = priv/quota/storage table = quota2 username_field = username value_field = bytes } map { pattern = priv/quota/messages table = quota2 username_field = username value_field = messages } # expires map { pattern = shared/expire/$user/$mailbox table = expires value_field = expire_stamp fields { username = $user mailbox = $mailbox } } # acl map { pattern = shared/shared-boxes/user/$to/$from table = user_shares value_field = dummy fields { from_user = $from to_user = $to } } map { pattern = shared/shared-boxes/anyone/$from table = anyone_shares value_field = dummy fields { from_user = $from } } -- e: ckubu(a)so36.net

2 3