June 2012 - dovecot - dovecot.org

[Dovecot] dsync replication available for testing
by Timo Sirainen 25 Aug '12

25 Aug '12

In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: - public namespace isn't replicated at all - shared namespace is replicated, but not private mail flags - I've only tested SSH replication setup now, not director replication setup (and director setup is still missing many things) - SSH replication setup uses aggregator process, which isn't really necessary and can probably be avoided in future Below is a configuration for virtual user setup. System user configuration works pretty much the same, except doveadm/ssh is run as root. Try first that dsync works successfully with ssh in host1: doveadm sync -u user@domain remote:vmail@host2.example.com and also in host2: doveadm sync -u user@domain remote:vmail@host1.example.com ------ mail_plugins = $mail_plugins notify replication service aggregator { # give enough permissions for mail processes fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup process_min_avail = 1 } plugin { # host1 replicates to host2 mail_replica = remote:vmail@host2.example.com # host2 replicates to host1 #mail_replica = remote:vmail@host1.example.com } #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service doveadm { # if you're using a single virtual user, set this to # start ssh as vmail (not root) user = vmail }

4 20

Re: [Dovecot] problem with virtual plugin/index files?
by Lutz Preßler 24 Aug '12

24 Aug '12

Hello, Timo: on Oct 29th I wrote ---- with latest 2.0.6 (601065674f74), the following virtual mailbox is not behaving correctly. I'm quite sure that the problem has not existed in some earlier 2.0.x (maybe only INBOX as mailbox then though): ~user/Maildir/virtual/.week/dovecot-virtual contains INBOX INBOX.* -INBOX.Trash inthread refs younger 604800 This works as expected for the first time. But later on, also older messages/ threads are included until I delete the virtual/.week/dovecot.index*. ---- Just want to state that this problem still persists with latest version. Any idea? Thanks and greetings, Lutz -- Lutz Preßler <Lutz.Pressler(a)SerNet.DE> http://www.SerNet.DE/ SerNet Service Network GmbH, Bahnhofsallee 1b, D-37081 Göttingen Tel.: +49-551-370000-2, FAX: +49-551-370000-9 AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen

2 4

[Dovecot] namespace from snapshots
by Karl Oulmi 21 Aug '12

21 Aug '12

hi all, I'm trying to give access to snapshots taken from by dell iscsi MD3200i to my maildirs users. snapshot are mounted in read only mode from my FreeBSD box. In my /usr/local/etc/dovecot/conf.d/10-mail.conf, I have : namespace inbox { inbox = yes } namespace da1 { prefix = INBOX.backup.da1. hidden = no list = yes inbox = no location = maildir:/da1/%u/Maildir type = private } I have two problems : - In my log, dovecot complain about the fact that my snapshot filsystem is read only Error: open(/da1/test2(a)toto.com/Maildir/.Trash/dovecot-uidlist) failed: Read-only file system The second problem is that through my thunderbird I can't see the Inbox saved in the snapshot (whereas I see Sent and Trash folders) If anyone could help me, It would be nice. Regard, Karl.

3 7

[Dovecot] Statistics scripts
by Timo Sirainen 08 Aug '12

08 Aug '12

I wrote two scripts, which you can use with the stats plugin enabled to get a better "doveadm stats top" than it currently is: http://dovecot.org/tools/stats-top.pl http://dovecot.org/tools/stats.pl

2 1

[Dovecot] mysql auth failover failing
by Paul B. Henson 29 Jul '12

29 Jul '12

We are running dovecot to provide authentication for postfix, using two mysql servers in a multi-master replication set as the password source: ---------------------------------------- # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.37-gentoo-r4 x86_64 Gentoo Base System release 2.0.2 auth_mechanisms = plain login digest-md5 cram-md5 auth_verbose = yes passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocols = none service auth-worker { unix_listener auth-worker { user = postfix } user = $default_internal_user } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = postfix } ssl = no userdb { driver = passwd } --------------------------------------- With an sql config of: ------------------------- driver = mysql connect = host=mysql-1.unx.csupomona.edu host=mysql-2.unx.csupomona.edu dbname=idmgmt user=postfix password=XXXXXXX default_pass_scheme = PLAIN password_query = XXXXXXXXX ------------------------- According to the sample SQL configuration file "HA / round-robin load-balancing is supported by giving multiple host settings, like: host=sql1.host.org host=sql2.host.org". However, as far as I can tell dovecot only connects to the first listed host, and processes all queries through it, there does not appear to be any load-balancing going on. That's not necessarily a dealbreaker; however, high-availability does not appear to be working either. If I shutdown the first mysql server, dovecot starts to log connection failures: Sep 9 15:47:34 tweak dovecot: auth: Error: mysql(mysql-1.unx.csupomona.edu): Connect failed to database (idmgmt): Can't connect to MySQL server on 'mysql-1.unx.csupomona.edu' (111) - waiting for 1 seconds before retry Sep 9 15:47:39 tweak dovecot: auth: Error: mysql(mysql-1.unx.csupomona.edu): Connect failed to database (idmgmt): Can't connect to MySQL server on 'mysql-1.unx.csupomona.edu' (111) - waiting for 25 seconds before retry And postfix starts to fail authentications: Sep 9 15:47:35 tweak postfix/smtpd[5119]: warning: bender.iitsys.csupomona.edu[134.71.250.134]: SASL DIGEST-MD5 authentication failed: Connection lost to authentication server Now and again the authentication process dies: Sep 9 15:47:39 tweak dovecot: auth: Panic: file auth-request-handler.c: line 697 (auth_request_handler_flush_failures): assertion failed: (auth_request->state == AUTH_REQUEST_STATE_FINISHED) Sep 9 15:47:39 tweak dovecot: auth: Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x3f71a) [0x7f25822ca71a] -> /usr/lib64/dovecot/libdovecot.so.0(+0x3f766) [0x7f25822ca766] -> /usr/lib64/dovecot/libdovecot.so.0(+0x198ca) [0x7f25822a48ca] -> dovecot/auth() [0x4137f4] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xd4) [0x7f25822d5fe4] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5b) [0x7f25822d6bcb] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f25822d5c48] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f25822c3de3] -> dovecot/auth(main+0x2be) [0x4179de] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f2581898bbd] -> dovecot/auth() [0x40bdc9] Sep 9 15:47:39 tweak dovecot: master: Error: service(auth): child 4154 killed with signal 6 (core dumps disabled) Requests start to pile up: Sep 9 15:51:46 tweak dovecot: auth: Warning: auth workers: Auth request was queued for 25 seconds, 45 left in queue Lookups time out: Sep 9 15:57:22 tweak dovecot: auth: Error: auth worker: Aborted request: Lookup timed out This occasionally pops up: Sep 9 15:58:38 tweak dovecot: auth: Fatal: net_connect_unix(auth-worker) failed: Resource temporarily unavailable And sometimes the auth process gets temporarily disabled: Sep 9 15:58:57 tweak dovecot: master: Error: service(auth): command startup failed, throttling Resulting in more postfix authentication failures: Sep 9 15:58:57 tweak postfix/smtpd[6531]: warning: bender.iitsys.csupomona.edu[134.71.250.134]: SASL DIGEST-MD5 authentication failed: Sep 9 15:59:08 tweak postfix/smtpd[6551]: fatal: no SASL authentication mechanisms To the point where postfix also temporarily throttles smtpd: Sep 9 15:59:21 tweak postfix/master[6526]: warning: /usr/lib64/postfix/smtpd: bad command startup -- throttling Resulting in a complete unavailability of smtp service, not just unavailability of authenticated services. I don't think all authentications fail during the scenario, but I think the majority do. Based on the network traffic, dovecot is almost continuously trying to connect to the first listed server. It sometimes connects to the second listed server, but when it does, the connection does not persist, it goes away almost immediately. Ideally, I would like no authentications to fail if one of the MySQL servers is unavailable. If a few fail just when the server dies, that would be undesirable but acceptable as long as they do not continuously fail while the server is down. Am I doing something wrong? Does the example sql config have incorrect information? We were previously running dovecot 1.2.11, we just recently upgraded to 2. In the previous version, we actually had two different passdb's configured, each one listing only one of the mysql servers. I seem to recall that was the recommendation at the time for high-availability. When that configuration did not seem to work under version 2, I found an updated recommendation to list both servers in the same passdb, which also does not appear to work correctly. I actually went back and tested the older version, and determined it seemed to work okay in the case where the server was up but the service was down, and connections were refused, but also failed a large number of authentication attempts when the server was completely down and connections were timing out. Thanks much... -- Paul B. Henson | (909) 979-6361 | http://www.csupomona.edu/~henson/ Operating Systems and Network Analyst | henson(a)csupomona.edu California State Polytechnic University | Pomona CA 91768

5 19

[Dovecot] SQLite dovecot query caching
by Dominic Malolepszy 19 Jul '12

19 Jul '12

Hi, I am wondering if Dovecot caches SQLite queries, and how well it works in high performance setups. I am particularly interested because in the below thread SQLite has been suggested as a means of Dovecot proxying connections to different ports. http://old.nabble.com/director%3A-non-standart-ports-at-backends-td33991991… Cheers, Dominic.

2 7

[Dovecot] director: non standart ports at backends
by Костырев Александр Алексеевич 16 Jul '12

16 Jul '12

hello, I'm trying to figure out how to proxy pop3 and pop3s that listens on non-standart ports at backends. For example, pop3 is at 1110 and pop3s at 1995 (on backend side). is it possible? how should I separate this ports in director's config? it's easy for one port: for example lmtp - you just use passdb in protocol lmtp {}

3 5

[Dovecot] dict Panic after upgrade to 2.1.7
by ckubu 03 Jul '12

03 Jul '12

Hello, after upgrade my mailsystem to dovecot version 2.1.7, dovecot doesn't work properly. something went wrong in dict service connecting the postgres backend. that happens not on every connection. the db connection data are correct, no difference connecting via tcp or linux socket. dovecot log entries: Jun 23 23:19:10 mx dovecot: dict: Panic: file driver-pgsql.c: line 84 (driver_pgsql_set_state): assertion failed: (state == SQL_DB_STATE_BUSY || db- >cur_result == NULL) Jun 23 23:19:10 mx dovecot: dict: Error: Raw backtrace: /usr/local/dovecot-2.1.7/lib/dovecot/libdovecot.so.0(+0x4478a) [0x7ffc7d8e578a] -> /usr/local/dovecot-2.1.7/lib/dovecot/libdovecot.so.0(+0x447d6) [0x7ffc7d8e57d6] -> /usr/local/dovecot-2.1.7/lib/dovecot/libdovecot.so.0(i_error+0) [0x7ffc7d8bc5ef] -> dovecot/dict() [0x40a9a6] -> dovecot/dict() [0x40aa01] -> dovecot/dict() [0x40be43] -> dovecot/dict() [0x409474] -> dovecot/dict(sql_db_cache_deinit+0x20) [0x4089d0] -> dovecot/dict(main+0x169) [0x4059f9] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7ffc7d335c8d] -> dovecot/dict() [0x404b59] Jun 23 23:19:10 mx dovecot: dict: Fatal: master: service(dict): child 13812 killed with signal 6 (core dumps disabled) Jun 23 23:23:09 mx dovecot: dict: Error: dict sql iterate failed: Not connected to database Jun 23 23:23:09 mx dovecot: pop3(xxx(a)yyy.zz): Error: acl: dict iteration failed, can't update dict Jun 23 23:23:09 mx dovecot: dict: Error: dict sql iterate failed: Not connected to database Jun 23 23:23:09 mx dovecot: pop3(xxx(a)yyy.zz): Error: acl: dict iteration failed, can't update dict Jun 23 23:23:17 mx dovecot: dict: Error: dict sql lookup failed: Not connected to database Jun 23 23:23:17 mx dovecot: imap(xxx(a)yyy.zz): Error: Internal quota calculation error Jun 23 23:23:19 mx dovecot: dict: Error: dict sql lookup failed: Not connected to database Jun 23 23:23:40 mx dovecot: dict: Error: dict sql lookup failed: Not connected to database maybe i have missconfigured the dovecot system, but i don't find the mistake. can anybody give me a hint ? best wiches christoph ----- doveconf -n # 2.1.7: /usr/local/dovecot-2.1.7/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-vserver-amd64 x86_64 Debian 6.0.5 ext3 auth_mechanisms = plain login digest-md5 cram-md5 apop auth_socket_path = /var/run/dovecot/auth-userdb auth_username_translation = %@ auth_verbose = yes auth_verbose_passwords = plain base_dir = /var/run/dovecot/ dict { acl = pgsql:/usr/local/dovecot/etc/dovecot/sql-dict.conf.ext expire = pgsql:/usr/local/dovecot/etc/dovecot/sql-dict.conf.ext quota = pgsql:/usr/local/dovecot/etc/dovecot/sql-dict.conf.ext } disable_plaintext_auth = no first_valid_gid = 5000 first_valid_uid = 5000 hostname = mx.warenform.de last_valid_gid = 5000 last_valid_uid = 5000 listen = 178.63.63.151 2a01:4f8:121:c5::2 mail_gid = vmail mail_location = maildir:/var/vmail/%d/%n/Maildir mail_plugins = autocreate quota expire acl mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { list = children location = maildir:/var/vmail/%%d/%%n/Maildir:INDEX=~/Maildir/shared/%%u prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { special_use = \Junk } mailbox Trash { special_use = \Trash } prefix = separator = / type = private } passdb { args = /usr/local/dovecot/etc/dovecot/sql-connect.conf.ext driver = sql } plugin { acl = vfile acl_shared_dict = proxy::acl autocreate = Spam autocreate2 = Sent autocreate3 = Trash autocreate4 = Drafts autosubscribe = Spam autosubscribe2 = Sent autosubscribe3 = Trash autosubscribe4 = Drafts expire = Trash expire2 = Trash.* expire3 = Spam expire_dict = proxy::expire quota = dict:User quota::proxy::quota quota_rule = *:storage=1G quota_rule2 = Trash:storage=+200M quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u recipient_delimiter = sieve = ~/.dovecot.sieve sieve_before = /usr/local/dovecot/etc/dovecot/sieve/move-spam.sieve sieve_dir = ~/sieve sieve_global_dir = /usr/local/dovecot/etc/dovecot/sieve/global/ } postmaster_address = admin(a)warenform.de protocols = imap pop3 sieve lmtp service auth { unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { inet_listener imap { address = 127.0.0.1 178.63.63.151 2a01:4f8:121:c5::2 } inet_listener imaps { address = 178.63.63.151 2a01:4f8:121:c5::2 } process_min_avail = 16 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service managesieve-login { inet_listener sieve { address = 127.0.0.1 port = 4190 } } service pop3-login { inet_listener pop3 { address = 178.63.63.151 2a01:4f8:121:c5::2 } inet_listener pop3s { address = 178.63.63.151 2a01:4f8:121:c5::2 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } shutdown_clients = no ssl_cert = </etc/postfix/ssl/servercert.pem ssl_key = </etc/postfix/ssl/servercert.pem syslog_facility = local1 userdb { args = /usr/local/dovecot/etc/dovecot/sql-connect.conf.ext driver = sql } protocol lmtp { mail_plugins = autocreate quota expire acl sieve } protocol lda { mail_plugins = autocreate quota expire acl sieve } protocol imap { imap_client_workarounds = delay-newmail mail_max_userip_connections = 24 mail_plugins = autocreate quota expire acl imap_quota imap_acl ssl_cert = </etc/postfix/ssl/imap_warenform_de.crt ssl_key = </etc/postfix/ssl/imap_warenform_de.key } protocol pop3 { mail_max_userip_connections = 10 pop3_client_workarounds = outlook-no-nuls oe-ns-eoh ssl_cert = </etc/postfix/ssl/pop_warenform_de.crt ssl_key = </etc/postfix/ssl/pop_warenform_de.key } ----- sql-dict.conf.ext: connect = host=/var/run/postgresql user=db_user password=db_passwd dbname=db_name # quota map { pattern = priv/quota/storage table = quota2 username_field = username value_field = bytes } map { pattern = priv/quota/messages table = quota2 username_field = username value_field = messages } # expires map { pattern = shared/expire/$user/$mailbox table = expires value_field = expire_stamp fields { username = $user mailbox = $mailbox } } # acl map { pattern = shared/shared-boxes/user/$to/$from table = user_shares value_field = dummy fields { from_user = $from to_user = $to } } map { pattern = shared/shared-boxes/anyone/$from table = anyone_shares value_field = dummy fields { from_user = $from } } -- e: ckubu(a)so36.net

2 3

[Dovecot] Problem with 'doveadm mailbox status -t' reporting cumulative vsizes after upgrading from v2.0.16 to v2.1.7
by Jase Thew 03 Jul '12

03 Jul '12

Hi, I upgraded from Dovecot v2.0.16 to v2.1.7 over night and I noticed this morning that one of my daily reports which lists summarised mailbox sizes per user has started listing nonsense for vsizes. The reporting script at its core calls : doveadm -f flow mailbox status -A -t 'messages vsize' '*' It appears that Dovecot 2.1.7 is not resetting the vsize after collating the sum total of mailboxes sizes for each user, so that vsize just constantly increases as it iterates over each user. Eg: # doveadm -f flow mailbox status -A -t 'messages vsize' '*' accounts(a)example.com messages=1 vsize=759 adam(a)example.com messages=0 vsize=759 amy(a)example.com messages=24 vsize=51699697 andy(a)example.com messages=5446 vsize=3220940815 anna(a)example.com messages=50 vsize=3224035563 careers(a)example.com messages=1 vsize=3224036311 craig(a)example.com messages=2471 vsize=4421343199 creative(a)example.com messages=189 vsize=4426884182 david(a)example.com messages=8 vsize=4440729729 davidw(a)example.com messages=0 vsize=4440729729 enquiries(a)example.com messages=1 vsize=4440730491 gemma(a)example.com messages=4109 vsize=6349098844 gin(a)example.com messages=86 vsize=6392599904 holly(a)example.com messages=2000 vsize=7200342663 ian(a)example.com messages=0 vsize=7200342663 info(a)example.com messages=4 vsize=7200558689 jackie(a)example.com messages=2 vsize=7200721146 jade(a)example.com messages=137 vsize=7210548548 jake(a)example.com messages=16667 vsize=15260532446 katie(a)example.com messages=1 vsize=15260533375 mark(a)example.com messages=0 vsize=15260533375 mike.a(a)example.com messages=9 vsize=15261474205 mike.s(a)example.com messages=296 vsize=15314352543 mike(a)example.com messages=6357 vsize=20631446344 nick(a)example.com messages=1184 vsize=21038046728 social(a)example.com messages=65 vsize=21038935461 will(a)example.com messages=85 vsize=21057572390 [SNIP] The same occurs with -u '*(a)example.com' in place of -A, and also for 'all' in place of 'messages vsize'. Is this expected behaviour in 2.1.x compared to 2.0.x, or have I stumbled upon a bug? Regards, Jase Thew.

2 2

[Dovecot] Additional passdb result status
by Jürgen Pabel 03 Jul '12

03 Jul '12

Dear Dovecot-Team, I am implementing a plugin (for the pop3/imap process) that requires some data to provided from the authentication phase (a derivative of the password). For that, I have now implemented a passdb plugin that generates this data and I would like to "pass" this data down to the mail process (pop3/imap) via extra_fields in the reply of the authentication. The general idea is that my custom passdb plugin calculates the data, sets the extra_field and returns some error (authentication was not successful) so that the "real" passdb backend can be invoked to "really" validate the authentication data. However, in auth_request_handle_passdb_callback() the extra_fields are reseted unless the return code is PASSDB_RESULT_USER_DISABLED. But if that return code is used then any following passdb's aren't invoked any more - which makes sense with respect to user authenticiation. I would therefore like to propose that some IGNORE/CONTINUE-status to be introduced in auth/passdb.h, that would be handled in that extra_fields and possible other values are not reseted in order to allow such propagation of data from authentication process down to the mail process (which could be extracted from the reply string by parsing it). As a further implementation alternative (to the parsing of the reply string), I also propose that some new "environment" item be introduced (in auth_request) in order to allow such data passing in a generic manner. I hope you consider my proposal to be reasonable. If desired, I could implement this myself and provide a patch for merging (based on 2.0.x). If my proposal is generally unfavored, it would be great if any alternative approaches for my situation were suggested. Thanks. Regards, Jürgen PS: please reply to my e-mail (or CC me), as I have not subscribed to the dovecot list

3 3