- dovecot - dovecot.org

[Dovecot] Mailbox with .broken files
by Michael Smith (DF) 19 Dec '13

19 Dec '13

We're using Dovecot 2.2.4 and mdbox storage with compression. I noticed yesterday that at least one of the accounts has unusual files in it's mail/storage directory. This account has approximately 17.5G of compressed mail, across about 750 storage files (m.###) using 8G of storage. Starting on Dec 11, storage files m.409 through m.731 have not only their m.### file, but also a m.###.broken file. At the same time .temp.<unix time>.<unique id>.<hostname> files also started showing up... ... -rw--w---- 1 user123 mail 3225 Dec 12 04:38 m.708 -rw--w---- 1 user123 mail 167106 Dec 12 03:35 m.708.broken -rw--w---- 1 user123 mail 46155 Dec 12 04:38 m.709 -rw--w---- 1 user123 mail 177267 Dec 12 03:40 m.709.broken -rw--w---- 1 user123 mail 20965889 Nov 26 19:22 m.71 -rw--w---- 1 user123 mail 133248 Dec 12 04:38 m.710 -rw--w---- 1 user123 mail 264361 Dec 12 03:40 m.710.broken -rw--w---- 1 user123 mail 103407 Dec 12 04:38 m.711 -rw--w---- 1 user123 mail 627736 Dec 12 03:40 m.711.broken -rw--w---- 1 user123 mail 5132 Dec 12 04:38 m.712 -rw--w---- 1 user123 mail 136244 Dec 12 03:40 m.712.broken -rw--w---- 1 user123 mail 187224 Dec 12 04:38 m.713 -rw--w---- 1 user123 mail 351104 Dec 12 03:45 m.713.broken -rw--w---- 1 user123 mail 435793 Dec 12 04:38 m.714 -rw--w---- 1 user123 mail 566906 Dec 12 03:45 m.714.broken -rw--w---- 1 user123 mail 60916 Dec 12 04:38 m.715 -rw--w---- 1 user123 mail 192028 Dec 12 03:50 m.715.broken -rw--w---- 1 user123 mail 38051 Dec 12 04:38 m.716 -rw--w---- 1 user123 mail 169163 Dec 12 03:50 m.716.broken -rw--w---- 1 user123 mail 226067 Dec 12 04:38 m.717 -rw--w---- 1 user123 mail 389948 Dec 12 04:00 m.717.broken ... -rw--w---- 1 user123 mail 20967506 Dec 2 17:32 m.96 -rw--w---- 1 user123 mail 20967131 Dec 2 18:40 m.97 -rw--w---- 1 user123 mail 20969078 Dec 2 18:38 m.98 -rw--w---- 1 user123 mail 20963978 Dec 2 19:23 m.99 -rw--w---- 1 user123 mail 294968 Dec 11 20:05 .temp.1386813944.P9142Q0M123686.myhost -rw--w---- 1 user123 mail 294968 Dec 11 20:05 .temp.1386813967.P9213Q0M387733.myhost -rw--w---- 1 user123 mail 294968 Dec 11 20:05 .temp.1386813970.P9231Q0M371190.myhost -rw--w---- 1 user123 mail 327736 Dec 11 20:05 .temp.1386813972.P9173Q0M932070.myhost -rw--w---- 1 user123 mail 327736 Dec 11 20:05 .temp.1386813972.P9239Q0M114754.myhost -rw--w---- 1 user123 mail 327736 Dec 11 20:05 .temp.1386813983.P9261Q0M640646.myhost -rw--w---- 1 user123 mail 1048632 Dec 11 20:06 .temp.1386814038.P9515Q0M530600.myhost I'm not really seeing anything that stands out in the maillog during this time. What caused this problem? How concerned should I be about possible lost email? This is a production environment. Below is the dovecot -n output. -- Michael Smith # dovecot -n # 2.2.4: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-279.19.1.el6.x86_64 x86_64 CentOS release 6.3 (Final) auth_debug = yes auth_debug_passwords = yes auth_default_realm = mydomain.com auth_master_user_separator = * auth_mechanisms = plain login auth_verbose = yes auth_verbose_passwords = plain disable_plaintext_auth = no dotlock_use_excl = no lda_mailbox_autocreate = yes lock_method = dotlock mail_access_groups = mail mail_debug = yes mail_fsync = always mail_location = mdbox:~/mail:INDEX=~/index mail_plugins = quota zlib mail_privileged_group = mail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mbox_read_locks = dotlock mbox_write_locks = dotlock mdbox_rotate_interval = 1 weeks mdbox_rotate_size = 20 M mmap_disable = yes namespace { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = type = private } passdb { args = /etc/dovecot/dovecot-sql-master.conf.ext driver = sql master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { autosubscribe = Trash autosubscribe2 = Junk autosubscribe3 = Drafts autosubscribe4 = Sent autosubscribe5 = INBOX mail_max_userip_connections = 10 managesieve_max_line_length = 65536 quota = dict:User quota::file:%h/dovecot.quota quota2_rule = Trash:storage=+10%% quota3_rule = Junk:storage=+20%% quota_rule = *:storage=100M:messages=100000 recipient_delimiter = + sieve_before = /var/opt/mail/global.sieve/ zlib_save = gz zlib_save_level = 9 } pop3_reuse_xuidl = yes protocols = imap pop3 sieve lmtp service auth { inet_listener auth { port = 113 } unix_listener auth-userdb { user = nobody } } service lmtp { inet_listener lmtp { port = 24 } process_min_avail = 10 service_count = 1 } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 1 vsz_limit = 64 M } ssl = no ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lda { mail_plugins = quota zlib sieve } protocol imap { mail_plugins = quota zlib imap_quota } protocol sieve { mail_max_userip_connections = 10 mail_plugins = quota zlib managesieve_max_line_length = 65536 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date } protocol lmtp { mail_plugins = quota zlib sieve }

2 1

Re: [Dovecot] Mailbox with .broken files
by Michael Smith (DF) 19 Dec '13

19 Dec '13

Anyone have any feedback, guidance or suggestions on this? Timo? -----Original Message----- From: Michael Smith (DF) [mailto:msmith@datafoundry.com] Sent: Friday, December 13, 2013 9:49 AM To: 'dovecot(a)dovecot.org' Subject: [Dovecot] Mailbox with .broken files We're using Dovecot 2.2.4 and mdbox storage with compression. I noticed yesterday that at least one of the accounts has unusual files in it's mail/storage directory. This account has approximately 17.5G of compressed mail, across about 750 storage files (m.###) using 8G of storage. Starting on Dec 11, storage files m.409 through m.731 have not only their m.### file, but also a m.###.broken file. At the same time .temp.<unix time>.<unique id>.<hostname> files also started showing up... ... -rw--w---- 1 user123 mail 3225 Dec 12 04:38 m.708 -rw--w---- 1 user123 mail 167106 Dec 12 03:35 m.708.broken -rw--w---- 1 user123 mail 46155 Dec 12 04:38 m.709 -rw--w---- 1 user123 mail 177267 Dec 12 03:40 m.709.broken -rw--w---- 1 user123 mail 20965889 Nov 26 19:22 m.71 -rw--w---- 1 user123 mail 133248 Dec 12 04:38 m.710 -rw--w---- 1 user123 mail 264361 Dec 12 03:40 m.710.broken -rw--w---- 1 user123 mail 103407 Dec 12 04:38 m.711 -rw--w---- 1 user123 mail 627736 Dec 12 03:40 m.711.broken -rw--w---- 1 user123 mail 5132 Dec 12 04:38 m.712 -rw--w---- 1 user123 mail 136244 Dec 12 03:40 m.712.broken -rw--w---- 1 user123 mail 187224 Dec 12 04:38 m.713 -rw--w---- 1 user123 mail 351104 Dec 12 03:45 m.713.broken -rw--w---- 1 user123 mail 435793 Dec 12 04:38 m.714 -rw--w---- 1 user123 mail 566906 Dec 12 03:45 m.714.broken -rw--w---- 1 user123 mail 60916 Dec 12 04:38 m.715 -rw--w---- 1 user123 mail 192028 Dec 12 03:50 m.715.broken -rw--w---- 1 user123 mail 38051 Dec 12 04:38 m.716 -rw--w---- 1 user123 mail 169163 Dec 12 03:50 m.716.broken -rw--w---- 1 user123 mail 226067 Dec 12 04:38 m.717 -rw--w---- 1 user123 mail 389948 Dec 12 04:00 m.717.broken ... -rw--w---- 1 user123 mail 20967506 Dec 2 17:32 m.96 -rw--w---- 1 user123 mail 20967131 Dec 2 18:40 m.97 -rw--w---- 1 user123 mail 20969078 Dec 2 18:38 m.98 -rw--w---- 1 user123 mail 20963978 Dec 2 19:23 m.99 -rw--w---- 1 user123 mail 294968 Dec 11 20:05 .temp.1386813944.P9142Q0M123686.myhost -rw--w---- 1 user123 mail 294968 Dec 11 20:05 .temp.1386813967.P9213Q0M387733.myhost -rw--w---- 1 user123 mail 294968 Dec 11 20:05 .temp.1386813970.P9231Q0M371190.myhost -rw--w---- 1 user123 mail 327736 Dec 11 20:05 .temp.1386813972.P9173Q0M932070.myhost -rw--w---- 1 user123 mail 327736 Dec 11 20:05 .temp.1386813972.P9239Q0M114754.myhost -rw--w---- 1 user123 mail 327736 Dec 11 20:05 .temp.1386813983.P9261Q0M640646.myhost -rw--w---- 1 user123 mail 1048632 Dec 11 20:06 .temp.1386814038.P9515Q0M530600.myhost I'm not really seeing anything that stands out in the maillog during this time. What caused this problem? How concerned should I be about possible lost email? This is a production environment. Below is the dovecot -n output. -- Michael Smith # dovecot -n # 2.2.4: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-279.19.1.el6.x86_64 x86_64 CentOS release 6.3 (Final) auth_debug = yes auth_debug_passwords = yes auth_default_realm = mydomain.com auth_master_user_separator = * auth_mechanisms = plain login auth_verbose = yes auth_verbose_passwords = plain disable_plaintext_auth = no dotlock_use_excl = no lda_mailbox_autocreate = yes lock_method = dotlock mail_access_groups = mail mail_debug = yes mail_fsync = always mail_location = mdbox:~/mail:INDEX=~/index mail_plugins = quota zlib mail_privileged_group = mail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mbox_read_locks = dotlock mbox_write_locks = dotlock mdbox_rotate_interval = 1 weeks mdbox_rotate_size = 20 M mmap_disable = yes namespace { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = type = private } passdb { args = /etc/dovecot/dovecot-sql-master.conf.ext driver = sql master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { autosubscribe = Trash autosubscribe2 = Junk autosubscribe3 = Drafts autosubscribe4 = Sent autosubscribe5 = INBOX mail_max_userip_connections = 10 managesieve_max_line_length = 65536 quota = dict:User quota::file:%h/dovecot.quota quota2_rule = Trash:storage=+10%% quota3_rule = Junk:storage=+20%% quota_rule = *:storage=100M:messages=100000 recipient_delimiter = + sieve_before = /var/opt/mail/global.sieve/ zlib_save = gz zlib_save_level = 9 } pop3_reuse_xuidl = yes protocols = imap pop3 sieve lmtp service auth { inet_listener auth { port = 113 } unix_listener auth-userdb { user = nobody } } service lmtp { inet_listener lmtp { port = 24 } process_min_avail = 10 service_count = 1 } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 1 vsz_limit = 64 M } ssl = no ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lda { mail_plugins = quota zlib sieve } protocol imap { mail_plugins = quota zlib imap_quota } protocol sieve { mail_max_userip_connections = 10 mail_plugins = quota zlib managesieve_max_line_length = 65536 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date } protocol lmtp { mail_plugins = quota zlib sieve }

1 0

[Dovecot] v2.2.9 released
by Timo Sirainen 19 Dec '13

19 Dec '13

http://dovecot.org/releases/2.2/dovecot-2.2.9.tar.gz http://dovecot.org/releases/2.2/dovecot-2.2.9.tar.gz.sig + Full text search indexing can now be done automatically after saving/copying mails by setting plugin { fts_autoindex=yes } + replicator: Added replication_dsync_parameters setting to pass "doveadm sync" parameters (for controlling what to replicate). + Added mail-filter plugin + Added liblzma/xz support (zlib_save=xz) - v2.2.8's improved cache file handling exposed several old bugs related to fetching mail headers. - v2.2.7's iostream handling changes were causing some connections to be disconnected before flushing their output (e.g. POP3 logout message wasn't being sent)

7 9

[Dovecot] zlib plugin question
by Przemysław Orzechowski 19 Dec '13

19 Dec '13

Hi Is it possible to compress incoming mails delivered via dovecots LDA when using dovecot --version 1.2.9 or do i have to compress them via cron?

2 2

[Dovecot] problem with password in a changing scenario
by Angel Luis Mateo Martinez 19 Dec '13

19 Dec '13

Hello, I'm having a problem with with password changing. According to http://wiki2.dovecot.org/Authentication/Caching, I understand "normal scenario" and "Using old cached password scenario", but I'm having a problem with the "Early change scenario". I understand that in this case, if in step 4 user tries password Y, then this password is not tried again and login fails. But if the user tries another password Z, then is it test it? Or does it directly fail? I think that it is directly failing. And here is my problem... I am using dovecot with a ldap backend and another pam backend. The first one is users connecting with an email client. The other one applies when the ldap backend fails and it is use for connections from our webmail, which is integrated in our CAS SSO environment (it has running an imap proxy too). Sometimes (the only scenari we were able to reproduce is with users that for some reason has two SSO sessions opened) dovecot receives an invalid CAS ticket, then authentication fails, and no newer tickets are tried until cache is clean. That's why I think that in this scenario another Z password is even tried. Any idea? -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868889150 Fax: 868888337

1 0

[Dovecot] letters local filter duplicated
by John Smith 18 Dec '13

18 Dec '13

two servers with replicated master/master. dovecot.conf = http://pastebin.com/jVnvyWZM with this setup letters processed local filter duplicated. ie when the mail client receives a letter begins processing using rules (filters) to move the letters in a different folder. summarized in this folder is two duplicate letters. if you disable the filter, the message is not duplicated. But does not move to another location. -- Best Regards, Evgeny Senior System Administrator OOO "Vaz Service Hurley"

3 4

[Dovecot] Size detection/replair does not work with zlib
by Roland Rosenfeld 18 Dec '13

18 Dec '13

Hi! Usually dovecot auto detects or repairs the size of a maildir message. So I can place a message named "foo" in the cur directory and dovecot uses it. Now I tried the same with a zlib compressed message but here dovecot doesn't recognize/repair the size of the message. When I access this folder via IMAP the connection is diconnected and in dovecot logs I see the following error messages: Error: Cached message size smaller than expected (805 < 2666) Error: Corrupted index cache file /somedir/dovecot.index.cache: Broken physical size for mail UID 23 Error: read() failed: Input/output error (FETCH for mailbox INBOX UID 23) Disconnected: Internal error occurred. Refer to server log for more information. [2013-12-12 10:54:18] in=321 out=1977 As you can see in the first line, dovecot does know the compressed size of the file (805) as well as the uncompressed size (2666), but it isn't able to repair its index for this. If I modify the setup a little with a standard file naming but with a wrong file size in S-flag (compressed size instead of uncompressed size), the log entries become stranger: Error: Cached message size smaller than expected (805 < 2666) Error: Maildir filename has wrong S value, renamed the file from /somedir/cur/1386772057.M152553P9709.host,S=805:2, to /somedir/cur/1386772057.M152553P9709.host,S=805:2, So here dovecot detects the wrong S value, but instead of fixing it by using the uncompressed size, it renames to the same file name as before... All the above was tested with dovecot 2.1.17. We did a short cross test with 2.2.9 which gives somewhat different error messages, but also isn't able to detect/repair the (uncompressed) file size: Error: Cached message size smaller than expected (805 < 2666) Error: Corrupted index cache file /somedir/dovecot.index.cache: Broken physical size for mail UID 23 Error: read(zlib(/somedir/cur/foo)) failed: Invalid argument We also noticed that on both dovecot versions after trying to access the above file, dovecot.index.cache is always deleted and not rebuild... Is all this intended behavior? It sounds different to the standard behavior of dovecot, that repairs broken folders if possible... Ciao Roland

4 3

[Dovecot] configure lmtp to deliver to email addresses case insensitively
by Adrian Zaugg 17 Dec '13

17 Dec '13

Dear List Using dovecot 2.1.7 with LMTP and exim4 I want to accept local parts regardless of their case. Exim does all virtual alias handling and delivers the messages to dovecot LMTP addressed to the right mailbox name. This works well except for addresses which do not need to be resolved by exim and which do have the "wrong" case in the local part. For example: There is a mailbox and a user on the mailsystem "example.org" called "user". Mails to "user(a)example.org" get delivered just as they should. If an incoming message is addressed to "USER(a)example.org" with upper case local part, exim passes the message to LMTP with the case unaltered (RFC 2821 conform), but LMTP fails with: LMTP error after RCPT TO:<USER(a)example.org>: 550 5.1.1 <USER(a)example.org> User doesn't exist: USER(a)example.org How can I tell dovecot to deliver "USER" to the mailbox "user" aswell? Thank you for your help. Best regards, Adrian. (I have tried to use %Ln and %Lu in mail_location setting with no success.)

5 9

[Dovecot] Dovecot proxy and Postfix SASL
by Andreas Kasenides 17 Dec '13

17 Dec '13

Can somebody please verify that currently (v 2.2.9) SMTP AUTH using SASL from Postfix with Dovecot proxy is still not supported as discussed in these threads (especially the first one)? http://www.dovecot.org/list/dovecot/2012-August/067977.html http://www.dovecot.org/list/dovecot/2011-May/059107.html As I understand it is possible to use saslauthd to do this by using the remote imap option (rimap). Such a facility is important since I am attempting to separate the outward facing servers (dovecot proxy, postfix relay) that have no knowledge of user databases from the backends. thanx Andreas

1 1

[Dovecot] Accessing plain text password from memory
by Stanislas SABATIER 17 Dec '13

17 Dec '13

Is there a way to retrieve the client's password in plain text from memory ? I don't store the password in plain text in my postgreSQL but I need it when the client is connected to make crypto computation. If I write a plugin to do the job, how could I retrieve the plain text password from master ? Thank you for your help, Regards, Stan. --------------------------------------------------------------- Stanislas SABATIER s.sabatier(a)pobox.com ---------------------------------------------------------------

6 9