August 2014 - dovecot

Assertion in istream.c::167 in POP3 mode when maildir content has wrong S value (using 2.2.13)
by Rolf Eike Beer 21 Aug '14

21 Aug '14

Hi, the affected machine is running 2.2.13 on Gentoo x86_64. For me it looks like the old size is still in some struct or whereever even after the file has been renamed. I see that there are many commits following 2.2.13, is that possibly already fixed or do I need to dig in the code? The original problem seems to be that maildrop uses the file size before piping the mail into spamassassin even if the mail is bigger afterwards. Not relevant here, but in case anyone wonders how this happens. Greetings, Eike Error: Cached message size smaller than expected (81118 < 81308) Error: Maildir filename has wrong S value, renamed the file from /var/vpopmail/domains/example.org/user/.maildir/cur/1408452738.16341.mail,S=81118:2, to /var/vpopmail/domains/example.org/user/.maildir/cur/1408452738.16341.mail,S=81308:2, Error: Corrupted index cache file /var/vpopmail/domains/example.org/user/.maildir/dovecot.index.cache: Broken physical size for mail UID 65 Panic: file istream.c: line 167 (i_stream_read): assertion failed: (old_size <= _stream->pos - _stream->skip) Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x6bbef) [0x7f50652a4bef] -> /usr/lib64/dovecot/libdovecot.so.0(+0x6bc4e) [0x7f50652a4c4e] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f506525e2be] -> /usr/lib64/dovecot/libdovecot.so.0(i_stream_read+0x214) [0x7f50652ad904] -> /usr/lib64/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f50652adfed] -> /usr/lib64/dovecot/libdovecot.so.0(message_get_body_size+0x9d) [0x7f5065299b3d] -> /usr/lib64/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x200) [0x7f506556f700] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x2dd0f) [0x7f506553cd0f] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_get_stream+0x4b) [0x7f506554915b] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x2dae7) [0x7f506553cae7] -> dovecot/pop3(client_create+0x56d) [0x40622d] -> dovecot/pop3() [0x404ea3] -> dovecot/pop3() [0x40508f] -> /usr/lib64/dovecot/libdovecot.so.0(+0x28346) [0x7f5065261346] -> /usr/lib64/dovecot/libdovecot.so.0(+0x28841) [0x7f5065261841] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x4e) [0x7f50652b58de] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xd7) [0x7f50652b68b7] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x9) [0x7f50652b5969] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f50652b59e8] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f50652635a3] -> dovecot/pop3(main+0x233) [0x404bf3] -> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5064eb6db5] -> dovecot/pop3() [0x404d49]

1 1

About network throughput between Director and backends
by Listas＠Adminlinux 21 Aug '14

21 Aug '14

I plan to migrate my Dovecot IMAP servers to a clustered structure. I'm planning a cluster with high availability and load balancing. I know I need instances of Dovecot Proxy / Director. But these instances can be on another network? There is need for high network throughput between the Director and the backends? For me it would be cheaper to run Director in VM's at Amazon AWS for example. Thanks! -- Thiago Henrique

1 0

doveadm proxy list
by Claus 21 Aug '14

21 Aug '14

Hi, i have configured my dovecot-proxy with "shutdown_clients = no" and "service_count = 1" for a "default_process_limit = 5000" and "default_client_limit = 25000". I don't know if this is a prefered setup for so many users/concurrent sessions but so i can reload my dovecot without disconnecting all sessions. The main problem are pop3 sessions - if i kill them, the mailbox remains locked and the client gets an error. Back to my problem: If i reload dovecot the "doveadm proxy list" shows me the list of the new processes. Is there any possibility to see the stats of the old sessions (user,source,destination) ? greet's Claus PS.: Could someone explain me why i have to set default_client_limit to 5x default_process_limit?

2 2

rebuilding indexes for dovecot pop3
by Arkadiusz Miśkiewicz 21 Aug '14

21 Aug '14

Hi. doveadm -D fetch -u login \ 'imap.envelope imap.bodystructure size.virtual date.received' '*' according to mail archives allows to regenerate indexes and precache specified headers for imap usage. What headers are needed for dovecot pop3 daemon? I have one server where dovecot works for imap only. For pop3 there is different software being used. I want to migrate to dovecot pop3 but I need to precache and regenerate indexes for dovecot pop3. Note, some users used pop3 only, so dovecot imap had no chance to generate any indexes. Thanks, -- Arkadiusz Miśkiewicz, arekm / maven.pl

2 2

Re: Frequent error: unknown namespace Trash
by Michael Ludwig 21 Aug '14

21 Aug '14

Hi Gedalya, I am very sorry for not having replied on-list. That was not my intention! I am not used to this behaviour of mailing lists, normally there is a reply-to field which points to the mailing list mail address. So do I have to use MAILBOX.Trash in 90-quota.conf ?? Or INBOX.Trash ? Cheers, Michael 2014-08-12 19:33 GMT+02:00 Gedalya <gedalya(a)gedalya.net>: > On 08/12/2014 09:28 AM, Michael Ludwig wrote: > >> prefix = INBOX. >> separator = . >> > Maybe that's it? It's therefore MAILBOX.Trash, not just Trash. > > You're supposed to reply on-list! Don't reply to gedalya(a)gedalya.net, > reply to dovecot(a)dovecot.org. If you want you can also cc the message > author, some people do that. > >

3 3

Fatal: lazy_expunge: Unknown namespace: '.EXPUNGED/'
by Philipp Faeustlin 21 Aug '14

21 Aug '14

Hello List, with dovecot version 2.2.13.9, I actually have a problem using lazy_expunge and acl together. Shared/Public Folders work fine with lazy_expunge disabled. If lazy_expunge is enabled and userA got a shared folder form userB, then there is the error shown in subject. My test case to reproduce the error. doveadm mailbox list -u userA@domain --> Output Fine. doveadm acl set -u userB@domain INBOX user=userA@domain lookup read write write-seen write-deleted insert expunge create --> Fine. doveadm mailbox list -u userA@domain --> Some output and then: --> doveadm(userA@domain): Fatal: lazy_expunge: Unknown namespace: '.EXPUNGED/' I think this is a bug in dovecot, can someone of the dovecot developers or someone else confirm this? If it is not please show me the fault in my configuration. In May 2014 there was a thread with the same issue on this mailing list, but with no solution. http://dovecot.org/pipermail/dovecot/2014-May/096355.html Best regards Philipp Fäustlin --------------------------------------------- # 2.2.13.9 (19befd72d9d8): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-431.11.2.el6.x86_64 x86_64 Scientific Linux release 6.5 (Carbon) auth_debug = yes auth_mechanisms = plain login auth_socket_path = /var/run/dovecot/auth-userdb auth_username_format = %Ln auth_verbose = yes listen = * mail_debug = yes mail_gid = vmail mail_location = maildir:~/maildir mail_plugins = " acl lazy_expunge mail_log notify zlib notify replication" mail_shared_explicit_inbox = yes mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate namespace { hidden = no ignore_on_failure = no inbox = no list = children location = maildir:%%h/maildir:INDEX=%h/Shared/%%u:CONTROL=%h/Shared/%%u prefix = Shared/%%u/ separator = / subscriptions = yes type = shared } namespace { location = maildir:/vmail/%d/public:INDEXPVT=~/index/%d/public prefix = Groups/ separator = / subscriptions = no type = public } namespace { hidden = yes list = no location = maildir:~/expunged prefix = .EXPUNGED/ separator = / subscriptions = no type = private } namespace inbox { inbox = yes list = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile:/etc/dovecot/global-acls acl_shared_dict = file:/var/lib/dovecot/db/shared-mailboxes.db lazy_expunge = .EXPUNGED/ lazy_expunge_only_last_instance = yes mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size mail_replica = tcps:mailhost1.rz.uni-hohenheim.de:12345 sieve = ~/.dovecot.sieve sieve_dir = ~/sieve zlib_save = gz zlib_save_level = 6 } protocols = imap pop3 lmtp sieve replication_dsync_parameters = -d -l 30 -U service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = vmail } unix_listener replication-notify { mode = 0666 user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0660 user = vmail } } service doveadm { inet_listener { port = 12345 ssl = yes } } service imap-login { process_min_avail = 16 service_count = 0 } service imap-postlogin { executable = script-login /usr/local/bin/postlogin.sh user = $default_internal_user } service imap { executable = imap imap-postlogin process_limit = 8192 } service lmtp { unix_listener /var/spool/postfix/private/lmtp-dovecot { group = postfix user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } } service pop3-login { process_min_avail = 16 service_count = 0 } service pop3 { process_limit = 8192 } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { group = vmail mode = 0660 } } ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_client_ca_file = /etc/openldap/cacerts/chain.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } verbose_proctitle = yes protocol lmtp { mail_plugins = " acl lazy_expunge mail_log notify zlib notify replication sieve" } protocol imap { mail_plugins = " acl lazy_expunge mail_log notify zlib notify replication imap_acl imap_zlib" }

1 1

New user mailbox permission
by Gionatan Danti 21 Aug '14

21 Aug '14

Hi all, I have a problem with my Dovecot installation. First, some informations: - OS: CentOS 6.5 x86_64 - Dovecot: dovecot-2.0.9-7.el6.x86_64 Config: mail_location = sdbox:~/dbox user_query = SELECT username, password, 'vmail' as uid, 'vmail' as gid, '/var/vmail/%d/%n' as home, CONCAT('*:storage=', (quota DIV 1000)) as quota_rule FROM mailbox WHERE username='%u' What I want to do is to create a backup user to rsync the mailstore to a remote location. If possible, I don't want to use the root user for this operation. I already added the backup user to "vmail" group. The problem is that when a new user is created, it's base directory permission are 0700 (vmail:vmail). From what I read here (http://wiki2.dovecot.org/SharedMailboxes/Permissions, "Permissions to new /domain/user directories"), simply using the setgid bit is not sufficient because I am not using an absolute path for mail_location. My questions are: 1) all I need to do is to switch to an absolute mail_location? 2) it is possibile to solve my problem _without_ altering mail_location? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti(a)assyoma.it - info(a)assyoma.it GPG public key ID: FF5F32A8

2 2

How to configure read-only access during a migration?
by Jeffrey Tunison 20 Aug '14

20 Aug '14

Hi. I would appreciate your advice about these TWO PROBLEMS: ======================================================= Find the simplest way to A) configure a Dovecot 2.2.10 IMAP server to access (existing) email users' (mbox-format) mail folders ****in read-only mode.**** B) Do the above while allowing a master_user to access mail on behalf of real users. (I've got master_user access working in read-write mode.) Simplest because while I'm proficient at UNIX systems administration, I'm under time pressure and dovecot has a lot of configuration flexibility. Identifying the right way to do this is proving time- consuming. Namespaces? ACL's? passdb's? I thought it was time to bring in the dovecot wizards. You! GOAL: ======================================================= Without knowing or changing users' passwords, migrate several hundred users of dovecot IMAP to MS Exchange using the commercial product Transend. I'm manging the dovecot side, someone else is doing the Transend/Exchange piece. Migration would occur in batches due to slowness of the Transend side when dealing with users with email collections over a few gigabytes. I have a production IMAP server (dovecot 1.0.13 on 32-bit RHEL 4.9). Let's call it PROD for short. Users INBOXes are stored locally on PROD. Their other mail folders are stored on a second machine and accessed via NFS. Those other folders contain the great majority of the email to be migrated. Let's call the NFS server MIGR for short. On MIGR, I've set up a migration IMAP server (dovecot 2.2.10 on 64-bit RHEL 5.10). MIGR is a dedicated NFS server with newer hardware than PROD. It has an unused Ethernet NIC I can tell Transend to use. Both PROD and MIGR use NIS for authentication. Therefore PAM is consulted for passdb information, normally. No SQL db's used. I set up the second server (MIGR) for several reasons: 1) there are some users mail folders larger than the 32-bit file size limit found on the PROD server. 2) the bulk of the email is stored locally on the MIGR server. 3) the version of dovecot is much newer and therefore more likely to have fewer bugs and more configuration options. 4) I don't want to modify the production environment except where really necessary. Pointing Transend at MIGR instead of PROD seems like it will be faster and better. Telling end-users to use MIGR instead of PROD would allow them to verify that their email came across okay (into MS Exchange) without being able to modify the original copy of email (in Dovecot). In the days and weeks after migration. I may need to re-enable write access on PROD for certain users if the migration does not succeed. Transend has been dying on a number of test users despite tweaking its parameters in consultation with the vendor. "Corrupt" messages seem to be the issue. Users (or sysadmins) might be required to identify and delete the offending messages before we retry their migration. (Which folders and which messages are causing problems will hopefully be discernible in Transend logs.) Ha! If you've read this far, then I'd also say that I welcome any advice on changes to increase performance on MIGR, such as whether trying to minimize indexing as I've done is useful. Current configurations of the two servers follow. =========================================================================== PRODUCTION: =========================================================================== # 1.0.13: /etc/dovecot.conf log_path: /var/log/dovecot.log info_log_path: /var/log/dovecot.log protocols: imaps imap listen: *:143 ssl_listen: *:993 ssl_cert_file: /etc/pki/dovecot/certs/dovecot.pem ssl_key_file: /etc/pki/dovecot/private/dovecot.key login_dir: /var/run/dovecot/login login_executable: /usr/libexec/dovecot/imap-login login_processes_count: 20 login_max_processes_count: 500 max_mail_processes: 1524 first_valid_uid: 100 mail_location: mbox:~/Mail:INBOX=/var/mail/%u:INDEX=/var/spool/dovecot-indexes/%u/ mbox_read_locks: dotlock mbox_write_locks: dotlock mbox_dotlock_change_timeout: 60 mbox_min_index_size: 2048 namespace: type: private separator: / inbox: yes namespace: type: private separator: / prefix: Mail/ hidden: yes namespace: type: private separator: / prefix: ~/Mail/ hidden: yes namespace: type: private separator: / prefix: mail/ hidden: yes namespace: type: private separator: / prefix: ~/mail/ hidden: yes auth default: master_user_separator: * verbose: yes debug: yes debug_passwords: yes passdb: driver: passwd-file args: /etc/dovecot.transend master: yes passdb: driver: shadow passdb: driver: pam userdb: driver: passwd =========================================================================== MIGRATION: =========================================================================== # 2.2.10: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-348.18.1.el5 x86_64 Scientific Linux release 5.10 (Boron) auth_debug = yes auth_debug_passwords = yes auth_master_user_separator = * auth_verbose = yes auth_verbose_passwords = plain first_valid_uid = 100 log_path = /var/log/dovecot.log mail_debug = yes mail_location = mbox:~/Mail:INBOX=/var/spool/concmail/%u:INDEX=MEMORY mbox_min_index_size = 50048 B mbox_write_locks = fcntl namespace four { hidden = yes location = prefix = ~/mail/ separator = / } namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } namespace one { hidden = yes location = prefix = Mail/ separator = / } namespace three { hidden = yes location = prefix = mail/ separator = / } namespace two { hidden = yes location = prefix = ~/Mail/ separator = / } passdb { args = /etc/dovecot/dovecot.transend driver = passwd-file master = yes } passdb { driver = pam } protocols = imap service imap-login { process_min_avail = 15 service_count = 10 } service imap { process_limit = 2048 } ssl = required ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { driver = passwd } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 1000 }

1 0

LMTP re-reading all messages
by Florent Bautista 20 Aug '14

20 Aug '14

Hi all, I have a little problem with Dovecot 2.1.17 and MailDir accounts. I was using a shared file system (MooseFS), and all was pretty stable. Now I'm switching to CephFS, and Dovecot LMTP have problem to deliver mails. Very often, on large accounts, LMTP processes are running for a few hours before delivering the message. When I strace it, I can see that they are reading (open and stat operations) ALL the messages of the account. I was thinking to an index problem, but I think that after hundreds of delivered mails, indexes are rebuilded and problem should be solved. But not. The same way, If user connects to IMAP, a lot of IMAP processes get stuck on "getquotaroot" command (multiple processes for a single account). The previous FS didn't support mmap, so I disabled it. The new one supports it, so I enabled it, but it does not solve the problem. Do you have an idea of why Dovecot (LMTP or IMAP) spend a lot of time on large accounts ? Thank you a lot. Florent

2 2

antispam plugin crash
by Thomas Witt 20 Aug '14

20 Aug '14

Hi, the dovecot antispam plugin crahses here on a nullpointer dereference in antispam_copy() at mailbox.c The crash tracks back to a call of antispam_classify_copy(asms->box_class, asmb->box_class) At that point in time asmb is set to 0x0, causing the process to crash with SIGSEGV. Regards, Thomas GDB output: Program received signal SIGSEGV, Segmentation fault. antispam_copy (ctx=0x14ed9f0, mail=0x14c6cc0) at mailbox.c:107 107 enum mailbox_copy_type copy_type = (gdb) bt #0 antispam_copy (ctx=0x14ed9f0, mail=0x14c6cc0) at mailbox.c:107 #1 0x00007fdadb627fcd in mailbox_copy () from /usr/lib64/dovecot/libdovecot-storage.so.0 #2 0x00007fdada6cdba0 in act_store_execute () from /usr/lib64/dovecot/libdovecot-sieve.so.0 #3 0x00007fdada6c59d9 in sieve_result_execute () from /usr/lib64/dovecot/libdovecot-sieve.so.0 #4 0x00007fdada6d4a2a in sieve_execute () from /usr/lib64/dovecot/libdovecot-sieve.so.0 #5 0x00007fdada9212cb in lda_sieve_deliver_mail () from /usr/lib64/dovecot/lib90_sieve_plugin.so #6 0x00007fdadb8cae1a in mail_deliver () from /usr/lib64/dovecot/libdovecot-lda.so.0 #7 0x0000000000405ebd in client_input_data_handle () #8 0x00007fdadb34e9bf in io_loop_call_io () from /usr/lib64/dovecot/libdovecot.so.0 #9 0x00007fdadb34f8b7 in io_loop_handler_run_internal () from /usr/lib64/dovecot/libdovecot.so.0 #10 0x00007fdadb34ea29 in io_loop_handler_run () from /usr/lib64/dovecot/libdovecot.so.0 #11 0x00007fdadb34eaa8 in io_loop_run () from /usr/lib64/dovecot/libdovecot.so.0 #12 0x00007fdadb2fe803 in master_service_run () from /usr/lib64/dovecot/libdovecot.so.0 #13 0x0000000000404995 in main () (gdb) p asms $3 = (struct antispam_mailbox *) 0x0

3 9