- dovecot - dovecot.org

[Dovecot] What is normal CPU usage of dovecot imap?
by Mikko Lampikoski 03 Jan '12

03 Jan '12

I got Dual Core Intel Xeon CPU 3.00GHz, over 1000 mailbox and almost 1 dovecot login / second (peak time). Server stats says that load is continually over 2 and cpu usage is 60%. top says that imap is making this load. virtual users are in mysql database and mysqld is running on another server (this server is ok). Do I need better CPU or is there something going on that I do not understand? # dovecot -n # 1.1.11: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-4-pve i686 Ubuntu 9.10 nfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s listen: *, [::] ssl_ca_file: /etc/ssl/**********.crt ssl_cert_file: /etc/ssl/**********.crt ssl_key_file: /etc/ssl/**********.key ssl_key_password: ********** disable_plaintext_auth: no verbose_ssl: yes shutdown_clients: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(pop3): no login_process_size: 128 login_processes_count: 10 login_max_processes_count: 2048 mail_max_userip_connections(default): 10 mail_max_userip_connections(imap): 10 mail_max_userip_connections(pop3): 3 first_valid_uid: 99 last_valid_uid: 100 mail_privileged_group: mail mail_location: maildir:/var/vmail/%d/%n:INDEX=/var/indexes/%d/%n fsync_disable: yes mail_nfs_storage: yes mbox_write_locks: fcntl mbox_min_index_size: 4 mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_process_size: 2048 mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 imap_client_workarounds(default): outlook-idle imap_client_workarounds(imap): outlook-idle imap_client_workarounds(pop3): pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls auth default: mechanisms: plain login cram-md5 cache_size: 1024 passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: static args: uid=99 gid=99 home=/var/vmail/%d/%n allow_all_users=yes socket: type: listen client: path: /var/spool/postfix/private/auth-client mode: 432 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 384 user: vmail

2 3

[Dovecot] Compressing existing maildirs
by Jan-Frode Myklebust 03 Jan '12

03 Jan '12

I've just enabled zlib for our users, and am looking at how to compress the existing files. The routine for doing this at http://wiki2.dovecot.org/Plugins/Zlib seems a bit complicated. What do you think about simply doing: find /var/vmail -type f -name "*,S=*" -mtime +1 -exec gzip -S Z -6 '{}' + I.e. find all maildir-files: - with size in the name ("*,S=*") - modified before I enabled zlib plugin - compress them - add the Z suffix - keep timestamps (gzip does that by default) It's of course racy without the maildirlock, but are there any other problems with this approach ? -jf

4 14

[Dovecot] Dsync fails on second sync for folders with dot in the name
by Jan-Frode Myklebust 03 Jan '12

03 Jan '12

I'm in the processes of running our first dsync backup of all users (from maildir to mdbox on remote server), and one problem I'm hitting that dsync will work fine on first run for some users, and then reliably fail whenever I try a new run: $ sudo dsync -u janfrode(a)example.net backup ssh -q mailbackup(a)repo1.example.net dsync -u janfrode(a)example.net $ sudo dsync -u janfrode(a)example.net backup ssh -q mailbackup(a)repo1.example.net dsync -u janfrode(a)example.net dsync-remote(janfrode(a)example.net): Error: Can't delete mailbox directory INBOX/a: Mailbox has children, delete them first The problem here seems to be that this user has a maildir named ".a.b". On the backup side I see this as "a/b/". So dsync doesn't quite seem to agree with itself for how to handle folders with dot in the name. -jf

3 7

[Dovecot] lmtp-postlogin ?
by Jan-Frode Myklebust 03 Jan '12

03 Jan '12

We have last-login tracking for imap and pop, and I intend to use this for deciding which users to backup daily. But, it would also be nice to backup users who has only received messages, but not logged in lately.. So is it possible to implement "last-login" tracking for lmtp ? I naively tried copying the settings from imap, but it didn't work: service lmtp-postlogin { executable = script-login /usr/local/sbin/lmtp-postlogin.sh unix_listener lmtp-postlogin { } } service lmtp { executable = lmtp lmtp-postlogin <snip> -jf

2 3

[Dovecot] Newbie: LDA Isn't Logging
by Michael Papet 03 Jan '12

03 Jan '12

Hi, I'm a newbie having some trouble getting deliver to log anything. Related to this, there are no return values unless the -d is missing. I'm using LDAP to store virtual domain and user account information. Test #1: /usr/lib/dovecot/deliver -e -f mpapet(a)yahoo.com -d zed(a)mailswansong.dom < bad.mail Expected result: supposed to fail, there's no zed account via ldap lookup and supposed to get a return code per the wiki at http://wiki2.dovecot.org/LDA. Supposed to log too. Actual result: nothing gets delivered, no return code, nothing is logged. Test #2: /usr/lib/dovecot/deliver -f mpapet(a)yahoo.com -d dude(a)mailswansong.dom < good.mail Expected result: deliver to dude and return 0. Actual result: delivers, but no return code. Nothing logged. The wiki is vague about the difficulties of getting deliver LDA to log, but I thought I had it covered in my config. I even opened permissions up wide (777) on my log files specified below. Nothing gets logged. The ONLY thing changed in 15-lda.conf is as follows. protocol lda { # Space separated list of plugins to load (default is global mail_plugins). #mail_plugins = $mail_plugins log_path = /var/log/dovecot/lda.log info_log_path = /var/log/dovecot/lda-info.log service auth { unix_listener auth-client { mode = 0600 user = vmail } } I'm running plain Debian Testing and used dovecot from Debian's repository. The end-goal is to write a qpsmtpd queue plugin, but I need to figure out what's the matter first. Thanks in advance. mpapet

2 1

[Dovecot] dsync / separator / namespace config-problem
by Jan-Frode Myklebust 03 Jan '12

03 Jan '12

Guess I don't quite get namespaces, but we have the following namespace configured: $ doveconf namespace namespace { hidden = no inbox = yes list = yes location = prefix = INBOX. separator = subscriptions = yes type = private } But when I try running dsync it complains: $ sudo -u atmail dsync -u janfrode(a)tanso.net mirror mdbox:/tmp/janfrodeboks dsync(atmail): Error: user janfrode(a)tanso.net: Initialization failed: namespace configuration error: list=yes requires prefix=INBOX. to end with separator dsync(atmail): Fatal: User init failed I can fix this by adding "separator = ." to the namespace and dsync is happy, but is changing this now safe, or will this affect existing clients? -jf

2 5

[Dovecot] Trouble with proxy_maybe and auth_default_realm
by Gedalya 01 Jan '12

01 Jan '12

Hello, I'm using proxy_maybe and auth_default_realm. It seems that when a user logs in without the domain name, relying on auth_default_realm, and the "host" field points to the local server, I get the Proxying loops to itself error. It does work as expected - log on to the local server without proxying, if the user does include the domain name in the login. (IP's and domain name masked below) No domain: Dec 29 11:49:07 imap01 dovecot: pop3-login: Error: Proxying loops to itself: user=<jedi(a)----.com>, method=PLAIN, rip=00.00.52.18, lip=00.00.241.140 Dec 29 11:49:27 imap01 dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<jedi(a)----.com>, method=PLAIN, rip=00.00.52.18, lip=00.00.241.140 With domain: Dec 29 11:52:13 imap01 dovecot: pop3-login: Login: user=<jedi(a)---.com>, method=PLAIN, rip=00.00.52.18, lip=00.00.241.140, mpid=19969 Dec 29 11:52:18 imap01 dovecot: pop3(jedi(a)---.com): Disconnected: Logged out top=0/0, retr=0/0, del=0/1, size=731 Otherwise, e.g. when the proxy host is indeed another host, auth_default_domain works fine, including or not including the domain seems to make no difference, and everything works. I'm using mysql, and I'm able to get around this problem including the following in the password query: IF(host='' or host='00.00.241.140', NULL, 'Y') as proxy_maybe # dovecot --version 2.0.15 # dovecot -n # 2.0.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.3 auth_default_realm = ----.com auth_mechanisms = plain login cram-md5 ntlm auth_username_format = %Lu auth_verbose = yes auth_verbose_passwords = plain dict { quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no login_greeting = How can I help you? mail_gid = vmail mail_uid = vmail passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocols = imap pop3 lmtp service lmtp { inet_listener lmtp { address = 0.0.0.0 port = 7025 } } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } verbose_proctitle = yes ----- dovecot-sql.conf.ext ---- driver = mysql connect = host=localhost dbname=email user=email default_pass_scheme = PLAIN password_query = SELECT password, \ IF('%s' = 'pop3', host_pop3, host) as host, \ IF(host='' or host='00.00.241.140', NULL, 'Y') as proxy_maybe, \ concat(userid, '@', domain) as destuser, \ password as pass, \ '/stor/mail/domains/%d/%n' AS userdb_home, \ 'maildir:/stor/mail/domains/%d/%n/Maildir' as userdb_mail, \ concat('*:storage=', quota_mb, 'M') as userdb_quota_rule, \ 'vmail' AS userdb_uid, 'vmail' AS userdb_gid \ FROM email WHERE userid = '%n' AND domain = '%d' user_query = SELECT '/stor/mail/domains/%d/%n' AS home, \ 'maildir:/stor/mail/domains/%d/%n/Maildir' as mail, \ concat('*:storage=', quota_mb, 'M') as quota_rule, \ 'vmail' AS uid, 'vmail' AS gid \ FROM email WHERE userid = '%n' AND domain = '%d'

1 1

[Dovecot] imap process limits problem
by Calvin Cochran 01 Jan '12

01 Jan '12

I am having a problem with the number of current processes that I cannot seem to diagnose adequately, and is a possible bug. This will be a bit long, but usually more info is better. I am running dovecot 2.0.16 on a CentOS 5 x86_64 server with the mailstore on gfs (output from dovecot -n at bottom). This is an imap issue. This is mostly to do with one client, but none of my tests indicate an issue with the client side. We have mail_max_userip_connections = 10 for imap, but they are not hitting the limit. We also have verbose_proctitle = yes to help in diagnosing the situation. Most of our clients, including this one, use SSL or TLS and connect on 993. As I understand it, that should have an imap-login process and an imap process per authenticated session. Based on some other diagnosis the client seems to have a PC using Outlook 2010 and an i-device (phone or pad, not sure), both on the office network, and both with imap connections to the server. Based on my analysis it seems like the client is connecting, authenticating, and then closing the session, but the imap-login process does not drop until it times out server side (I don't know a way to tell which device, the PC or i???). One odd thing is that the tcp sessions time out at 2 hours and 11 minutes (this is where the possible bug aspect comes in). I have put a strace on the process, and there does not appear to be any traffic, so I don't understand why the 30 min timeout isn't happening. Based on netstat and verbose_proctitle, at this moment there are 99 connections from the IP in question, all of which show in ps output as: dovecot/imap-login [1 connections (1 TLS)] My understanding is that means they have successfully authenticated, and that there should be line with dovecot/imap [username ip TLS] in ps output, but there isn't, so I am taking that to mean the client closed the imap session. The client ip address puts them on comcast (tcp resets?) and we do have a load balancer in front of two servers, just to add a little challenge to the diagnosis fun. The short term fix has been to increase the process limits. However, it is clearly not a workable solution to increase the limits by 100 every time someone starts accessing the server with their new i??? device. I appreciate your thoughts on this, and I am happy to provide additional useful debug info if I have missed something. Thanks, Calvin output from dovecot -n auth_mechanisms = plain apop cram-md5 auth_verbose = yes auth_verbose_passwords = plain auth_worker_max_count = 400 default_process_limit = 400 disable_plaintext_auth = no first_valid_uid = 89 last_valid_uid = 89 login_greeting = EMCS ready. mail_fsync = never maildir_very_dirty_syncs = yes mmap_disable = yes passdb { driver = vpopmail } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size mail_log_group_events = no } protocols = imap pop3 shutdown_clients = no ssl_cert = </var/qmail/control/servercert.pem ssl_key = </var/qmail/control/servercert.pem userdb { driver = vpopmail } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 10 } protocol pop3 { mail_max_userip_connections = 3 pop3_no_flag_updates = yes pop3_uidl_format = %f }

4 4

[Dovecot] fts_solr break-imap-search 2.1
by Daniel L. Miller 31 Dec '11

31 Dec '11

Is "break-imap-search" still required using 2.1 with Solr to index TEXT & BODY searches? -- Daniel

2 1

[Dovecot] crashes on 2.0.16
by Cor Bosman 31 Dec '11

31 Dec '11

Hey all, I upgraded some servers today from Debian Lenny to Debian Squeeze, and after the upgrade I started getting dovecot crashes. I was on 2.0.13 but got these there as well, and hoped upgrading to 2.0.16 would help. It didn't. Anyone have an idea? Cor Dec 18 23:32:21 userimap1 kernel: [263492.245530] __ratelimit: 122 callbacks suppressed Dec 18 23:32:21 userimap1 kernel: [263492.245536] imap[19665]: segfault at 7f108efb7f60 ip 00007f108ecd37e0 sp 00007fff73c3d658 error 5 in libc-2.11.2.so[7f108ec5d000+158000] Dec 18 23:32:21 userimap1 kernel: [263492.245663] BUG: Bad page map in process imap pte:00000001 pmd:09767067 Dec 18 23:32:21 userimap1 kernel: [263492.252563] page:ffffea0000000000 flags:(null) count:-84 mapcount:-84 mapping:(null) index:0 Dec 18 23:32:21 userimap1 kernel: [263492.261186] addr:00007f108ef46000 vm_flags:08000070 anon_vma:(null) mapping:ffff88012b6e5238 index:2e9 Dec 18 23:32:21 userimap1 kernel: [263492.270804] vma->vm_ops->fault: filemap_fault+0x0/0x460 Dec 18 23:32:21 userimap1 kernel: [263492.276172] vma->vm_file->f_op->mmap: generic_file_mmap+0x0/0x60 Dec 18 23:32:21 userimap1 kernel: [263492.282424] Pid: 19665, comm: imap Tainted: G B 2.6.32.36-xsserver #1 Dec 18 23:32:21 userimap1 kernel: [263492.282429] Call Trace: Dec 18 23:32:21 userimap1 kernel: [263492.282441] [<ffffffff81103ee5>] print_bad_pte+0x1d5/0x280 Dec 18 23:32:21 userimap1 kernel: [263492.282448] [<ffffffff8110568a>] unmap_vmas+0xa1a/0xa20 Dec 18 23:32:21 userimap1 kernel: [263492.282458] [<ffffffff8137e501>] ? sch_direct_xmit+0x81/0x1d0 Dec 18 23:32:21 userimap1 kernel: [263492.282465] [<ffffffff8110b60d>] exit_mmap+0xbd/0x190 Dec 18 23:32:21 userimap1 kernel: [263492.282473] [<ffffffff81059cbd>] mmput+0x4d/0x120 Dec 18 23:32:21 userimap1 kernel: [263492.282481] [<ffffffff8105f282>] exit_mm+0x112/0x150 Dec 18 23:32:21 userimap1 kernel: [263492.282488] [<ffffffff8106104d>] do_exit+0x13d/0x800 Dec 18 23:32:21 userimap1 kernel: [263492.282496] [<ffffffff81072f31>] ? __dequeue_signal+0xe1/0x200 Dec 18 23:32:21 userimap1 kernel: [263492.282503] [<ffffffff81061765>] do_group_exit+0x55/0xd0 Dec 18 23:32:21 userimap1 kernel: [263492.282510] [<ffffffff81075251>] get_signal_to_deliver+0x1e1/0x3e0 Dec 18 23:32:21 userimap1 kernel: [263492.282517] [<ffffffff8100b315>] do_signal+0x75/0x7d0 Dec 18 23:32:21 userimap1 kernel: [263492.282523] [<ffffffff81103203>] ? print_vma_addr+0xb3/0x120 Dec 18 23:32:21 userimap1 kernel: [263492.282533] [<ffffffff8143856e>] ? printk+0x41/0x43 Dec 18 23:32:21 userimap1 kernel: [263492.282541] [<ffffffff81036436>] ? __bad_area_nosemaphore+0xf6/0x1e0 Dec 18 23:32:21 userimap1 kernel: [263492.282547] [<ffffffff810365ee>] ? bad_area_access_error+0x4e/0x60 Dec 18 23:32:21 userimap1 kernel: [263492.282553] [<ffffffff8100bac7>] do_notify_resume+0x57/0x60 Dec 18 23:32:21 userimap1 kernel: [263492.282559] [<ffffffff8100cadc>] retint_signal+0x48/0x8c # 2.0.16: /usr/local/etc/dovecot/dovecot.conf # OS: Linux 2.6.32.36-xsserver x86_64 Debian 6.0.3 auth_cache_size = 1 M auth_verbose = yes base_dir = /var/run/dovecot/ default_vsz_limit = 2 G disable_plaintext_auth = no first_valid_uid = 20 lock_method = dotlock login_greeting = User-IMAP ready. login_trusted_networks = 194.109.26.128/27 mail_fsync = always mail_nfs_index = yes mail_nfs_storage = yes mailbox_idle_check_interval = 1 mins maildir_stat_dirs = yes mmap_disable = yes namespace { inbox = yes location = maildir:%Nu:INDEX=/var/spool/mail/dovecot-control/indexes/%1u/%2u/%u:INBOX=%Nu:CONTROL=/var/spool/mail/dovecot-control/%1u/%2u/%u/INBOX prefix = separator = / type = private } passdb { args = /usr/local/etc/dovecot/dovecot.masterusers driver = passwd-file master = yes } passdb { args = cache_key=%u%r dovecot driver = pam } plugin { quota = fs:User quota } protocols = imap service auth { client_limit = 5000 } service imap-login { process_min_avail = 8 service_count = 0 vsz_limit = 2 G } service imap { process_limit = 4096 vsz_limit = 2 G } shutdown_clients = no ssl_cert = </etc/ssl/certs/xx ssl_key = </etc/ssl/private/xx userdb { driver = passwd } verbose_proctitle = yes protocol imap { imap_capability = IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS imap_client_workarounds = delay-newmail mail_max_userip_connections = 24 mail_plugins = " quota imap_quota" }

16 30