May 2006 - dovecot - dovecot.org

[Dovecot] Server settings for a busy server
by Pat Hastings 11 Jun '06

11 Jun '06

Hi, We are migrating from a windows server and i have very few stats as to what our current concurrent usage stats are other than about 50,000 active users and 90,000 email messages a day inbound 20,000 outbound. Anyone got some tips for tuning dovecot to handle this load? Below is my config - do the values of login_process_size etc sound correct? Also my logs seem to be written out as root which seems kind of scary do how do I / should I have these written out by the dovecot user. Any feedback greatfully accepted. Regards, Pat protocols = imap pop3 listen = * ssl_disable = no disable_plaintext_auth = no shutdown_clients = yes log_path = /var/log/dovecot/mainlog info_log_path = /var/log/dovecot/infolog log_timestamp = "%Y-%m-%d %H:%M:%S " login_dir = /var/run/dovecot/login login_chroot = yes login_user = dovecot login_process_size = 256 login_process_per_connection = no login_processes_count = 48 login_max_logging_users = 2048 login_greeting = Email server ready. login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c login_log_format = %$: %s first_valid_uid = 100 mail_extra_groups = mail valid_chroot_dirs = /var/vmail mail_debug = no default_mail_env = maildir:/var/vmail/dovecoterror/%domain/%u maildir_copy_with_hardlinks = yes protocol imap { } protocol pop3 { pop3_reuse_xuidl = no pop3_uidl_format =%u pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_client_workarounds = outlook-no-nuls, oe-ns-eoh } auth_executable = /usr/lib/dovecot/dovecot-auth auth_verbose = yes auth_debug = yes auth_debug_passwords = yes auth default { mechanisms = plain passdb pam { } passdb ldap { args = /etc/dovecot/dovecot-ldap.conf } userdb passwd { } userdb ldap { args = /etc/dovecot/dovecot-ldap.conf } user = root } --- [This E-mail has been scanned for viruses but it is your responsibility to maintain up to date anti virus software on the device that you are currently using to read this email. ]

2 1

[Dovecot] Dovecot 1.0beta8 dovecot-auth consumes 100% CPU time on Solaris 10 amd64
by Tore André Klock 11 Jun '06

11 Jun '06

Hello. I hope someone out there can help with this. It is getting pretty urgent. I am running a Solaris 10 server on Opteron (amd64) hardware and have compiled Dovecot 1.0beta8 from source. It has openssl compiled in (after much mucking around with various environment variables and modifying the Makefile), and was built with: $ ./configure --with-ldap --with-ssl-dir=/etc/ssl $ make # make install The variables: CC=/usr/sfw/bin/gcc CFLAGS='-m64 -I/usr/local/ssl/include:/usr/sfw/include:/opt/sfw/include' CPP=/usr/sfw/bin/cpp CPPFLAGS=-I/usr/local/ssl/include/openssl:/usr/sfw/include:/opt/sfw/include CXX=/usr/sfw/bin/g++ LDFLAGS='-m64 -L/usr/local/ssl/lib -R/usr/local/ssl/lib:/usr/sfw/lib:/opt/sfw/lib' LD_LIBRARY_PATH=/usr/local/ssl/lib:/usr/sfw/lib:/opt/sfw/lib It was compiled with gcc version 3.4.3 (csl-sol210-3_4-branch+sol_rpath). I modified the SSL_CFLAGS and SSL_LIBS variables in the Makefile to make it find my openssl install in /usr/local/ssl. The one that came with the OS was broken[1], so I compiled 0.9.8b from source. The configure script for some reason didn't pick this up, even when using CFLAGS, LDFLAGS, etc. At any rate, I successfully set it up to accept SSL/TLS connections and authenticate against an LDAP server. I use the auth_bind authentication method and only one LDAP host. I have confirmed that this works. In fact, dovecot runs just fine with the exception of the dovecot-auth proces, which is thrown into an endless loop. It starts off at maybe 1-2% CPU time, but after a few minutes it is back up to 100%. This is the output from truss(1) pointed at the dovecot-auth process: --cut-- pollsys(0xFFFFFD7FFFDFF6F0, 5, 0xFFFFFD7FFFDFF6D0, 0x00000000) = 0 pollsys(0xFFFFFD7FFFDFF6F0, 5, 0xFFFFFD7FFFDFF6D0, 0x00000000) = 0 pollsys(0x00467DC0, 10, 0xFFFFFD7FFFDFFB10, 0x00000000) = 1 [same sequence repeats ad nauseum] --cut-- This is a sample of top(1) output: --cut-- load averages: 1.00, 1.00, 1.00 08:51:38 60 processes: 55 sleeping, 3 running, 1 zombie, 1 on cpu CPU states: 0.0% idle, 68.3% user, 31.7% kernel, 0.0% iowait, 0.0% swap Memory: 1024M real, 680M free, 71M swap in use, 591M swap free PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND 22875 nobody 1 0 0 0K 0K run 726:22 99.68% dovecot-auth --cut-- Here is a condensed version of my ldap-config: (comments removed, names changed to protect the innocent) --cut-- hosts = ldap.example.com auth_bind = yes auth_bind_userdn = uid=%u,ou=People,o=example.com,o=example.com ldap_version = 3 base = uid=%,ou=People,o=example.com,o=example.com --cut-- This is my condensed dovecot.conf: --cut-- base_dir = /var/run/dovecot/ protocols = imap pop3 listen = * ssl_disable = no ssl_cert_file = /etc/ssl/certs/host.cert ssl_key_file = /etc/ssl/private/host.key ssl_cipher_list = ALL:!LOW disable_plaintext_auth = no shutdown_clients = yes log_path = /var/log/dovecot log_timestamp = "%b %d %H:%M:%S " login_user = dovecot login_process_per_connection = yes verbose_proctitle = yes verbose_ssl = yes first_valid_uid = 100 last_valid_uid = 60000 mail_debug = yes default_mail_env = maildir:/var/mail/%u/Maildir mail_full_filesystem_access = no protocol imap { login_executable = /usr/local/libexec/dovecot/imap-login mail_executable = /usr/local/libexec/dovecot/imap login_greeting_capability = yes imap_client_workarounds = outlook-idle tb-extra-mailbox-sep } protocol pop3 { login_executable = /usr/local/libexec/dovecot/pop3-login mail_executable = /usr/local/libexec/dovecot/pop3 pop3_uidl_format = %08Xu%08Xv pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s mail_plugin_dir = /usr/local/lib/dovecot/pop3 pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } auth_executable = /usr/local/libexec/dovecot/dovecot-auth auth_cache_size = 0 auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ auth_verbose = yes auth_debug = no auth default { mechanisms = plain passdb ldap { args = /usr/local/etc/dovecot-ldap.conf } userdb passwd { } user = nobody } --cut-- I may have commented out more than necessary in an attempt to be explicit while I was trying to narrow this down. This is the output of ldd dovecot-auth: --cut-- libpam.so.1 => /lib/64/libpam.so.1 libldap.so.5 => /usr/lib/64/libldap.so.5 libsocket.so.1 => /lib/64/libsocket.so.1 libnsl.so.1 => /lib/64/libnsl.so.1 librt.so.1 => /lib/64/librt.so.1 libsendfile.so.1 => /lib/64/libsendfile.so.1 libc.so.1 => /lib/64/libc.so.1 libcmd.so.1 => /lib/64/libcmd.so.1 libsasl.so.1 => /usr/lib/64/libsasl.so.1 libmd5.so.1 => /lib/64/libmd5.so.1 libnspr4.so => /usr/lib/mps/64/libnspr4.so libplc4.so => /usr/lib/mps/64/libplc4.so libnss3.so => /usr/lib/mps/64/libnss3.so libssl3.so => /usr/lib/mps/64/libssl3.so libmp.so.2 => /lib/64/libmp.so.2 libscf.so.1 => /lib/64/libscf.so.1 libaio.so.1 => /lib/64/libaio.so.1 libpthread.so.1 => /lib/64/libpthread.so.1 libthread.so.1 => /lib/64/libthread.so.1 libdl.so.1 => /lib/64/libdl.so.1 libsoftokn3.so => /usr/lib/mps/amd64/libsoftokn3.so libplds4.so => /usr/lib/mps/amd64/libplds4.so libdoor.so.1 => /lib/64/libdoor.so.1 libuutil.so.1 => /lib/64/libuutil.so.1 libm.so.2 => /lib/64/libm.so.2 --cut-- Notes: - The problem *seems* to occur only when I use LDAP authentication. - Disabling SSL in the config has no effect on the CPU usage. - I am not using NFS for anything. - I use Thunderbird 1.5.0.2 and another client runs Outlook. - There is no indication of any problem in the dovecot or system logs. - I suspected this had to do with the poll method dovecot is compiled with (poll), but it happens even when I compiled it with the select method. As far as I know, none of the other methods are available on Solaris. - I noted that the poll(2) call in Solaris will return immediately given a timeout of zero (-1 is 'forever'). I tried tacking down the use of that system call, but I had to give up on that. I only mention it in case it is a dovecot bug and of any help to Timo. Any help is greatly appreciated. Thank you. - Tore [1] If you use the factory-installed version, you get messages like "Warning: imap-login: SSL_accept() failed: error:1409D08A:SSL routines:SSL3_SETUP_KEY_BLOCK:cipher or hash unavailable" in your dovecot log.

2 7

[Dovecot] vpopmail
by C J Kenneth Tan -- OptimaNumerics 06 Jun '06

06 Jun '06

I was running 1.0.beta3 code with no problems. I recently switched to 1.0.beta8 and I noticed that vpopmail's pop-before-smtp functionality stopped working. I am using vpopmail authentication. Has anyone else encountered this problem? Thank you. Kenneth Tan ----------------------------------------------------------------------- C J Kenneth Tan, PhD OptimaNumerics Ltd Telephone: +44 798 941 7838 E-mail: cjtan(a)OptimaNumerics.com Telephone: +44 207 099 4428 Web: http://www.OptimaNumerics.com Facsimile: +44 207 100 4572 -----------------------------------------------------------------------

8 24

[Dovecot] imaps multiple domains proposal
by Jim Flowers 04 Jun '06

04 Jun '06

There was a thread earlier this year that noted a difficulty in modifying dovecot to handle multiple domain certificates. This is a proposal that may require only minor modifications. Virtual domains can be set up with a unique ip address for each server; mail.domain1.tld ###.###.###.1, mail.domain2.tld ###.###.###.2, etc. Users are instructed to enter 'mail.domain?.tld' for their imap server. When an SSL TCP connection is established on port 993, the server program will know the IP # that was used for the connection and can use getnameinfo to extract the domain name (%d). This can then be used to build a specific reference to the certificate to be used, prior to responding to CLIENT_HELLO. Something like /etc/ssl/%d/[certs|private]/dovecot.pem One technique would be to include this in multiple dovecot.conf listen parameters read on startup, similar to tpop3d: listen = ###.###.###.1:993(domain1.tld);tls=stls,\ /etc/ssl/domain1.tld/certs/dovecot.pem,\ /etc/ssl/domain1.tld/private/dovecot.pem listen = ###.###.###.2:993(domain2.tld);tls=stls,\ /etc/ssl/domain2.tld/certs/dovecot.pem,\ /etc/ssl/domain2.tld/private/dovecot.pem Alternatively, as virtual servers almost always use some form of lookup (in my case mysql) the certificate used could be determined at time of connection, similar to the way ssl_ca is handled. For example: user_query = SELECT '125'as uid, '125' as gid, maildir as home, '/etc/ssl/%d/certs/dovecot.pem' as cert, '/etc/ssl/%d/private/dovecot.pem as priv FROM mailbox WHERE username = '%u' In this manner the certificate returned would match the server ip/FQDN and can also be customized for the specific domain involved. Seeing their own domain name as the CN for both the certificate and CA goes a long way to establish credibility for most secure email users. Could this be done? -- Jim Flowers<jflowers(a)ezo.net>

4 5

Re[2]: [Dovecot] dovecot can't start!
by Oleg D. 02 Jun '06

02 Jun '06

MB> Hmmm... I was seeing a problem rather like this with 1.0beta8 under Solaris MB> with Sun's compiler: only it was with the IMAP protocol rather than POP. MB> The problem there was an assumption Tio had made that everyone was using a MB> C99-compatible compiler. Apparently this handles some aspects of variable MB> initialisation differently from earlier C compilers. MB> In my case adding a command-line option to the C compiler to flip it into MB> C99 mode seemed to solve the problem. The option for our compiler was MB> "-xc99" -- however if your compiler has such an option it may have a MB> different name. If you check the manual page for your C compiler you may MB> find an equivalent option to try. MB> However another solution was provided by Timo in the form of a simple patch MB> that made changes to a couple of source files. I hope Timo doesn't mind: MB> I'm re-sending his patch as an attachment to this message as it is tiny. MB> You could try applying it and rebuilding Dovecot. MB> Cheers, MB> Mike Brudenell But I use FreeBSD 4.11-RELEASE-p15 with native compiler. Really nobody has this problem on freebsd? And the main - Why I can't start dovecot without SSL :) Why I see in log "pop3-login: BUG: SSL initialization parameters not given while they should have been" even if I have "ssl_disable = yes" in dovecot.conf 1.0b3 works fine. This problem is main for me because I don't want to use SSL. -- Best regards, Oleg mailto:dzoleg@rsute.ru

4 6

[Dovecot] Outlook causes BIG dovecot index problem (beta 8, outlook 2003)
by Casey Allen Shobe 01 Jun '06

01 Jun '06

Hey all, Well, I typed up all the below, then managed to fix the problem before sending this to the list. I'll send it anyways, in case anybody else is affected by it (and because it does indicate a definite bug in Dovecot's experimental Kerberos support). The fix is to recompile without kerberos enabled, and to take gssapi out of the auth methods in the config (the latter alone may or may not be enough of a fix on it's own, I didn't try it without recompiling). I'm guessing that Outlook attempts to use Kerberos authentication by default or something and silently fails and uses other auth methods, and that any attempt to use Kerberos is in fact what hoses the server. However I have not done any further testing against beta8 with a Kerberos-enabled client such as KMail, so I leave this as an exercise for the next adventurer. I'm also uncertain as to why this problem does not manifest on our main servers, which also have Kerberos enabled (and Outlook 2003 clients). Or /had/ anyways, as I'm disabling Kerberos support as we speak as a preventative measure. Here's the problem description: I've been running dovecot happily for many months on our production mail servers hosting about 600 accounts, with no problems. People use every mail client under the sun - Apple Mail, Outlook 97|2000|XP|2003, Outlook Express, Mutt, Thunderbird, Mozilla Mail, Eudora, and other esoteric things with no problem. So today I installed a new mail server on a client site dedicated to their domain. The MTA is exim, running with almost the same configuration we use on our main servers. The dovecot.conf is 100% identical. No mail was moved by hand to the new mail server - to migrate we copied the mail setting up the old and new accounts in Thunderbird and dragging things over (just to rule out that possibility, we have also tested using brand new empty accounts to reproduce the issue). Everything is golden using Thunderbird - email is send, and shows up in the inbox immediately. Enter Outlook 2003 (SP1): It shows all folders and mail fine. It sends mail fine. Nothing shows up in the inbox. At this moment, Thunderbird also stops receiving new mail in the inbox. I check the server, and there's a mail sitting in the /new directory in the maildir. Not only that, but suddenly NONE of the accounts on the box show any new mail, exhibiting the same signs. I tried a variety of things, including 'sync', removing all the dovecot* index files in the maildir, and restarting dovecot (being sure to killall imap before bringing it back up). Sometimes, these steps would magically cause everything to work again, until I did a mail poll in Outlook which causes the problem to re-occur. Other times, these steps didn't seem to do anything. A couple times, for no apparent reason, without doing anything, mail would show up. Too hopeful, we sat around for 30 minutes sending more mail to an account and then just idling...nothing showed up. The hardware is good, we've compiled a full glibc and gcc just to ensure that there are no CPU/memory issues, and this box has a solid good history behind it in it's prior uses. Until Outlook is used, everything works fine. We do have the outlook-idle workaround enabled, and tried both with and without delay-newmail, which didn't make any difference. Cheers, -- Casey Allen Shobe | cshobe(a)seattleserver.com | 206-381-2800 SeattleServer.com, Inc. | http://www.seattleserver.com

3 3

[Dovecot] Cannot remove Maildir folder
by Marcin Deranek 01 Jun '06

01 Jun '06

Hi, I have been using dovecot 1.0 beta 8 on Debian Sarge for a couple of days and I experienced some problems when removing folders.. As an IMAP client I use Thunderbird 1.5 and IMAP folders are on NFS. Here are the steps to re-produce the problem: 1. Run Thunderbird as usual 2. Create folder 3. Remove folder - In my case folder is not removed but moved to Trash (Thunderbird setting) 4. I go to Trash and remove it from there 5. Folder is removed but I get additional ..DOVECOT-TRASHED directory - next time it won't be possible to remove it Some more info: * Just before removal I run lsof to see which files are in use: bash# lsof /home/marcin ... imap 21360 marcin cwd DIR 0,15 344064 3307008 /home/marcin (filer:/vol/vol0/home/marcin) imap 21360 marcin 8u DIR 0,15 4096 4223651 /home/marcin/Maildir/.Trash.test/new (filer:/vol/vol0/home/marcin) imap 21360 marcin 9u DIR 0,15 4096 4223650 /home/marcin/Maildir/.Trash.test/cur (filer:/vol/vol0/home/marcin) imap 21360 marcin 11r REG 0,15 144 4223663 /home/marcin/Maildir/.Trash.test/dovecot.index (filer:/vol/vol0/home/marcin) imap 21360 marcin 12r REG 0,15 104 4223647 /home/marcin/Maildir/.Trash.test/dovecot.index.log (filer:/vol/vol0/home/marcin) imap 21360 marcin 13u REG 0,15 10272 4223657 /home/marcin/Maildir/.Trash.test/dovecot.index.cache (filer:/vol/vol0/home/marcin) ... There are more files in use but in this case I'm interested with PID 21360 which has opened Trash/test (the one which I'm going to remove) * Just after removal I see error message in Syslog: dovecot: IMAP(marcin): unlink_directory(/home/marcin/Maildir/..DOVECOT-TRASHED) failed: Device or resource busy Running lsof again gives the following: imap 21360 marcin cwd DIR 0,15 344064 3307008 /home/marcin (filer:/vol/vol0/home/marcin) imap 21360 marcin 11r REG 0,15 144 4223663 /home/marcin/Maildir/..DOVECOT-TRASHED/.nfs004072af0002cb1b (filer:/vol/vol0/home/marcin) imap 21360 marcin 12r REG 0,15 104 4223647 /home/marcin/Maildir/..DOVECOT-TRASHED/.nfs0040729f0002cb1a (filer:/vol/vol0/home/marcin) imap 21360 marcin 13u REG 0,15 10272 4223657 /home/marcin/Maildir/..DOVECOT-TRASHED/.nfs004072a90002cb1c (filer:/vol/vol0/home/marcin) Looking into some NFS docs we can read: "A client has a file opened. Another process on that client deletes the file. So the client NFS stack rather than deleting the file and potentially hurting the other process, it renames the file to .nfs<file_handle>." To me it looks like one imap process tries to delete .Trash.test directory (first rename and then unlink) whereas other imap process has still opened dovecot.* files. Any workaround/fix for that ? Regards, Marcin Deranek

2 5

[Dovecot] beta8 starts but no POP3 access
by Obantec Support 31 May '06

31 May '06

Hi Timo i see you just answered a posting less than 1 hour old! any chance you can comment on mine (29/05/06) repeated below and very urgent i get dovecot beta8 up. I just compiled stock beta8 on FC3 with ./configure make make install all looked ok and i changed my dovecot.conf to suit. start dovecot and i see it running May 29 08:18:00 Info: Dovecot v1.0.beta8 starting up POP3 to 1 account and May 29 08:18:06 Info: auth(default): client in: AUTH 1 PLAIN service=POP3 lip=xx.xx.xx.xx rip=yy.yy.yy.yy resp=<hidden> May 29 08:18:06 Info: auth(default): client out: OK 1 user=obantec_com May 29 08:18:06 Info: auth(default): master in: REQUEST 1 19528 1 May 29 08:18:06 Info: auth(default): master out: USER 1 obantec_com system_user=obantec_com uid=501 gid=10001 home=/home/domain/domain17 May 29 08:18:06 Info: pop3-login: Login: user=<obantec_com>, method=PLAIN, rip=yy.yy.yy.yy, lip=xx.xx.xx.xx May 29 08:18:06 proteus2 dovecot: POP3(obantec_com): Corrupted index cache file /home/domain/domain17/mail/.imap/INBOX/dovecot.index.cache: indexid changed May 29 08:18:06 proteus2 dovecot: child 19536 (pop3) killed with signal 11 yy.yy.yy.yy & xx.xx.xx.xx are the ip's just masked out. OE just immediately say error , what is also worrying is the Corrupted index message even though i am using POP3. any ideas what to look for. I might add a test box running FC3 and the same .conf works! Mark -- Obantec Support www.obantec.net 0845 458 3121 WebHosting and Domains Nominet UK Member & IPStag Holder CentralNic Accredited Reseller

2 2

[Dovecot] dovecot.index.cache
by Fintec 31 May '06

31 May '06

Hopefully someone can explain/help me with the dovecot.index.cache file. I am wanting to move a dovecot server (v1.0b7) onto a new box running the same version (1.0b7) of dovecot. To do this I rsync'd the maildir files to the new server but when I logged into the new machine through IMAP dovecot reported: IMAP(username): file index-mail.c: line 105 (index_mail_get_fixed_field): assertion failed: (buffer_get_used_size(buf) == data_size) As soon as I delete the dovecot.index.cache file within the users mailbox this error goes away. Great, but that now I have some questions: 1) Does the dovecot.index.cache file store/cache server specific settings? If not why did it cause this error on the new box with exactly the same dovecot version and dovecot.conf? 2) What will happen if I delete the dovecot.index.cache file for all users? Will POP3 users have to re-download all their mail or will dovecot simply reference the dovecot.index file in the absence of the cache file? If I've missed something on the wiki explaining all this please point me in the right direction. Any help would be greatly appreciated. Gavin

3 4

[Dovecot] Apple Mail and too many open files?
by Alan Schmitt 31 May '06

31 May '06

Hello, For two days in a row, I've noticed that my dovecot server (hosted on a PowerMac running OS X) has stopped working, with a bunch of error messages in /var/log/mail.log like: May 24 07:31:10 top dovecot: pipe() failed: Too many open files To solve this, I have to kill -9 dovecot and the (many) imap processes running. The only client accessing the server is a machine running Apple Mail. I tried to increase the limit in the number of files (I did a "launchctl limit maxfiles 1024 unlimited", but the problem still happened. Is this a known problem? Should I allow still more files? Thanks, Alan -- Alan Schmitt <http://alan.petitepomme.net/> The hacker: someone who figured things out and made something cool happen. .O. ..O OOO

4 5