- dovecot - dovecot.org

[Dovecot] Indexing Performance Question (was tpop3d vs dovecot)
by Nate 06 Feb '07

06 Feb '07

Since posting the previous thread we've setup a new system (Opteron 2.0ghz, 1gb ram, Fedora 6) for testing. I am hoping somebody very familiar with the indexing portion of dovecot can shed some light on this for me. After much testing, I've come to one primary conclusion that dovecot is possibly unnecessarily scanning or reading files within the maildir directories. Take a mailbox for instance (my test one has 71570 emails in the cur/ folder so the process runs long enough to get data of what it's doing). Follow these steps: 1. First pop3 connection. With no index files whatsoever, it takes nearly a full 10 minutes at 100% io load to scan the 71570 files (748M) directory and build the index files from scratch which look like this: -rw------- 1 postfix postdrop 839K Feb 1 00:43 dovecot.index -rw------- 1 postfix postdrop 1.9M Feb 1 00:57 dovecot.index.cache -rw------- 1 postfix postdrop 560K Feb 1 00:43 dovecot.index.log -rw------- 1 postfix postdrop 3.1M Feb 1 00:43 dovecot-uidlist While it's building index i see a lot of these fly by in lsof pop3 2023 postfix 8r REG 8,3 14425 285947 /var/spool/mail/test.visp.net/natetest/cur/1128488047.V902I7d42e.s3.visp.net:2, No problem here, the disk can only go as fast as the disk can go and thats a *lot* of mail. Although, one curious thing here, it appears that dovecot is reading or at least scanning the entire contents of the files within the cur/ directory to generate an index. I assume this by watching my disk cache go from 100MB to 600MB. Is it necessary to do that or only read the top X lines of each file, is that even possible? 2. Second connection. Near instantaneous results from a login and LIST. Cannot perform an lsof fast enough to see what files it's opening. This step can be repeated with nearly the same results. *rebooted machine to clear all cache* 3. Third connection. It takes ~40 seconds to display a LIST after login. The index files are untouched; however, during the 40 seconds I'm waiting, I can catch a few of these in lsof. pop3 1971 postfix 10r REG 8,3 4196 205963 /var/spool/mail/test.visp.net/natetest/cur/1157863512.V902I1f3867.s3.visp.net:2,S This tells me even though there is no new email, it is re-scanning the cur/ directory somewhat. It doesn't take near as long as the initial index build, but certainly takes longer than it would for the system to read any or all of the listed index files above. Why would dovecot need to even enter cur/ if the index file is up to date? Is it verifying that the index file is up to date by re-reading some or all of the many files in cur/? If so, could it not get the info it needs by simply doing the equivalent of a ls, which would take considerably less time? The disk cache is only filled up an additional 30meg vs an additional 500meg from connection 1, so it's certainly not reading as much data. (also makes me wonder why the initial scan/index build could not use this lesser time consuming function if thats what it's doing). Additional information: Running strace on the initial indexing looks like this... ------------------------------------------- open("/var/spool/mail/test.visp.net/natetest/cur/1115929738.V902I67c13.test.visp.net:2,", O_RDONLY) = 8 fstat(8, {st_mode=S_IFREG|0600, st_size=4667, ...}) = 0 pread(8, "Return-Path: <mehl8nl3n8osm3r83a"..., 4096, 0) = 4096 pread(8, "zdik8dyrnffv6bbrue.itownshipno12"..., 4095, 4096) = 571 pread(8, "", 3524, 4667) = 0 close(8) = 0 open("/var/spool/mail/test.visp.net/natetest/cur/1115931742.V902I67c42.test.visp.net:2,", O_RDONLY) = 8 fstat(8, {st_mode=S_IFREG|0600, st_size=3881, ...}) = 0 pread(8, "Return-Path: <sender-5-8759012-9"..., 4096, 0) = 3881 pread(8, "", 215, 3881) = 0 close(8) = 0 ------------------------------------------- Running strace on the 3rd connection looks like this... ------------------------------------------- open("/var/spool/mail/test.visp.net/natetest/cur/1114490671.V902I6011f.test.visp.net:2,", O_RDONLY) = 10 fstat(10, {st_mode=S_IFREG|0600, st_size=5367, ...}) = 0 pread(10, "Return-Path: <qflfmq82vz8zg-u6ci"..., 4096, 0) = 4096 pread(10, "gs/rpoc_District6Spauldings.php?"..., 4095, 4096) = 1271 pread(10, "", 2824, 5367) = 0 close(10) = 0 open("/var/spool/mail/test.visp.net/natetest/cur/1114496568.V902I60bf5.test.visp.net:2,", O_RDONLY) = 10 fstat(10, {st_mode=S_IFREG|0600, st_size=2047, ...}) = 0 pread(10, "Return-Path: <removed(a)aaa-calif.c"..., 4096, 0) = 2047 pread(10, "", 2049, 2047) = 0 close(10) = 0 Thank you for the help! - Nate

8 27

[Dovecot] Problem with vpopmail auth
by Fred Moyer 06 Feb '07

06 Feb '07

Hello, I've install 1.021 and am having some problems getting vpopmail authentication to work correctly (running 0.99 in production, upgrading to 1.021). I'm using vpopmail-5.4.16. Here are the authentication debug statements in the log, my dovecot.conf changes from default, and the schema of my vpopmail database. Any ideas / suggestions? I've commented out all authentication except for vpopmail. I took a look around the source but am not that familiar with the vpopmail api. Thanks dovecot: Feb 05 22:23:31 Info: auth(default): client in: AUTH 1 PLAIN service=IMAP lip=10.0.0.2 rip=10.0.0.1 resp=<hidden> dovecot: Feb 05 22:23:31 Info: auth(default): vpopmail(fred(a)redhotpenguin.com,10.0.0.1): crypted password=rightpasswd dovecot: Feb 05 22:23:31 Info: auth(default): client out: OK 1 user=fred(a)redhotpenguin.com dovecot: Feb 05 22:23:31 Info: auth(default): master in: REQUEST 1 13457 1 dovecot: Feb 05 22:23:31 Info: auth(default): vpopmail(fred(a)redhotpenguin.com,10.0.0.1): vget_assign(redhotpenguin.com) failed dovecot: Feb 05 22:23:31 Info: auth(default): master out: FAIL 1 dovecot: Feb 05 22:23:31 Info: imap-login: Internal login failure: user=<fred(a)redhotpenguin.com>, method=PLAIN, rip=10.0.0.1, lip=10.0.0.2 852 # vpopmail authentication 853 # http://wiki.dovecot.org/AuthDatabase/VPopMail 854 passdb vpopmail { 855 # args = cache_key=%u dovecot 856 # [cache_key=<key>] - See cache_key in PAM for explanation. 857 # args = cache_key=%u 858 } 907 # vpopmail 908 # http://wiki.dovecot.org/AuthDatabase/VPopMail 909 userdb vpopmail { 910 } +-----------+-----------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------+-----------+------+-----+---------+-------+ | pw_name | char(32) | | PRI | | | | pw_domain | char(64) | | PRI | | | | pw_passwd | char(40) | YES | | NULL | | | pw_uid | int(11) | YES | | NULL | | | pw_gid | int(11) | YES | | NULL | | | pw_gecos | char(48) | YES | | NULL | | | pw_dir | char(160) | YES | | NULL | | | pw_shell | char(20) | YES | | NULL | | +-----------+-----------+------+-----+---------+-------+

1 1

[Dovecot] Big problems after upgrade from beta8 (FC5) to rc21
by Jim Garrison 06 Feb '07

06 Feb '07

After upgrading a working system from FC5's beta8 to rc21, one vhost is failing with: > dovecot: Feb 05 09:54:30 Error: POP3(user(a)mydomain.com): > open(/var/mail/vhosts/mydomain.com/david/user, O_CREAT) failed: Not a > directory > dovecot: Feb 05 09:54:30 Error: POP3(user(a)mydomain.com): > mkdir_parents(/var/mail/vhosts/mydomain.com/user/.imap/INBOX) failed: > Not a directory > dovecot: Feb 05 09:54:30 Error: POP3(user(a)mydomain.com): Couldn't open > INBOX: Internal error occurred. Refer to server log for more > information. [2007-02-05 09:54:30] > dovecot: Feb 05 09:54:30 Info: POP3(user(a)mydomain.com): No INBOX for > user top=0/0, retr=0/0, del=0/0, size=0 > dovecot: Feb 05 09:54:30 Info: pop3-login: Login: > user=<user(a)mydomain.com>, method=PLAIN, rip=::ffff:4.242.120.235, > lip=::ffff:216.75.55.45, TLS Here's the entire dovecot.conf: protocols = pop3s ssl_cert_file = /etc/pki/dovecot/certs/popcert.pem ssl_key_file = /etc/pki/dovecot/private/popkey.pem ssl_ca_file = /etc/pki/tls/certs/ca-bundle.crt ssl_parameters_regenerate = 48 disable_plaintext_auth = yes log_path = /var/log/dovecot.log mail_extra_groups = mail mbox_read_locks = fcntl mbox_write_locks = fcntl dotlock protocol imap { } protocol pop3 { } auth_verbose = no auth_debug = no auth_debug_passwords = no auth default { mechanisms = plain passdb passwd-file { args = /etc/dovecot/%d.passwd } passdb pam { } userdb passwd { } userdb static { args = uid=502 gid=502 home=/var/mail/vhosts/%d mail=mbox:/var/mail/vhosts/%d/%n } user = root } plugin { } Did something changed between beta8 and rc21 in the index format and/or mbox location? If so, how do I recover so users can retrieve mail again? Thx.

2 4

[Dovecot] LDAP auth_bind=yes in rc21 = goodness
by Adrian Close 06 Feb '07

06 Feb '07

Hi all, I finally got a chance to try out the fix for the LDAP auth_bind=yes issues in production (using rc21). It seems to be working just fine even after 24 hours of severe abuse. Incidentally, I had switched to bsdauth+login_ldap (on OpenBSD) which was often OK, but occasionally I would see Postfix smtpd throttled with SASL errors due to "connection refused" on the Dovecot authentication socket. This may just have been due to increased LDAP server load (Dovecot's internal LDAP generates much less). Thanks Timo! (And thanks Brad for updating the port) Adrian.

2 1

[Dovecot] minor kerberized dovecot build issue
by Dan Price 05 Feb '07

05 Feb '07

Hi all, again, I'm not knowledgeable about Kerberos, but we have a few users who use it to make sure that it works well on Solaris... so, they would like a Kerberized IMAPd. I tried compiling dovecot --with-gssapi on Solaris Nevada (the current development release), and the configure script choked for a couple of reasons. First, script uses 'grep -qv' in the case that you specify --with-gssapi to run some test. Unfortunately, Solaris doesn't have 'grep -q'. There's a note in GNU grep's manpage about this problem and a note that portable scripts should just redirect to /dev/null. Once I fixed this problem, I got: configure: WARNING: gssapi/gssapi.h: present but cannot be compiled configure: WARNING: gssapi/gssapi.h: check for missing prerequisite headers? configure: WARNING: gssapi/gssapi.h: see the Autoconf documentation configure: WARNING: gssapi/gssapi.h: section "Present But Cannot Be +Compiled" configure: WARNING: gssapi/gssapi.h: proceeding with the preprocessor's result configure: WARNING: gssapi/gssapi.h: in the future, the compiler will take +precedence configure: WARNING: ## ---------------------------------- ## configure: WARNING: ## Report this to dovecot(a)dovecot.org ## configure: WARNING: ## ---------------------------------- ## For whatever reason, on Solaris /usr/bin/krb5-config does not like the syntax 'krb5-config --cflags gssapi' (ditto with --libs). I found that removing the 'gssapi' part from the configure script fixed the problem, but I imagine this must break some other OS. If need be, I can follow up with the Kerberos team about this. Once I made these minor changes, I also had to add -lgss to the src/auth makefile's compile line. After that, I had no problem getting this to compile, although we have not yet tested it. -dp -- Daniel Price - Solaris Kernel Engineering - dp(a)eng.sun.com - blogs.sun.com/dp

2 3

[Dovecot] Nitpicking: rc21 startup banner says rc19
by Jim Garrison 05 Feb '07

05 Feb '07

> [jhg@helios RPMS]$ rpm -q dovecot > dovecot-1.0-1_42.rc21.fc5.at but: > dovecot: Feb 05 10:49:21 Info: Dovecot v1.0.rc19 starting up

4 5

[Dovecot] Problems enabling quota on MacOS X
by Lars Skovgaard 05 Feb '07

05 Feb '07

Hi, I'm trying to activate quotas for my users, preferably maildir++ quotas as all users are using maildir. I have compiled dovecot with quota-support, but when I try to enable quotas in the config, dovecot refuses to start up with a (to me, at least) strange message: Fopendir(/usr/local/lib/dovecot/imap) failed: No such file or directory Error: imap dump-capability process returned 89 In the logs I get the following: dovecot: read (imap dump-capability process) failed: Interrupted system call The directory "/usr/local/lib/dovecot/imap" is not missing, which would be the obvious explanation... Dovecot-version is 1.0rc21 (which by the way is running ***great*** without quotas). Thanks, /Lars

2 2

[Dovecot] Compile without GSSAPI disables SSL?
by Troy Engel 05 Feb '07

05 Feb '07

Perchance this is something unique to Red Hat -- I'm compiling/running 1.0rc?? on a CentOS 4 machine (i386), building RPMs and trimming down things I don't need (SQL e.g.). If I compile with the flag --with-gssapi, the resulting dovecot-auth binary links to libcrypto and libssl, the standard OpenSSL libs. Life is happy even though I don't need gssapi/krb5 support. If I remove --with-gssapi (but leave --with-ssl=openssl as before), the resulting dovecot-auth binary is no longer linked to the OpenSSL libraries (so I'm assuming no TLS/SSL support, but can't run the binary to test - I have SSL betatest users). So the question is, is this a ./configure problem in Dovecot, or some strangeness with Red Hat that you must link with gssapi/krb5 to get OpenSSL? Anyone care to hazard a guess? -te == details == The full configure line generated by the build is: + ./configure --build=i686-redhat-linux-gnu --host=i686-redhat-linux-gnu --target=i386-redhat-linux-gnu --program-prefix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info 'INSTALL_DATA=install -c -p -m644' --with-doc --with-ssl=openssl --with-ssldir=/etc/pki/dovecot --with-ldap --with-gssapi --with-linux-quota=2 With gssapi: Install prefix ...................... : /usr File offsets ........................ : 64bit I/O loop method ..................... : poll File change notification method ..... : dnotify Building with SSL support ........... : yes (OpenSSL) Building with IPv6 support .......... : yes Building with pop3 server ........... : yes Building with mail delivery agent .. : yes Building with GSSAPI support ........ : yes Building with user database modules . : static prefetch passwd passwd-file checkpassword (modules) Building with password lookup modules : passwd passwd-file shadow pam checkpassword (modules) Building with SQL drivers ............: Without gssapi: Install prefix ...................... : /usr File offsets ........................ : 64bit I/O loop method ..................... : poll File change notification method ..... : dnotify Building with SSL support ........... : yes (OpenSSL) Building with IPv6 support .......... : yes Building with pop3 server ........... : yes Building with mail delivery agent .. : yes Building with GSSAPI support ........ : no Building with user database modules . : static prefetch passwd passwd-file checkpassword (modules) Building with password lookup modules : passwd passwd-file shadow pam checkpassword (modules) Building with SQL drivers ............: -- Troy Engel | Systems Engineer Fluid, Inc | http://www.fluid.com

2 2

[Dovecot] URGENT: specify mailboc location
by T. Horsnell 05 Feb '07

05 Feb '07

Is there a way I can tell dovecot to only use the .subscriptions file, if it exists, as the source of mailfolder paths, rather than have it look at every file in a user's directory to see if it's a mail folder? I'm on RHEL4, still on dovecot dovecot-0.99.11 Thanks, Terry.

2 1

[Dovecot] pop3 not working properly in 1.0.r19
by Charlie Root 05 Feb '07

05 Feb '07

Hi, I often see this message: Feb 1 14:42:44 vlk dovecot: POP3(obec(a)nizkov.cz): Mailbox init failed top=0/0, retr=0/0, del=0/0, size=0 Feb 1 14:42:44 vlk dovecot: POP3(obec(a)nizkov.cz): mbox sync: UID inserted in the middle of mailbox /var/mail/obec(a)nizkov.cz (68386 > 68346, seq=2, idx_msgs=3) this is on FreeBSD@i386 runnin 1.0.r19 (but I have seen this with all versions of 1.0.x), filesystem in use is UFS (ie. local filesystem). THe mailing client in question seems to be outlook express. I can provide the mbox in question. locking method is #lock_method = flock and the dovecot was compiled with kqueue thnx roman

2 1