- dovecot - dovecot.org

[Dovecot] deliver+quota assertion
by Andrew Garner 30 Nov '07

30 Nov '07

When stressing dovecot's delivery agent, I sporadically encountered the following assertion failures and the associated errors. I was trickling in ~50 messages per minute to this mailbox via postal. Nov 30 09:38:41 mail deliver(example(a)example.com): close(/home/vmail/domains//+/example.com/k/example//Maildir/maildirsize) failed: Bad file descriptor Nov 30 09:38:41 mail deliver(example(a)example.com): rename(/home/vmail/domains//+/example.com/k/example//Maildir/maildirsize.lo…, /home/vmail/domains//+/example.com/k/example//Maildir/maildirsize) failed : No such file or directory Nov 30 09:38:41 mail deliver(example(a)example.com): file_dotlock_replace(/home/vmail/domains//+/example.com/k/example//Maildir/… failed: No such file or directory Nov 30 09:38:41 mail deliver(example(a)example.com): Internal quota calculation error Nov 30 09:38:41 mail deliver(example(a)example.com): file index-transaction.c: line 54 (index_transaction_finish_rollback): assertion failed: (t->mail_ref_count == 0) Nov 30 09:38:41 mail deliver(example(a)example.com): Raw backtrace: /usr/lib/dovecot/deliver(i_syslog_fatal_handler+0x2b) [0x80bf0ab] -> /usr/lib/dovecot/deliver [0x80beeba] -> /usr/lib/dovecot/deliver [0x 808f243] -> /usr/lib/dovecot/deliver(index_transaction_rollback+0x1a) [0x808f06a] -> /usr/lib/dovecot/modules/lda/lib10_quota_plugin.so [0xb7ea2f13] -> /usr/lib/dovecot/deliver(deliver_save+0x212) [0x8059412] -> /usr/lib/dovecot/deliver(main+0x12cb) [0x805a79b] -> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xc8) [0xb7ebcea8] -> /usr/lib/dovecot/deliver [0x8058611] # dovecot -n # 1.1.beta9: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ protocols: imap imaps pop3 pop3s login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_greeting: Nationwide Internet Mail Server ready. verbose_proctitle: yes first_valid_uid: 89 last_valid_uid: 89 first_valid_gid: 89 last_valid_gid: 89 mail_uid: 89 mail_gid: 89 mail_location: maildir:/home/vmail/domains/%h/Maildir mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes lock_method: dotlock mail_executable(default): /usr/lib/dovecot/rawlog /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/rawlog /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 imap_client_workarounds(default): delay-newmail imap_client_workarounds(imap): delay-newmail imap_client_workarounds(pop3): pop3_uidl_format(default): %08Xu%08Xv pop3_uidl_format(imap): %08Xu%08Xv pop3_uidl_format(pop3): %f pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: mechanisms: login plain cram-md5 digest-md5 username_format: %Lu verbose: yes debug: yes passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 384 user: vmail group: vmail plugin: quota: maildir quota_rule: *:storage=102400 This is the latest hg sources, somewhat after beta9: $ hg tip changeset: 6877:a5a7d38b6df7 branch: HEAD tag: tip user: Timo Sirainen <tss(a)iki.fi> date: Thu Nov 29 19:38:06 2007 +0200

1 0

[Dovecot] Zimbra benchmarking
by Timo Sirainen 30 Nov '07

30 Nov '07

Now that I have a working kvm setup, I thought I'd finally try how Zimbra works. This is mainly some microbenchmarking, so it may not have much to do with actual performance in real life. Setup: - 1GB memory given to kvm (from host's 2GB) - Intel Core 2 6600 (kvm uses only one CPU) - CentOS 5 - 15GB qcow2 image on XFS filesystem - Zimbra 5.0 RC2 RHEL5 x86_64 - Dovecot latest hg, dbox format Both Zimbra and Dovecot were tested running on the same kvm image. All data could be cached in memory, so this didn't really test disk I/O. Dovecot was run with fsync_disable=no. Running with =yes would have made its performance even better. Zimbra features/bugs -------------------- Zimbra's SEARCH command doesn't support substring searching as IMAP requires: 1 search text performance * SEARCH 136 183 227 231 232 233 245 1 OK SEARCH completed 2 search text erformance * SEARCH 2 OK SEARCH completed 3 search text performanc * SEARCH 3 OK SEARCH completed It's possible to search "performanc" in a non-standard way though: 4 search text "performanc*" * SEARCH 136 183 227 231 232 233 245 4 OK SEARCH completed This means that the upcoming Dovecot v1.1 is probably the only IMAP server that supports IMAP-compatible full text search indexes with incremental updates. (Cyrus Squat requires rebuilding the whole index from scratch just for adding one mail, which makes it kind of impractical.) Zimbra doesn't support SORT or THREAD extensions. I had trouble benchmarking more than 5 simultaneous connections in one mailbox. I'm not sure if this is a configurable setting somewhere, or if there's just some bug. The error messages that my imaptest gave looked like Zimbra was buggy, but I didn't look at them too closely. STORE command also seemed to be buggy, giving lots of "STORE failed" errors, even with just one connection. Zimbra uses 500MB of memory just to start up, so it's not exactly for hosting small installations. Login+Logout ------------ Dovecot: ./imaptest - select=0 secs=10 seed=0 clients=1 Logi Logo 100% 100% 3439 6878 ./imaptest - select=0 secs=10 seed=0 clients=10 Logi Logo 100% 100% 4415 8830 Zimbra: ./imaptest - select=0 secs=10 seed=0 clients=1 Logi Logo 100% 100% 18032 36064 ./imaptest - select=0 secs=10 seed=0 clients=10 Logi Logo 100% 100% 25519 51056 Looks like Dovecot's imap process creation hurts it a lot compared to Zimbra's thread creation. Luckily IMAP connections are usually long-living, so this shouldn't matter much, except for webmails for which you can use imapproxy in the middle. LIST "" * --------- Dovecot: ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1 Logi List 100% 100% 1 181411 ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10 Logi List 100% 100% 10 133451 Zimbra: ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1 Logi List 100% 100% 1 1021 ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10 Logi List 100% 100% 10 972 Yes, Dovecot's LIST is over 100 times faster. STATUS INBOX (MESSAGES UNSEEN RECENT) ------------------------------------- 100 messages in INBOX dovecot: ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1 Logi Stat 100% 100% 1 191003 ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10 Logi Stat 100% 100% 10 171171 Zimbra: ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1 Logi Stat 100% 100% 1 3697 ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10 Logi Stat 100% 100% 10 7009 FETCH 1:* (UID FLAGS ENVELOPE INTERNALDATE BODYSTRUCTURE) --------------------------------------------------------- 100 messages in INBOX. dovecot: ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1 Logi Sele Fetc 100% 100% 100% 1 1 2769 ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5 Logi Sele Fetc 100% 100% 100% 5 5 2902 Zimbra: ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1 Logi Sele Fetc 100% 100% 100% 1 1 203 ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5 Logi Sele Fetc 100% 100% 100% 5 5 258 FETCH ? (BODY[]) ---------------- Randomly fetch message body from 100 messages Dovecot: ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1 Logi Sele Fet2 100% 100% 100% 30% 1 1 51958 ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5 Logi Sele Fet2 100% 100% 100% 30% 5 5 141397 Zimbra: ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1 Logi Sele Fet2 100% 100% 100% 30% 1 1 410 ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5 Logi Sele Fet2 100% 100% 100% 30% 5 5 2291 STORE ? FLAGS.SILENT (random flags and keywords) ------------------------------------------------ Dovecot: ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1 Logi Sele Stor 100% 100% 100% 1 1 15236 ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5 Logi Sele Stor 100% 100% 100% 5 5 5533 Zimbra: ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1 Logi Sele Stor 100% 100% 100% 1 1 308 ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5 Logi Sele Stor 100% 100% 100% 5 5 319 Zimbra gave lots of errors: Error: STORE failed: STORE failed APPEND ------ Appends to empty mailbox: Dovecot: 1 client, 5 seconds: 3959 mails 1 client, 5 seconds, fsync_disable=yes: 21889 mails Zimbra: 1 clients, 5 seconds: 222 mails 5 clients, 5 seconds: 322 mails Zimbra is apparently building full text search indexes while appending, so this test doesn't mean much until I can test Dovecot's performance with Squat indexing.

5 6

[Dovecot] squat plugin
by Joe Wong 29 Nov '07

29 Nov '07

Hello Timo, I have installed 1.1 beta 8 with fts-squat enabled. I have added the plugin section in dovecot.conf. How can I actually make use of the full text index? Regards, - Joe

2 2

[Dovecot] old mails
by Timo Sirainen 29 Nov '07

29 Nov '07

I just got a 6 months old mail to this list and wondered for a while why it happened. Looks like the old dovecot.org was rebooted and Postfix went up and started sending queued mails. So just ignore if you got those.

1 0

Re: [Dovecot] 1.1beta9 'make' fails on osx/Tiger
by Jerry Yeager 29 Nov '07

29 Nov '07

On Nov 28, 2007, at 1:39 PM, dovecot-request(a)dovecot.org wrote: > ------------------------------ > > Message: 7 > Date: Wed, 28 Nov 2007 10:38:19 -0800 > From: snowcrash <schneecrash+dovecot(a)gmail.com> > Subject: Re: [Dovecot] 1.1beta9 'make' fails on osx/Tiger > To: "Jerry Yeager" <jerry(a)scene-naturally.dyndns.org> > Cc: dovecot(a)dovecot.org > Message-ID: > <70f41ba20711281038q12499454o94a9a221e61c54b2(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > (stuff cut out) > > i'd gather tss would like to know about issues. (it shouldn't be > terribly surprising for a projects' beta release to at least be > _tried_ with _other_ projects' beta releases ...) > > cheers. > > That is usually a good idea, especially if you mention that you are using non standard stuff so that folks who are not aware of that can then get a better understanding of the issues. Or for the elucidation of the folks that are following the list trying to decide what efforts they will have to undertake to make the switch. p.s. The folks at AT&T also recommend that "you" use the newer version of gcc than the older-hacked-to-get-something-for-Tiger version of the Apple Leopard version they (AT&T) are providing -- it is further down the page where they provide links. Jerry

2 2

[Dovecot] autogen.sh fails -> "required file `./config.rpath' not found"
by snowcrash 29 Nov '07

29 Nov '07

if i run autogen.sh on fresh 1.1.beta9 source, it fails @ ./autogen.sh ... 100%[====================================>] 154,427 115.09K/s 08:45:56 (114.82 KB/s) - `wiki-export.tar.gz' saved [154427/154427] configure.in:16: required file `./config.rpath' not found configure.in:5: installing `./missing' configure.in:5: installing `./install-sh' src/auth/Makefile.am: installing `./depcomp' autoreconf: automake failed with exit status: 1 per advice, "gettext-related test failures: config.rpath presence" http://www.mail-archive.com/automake-patches@gnu.org/msg00426.html a simple touch config.rpath ./autogen.sh fixes the problem. hth

2 4

[Dovecot] spf record
by Jim Flowers 28 Nov '07

28 Nov '07

Your spf record is broken: dovecot.org. 39942 IN TXT "v=spf1 a -all" -- Jim Flowers <jflowers(a)ezo.net> -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

12 20

Re: [Dovecot] 1.1beta9 'make' fails on osx/Tiger
by snowcrash 28 Nov '07

28 Nov '07

(Yes, Really .... w/ gcc v4.2.1, like i said in the first place) > Actually, the 1.1 beta v9 builds just fine under Tiger (PPC and > PPC64) if you use the official Apple sanctioned gcc v 4.01 they ship > with both Tiger and Leopard. If you do download and use gcc 4.2.1 from > the GNU folks you might want to pass along the error / bug reports to > them instead of Apple, but I suspect the GNU folks will tell you to > try the current 4.2.2 version or the new 4.3 version. i'm using Apple's gcc 4.2.1: http://www.opensource.apple.com/darwinsource/10.5/gcc_42-5531/ and the v4.2.1 Tiger equiv from the R-project provided at http://r.research.att.com/tools/ both Apple & ATT are _already_ well plugged into both .. er .. Apple & GNU. yes, 4.2.1 is "not distro" ... but it *is* up-n-coming, as evienced by the fact that it *is* offered on the darwinsource page. i'd gather tss would like to know about issues. (it shouldn't be terribly surprising for a projects' beta release to at least be _tried_ with _other_ projects' beta releases ...) cheers.

1 0

Re: [Dovecot] 1.1beta9 'make' fails on osx/Tiger (Not Really)
by Jerry Yeager 28 Nov '07

28 Nov '07

On Nov 28, 2007, at 10:59 AM, dovecot-request(a)dovecot.org wrote: > Message: 8 > Date: Wed, 28 Nov 2007 07:49:13 -0800 > From: snowcrash <schneecrash+dovecot(a)gmail.com> > Subject: Re: [Dovecot] 1.1beta9 'make' fails on osx/Tiger, but OK on > osx/Leopard (multiple definitions of symbol _hash_create) > To: "Timo Sirainen" <tss(a)iki.fi> > Cc: Dovecot Mailing List <dovecot(a)dovecot.org> > Message-ID: > <70f41ba20711280749k63a6b099obf09b687e5733422(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > >> (lots of messages from the digest snipped out) Actually, the 1.1 beta v9 builds just fine under Tiger (PPC and PPC64) if you use the official Apple sanctioned gcc v 4.01 they ship with both Tiger and Leopard. If you do download and use gcc 4.2.1 from the GNU folks you might want to pass along the error / bug reports to them instead of Apple, but I suspect the GNU folks will tell you to try the current 4.2.2 version or the new 4.3 version. Jerry

1 0

[Dovecot] 1.1beta9 'make' fails on osx/Tiger, but OK on osx/Leopard (multiple definitions of symbol _hash_create)
by snowcrash 28 Nov '07

28 Nov '07

i'm building dovecot 1.1.beta9 on both osx 10.4.11 osx 10.5.1 using Using built-in specs. Target: powerpc-apple-darwin8 Configured with: /Builds/apple/gcc_42-5531/obj/src/configure --disable-checking -enable-werror --prefix=/usr --mandir=/usr/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-gxx-include-dir=/usr/include/c++/4.0.0 --with-slibdir=/usr/lib --build=i686-apple-darwin8 --program-prefix= --host=powerpc-apple-darwin8 --target=powerpc-apple-darwin8 Thread model: posix gcc version 4.2.1 (Apple Inc. build 5531) on both boxes. configuring as Install prefix ...................... : /usr/local/dovecot File offsets ........................ : 64bit I/O loop method ..................... : kqueue File change notification method ..... : kqueue Building with SSL support ........... : yes (OpenSSL) Building with IPv6 support .......... : yes Building with pop3 server ........... : yes Building with mail delivery agent .. : yes Building with GSSAPI support ........ : no Building with user database modules . : static prefetch passwd passwd-file checkpassword sql Building with password lookup modules : passwd passwd-file pam checkpassword sql Building with SQL drivers ............:mysql sqlite is OK on both boxes. on Leopard, make completes without error. but, on Tiger, it fails @ make ... make[3]: Nothing to be done for `all'. Making all in master /bin/sh ../../libtool --tag=CC --mode=link gcc -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -Wstrict-aliasing=2 -I/usr/local/ssl/include -L/usr/local/db46/lib -L/usr/local/sqlite/lib -lsqlite3 -L/usr/local/mysql/lib/mysql -L/usr/local/lib -liconv -o dovecot auth-process.o askpass.o capabilities-posix.o child-process.o dict-process.o dup2-array.o listener.o log.o login-process.o mail-process.o main.o master-settings.o syslog-util.o ssl-init.o ../lib-settings/libsettings.a ../lib/liblib.a gcc -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -Wstrict-aliasing=2 -I/usr/local/ssl/include -o dovecot auth-process.o askpass.o capabilities-posix.o child-process.o dict-process.o dup2-array.o listener.o log.o login-process.o mail-process.o main.o master-settings.o syslog-util.o ssl-init.o -L/usr/local/db46/lib -L/usr/local/sqlite/lib /usr/local/sqlite/lib/libsqlite3.dylib -L/usr/local/lib -lreadline -lncurses -lpthread -L/usr/local/mysql/lib/mysql /usr/local/lib/libiconv.dylib ../lib-settings/libsettings.a ../lib/liblib.a /usr/libexec/gcc/powerpc-apple-darwin8/4.2.1/ld: multiple definitions of symbol _hash_create /usr/lib/gcc/powerpc-apple-darwin8/4.2.1/../../../libpthread.dylib(strhash.So) definition of _hash_create ../lib/liblib.a(hash.o) definition of _hash_create in section (__TEXT,__text) /usr/libexec/gcc/powerpc-apple-darwin8/4.2.1/ld: multiple definitions of symbol _hash_destroy /usr/lib/gcc/powerpc-apple-darwin8/4.2.1/../../../libpthread.dylib(strhash.So) definition of _hash_destroy ../lib/liblib.a(hash.o) definition of _hash_destroy in section (__TEXT,__text) collect2: ld returned 1 exit status make[3]: *** [dovecot] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 looks like problems with ot at all sure what in Tiger's causing this ... hints/suggestions? thanks!

3 14