- dovecot - dovecot.org

[Dovecot] 1.0-test24 and some mbox benchmarking
by Timo Sirainen 02 Jul '04

02 Jul '04

http://dovecot.org/test/ Again only mbox fixes. I found some more bugs which could have also caused some of the mbox problems that people reported. They were found when I today thought I'd again do a bit of testing with my favourite 1.4GB mbox. Then I thought I might as well see how it compares against UW-IMAP. First a bit of explanations how they work internally: Dovecot 1.0-test24 ------------------ Dovecot works by trying to do everything in one "sync" function. It reads new mails, inserts missing headers, writes header modifications and expunges messages. Dovecot leaves 100 bytes of padding in every mail's headers, which it can use to avoid moving the rest of the file forward when it needs to insert space. However, if there's not enough space (and with new mails there isn't), it needs to do the moving. Moving is done by first reading messages and counting how many bytes we're short. Once we've seen enough padding we'll rewrite those messages. If we read until end of file, we'll grow the file and rewrite the messages. Rewriting is made backwards, so we don't have to do much buffering, and in case the rewriting gets interrupted (crash, power loss, etc.) the data loss is very small, a few kilobytes maximum. UW-IMAP 2004 ------------ UW-IMAP first only reads the mbox with SELECT, rewrite is only triggered by LOGOUT, EXPUNGE and CHECK commands. UW-IMAP also inserts padding, but less than Dovecot. It tries to get added headers to fit into 50 bytes, and what's not used is left as padding. Normally this seems to get it around 15-20 bytes of padding. Perhaps I should shrink it from Dovecot too. Rewriting works by reading the file forward into buffer and writing the changes as needed. This is quite fast, but it means the buffer can grow large, and if the rewrite gets interrupted everything in the buffer gets lost. In my test mbox this would have been 23MB of lost data, but normally much less. Benchmarks ---------- I simply rewrote a 1.4GB mbox containing 361052 mails, Linux kernel mailing list archives from years 96-02. Computer is Athlon XP 2700+ with 1GB of memory. reads/writes were counted using Linux's iostat command. Nothing else was being used in that partition, so the numbers should be accurate. Except UW-IMAP's read count is a few blocks too much because I got tired of waiting it and started looking into the mbox to see how far it had gotten. Read counts could also be somewhat wrong if some of the mbox was already in buffer cache. I tried to trash it anyway by catting 2x4GB of data into /dev/null. Kernel used around 900MB of memory for caching. Total CPU times may also be a bit off, as the computer was being used at the same time. Dovecot 1.0-test24 ------------------ reads : 4007432 blocks = 1956 MB writes: 2947381 blocks = 1439 MB original mbox : 1420611590 B = 2774632 blocks = 1354 MB rewritten mbox: 1472684487 B = 2876336 blocks = 1404 MB indexes : 14452732 B = 28227 blocks = 14 MB 7221164 dovecot.index 10436 dovecot.index.cache 7221132 dovecot.index.log (this will be truncated after a while) - 16064 VSZ, 8012 RSS after SELECT completed - 14MB is mmaped index files - 216kB heap left of which 70kB actually in use - heap usage was 25MB VSZ/RSS constantly while syncing, but allocations were so large that libc used anonymous mmap()s so they got dropped after sync - VSZ peaked at 32MB, most likely because index file(s) were temporarily being mmap()ed more than once - 63.37s user - 16.56s system - 24% cpu - 5:22.75 total UW-IMAP 2004 ------------ Reading: - memory: 70044 VSZ, 67724 RSS - CPU: 7s user, 53s total Totally: reads : 5549640 blocks = 2709 MB writes: 2875581 blocks = 1404 MB original mbox : 1420611590 B = 2774632 blocks = 1354 MB rewritten mbox: 1444411190 B = 2821115 blocks = 1377 MB - 93416 VSZ, 91120 RSS - 2295.81s user - 23.99s system - 96% cpu - 39:57.90 total Notes ----- Not counting Dovecot's indexes UW-IMAP wrote 21MB less. But Dovecot wrote 27MB more padding, so just by shrinking it Dovecot would have written 6MB less data. Also because Dovecot writes the file backwards, it needs to do some extra jumping around and overlapping writes, but I guess OS nicely merged them. I'm not exactly sure why UW-IMAP uses so much CPU for rewriting, but it does and so Dovecot is over 7x faster in total (with 36x less CPU). Dovecot should also support delaying the rewrite. This is mostly useful for POP3 clients which deletes all the mail at logout, so they won't need the rewriting at all. Dovecot also writes all flag changes to disk immediately while UW-IMAP leaves it later to do more at once. That results in less total I/O as well.

1 0

[Dovecot] 1.0-test22 - maildir fixed?
by Timo Sirainen 02 Jul '04

02 Jul '04

http://dovecot.org/test/ Well, remember when I said: > Here you go, maildir syncing problems finally fixed (I hope). Well, > except there's this "new-dir-only syncing" optimization which I'm not > really sure if it works as it should. Just before I found the expunge bug in index, I was trying to fix the new-dir-only syncing because the code looked wrong. Then I figured out the indexing problem and thought maybe this one wasn't broken after all, and then forgot about it. Finally after lots of debugging, I came to same conclusion again. The code was wrong, and cause of the remaining "Maildir sync: UID < next_uid (312 < 3)" errors. So, please try and see if you still can break this release. Other changes include some crashfixes for cache file and fix for mbox which kept constantly rewriting headers for \recent messages if we had selected the mailbox as read-only (EXAMINE, STATUS).

11 30

[Dovecot] building dovecot 1.0 test19 over fc1
by Luis Miguel Cruz 02 Jul '04

02 Jul '04

[root@fc1-devel tmp]# rpmbuild -ta dovecot-1.0-test19.tar.gz error: line 7: Illegal char '-' in version: Version: 1.0-test19 Seems to be an error into the .spec file I am not sure. I will check it. Did anyone get this problem in the past? Thanks.

2 1

[Dovecot] 1.0-test23 and caching decisions
by Timo Sirainen 30 Jun '04

30 Jun '04

http://dovecot.org/test/ mbox fixes mostly. Began some other things, but none of them actually work yet: - Use X-UIDL header in mboxes for POP3 UIDL list - Cache file compression - Some cleverness for cache file for figuring out what to cache Well, I might as well explain the last one now. Users can be divided to three groups: 1. Most users will use only a single IMAP client which caches everything locally. For these users it's quite pointless to do any kind of caching as it only wastes disk space. That might also mean more disk I/O. 2. Some users use multiple IMAP clients which cache everything locally. These could benefit from caching until all clients have fetched the data. After that it's useless. 3. Some clients don't do permanent local caching at all. For example Pine and webmails. These clients would benefit from caching everything. Some locally caching clients might also access some data from server again, such as when searching messages. They could benefit from caching only these fields. After thinking about these a while, I figured out that people who care about performance most will be using the upcoming Dovecot LDA anyway which updates the indexes/cache immediately. In that case even the first user group would benefit from caching the same way as second group. LDA reads the mail anyway, so it might as well extract some information about it and store them into cache. So, group 1. and 2. could be optimally implemented by keeping things cached only for a while. I thought a week would be good. When cache file is compressed, everything older than week will be dropped. But how to figure out if user is in group 3? One quite easy rule would be to see if client is accessing messages older than a week. But with only that rule we might have already dropped useful cached data. It's not very nice if we have to read and cache it twice. Most locally caching clients always fetch new messages (all but body) when they see them. They fetch them in ascending order. Noncaching clients might fetch messages in pretty much any order, as they usually don't fetch everything they can, only what's visible in screen. Some will use server side sorting/threading which also makes messages to be fetched in random order. Second rule would then be that if a session doesn't fetch messages in ascending order, the fetched field type will be permanently cached. So, we have three caching decisions: 1. Don't cache: Clients have never wanted the field 2. Cache temporarily: Clients want this only once 3. Cache permanently: Clients want this more than once Different mailboxes have different decisions. Different fields have different decisions. There are some problems, such as if a client accesses message older than a week, we can't know if user just started using a new client which is just filling it's local cache for the first time. Or it might be a client user hasn't just used for over a week. In these cases we shouldn't have marked the field to be permanently cached. User might also switch clients from non-caching to caching. So we should re-evaluate our caching decisions from time to time. This is done by checking the above rules constantly and marking when was the last time the decision was right. If decision hasn't matched for two months, it's changed. I picked two months because people go to at least one month vacations where they might still be reading mails, but with different clients.

1 0

[Dovecot] Locked index file
by Kevin Cramer 30 Jun '04

30 Jun '04

I have docevot 0.99.10.6 installed on OpenBSD 3.5 and I'm using it with Squirrelmail. I tried to empty my Trash folder and it timed out. It looks like the process or thread died but the lock wasn't removed: Jun 25 09:16:30 plato imap(kevin): Timeout while waiting for release of shared fcntl() lock for index file /home/kevin/Maildir/.Trash/.imap.index How can I free this lock? I have tried stopping and re-starting dovecot already but that didn't help. Also, is there something I should do to prevent this in the future? The OpenBSD package sets the default locking to fcntl for mbox. I'm not sure if that is the setting that also controls the locking mechanism on the index files. Thanks, Kevin

2 3

[Dovecot] ldap auth problems
by Karl Latiss 30 Jun '04

30 Jun '04

I have previously configured dovecot with ldap auth on a fedora setup with no problems. Now, using RHEL 3 with the same config I get: dovecot: Jun 30 17:46:07 Error: Auth process died too early - shutting down dovecot: Jun 30 17:46:07 Error: child 16780 (auth) killed with signal 11 pop3-login: Jun 30 17:46:07 Fatal: fd_send(-1) failed: Broken pipe imap-login: Jun 30 17:46:07 Fatal: fd_send(-1) failed: Broken pipe This doesn't happen when using other kinds of authentication, eg pam The ldap directory is working fine for other apps so I'm pretty sure that's not it. I'm using the rpm from dag's repository and even re-compiled to ensure ldap support is there. Any clues? -- Karl Latiss <karl(a)designit.com.au> designIT

2 1

[Dovecot] 1.0 Maildir/mbox
by Paul Adams 29 Jun '04

29 Jun '04

Just to confirm, and sorry if I've missed this info somewhere in the flurry of messages: dovecot 1.0 will be able to support maildir and mbox users simultaneously? Paul

2 1

Re: [Dovecot] phpGroupWare
by Avery 29 Jun '04

29 Jun '04

I am currently using SQ mail also. Was wondering though about phpGroupWare. Brook Humphrey said: > On Friday 25 June 2004 08:46 pm, Avery wrote: >> Anyone here using phpGroupWare with dovecot? Does anyone know if it is >> possible? >> >> Thanks > I use squirllmail and it works great. > > -- > -~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~- > Brook Humphrey > Mobile PC Medic, 420 1st, Cheney, WA 99004, 509-235-9107 > http://www.webmedic.net, bah(a)webmedic.net, bah(a)linux-mandrake.com > Holiness unto the Lord > -~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~-~`'~- > > -- Avery

1 0

[Dovecot] Dovecot on Redhat GFS
by Tomi Hakala 28 Jun '04

28 Jun '04

Now that Redhat released GFS to GLP has anyone gotten around to test Dovecot on top of that? -- Tomi Hakala

1 0

[Dovecot] 0.99.10.6 -imap flag update problem still present
by Hauke Fath 28 Jun '04

28 Jun '04

Hi, the email status problem that I reported for Dovecot 0.99.10.5 on June 3rd does still show up in 0.99.10.6. Quoting myself: (1) When new mail is delivered to the inbox, the last read mail(s) change(s) to "unread". Clients: Eudora 6 (Mac), Mozilla 1.6 (NetBSD, Linux, Win XP), MS Outlook. (2) When you attempt to move mails from the inbox to another folder with Mozilla (1.6 on win32, Linux, NetBSD), they are not removed from the inbox, effectively performing a copy instead of a move. And, while I am at it, (3) I see a regular log entry from a MS Outlook 2000 SP 3 Jun 3 10:26:23 bounce imap-login: SSL_read() syscall failed: EOF [130.83.XXX.XXX] The setup here: OS NetBSD 2.0beta, MTA sendmail 8.12.11, LDA maildrop 1.3.9, dovecot 0.99.10.6, OpenSSL 0.9.7d Any ideas? My users are starting to nag... hauke -- Institut für Nachrichtentechnik /~\ The ASCII Ribbon Campaign FG Signalverarbeitung \ / No HTML/RTF in email TU Darmstadt X No Word docs in email Ruf +49-6151-16-3281, Fax -3778 / \ Respect for open standards

2 7