- dovecot - dovecot.org

[Dovecot] New mailbox format
by Timo Sirainen 06 Dec '05

06 Dec '05

I've been doing some thinking on a new better mailbox format that'd be faster than mbox and maildir for most common uses. Comments welcome. It won't come to Dovecot v1.0, but I'll probably create another branch to Dovecot CVS where I'll start implementing it and some other possibly destabilizing changes. Generic ------- The point is to have a mailbox format where the mailbox can consist of one or more files. Grouping multiple files in a single file makes it faster to read, but it's slower to expunge mails from the beginning of the file. So this format would allow sysadmin to specify rules on how large the files would be allowed to grow. This format is mostly designed to work nicely with separate index files such as Dovecot has. The flags are stored in the mailbox files mostly as a backup in case the index gets lost or corrupted. Index File ---------- Named "index". Index file gives a mapping between UIDs <-> filenames: Header: <uidvalidity> <last-used-uid> Record: <uid-first> <uid-last> <filename> Record: .. eg.: 1127468129 5 1 3 msg.1 4 4 msg.4 All of the UIDs don't have to actually exist in the file if they have been expunged. The record should be removed once all of the UIDs have been removed from it (and the file itself been deleted). The same file may appear multiple times in the index. For example: 1 2 msg.1 3 3 msg.3 4 5 msg.1 This means that UIDs 1, 2, 4 and 5 are in file msg.1, but UID 3 is in file msg.3. The filename doesn't matter as long as it's unique, but probably wouldn't hurt to include its first UID in the filename so humans can find mails more easily. The filename however must not end with ".lock". The file is modified by creating first exclusively owned "index.lock" file, updating it and then rename()ing it over the index file. This lock file shouldn't be held for a long time, so if it exists, and it doesn't get modified at all within 30 seconds, it can be overwritten. Before rename()ing over the index file, you should check with stat() that the index.lock file is still the same as you expect it to be (in case your process was temporarily stopped for over 30 secs). The index file's mtime timestamp must grow after each modification. This makes it easy to check if new mail has been added to the mailbox. Mail File --------- File header: Bytes Format Description 8 hex Header size 16 hex Append offset 4 hex Mail header size 4 hex Mail header padding 4 hex Keyword count in mail headers Mail header: Bytes Format Description 8 hex UID 16 hex Mail size 1 1/0 Answered flag 1 1/0 Flagged flag 1 1/0 Deleted flag 1 1/0 Seen flag 1 1/0 Draft flag 1 1/0 Expunged flag n 1/0 Keywords The file layout then goes: <File header> - possible padding due to Header size <Mail header #1> - possible padding due to Mail header size <Mail data #1> - possible padding due to Mail header padding (ie. next Mail header's offset is divisible with Mail header padding) <Mail header #2> etc.. The UIDs must be ascending within the file (but as said above, some UIDs between them may exist in other files). When the file is opened, it must be shared locked with flock(). Expunging mails requires exclusive flock(), so expunges can't happen while someone is still reading the file. Mails can however be marked with expunged-flag, and such mails should be treated as if they are already expunged. Expunged flag must not be removed. If the mail for some reason is wanted to be un-expunged, it must be copied as a new mail with new UID to the mailbox. Modifying flags doesn't require any locks. You'll simply write over those flags you wish to change. That's why each flag takes one byte. Safest way to expunge is to copy all non-expunged mails after the first expunged one to new file. Then update the index file with the new location for the UIDs, and at last ftruncate() at the position of first expunged mail in the original file. Alternative way to expunge is to just move the data over the expunged mails and fruncate() at the end. No index file updates necessary. This is more filesystem quota friendly, but if the expunging gets aborted in the middle it will corrupt the mails. Appending mails can be done by either locking existing file with a dotlock and appending to it, or creating a new file. If appending to existing file, also flock() it so expunging won't happen in the middle of append. If appending to new file, create it with .lock suffix and then check that the file without .lock wasn't already just created. When appending to existing file, check "Append offset" from file's header. If that offset is at the end of file, just append there. Otherwise read the mail's header from there. If it has a non-zero UID and index file says that the same UID exists in this file, jump over the mail and do the same checks for all the rest of the mails in the file. Once the check doesn't match or you reach EOF, update the append offset in the header to that offset and write your mail there. Initially write the UID in mail's header as zeroes ('0', not NUL). Once the append is completely finished, you'll have to update it with the new UID. You'll do this by locking index file, reading the latest last-used-uid and using the next value for your UID. Once you've written it, rewrite the index file with updated last-used-uid field and your UID added to it. After this, update the append offset in the mail file and unlock it. Keywords file ------------- Named "index.keywords". This contains mappings between keyword numbers and names. TODO: may be problematic with lockless flag writes.. Recent file ----------- "index.recent.<uid>" file tells what the last non-recent UID was. The file doesn't have any data inside it. When opening the mailbox you should find and rename() the existing index.recent.<uid> file to the new index.recent.<last-uid>. A successful rename() means the mails between <uid>+1 .. <last-uid> are recent for you. A failed rename() means that someone else managed to do it. You should look for the file again and if it was renamed to less than <last-uid>, you should try again. Nightly cleanups ---------------- Because expunged mails may be left lying around for a while, there probably should be some nightly check that cleans them up. Other cleanups that may be necessary after crashes are: - .lock files - expunged data at the end of files (can be checked with same rules as how appending determines the offset where to append) One problem with nightly expunging is how to expunge mails if the file is still locked by some process. Forced shutdowns may be annoying to users and can be difficult to implement.. Alternative locking strategy ---------------------------- The locking strategy as described above is optimized for high concurrency: Only if mail file is in the middle of being expunged, we're waiting on locks. With appending we may be waiting for existing appends to finish if we don't wish to create a new file for some reason. Other than those, reading and flag updating are completely lockless. The two downsides to this are: 1. flock() isn't available everywhere. It could however be replaced with fcntl() if that's available. NFS may not have either. 2. Expunges can't happen while multiple processes are accessing the mailbox at a same time. This may be a problem if the mailbox never has less than two processes accessing it. The alternative to flock()ing would be to simply handle unexpected end-of-file reads as a sign of expunge, which would require re-reading index file to see where the mails went (or if they were really expunged). This has two problems however. One is that flag writes couldn't catch expunges without locking, and another is that if you've already sent half a mail to IMAP client, you have no other choice than to disconnect the client if you can't send the rest of the mail. It might be worth it optionally. Especially for NFS people. Then you'd basically have the dotlock file that locks appends, expunges and flag updates. Reading would still be lockless. Also for the best performance the mail flags could be stored only in Dovecot's index files. In that case there's no need to update flags and the only downside with this locking strategy would be the potential disconnections, which probably aren't all that bad. Compatibility ------------- If needed, it would be possible to create new/ and tmp/ directories under the mailbox to allow mail deliveries to be in maildir format. The maildir files would be then either just moved or maybe be appended to existing mail files.

19 52

[Dovecot] Still weird UID ordering issues with maildir
by Todd Vierling 06 Dec '05

06 Dec '05

Nightly 20050829 (includes everything currently in nightly/ChangeLog): Sep 1 09:25:02 server dovecot: imap-login: Login: user=<tv>, method=PLAIN, rip=192.168.1.1, lip=192.168.1.3, TLS Sep 1 09:25:02 server dovecot: IMAP(tv): Maildir /home/tv/.maildir sync: UID < next_uid (1187 < 1188, file = 1125581033.7788_2.server.duh.org:2,) repeated every time I try to login. I had to nuke the dovecot indexes and let it reindex (which throws off the ordering of some messages). It had been working without incident since updating to 20050829, until now. -- -- Todd Vierling <tv(a)duh.org> <tv(a)pobox.com> <todd(a)vierling.name>

6 18

[Dovecot] 1.0-alpha2: assertion failed
by Wolfram Schlich 06 Dec '05

06 Dec '05

Hi, I am experiencing some problems with Dovecot 1.0-alpha2: --8<-- Sep 20 13:54:45 mailgateway dovecot: imap(support): file mail-index-view-sync.c: line 168 (mail_index_view_sync_begin): assertion failed: (!view->syncing) Sep 20 13:54:45 mailgateway dovecot: child 27047 (imap) killed with signal 6 --8<-- I tried to make it core as described in http://www.dovecot.org/bugreport.html, but without luck. The problem only happens to two accounts/users, which are used by more than one remote user/client at once. Dovecot OS is Linux 2.4.30 on RedHat 7.3. Clients are Thunderbird 1.0.6 on Linux and Windows. The mail storage format is Maildir. The indexes are stored in memory or in a separate directory (tried both). Any hints? -- Wolfram Schlich

2 3

[Dovecot] Segfault when using multiple servers
by Marc Maurer 06 Dec '05

06 Dec '05

Should it be possible to run multiple dovecot servers on the same mailboxes (Maildir in my case)? I've a setup where 2 dovecot 0.99.13 servers write to the same mailboxes, which dovecot apparently does not like too much. Example (using imap): - Login with mail client on server1 - Login with webmail on server2 - Delete a mail using webmail on server2 - Hit send/receive in the mail client Result: server1 segfaults. Is such a setup, which is basically a loadbalancing setup, allowed at all by dovecot? Cheers, Marc

2 1

[Dovecot] mbox separators ignored
by Kenneth Porter 06 Dec '05

06 Dec '05

Using dovecot-0.99.13-4.FC2 I grabbed a bunch of Mailman archive files from the Weather mailing list here: <http://www.buoy.com/pipermail/weather/> I gunzipped the older ones, then concatenated them all in correct order to create one big mbox file. I dropped it into my ~/mail folder and started my IMAP client (Mulberry) and opened the new Weather folder. It shows one gargantuan concatenated message, instead of lots of more reasonable-sized ones. The messages appear to be properly separated by "<linefeed><linefeed>From<space>". Why doesn't dovecot parse them into separate messages? I double-checked and removed any header lines with Content-Length, X-UID, X-IMAPbase, and X-Keywords. So this should look like a new mbox file with new content delivered by procmail. What's confusing dovecot?

2 1

[Dovecot] Maildir quota's
by Lieven Van Acker 05 Dec '05

05 Dec '05

Hey, is http://www.dovecot.org/patches/1.0/quota.tar.gz supposed to work with maildir quota's for virtual mailboxes? Using it with dovecot-lda (cvs) results in DSN Your message was automatically rejected by Sieve, a mail filtering language. The following reason was given: Internal quota calculation error Thanks, Lieven

4 3

[Dovecot] Dovecot-auth <defunct>
by Son Nguyen Truong 05 Dec '05

05 Dec '05

We have used qpopper on our email server RH 9 for long. But recently as the number of people using the server has increased too fast we have to switch to dovecot. Things seem to be OK with the rate users post and receive emails. But from that time very often I see this when top: . dovecot-auth <defunct> And some people sometimes have to restart their outlook to receive emails Could any give us a hand to solve this problem. Thanks and regards,

1 0

[Dovecot] Virtual domain support using more than one MySQL database
by Dragiša Durić 04 Dec '05

04 Dec '05

For historical reasons I am stuck with situation where hosted domain users have their Mailbox info in mysql enabled vpopmail and our ISP's users have their info in nss_mysql. While I can easily move my users from vpopmail's mysql database to something more "native" to Dovecot, I see no problem to configure Dovecot, as I've already configured Postfix, to use this database directly and access mailboxes without problem. And I would not like to make separate auth database for ISP's users, as I don't like to have two watches when one suffice. Under courier-imap, I've configured my vpopmail domains to authenticate using it's vchkpw module, and mysql was left to me to be configured to use nss_mysql. It works well. Is it possible under Dovecot, and how? Is it possible to to MySQL auth using two different databases? It would be the best. TIA, dd

1 0

[Dovecot] Dovecot strange behaviors on Solaris
by Mario Nigrovic-rvvk40 03 Dec '05

03 Dec '05

Folks - I've been using Dovecot for some time now, but have only recently upgraded to 1.0alpha4. I'm struggling to get my configuration right and I'm seeing some truly bizarre issues. My Setup: Solaris 2.8 / Thunderbird / Procmail / Dovecot 1.0alpha4 My mail setup is only for me, so Dovecot runs at an odd port and under my userID. All my mail is in my home directory, which is on NFS (NetApp). The main mail receiver is a different machine which mounts my home directory. Typical entry from .procmailrc: :0 * ^(From|To|CC):.*@.*\<si2.org Maildir/.SI2/ Highlights from dovecot.conf: #default_mail_env = maildir:%h/.mail/Maildir:INDEX=/var/tmp/mario/dovecot default_mail_env = maildir:%h/.mail/Maildir:INDEX=MEMORY mmap_disable = yes #lock_method = flock lock_method = fcntl #lock_method = dotlock protocol imap { listen = *:2143 } auth_worker_max_count = 0 auth default { mechanisms = plain userdb passwd { } } When I run as above (INDEX=MEMORY), I have access to all my mail, but I'm seeing literally thousands of .nfs files created all throughout my .mail/Maildir directories, and Thunderbird periodically locks up. As the filesystem fills with useless .nfs files (I deleted 10000 from ONE mailbox directory this morning), everything gets really, really slow, and I suspect this contributes to Thunder bird's problem. But, if I run with the INDEX on a local disk, many mailboxes will not load (the most critical ones, of course...), but also some do. My inbox will list messages, but cannot display any. Others won't even list messages. I'm at a total loss how to proceed, here. I've tried everything I can think of to debug what's going on, but have gotten nowhere (well, I did discover the whole lock_method=fcntl trick). I've tried enabling rawlog, but am just not familiar enough with IMAP to get help from those logs. Here's what's happening in a typical mailbox: % pwd /home/mario/.mail/Maildir/.SI2 % ls -al total 544 drwx--S--- 5 rvvk40 lowv 425984 Dec 2 14:42 ./ drwx--S--- 64 rvvk40 lowv 16384 Dec 2 14:43 ../ -rw------- 1 rvvk40 lowv 15 Nov 16 07:16 .customflags -rw------- 1 rvvk40 lowv 24 Dec 2 14:38 .nfs00986 -rw------- 1 rvvk40 lowv 24 Dec 2 14:40 .nfs08986 -rw------- 1 rvvk40 lowv 312 Dec 2 14:11 .nfs16686 -rw------- 1 rvvk40 lowv 24 Dec 2 14:32 .nfs1A886 -- 40+ deleted here for brevity -- -rw------- 1 rvvk40 lowv 24 Dec 2 14:39 .nfsE3986 -rw------- 1 rvvk40 lowv 312 Dec 2 14:38 .nfsEF886 -rw------- 1 rvvk40 lowv 24 Dec 2 14:26 .nfsF3886 -rw------- 1 rvvk40 lowv 312 Dec 2 14:32 .nfsF9886 -rw------- 1 rvvk40 lowv 312 Dec 2 14:17 .nfsFF686 drwx--S--- 2 rvvk40 lowv 4096 Dec 2 07:58 cur/ drwx--S--- 2 rvvk40 lowv 4096 Dec 2 07:58 new/ drwx--S--- 2 rvvk40 lowv 4096 Dec 2 07:58 tmp/ % pwd /home/mario/.mail/Maildir/.SI2 Help! Mario

2 1

[Dovecot] Silly newbie questions...
by Michael Segel 03 Dec '05

03 Dec '05

Hi, I'm relatively new to dovecot, less than 48 hours, and I've ran in to a few bugs in the installation/configuration process. Here's my set up. I have my own domain. Within the domain, I have a couple of Linux boxes. So I control both my primary and secondary mail servers. My Linux boxes are a mixture of SuSE 9.2, 9.3 and 10.0 (All based on age of the hardware.) I'd like to use Postfix for my MTA, which I have up and running, but I'd also like to add an IMAP server (hence dovecot) so I can get my mail from any of my PCs and my blackberry.) On my main mail server, I have KDE running and it uses ~/Mail/inbox/(cur,new,...) When I tried to use KDE on my other Linux server, I got it to authenticate and I could see the folders, but I couldn't see any of the mail. (Its all in cur subdirectory.) Now I did notice that when Postfix gets a mail message, its placed in /var/spool/mail/msegel which would imply its using an mbox format. So what am I missing? TIA! -Mike -- Michael Segel Principal MSCC

3 5