September 2005 - dovecot

[Dovecot] new mailbox format v2
by Timo Sirainen 27 Sep '05

27 Sep '05

Updated version. The previous one was a bit too complicated with flag updates for pretty minimal benefits. Changes: - writing/reading mail flags can be disabled to get better performance without it requiring any changes to how the files are normally handled (so mixed more works fine). - writing flags now requires getting shared locks. so there's the overhead of locking but flag writers don't have to wait for each others. - reading requires no locking anymore. the comment in the previous version about reading only half a mail being a problem doesn't actually matter, since if a client tries to read an expunged mail Dovecot already disconnects the client immediately (since most clients can't handle such situation well anyway). - no more problems with mailbox readers locking out expungers - note about fcntl() locking being able to provide even higher concurrency - works just fine with plain dotlocks, so works fine with NFS And looks like I forgot to explain the "mail header padding" before. I'm not exactly sure if it's useful or not, but it doesn't hurt to have it there if it's wanted. The idea was anyway that if you're writing more than one flag at a time, having a 512 byte padding would make sure that when writing the few bytes the OS will only have to read one sector from disk (512 bytes in all todays OSes I believe?) to get the non-written parts of the sector instead of possibly reading two sectors. So it might help with trading disk reads for some wasted disk space. Generic ------- The point is to have a mailbox format where the mailbox can consist of one or more files. Grouping multiple files in a single file makes it faster to read, but it's slower to expunge mails from the beginning of the file. So this format would allow sysadmin to specify rules on how large the files would be allowed to grow. Index File ---------- Named "index". Index file gives a mapping between UIDs <-> filenames: Header: <uidvalidity> <last-used-uid> Record: <uid-first> <uid-last> <filename> Record: .. eg.: 1127468129 5 1 3 msg.1 4 4 msg.4 All of the UIDs don't have to actually exist in the file if they have been expunged. The record should be removed once all of the UIDs have been removed from it (and the file itself been deleted). The same file may appear multiple times in the index. For example: 1 2 msg.1 3 3 msg.3 4 5 msg.1 This means that UIDs 1, 2, 4 and 5 are in file msg.1, but UID 3 is in file msg.3. The filename doesn't matter as long as it's unique, but probably wouldn't hurt to include its first UID in the filename so humans can find mails more easily. The filename however must not end with ".lock". The file is modified by creating first exclusively owned "index.lock" file, updating it and then rename()ing it over the index file. This lock file shouldn't be held for a long time, so if it exists, and it doesn't get modified at all within 30 seconds, it can be overwritten. Before rename()ing over the index file, you should check with stat() that the index.lock file is still the same as you expect it to be (in case your process was temporarily stopped for over 30 secs). The index file's mtime timestamp must grow after each modification. This makes it easy to check if new mail has been added to the mailbox. Mail File --------- File header: Bytes Format Description 8 hex Header size 16 hex Append offset 4 hex Mail header size 4 hex Mail header padding 4 hex Keyword count in mail headers Mail header: Bytes Format Description 8 hex UID 16 hex Mail size 1 1/0 Answered flag 1 1/0 Flagged flag 1 1/0 Deleted flag 1 1/0 Seen flag 1 1/0 Draft flag 1 1/0 Expunged flag n 1/0 Keywords The file layout then goes: <File header> - possible padding due to Header size <Mail header #1> - possible padding due to Mail header size <Mail data #1> - possible padding due to Mail header padding (ie. next Mail header's offset is divisible with Mail header padding) <Mail header #2> etc.. The UIDs must be ascending within the file (but as said above, some UIDs between them may exist in other files). If mail has been marked with expunged-flag, it should be treated as if it was already expunged. The mail will really be expunged later in some nightly run. Reading the files requires no locking. If while reading you happen to hit unexpected end of file, it means that mails were expunged from the file. After that you'll look at the index file again to see which file the mail went and continue from there on. If the mail isn't found, it means it got expunged. Expunging happens by locking the file and then copying all non-expunged mails after the first expunged one to new file. Then update the index file with the new location for the UIDs, and at last ftruncate() at the position of first expunged mail in the original file. Flag updates require locking the file as well, otherwise we might try to write past the end of file when another process is expunging the mails. Shared locking can be used for flag update locking since each flag update can be atomically changed by writing one byte. Appending mails can be done by either locking existing file and appending to it, or creating a new file. If appending to a new file, create it with .lock suffix and then check that the file without .lock wasn't already just created to prevent race conditions. When appending to existing file, check "Append offset" from file's header. If that offset is at the end of file, just append there. Otherwise read the mail's header from there. If it has a non-zero UID and index file says that the same UID exists in this file, jump over the mail and do the same checks for all the rest of the mails in the file. Once the check doesn't match or you reach EOF, update the append offset in the header to that offset and write your mail there. Initially write the UID in mail's header as zeroes ('0', not NUL). Once the append is completely finished, you'll have to update it with the new UID. You'll do this by locking index file, reading the latest last-used-uid and using the next value for your UID. Once you've written it, rewrite the index file with updated last-used-uid field and your UID added to it. After this, update the append offset in the mail file and unlock it. If fcntl() locking is used, locking only specific ranges can give more concurrency. Flag updates only need to lock the part of the file they're modifying. Expunges need to lock the part which it's removing. Appending needs to lock from <end of file-1> to end of file (ie. l_len=0). Keywords file ------------- Named "index.keywords". This contains mappings between keyword numbers and names. TODO: write how it's updated Recent file ----------- "index.recent.<uid>" file tells what the last non-recent UID was. The file doesn't have any data inside it. When opening the mailbox you should find and rename() the existing index.recent.<uid> file to the new index.recent.<last-uid>. A successful rename() means the mails between <uid>+1 .. <last-uid> are recent for you. A failed rename() means that someone else managed to do it. You should look for the file again and if it was renamed to less than <last-uid>, you should try again. Nightly cleanups ---------------- Because expunged mails may be left lying around for a while, there probably should be some nightly check that cleans them up. Other cleanups that may be necessary after crashes are: - .lock files - expunged data at the end of files (can be checked with same rules as how appending determines the offset where to append) Alternative ways to use ----------------------- Expunging mails isn't very filesystem quota friendly since it requires creating another file to copy the expunged mails to. This could also be done by just moving the data within the file. This however requires that all file reads must also be shared-locked. Also it has the problem of leaving the mailbox corrupted if the copying gets aborted in the middle. For the best performance the mail flags could be stored only in Dovecot's index files. In that case the flag update locking isn't slowing down expunges (and if fcntl() locks can't be used, doesn't slow down appends either). Also there's no need for keywords/recent files. Compatibility ------------- If needed, it would be possible to create new/ and tmp/ directories under the mailbox to allow mail deliveries to be in maildir format. The maildir files would be then either just moved or maybe be appended to existing mail files.

1 0

[Dovecot] Mail directory funkiness
by Ted Kaczmarek 27 Sep '05

27 Sep '05

I am using dovecot-0.99.14-4.fc4 on FC4, was testing using mbox and than switched to using Maildir. default_mail_env = maildir:~/Maildir >From evolution when I create a folder the folder is created in ~/mail instead of ~/Maildir I am still rather new to dovecot, but this does not make sense. I made sure to restart dovecot, even added a .procmailrc. cat .procmailrc MAILDIR=/home/tedkaz/Maildir :0 ./ I am going to grab the latest version and test with that, but if if anyone has insight into how I got here it would be appreciated :-) Likely a pilot error. Regards, Ted

2 3

Re: Re: [Dovecot] virtual domain
by Barry Kwok 27 Sep '05

27 Sep '05

>On Mon, 2005-09-26 at 21:33 +0800, Barry Kwok wrote: >> we are currently using vm-pop3d to implement virtual domains. But due to some locking problem in vm-pop3d, we want to test the possibility to change to dovecot. >> Currently, in vm-pop3d, the passwd file is located in >> /etc/virtual/domain1.com/passwd >> /etc/virtual/domain2.com/passwd > >passdb passwd-file { > args = /etc/virtual/%d/passwd >} > >This requires Dovecot 1.0.alpha2 or later. > >> and the passwd format is: >> user: md5 encrpted password > >In MD5crypt format? ie. $1$something? Then it should work directly. > >> and the email is store in >> /var/spool/virtual/domain1.com/user >> /var/spool/virtual/domain2.com/user > >As mboxes? For IMAP you'll need also some directory where non-INBOXes >are stored, and with POP3 Dovecot also wants a directory where it can >store index files. > >If you don't want either, you could do something like: > >default_mail_env = mbox:/tmp:INBOX=/var/spool/virtual/%d/%u:INDEX=MEMORY Thank you. Timo. I have installed Dovecot 1.0 alpha3 and tested. It is a mbox POP3 only server. The relevent part of dovecot.conf I have changed: default_mail_env = mbox:/tmp:INBOX:/var/spool/virutal/%d/%u:INDEX=MEMORY passdb passwd-file { # Path for passwd-file args = /etc/virtual/%d/passwd } userdb static { args = uid=503 gid=503 mail=mbox:/var/spool/virtual/%d/%n } I can login with user(a)domain1.com but there is error in logfile: Sep 27 13:15:16 ns2 dovecot: pop3(user(a)domain1.com): mbox: Can't create root IMAP folder /mail: Permission denied Sep 27 13:15:16 ns2 dovecot: pop3(user(a)domain1.com): Failed to create storage with data: mbox:/var/spool/virtual/domain1.com/user Sep 27 13:15:16 ns2 dovecot: pop3-login: Login: user=<user(a)domain1.com>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Sep 27 13:15:16 ns2 dovecot: child 19147 (pop3) returned error 89 I then created a /mail directory with file mode 777, then it is OK. Why I need a IMAP folder in a POP3 only server and can I change that folder to /tmp directory.

2 1

[Dovecot] Misbehavior with Dovecot and Mulberry
by Jon Roma 27 Sep '05

27 Sep '05

I'm having a bit of misbehavior wherein Dovecot seems to refuse to cooperate with my Mulberry MUA. By and large, everything works great. I can move mail back and forth happily. I can compose a note and copy the outgoing mail to my Dovecot "Sent" folder using my default Mulberry settings. But if I reply or forward a mail, I get a Mulberry error popup saying that it "couldn't decode the response from the server" and that the operation was unsuccessful. This was somewhat mystifying (especially since this does work for a newly composed message as opposed to a reply/forward), so I turned on Mulberry logging. This is what I see (with unrelated operations clipped from the log). First, the first case where I create a new message, send it, and save a copy in my Sent folder: --> #6.3408 Mon Sep 5 23:24:27 2005 A00085 APPEND Sent (\Seen) {418+} * 5544 EXISTS * 1 RECENT A00085 OK Append completed. All's well there. Mulberry does a FETCH to grab the new message (reply) from my Sent folder and then does a SORT so that it can order it properly in the display. This too succeeds but is probably irrelevant to us. After a couple NOOP commands come and go in the background, I try to reply to a message in one of my mailboxes. In this case it happens to be the one in my Sent folder. So Mulberry issues a FETCH (you can see the original mail there in the FETCH response from Dovecot). It then does a STORE of the original mail, apparently to update the \Answered flag. It then tries to APPEND the reply to the Sent folder, but this time, the server comes back with a NO. See below --> #6.3408 Mon Sep 5 23:24:43 2005 A00089 FETCH 5544 (BODY[1]) * 5544 FETCH (BODY[1] {63} ) A00089 OK Fetch completed. --> #6.3408 Mon Sep 5 23:25:13 2005 A00090 FETCH 5544 (RFC822.HEADER) * 5544 FETCH (RFC822.HEADER {355} Date: Mon, 05 Sep 2005 23:24:27 -0500 From: Jon Roma <roma(a)uiuc.edu> To: Jon R Roma <roma(a)uiuc.edu> Subject: Test 1 Message-ID: <DDDD999F0F4E38F8C1510CC8(a)jrr2.isdn.uiuc.edu> X-Mailer: Mulberry/3.1.6 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline ) A00090 OK Fetch completed. --> #6.3408 Mon Sep 5 23:25:13 2005 A00091 STORE 5544 +FLAGS (\Answered) * 5544 FETCH (FLAGS (\Answered \Seen \Recent)) A00091 OK Store completed. --> #6.3408 Mon Sep 5 23:25:13 2005 A00092 APPEND Sent (\Seen) {610+} A00092 NO I am having similar problem saving a draft to my Drafts folder and suspect the same thing is behind both of these. I didn't create a Mulberry trace log for this but can do so if need be. I am running 1.0-test80 with Timo's patch to fix the "time off by 200 minutes" error with mbox. All my folders are maildir at this point -- I've converted them out of my UW mbx format store. My client is Mulberry 3.1.6 for Windows and the OS is AIX 5.1. This weird behavior is a mystery to me!

3 13

[Dovecot] Problems with alpha3 + openssl-0.9.8
by Andy Cravens 27 Sep '05

27 Sep '05

Has anybody successfully compiled alpha 3 with the newly released openssl-0.9.8? I just wasted the weekend trying to get dovecot to compile using the new openssl libs. Openssl-0.9.7d works like dream. Just wanted to know if anybody had this combo working.

3 2

[Dovecot] 1.0alpha3 packages for debian
by Jaldhar H. Vyas 26 Sep '05

26 Sep '05

Sorry this took even longer than I thought but they should be arriving in unstable this afternoon. I also have a backport for stable available at http://www.braincells.com/backports This directory is not aptable so manually install the packages for now. (Hopefully backports.org will have their sarge repository open soon. When that happens they will move there.) The configuration file format has changed so you will want to carefully check /etc/dovecot/dovecot.conf when upgrading. I forgot to put a note about that in NEWS.debian sorry. I think I might have forgotten a couple of config file tweaks people have suggested and in general, if you sent me mail in the past month or so and I didn't respond or the problem still isn't fixed, please give me a nudge. This is why you really ought to use the Debian bug tracking system rather than mailing the maintainer. Anyway, enjoy. -- Jaldhar H. Vyas <jaldhar(a)debian.org> La Salle Debain - http://www.braincells.com/debian/

1 0

[Dovecot] Dovecot-Pop3d
by Mark Smith 26 Sep '05

26 Sep '05

What port does Dovecot-Pop3d run on? 110? If so I can't get it to work "netstat -ltnp | grep ':110'" shows nothing. and dovecot-pop3d is installed. Regards, _________________________________________________________________ The new MSN Search Toolbar now includes Desktop search! http://toolbar.msn.co.uk/

2 1

[Dovecot] virtual domain
by Barry Kwok 26 Sep '05

26 Sep '05

we are currently using vm-pop3d to implement virtual domains. But due to some locking problem in vm-pop3d, we want to test the possibility to change to dovecot. Currently, in vm-pop3d, the passwd file is located in /etc/virtual/domain1.com/passwd /etc/virtual/domain2.com/passwd and the passwd format is: user: md5 encrpted password and the email is store in /var/spool/virtual/domain1.com/user /var/spool/virtual/domain2.com/user and we login as user(a)domain1.com, user(a)domain2.com . We want to maintain this format as login id and the password format. What should be changed in dovecot.conf? Thanks.

3 2

[Dovecot] DSA support for TLS?
by OpenMacNews 25 Sep '05

25 Sep '05

hi all, i've dovecot TLS working correctly w/ locally generated *RSA* CA cert, domain privkey & self-signed domain cert. to that end, my dovecot.conf includes: ssl_key_file = /var/Security/mail.testdomain.com.privkey.rsa.pem ssl_cert_file = /var/Security/mail.testdomain.com.cert.rsa.pem ssl_ca_file = /var/Security/MyCertificateAuthority.CA.cert.rsa.pem ssl_verify_client_cert = no ssl_cipher_list = ALL:!SSLv2:!aNULL:!NULL:!EXPORT:!DES:!LOW:@STRENGTH however, if I try to setup for DSA use: ssl_key_file = /var/Security/mail.testdomain.com.privkey.dsa.pem ssl_cert_file = /var/Security/mail.testdomain.com.cert.dsa.pem ssl_ca_file = /var/Security/MyCertificateAuthority.CA.cert.dsa.pem ssl_verify_client_cert = no ssl_parameters_file = /var/Security/dsaparam.pem ssl_parameters_regenerate = 0 an attempt @ dovecot launch results in a logged error of: dovecot: Sep 11 11:58:43 Error: imap-login: Can't load private key file /var/Security/mail.testdomain.com.privkey.dsa.pem: error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key wherein it looks like dovecot is _still_ seeking an RSA key. fwiw, % cat /var/Security/mail.testdomain.com.privkey.dsa.pem -----BEGIN DSA PRIVATE KEY----- ... i've searched the dovecot wiki, and although the _only_ reference i find to dsa/diffie is @: http://wiki.dovecot.org/moin.cgi/MainConfig " ... SSL parameter file. Master process generates this file for login processes. It contains Diffie Hellman and RSA parameters. ssl_parameters_file = /var/run/dovecot/ssl-parameters.dat ... " it does seem to imply that DSA certs are, at least, supported. comments? cheers, richard

2 4

Re: [Dovecot] ManageSIEVE services with Dovecot cvs-LDA?
by OpenMacNews 25 Sep '05

25 Sep '05

i can't find a trace of any sieve mgmt services in dovecot as yet, and haven't found anything in this list ... so, if i *am* missing it, if someone would be kind enuf to point me in the right direction, i'd appreciate it! 8-} in the meantime, a little more info for those interested ... the latest spec for sieve management, "A Protocol for Remotely Managing Sieve Scripts", can be found at: <http://www.ietf.org/internet-drafts/draft-martin-managesieve-05.txt> iiuc, one of the authors, Tim Martin / Mirapoint Inc., is the "Tim" of "timsieved" -- Cyrus' ManageSIEVE 'server' bundled with cyrus-imap. i can strip timesieved it out of a successful build of cyrus-imap, but it seems to be dependent on cyrus-sasl ... as such, i'm not sure how to integrate it with dovecot's SASL auth methods. i have a feeling that such fuctionality would have to be re-implemented in dovecot alongside of sieve itself ... and probably should, if it were to be included. the courier-imap mailing lists also have folks searching for an alternative to timsieved ... richard

1 0