- dovecot - dovecot.org

[Dovecot] v1.1 plans
by Timo Sirainen 14 May '07

14 May '07

I think I won't do any actual releases until it's mostly feature complete. Then maybe v1.1.alpha1 or v1.1.beta1. So if you want to test it before then, use CVS or the nightly snapshots. I'm planning on keeping v1.1 almost completely compatible with v1.0. There could be some minor configuration file changes, but for most people v1.0's dovecot.conf should work with v1.1. I want to update index file handling to use less locks and update dovecot.index file less often. The file formats would still stay backwards compatible with v1.0. After this I'm hoping to get index files working well with NFS even with attribute cache enabled. Dovecot v2.0 then brings incompatible index files and more or less rewritten configuration file handling. It could still support old config file format and settings, but the default will be somewhat different. I'm hoping to release the first alphas/betas in 2-3 months, with v1.1.0 maybe even as early as next summer. The good thing here is that index file code hasn't changed much since v1.0, so it should be stable. The bad thing is that I've added a couple of completely new indexes. I'm also quite busy for the next month, but I'm not sure how much it affects my coding. :) The largest already implemented new features are: + Added support for IMAP UIDPLUS extension + IMAP SORT: Sort keys are indexed, which makes SORT commands faster. + IMAP THREAD: Threads are indexed into a dovecot.index.thread file. It can usually be updated incrementally, so this makes THREAD fast. + Added Dovecot-specific X-REFERENCES2 threading algorithm. It's similar to REFERENCES, but it doesn't do base subject merging and it sorts the threads by their newest message. + When saving messages, update cache file immediately with the data that we expect client to fetch later. Maildir-only currently. + Mailbox list indexes. Mailbox metadata is stored there, so STATUS commands can return synchronized mailboxes' metadata simply by reading a single mailbox list index file. - FIXME: Currently works only with Maildir++ layout. + Expire plugin can be used to keep track of oldest messages in specific mailboxes. A nightly run can then quickly expunge old messages from the mailboxes that have them. The tracking is done using lib-dict, so you can use either Berkeley DB or SQL database. + Namespaces are supported everywhere now. - FIXME: except by convert plugin + Full text search indexing support with Lucene and Squat backends. + OTP and S/KEY authentication mechanisms (by Andrey Panin). + mbox and Maildir works with both Maildir++ and FS layouts. You can change these by appending :LAYOUT=maildir++ or :LAYOUT=fs to mail_location. Features that I'm planning on implementing: - Fully supported shared mailboxes and IMAP ACL extension - Replace Squat FTS indexes with my new design - Case-insensitive searches with non-ASCII text as well - Maybe add support for all kinds of IMAP extensions that can be easily supported. LEMONADE extensions especially: CONDSTORE, CATENATE and maybe even URLAUTH if I can figure out how it should work. If a feature can be implemented completely as a plugin, it could still come in v1.1.beta stage, because it won't destabilize the rest of the Dovecot anyway.

17 36

[Dovecot] Feature Request: content-filter / MOVE scripts
by Daniel L. Miller 14 May '07

14 May '07

Since your "1.1 plans" announcement somehow turned into everybody handing in a wish list . . . thought I'd start one by itself - since I expect a few people to have opinions on this matter. I'm aware of a plug-in functionality in Dovecot, and that a plug-in exists for dspam processing - but my review of the plug-ins indicate they require compiling your own Dovecot - the plug-in setup isn't really suitable for packaged distros. As a packaged software user - I really really really don't like compiling my own software, either for features or bugfixes - I want leave that to the experts. On learning of 1.0's official release, I nudged the Debian package maintainers to check - and they were already on it and it updated. Thanx Timo, Jaldhar, and Fabio. I have some thoughts on an interface that would eliminate the need for compile-time plug-ins, but provide some major power to sysadmins. Basically, I want a way of telling an external program the following: 1. A message was moved. 2. The folder is moved from was XXX. 3. The folder it moved to was YYY. 4. The message moved was ZZZ. Armed with this information, just about anybody's spam fighting learn scripts can be called to do their thing. A few different ways have occurred to me: 1. Have a dedicated "content-filter" functionality built-in to Dovecot. Under this model, Dovecot.conf would include at least three new parameters: FILTER-FOLDER-SUFFIX, FILTER-FOLDER-IN-SCRIPT, and FILTER-FOLDER-OUT-SCRIPT. Whenever a message is moved into or out of a folder ending in SUFFIX the appropriate SCRIPT is called, with the arguments above being passed. 2. A bit more generic, less processing for Dovecot. Dovecot.conf parameter ACTION-SCRIPT (hey, feel free to use other names - I'm just brainstorming here). Any time any message is moved anywhere, call SCRIPT with the arguments. Having never looked at Dovecot's code, I'm just guessing, but this probably wouldn't take much to implement - but might be the most performance affecting. 3. Have a definable named-pipe to send the same information to. 4. Have a definable TCP socket to send the same information to. In any case, I figure it's best to prevent any message processing by Dovecot itself - so while it might be nice to define a list of headers to send to the scripts, instead of the whole message, that's probably not appropriate for Dovecot to focus on. On the other hand, any of the above options would probably benefit from a MAXIMUM_FILTER_SIZE parameter to limit the maximum message size. So what's the world got to say about this? (More importantly, how does TIMO feel about this?) ;). -- Daniel

3 2

[Dovecot] Dovecot, Postfix and SMTP AUTH....
by Bartosz Toczek 13 May '07

13 May '07

Hello. I've notice a problem with SMTP AUTH in postfix with dovecot. My configuration is based on Postgresql. I've created a function in postgresql which returns password and username and attached it to password_query. Query looks like this password_query = SELECT username_out as username, password_out as password FROM get_password(lower('%n'),lower('%d')); Problem is when get_password returns something like this... vmail=# SELECT username_out as username, password_out FROM get_password(lower(''),lower('')); LOG: statement: SELECT username_out as username, password_out FROM get_password(lower(''),lower('')); LOG: duration: 2.342 ms statement: SELECT username_out as username, password_out FROM get_password(lower(''),lower('')); username | password_out ----------+-------------- | (1 row) It returns 1 empty row.... When this is attached to postfix with smtp auth with such configuration # SMTP AUTH broken_sasl_auth_clients = yes smtpd_sasl_auth_enable = yes smtpd_sasl_security_options = noanonymous smtpd_sasl_authenticated_header = yes smtpd_sasl_local_domain = smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth my serwer starts to be open relay... I now that it returns empty string for username and password... so where is the password checked... if I dont send password as a parameter to my function. Definition of the function looks like this: CREATE FUNCTION get_password("login" character varying, "domain" character varying, OUT username_out character varying, OUT password_out character varying) RETURNS record Can you help with that... BT ---------------------------------------------------------------------- Zrob numer kumplom >> http://link.interia.pl/f1a5d

4 4

[Dovecot] Public folders using NAMESPACE
by Joakim Ryden 13 May '07

13 May '07

Howdy folks! I've been trying to research how to (or if possible at all) implement public folders using the namespace feature on a server with multiple virtual domains. After about an hour of looking around I'm a little confused. :) I guess there are two questions: 1) Is it possible / does it work? 2) If yes, does anyone have a sample configuration or write-up of How You Did It[TM] to share? I've been mainly focusing on the thread from here: http://www.dovecot.org/list/dovecot/2006-June/013683.html and the wiki ACL documentation: http://wiki.dovecot.org/ACL but I'm having a hard time piecing it all together... Thanks for any and all tips! --Jo

2 5

[Dovecot] mailquota in ldap attribute 'mailQuota'
by Shahmat Dahlan 13 May '07

13 May '07

I've tried having the ldap quota in an ldap attribute, which still doesn't work. /etc/dovecot.conf protocols = imap pop3 mail_location = maildir:/data/mail/%u/Maildir protocol imap { mail_plugins = quota imapquota } plugin { quota = maildir:ignore=Trash } the rest are defaults added an attribute 'mailQuota' with value '10240' kbytes or 10Mbytes as for /etc/dovecot-ldap.conf user_attrs = mail,homeDirectory=home,uidNumber=uid,gidNumber=gid,quotaStorage=mailQuota Would appreciate if anyone could assist on this. I'd prefer to use the above than setting the mail quota manually: quota = maildir:storage=10240:ignore=Trash We are actually in the planning stage of migrating all of current our mailboxes from Netscape to Dovecot, instead of upgrading it to those new Sun.

2 1

[Dovecot] Deferring, instead of rejecting, messages when at quota
by Mike Brudenell 13 May '07

13 May '07

Greetings - I'm having trouble working out where to alter this behaviour (so please forgive me if it turns out to be an Exim thing rather than Dovecot!)... My test service is using Exim as the MTA and Dovecot's "deliver" for the LDA. We are using Maildir mailboxes with filestore quotas. When I send a large test message to myself that cannot be delivered because I don't have sufficient available quota remaining an error message is immediately returned to the sender and the incoming message discarded. The error message says this: Your message was automatically rejected by Dovecot Mail Delivery Agent. The following reason was given: Not enough disk space Instead of Exim rejecting the incoming message and generating the error response because of deliver's failure to, umm, deliver I'd like the message to be retained in Exim's queue so that it can be retried periodically. With our old/live system (Sendmail + UW IMAP's "tmail" LDA) I edited the tmail source code to return a different exit code: one that caused Sendmail to defer (re-queue) the message rather than reject it. How do I achieve this effect with Exim + deliver please? Is it a case of: * Setting something in Dovecot's configuration file (I can't spot what!); * Changing a return code in deliver's source (if so, whereabouts and to what?); * Or is there an Exim directive I need to add to the transport definition in its configuration file? With thanks, Mike B-) -- The Computing Service, University of York, Heslington, York Yo10 5DD, UK Tel:+44-1904-433811 FAX:+44-1904-433740 * Unsolicited commercial e-mail is NOT welcome at this e-mail address. *

3 3

[Dovecot] wierd ssl-parameters.dat regeneration error
by Kyle Wheeler 13 May '07

13 May '07

I just had a strange problem with my Dovecot 1.0 installation. After about two weeks, it logged these messages: dovecot: Error: ssl-build-param: rename(/usr/local/var/lib/dovecot/ssl-parameters.dat.tmp, /usr/local/var/lib/dovecot/ssl-parameters.dat) failed: No such file or directory dovecot: Error: child 30689 (ssl-build-param) returned error 89 And thereafter, all login attempts resulted in this: dovecot: Error: imap-login: read(ssl-parameters.dat) failed: Unexpected EOF dovecot: Error: child 19036 (login) returned error 89 Thankfully, deleting the ssl-parameters.dat file (/var/run/dovecot/memoryhole.net/login/ssl-parameters.dat) and restarting dovecot fixed the problem, but it seems odd to me that dovecot expects /usr/local/var/lib/dovecot to exist (because, of course, it doesn't). I'm not defining that directory in the config file anywhere. Does anyone have any idea what's going on? ~Kyle -- I'm sick of following my dreams. I'm just going to ask them where they're going and hook up with them later. -- Mitch Hedberg

2 10

[Dovecot] Mailbox creation with Spam
by Master Abi 13 May '07

13 May '07

Hi all, I got dovecot working, but I would like to know when and what creates the mailbox (cur, new, trash, etc ). I noticed that when a client connects for the first time that mailbox is created or when a mail arrives for the first time (i think). I would like to give each mailbox a Spam folder, and possibly doing this automatically. Is this possible with some dovecot setting and not via scripting. Thanks master

2 1

[Dovecot] fs quota plugin and NFS
by Nicolas STRANSKY 13 May '07

13 May '07

Hi, I'm trying to use the Dovecot v1 fs quota plugin. The server uses NFS mounted volumes for INBOX and other maildir folders. The /usr/bin/quota command is working seamlessly but I get errors with the quota plugin, which gives the following logs: Apr 19 17:46:15 rouge dovecot: IMAP(xyxyxyx): quotactl(Q_GETQUOTA, nfs.xxx.yyy.org:/home) failed: No such file or directory Apr 19 17:46:18 rouge dovecot: IMAP(yxyxyxy): quotactl(Q_GETQUOTA, nfs.xxx.yyy.org:/home) failed: No such file or directory Is there a way to get it work ? Thanks, -- Nico Il faut s'entraider, c'est la loi de nature. -+- Jean de La Fontaine (1621-1695), L'Âne et le Chien (Fables VIII.17) -+-

2 3

[Dovecot] mailquota in ldap attribute 'mailQuota'
by Shahmat Dahlan 13 May '07

13 May '07

I've tried having the ldap quota in an ldap attribute, which still doesn't work. /etc/dovecot.conf protocols = imap pop3 mail_location = maildir:/data/mail/%u/Maildir protocol imap { mail_plugins = quota imapquota } plugin { quota = maildir:ignore=Trash } the rest are defaults added an attribute 'mailQuota' with value '10240' kbytes or 10Mbytes as for /etc/dovecot-ldap.conf user_attrs = mail,homeDirectory=home,uidNumber=uid,gidNumber=gid,quotaStorage=mailQuota Would appreciate if anyone could assist on this. I'd prefer to use the above than setting the mail quota manually: quota = maildir:storage=10240:ignore=Trash We are actually in the planning stage of migrating all of current our mailboxes from Netscape to Dovecot, instead of upgrading it to those new Sun.

1 0

[Dovecot] Migrating to Maildir -> path
by Wouter de Jong 13 May '07

13 May '07

Hello, I'm in the process of migrating to Maildir, and I wondered what would be an acceptable path. I want to do virtual-accounts only, so $HOME/Maildir/ is not an option. I was thinking of /some/path/username(a)domain.tld/Maildir/ and later perhaps /some/path/(firstletter)/username(a)domain.tld/Maildir/ but perhaps Maildir/ is a bit too redundant here.... Anyone who could advice ? Many thanks, Wouter

3 2

[Dovecot] Mixing MBOX and Maildir?
by Marc Perkel 13 May '07

13 May '07

Is it possible to have some folders be mbox and some be maildir format? Why you might ask? Sometime for diagnostics I have the Exim MTA archive certain messages in MBOX format and then I want to copy them into an imao folder to look at them with Thunderbird. When I used MBOX all I had to do was copy the file into a folder and it worked. So what I'd like to do is have a folder with a special name that is MBOX and the rest be maildir. Ideally have Dovecot be smart so that if the file exists then it is assumed to be MBOX and if the directory exists then it is assumed to be maildir. Has anyone done anything like this?

6 10

[Dovecot] Time adjusting
by Robert McKenzie 13 May '07

13 May '07

Would it be possible to have an option/variable in the config file to allow for easier setting of the maximum time to sleep for rather than having it hard coded or allow for adjustment up to an amount greater than 5 as it is at present but still have an upper limit hard coded. My system is regularly out on boot up by up to 7 seconds and consequently dovecot kills itself, if I restart it is fine and keeps going without a problem until the next reboot. Regards, Robert McKenzie. -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.467 / Virus Database: 269.7.0/801 - Release Date: 12/05/2007 18:40

4 3

[Dovecot] Writing a new mailbox access module
by Tom Alsberg 13 May '07

13 May '07

Having finished a few other projects, I'm contemplating trying to write a new mailbox module which could be useful for us here. I would appreciate any guidance. Basically, I need (or rather, could use) a module that provides a merge of two mailboxes (which are themselves accessed by other modules), cascading them, so that: * The user sees one mailbox M, which contains all the messages in the mailboxes A and B. * A and B are both already accessible by existing Dovecot mailbox access modules, but possibly by different modules (e.g. A is an mbox, B is a Maildir). * Any write operation (creation of a new message, etc.) is eventually performed on B. * As soon as a message that was in A is accessed (either for read or for write), it is transferred to B. I would use such a module to combine the spool file in /var/mail with some INBOX in the user's home directory. The incentive is the following: * Traditionally, users receive their mail in the spool, commonly in /var/spool. Mailers, like mail, elm, pine, mutt, etc. move each read message to some record mailbox (commonly ~/mbox, but with the more advanced mailers it can be configured to be some other mailbox, even a Maildir). * Users keep working this way. IMAP will not substitute the regular mail access when inside the network here. But it should be as convenient as possible to use it from the outside (e.g. at home, or from an Internet Café, or using some Webmail interface). * Mainly for administrative reasons, it is desirable that mail keeps being delivered to the spool, and not directly to the user's home directory. One of the reasons is that mail never fails to be delivered as long as there is space in the spool. When the user is out of quota on his home directory, the mail does not get rejected, and a failure only occurs when the user initiates some operation (like reading the message). Another reason is simply to be compatible with the way traditional mailers and users work. * IMAP, and IMAP clients (e.g. Thunderbird, the most common one here), on the other hand, usually do not recognize the concept of a spool. There is an INBOX, where all mail arrives, and this is where all new messages should arrive, and remain, unless they are deleted moved elsewhere. * It would of course not be desirable to just move messages automatically after the user reads them, because it would confuse users when a message suddenly disappears from a mailbox all by itself. * The best way I can think of to be consistent with both the traditional way mailers work (moving messages from spool to ~/mbox after they are read) and the way IMAP is commonly used (INBOX contains all received messages not deleted or moved, including messages that just arrived), would be using such a module as I suggested, with (using the previous terminology) e.g. M=INBOX, A=spool, and B=~/mbox. I have just skimmed again through the mailbox code (which I last looked at over a year ago, back when it was in alpha releases), not going too much into details. From past experience and current skimming, working with that code doesn't seem too straightforward, though. I can think of how to implement the logic I described on its own (i.e. as a library with a few functions), but not sure how difficult it will be to interface it with Dovecot. Is there any developer/module writer's guide somewhere that explains the mailbox access module interface? I'd appreciate some basic guidance to start with. If I get to working on it seriously, expect lots of questions from me about the code :-) Regards, -- Tom -- Tom Alsberg - hacker (being the best description fitting this space) Web page: http://www.cs.huji.ac.il/~alsbergt/ DISCLAIMER: The above message does not even necessarily represent what my fingers have typed on the keyboard, save anything further.

2 1

[Dovecot] dovecot deliver tries to setuid but why?
by Daniel 13 May '07

13 May '07

Hi! I've set up postfix(2.4.1) + dovecot(1.0-cvs) + dovecot-lda(1.0-cvs). Dovecot's deliver is running as vmail:vmail (according to postfix's master.cf). Now the problem (when receiving mail): deliver(leva): auth input: leva deliver(leva): auth input: uid=8006 deliver(leva): auth input: gid=8000 deliver(leva): auth input: home=/var/mail/virtual/leva deliver(leva): setuid(8006) failed: Operation not permitted Every virtual user has different uids, but the gid is the *same* for everyone. The gid is 8000(vmail) for everybody, and deliver runs as 8000:8000 (vmail:vmail). vmail group has write access to the users' maildirs. So why does deliver try to setuid when it has group write permission to the destination directory? Thanks! Daniel

2 1

[Dovecot] Virtual User Home Directory, and APOP Clear Text Passwords
by imacat 13 May '07

13 May '07

Dear all, Hi. This is imacat from Taiwan. I'm new to this list. I was migrating from Qpopper to Dovecot. (Qpopper copies the mailbox when POP3 logged in, which causes quota problem.) Generally Dovecot is nice. However, I have a few questions: 1. I have a few virtual users, and I'm not using IMAP. Do I still have to give them a writable home directory? I mean, they exists only for me to receive mails from fetchmail. They really have nothing but their inbox. But if I do not give them a writable home directory, I got this error in my syslog all the time: mbox: Can't create root mail directory /home/homeless/mail: Permission denied 2. I would like to use APOP in addition to SSL/TLS. Currently Dovecot saves APOP passwords as clear text. I understand this. But is it possible to have some sort of encoding, for example, Base64? Just to prevent casual reads. I have several users. I do not want to know their passwords just to add them to the APOP database. I do not want other roots to read my passwords, intended or not. Encoded passwords also have the advantage that I can use passwords that contain the colon sign ":". It would be nice if something a little harder than Base64 can be used. Thank you very much in advance. -- imacat ^_*' imacat(a)mail.imacat.idv.tw PGP Key: http://www.imacat.idv.tw/me/pgpkey.txt Tavern IMACAT's http://www.imacat.idv.tw/ Woman's Voice http://www.wov.idv.tw/ TLUG List Manager http://www.linux.org.tw/mailman/listinfo/tlug

2 6

[Dovecot] SASL using different auth database?
by Dave McGuire 13 May '07

13 May '07

Hi folks. Is it possible to configure Dovecot to authenticate against one mechanism (i.e., /etc/passwd) for POP/IMAP connections, while providing a SASL authentication socket that satisfies authentication requests against a different mechanism, say MySQL? The goal here is to allow users to have different passwords for SMTP authentication and POP/IMAP connections. I am converting an existing installation to Dovecot...this installation currently has quite a few users who are configured with SMTP authentication (Cyrus SASL/saslauthdb) using different passwords than are configured for their POP connections (via cucipop, /etc/passwd). If someone can think of any better way to accomplish this goal, I'm all ears. Thanks, -Dave -- Dave McGuire Port Charlotte, FL

3 5

[Dovecot] logging IMAP size?
by Armijn Hemel 12 May '07

12 May '07

hi all, is there a way to somehow log the size of an IMAP session with Dovecot? With POP3 there is this pop3 format string I can set in dovecot.conf, but I don't see an equivalent for IMAP. If so, are there any plans to add this functionality? thanks, armijn -- --------------------------------------------------------------------------- armijn(a)uulug.nl | http://www.uulug.nl/ | UULug: Utrecht Linux Users Group ---------------------------------------------------------------------------

3 3

[Dovecot] dbox redesign
by Timo Sirainen 12 May '07

12 May '07

I don't think anyone uses dbox currently, so the whole format could still be redesigned. So I was thinking about doing two major changes: 1. Rely on index files a lot more. The flags are already stored in index files, so there's no need to waste I/O updating them to dbox files all the time. They could still be updated (if indexes get deleted, the flags aren't all gone), but less often. 2. Require fcntl() locking. Currently dbox uses dotlocks which is slow. Cydir could be a good alternative also once index file code is made a bit more robust. Perhaps I could implement single instance attachments for cydir too.. Locking ======= The current dbox "index" file would be gone. It's pretty useless. Replace it with a whole new index file. Or perhaps it should be called "locks" file or something. The locks file would contain records: <file id> <lock timestamp> <status>. So something like: <header> 1 4645a60f 0 2 00000000 N 3 00000000 D That would mean that the first file is locked by some process, either for appending or expunging. The 2nd and 3rd files aren't locked. New messages can't be appended to 2nd file anymore. 3rd file is already deleted and this record needs to be removed when rewriting the file. Locking a dbox file for either appends or expunges is done like: 1. See if timestamp is zero - If not, see if it's older than .. let's say a day or so .. a) Yes: Continue to 2. b) No: Assume the file is locked 2. Do fcntl() byte range lock over the record line in the locks file. - If it failed, the record is locked 3. Write the timestamp. 4. Compare stat() and fstat() inodes to see if the file was rebuilt - If yes, reopen the file and goto 1 5. File is now locked. Do the append/expunge. 6. Write timestamp to zero. 7. Unlock the byte range. If a file is locked, append will try another file and expunge will mark the message as expunged instead of actually expunging it yet. Note that the locks file is read without locking. This is safe because data is never moved within the file, and it doesn't matter if the timestamp isn't read correctly always. The timestamp check is only an optimization. Actually I'm not sure if it would be better not to have the timestamp at all. Deleted records will stay in the file until the file is rebuilt. If a deleted record is noticed in the file, the process tries to lock the whole locks file. If it succeeds, it proceeds with writing the non-deleted records to a temporary file and rename()ing it over the locks file. Appending ========= 1. Find the first file in the locks file that has appendable=0 - If no such file was found, go to "create a new file" logic as described below 2. Lock the file record 3. Verify from the file's headers that this file can actually be appended to - If messages have been expunged from a dbox file, it can't be safely appended to anymore. - Other reasons include eg. configurable max. file size and daily rotations 4. Write the mail 5. Lock locks file's header 6. Get the UID from "next uid" field and update it 7. Unlock the header 8. Unlock the file record 9. Update index file Create a new file logic: 1. Create a temporary file 2. Write the messages there 3. Lock locks file's header (including fstat() / stat() rebuild check) 4. See what the latest file ID is in the file 5. rename() temp fail to msg.<latest file id+1> 6. Lock locks file for the range of the to-be-written record below 7. Write the new record to locks file 8. Go to step 6 in the original append logic Syncing / expunging =================== If locks file's header's "next uid" doesn't match the one currently in index file, the appending crashed between steps 6 and 9. Find the new message(s) and append them to index file. If the locks file is completely gone , rebuild it by going through all the msg.* files in the directory. If "expunge counter" (see below) doesn't match in locks file's header vs. index file header, go through all the msg.* files to see if a message exists in multiple files. If found, remove the duplicates. Typically neither of the above happens, so the only thing to do here is to write changes from index file to dbox files. This may mean flag changes once in a while, but most importantly expunges will always be synced. Initially figure out what files require expunging. Try to create a single lock range that includes all of them. If there are non-zero lock timestamps in that range, create multiple ranges. If a file couldn't be locked, the expunge is done by updating expunge-flag in the file. This can be done without locking (see below for flag updates). If a file was successfully locked, the expunging is done by: 1. Update "expunge offset" in file header to the offset of the first expunged message. If expunge offset is non-zero, the file is treated as non-appendable. Also when rebuilding and finding the same message from multiple files, this field is used to figure out which file should be truncated. 2. Copy the rest of the non-expunged messages to a new temporary file and add the file to locks file using the "Create a new file" logic described for appends. Instead of updating "next uid" field, update "expunge counter" field. 3. ftruncate() the old file. 4. Update "expunge counter" in index file. Sync locking ============ If there is one big "sync lock", then flags can be updated in any file with relying only on that lock. Some files may be locked, but they're locked for appends so flag updates (and other record updates as well) are safe. It would be interesting to make it work without that sync lock though, so that multiple processes could be syncing dbox at the same time. But there probably isn't much point. If the sync is locked, there's no real requirement to do the syncing at all then, it's just left to be done later. If multiple processes were syncing at the same time, the only problem would come from flag updates. Either the files would have to be locked for flag updates (with waiting locks, bad bad), or it would need to be able to handle the situation when a flag was written just when another process expunged messages from the file and truncated it. That could be done by checking (one way or another) after the flag write if a truncation had happened, and if so do the truncation again. Preventing out of sync disconnects ================================== Reading messages from dbox files requires no locking, so it's possible that while one session is reading a message another session goes and expunges the message by truncating the file. The only way to handle this situation is to disconnect the client. Also even if the message wasn't in the middle of being read, this kind of a out-of-sync could happen if a session hasn't synced itself yet (no NOOP/IDLE/etc. for a while) and tries to begin reading an expunged message. These could be avoided if the expunging process knew that there are other processes that still know about the message, so it could just mark the message as expunged instead of actually doing it. I think this could be done with yet another file that tracks what messages are expunged. The file would contain "dovecot.index.log file sequence + offset" records. Whenever message(s) are expunged, a new seq +offset is written containing the position in the log file for that transaction. All the processes accessing the mailbox would then read lock the file from eof to infinity. The lock is updated whenever mailbox is synced (and being careful not to have race conditions there). So syncing process would then try if it can write lock the file from "last real expunge" log position to end of file. If not, then using some binary search find the position how far it can be locked. If it managed to update the write lock position at least some, the messages up to that far in the transaction log can be expunged safely. One problem with this is that the file needs to be rotated once in a while so that it won't grow infinitely. This can be done only if the write lock can include the whole file (including infinity). And that can be done only if there's a single process having the mailbox open, which may never happen with shared mailboxes. I think I'll leave this feature to be implemented later. It's extra overhead that's probably rarely useful. Single instance attachments =========================== I'll change the dbox messages to have variable sized headers. The header could then contain information such as "at offset 5000 insert 11241242 bytes from file AB123427852". Fast copying ============ Would be nice if copying a message from one mailbox to another wouldn't require actually reading+writing the whole message contents. But I can't really figure out how to implement this without requiring that there is only a single dbox storage which contains the mails for all the mailboxes, and the mailboxes themselves are just Dovecot's index files containing pointers to the dbox storage. The problem with having everything in one storage is that if the index files are broken, the messages can't be placed into correct mailboxes anymore. Although one possibility would be treat mailboxes a bit similarly than keywords. So that when a message is copied to another mailbox, the message in dbox file is updated to contain information that it exists in such and such mailboxes. Hmm. Perhaps that would be good enough, yes.

1 0

[Dovecot] Problems with BEFORE searches.
by Trever L. Adams 12 May '07

12 May '07

Ok, I am trying to modify some code given to me by the author of the dspam plugin for dovecot (sorry the name eludes me). I am modifying it to handle UNSEEN differently than SEEN. Below is the python code. I am not a python programmer, but I get the basic idea. ok, uids = conn.search(None, 'BEFORE', unreadsentbefore, 'SEEN') "11-May-2007 06:18:29 -0600" is the value of unreadsentbefore. However, this doesn't match any emails even though they are all dated as 5/10 and they have all been read for several hours (the time was the current to figure out why it isn't working). Is there a problem with BEFORE and SEEN together? Is there a BEFORE bug in the system? Some of the messages are -500 and others, I am not sure. Thanks, Trever Adams

4 6

[Dovecot] read only maildir subscriptions
by Nagyon Almos 11 May '07

11 May '07

Hello, I have some (possibly very simple) questions about subscriptions: I store my old mails in /home/oldmails/ in one Maildir where I have only read-only permissions (I cannot delete, create or modify any files). Under thunderbird, I can successfully log in and read my new mails, but I cannot subscribe (?) within thunderbird to oldmails (it is grey not black as other folders). What did I do wrong? What other permissions does dovecot need? Here's the relevant part of dovecot.conf: namespace private { separator = / prefix = oldmails/ location = maildir:/home/oldmails/:CONTROL=~/Maildir/control-oldmails:INDEX=~/Maildir/index-oldmails } and when I try to subscribe to it, dovecot says: dovecot: 2007-04-30 17:58:53 Info: IMAP(xxx): Effective uid=7001, gid=1000 dovecot: 2007-04-30 17:58:53 Info: IMAP(xxx): Namespace: type=private, prefix=regi/, sep=/, inbox=no, hidden=no, subscriptions=no dovecot: 2007-04-30 17:58:53 Info: IMAP(xxx): maildir: data=/home/oldmails/:CONTROL=~/Maildir/control-oldmails:INDEX=~/Maildir/index-oldmails dovecot: 2007-04-30 17:58:53 Info: IMAP(xxx): maildir: root=/home/oldmails, index=~/Maildir/index-oldmails, control=~/Maildir/control-oldmails, inbox= Thanks for any help in advance. Regards, A. <br><br>_______________________________________________________________<br> <a href="http://cthandler.adverticum.net/?cturl=http%3A%2F%2Fwww.klikkbank.hu%2Flako… " target="new">INTEZZE PÉNZÜGYEIT AZ INTERNETEN!</a> Lakossági számlacsomag havi 199 Ft-ért! Bankkártya éves kártyadíj nélkül! <a href="http://cthandler.adverticum.net/?cturl=http%3A%2F%2Fwww.klikkbank.hu%2Flako… " target="new">www.klikkbank.hu</a>

2 2

[Dovecot] Dovecot not handling r/o mailboxes completely, and problem with ACL as a workaround
by Adam McDougall 11 May '07

11 May '07

First of all, I think dovecot is really fantastic and I have thanks for all the hard work. I think it will be the best fit for my ~5000 users when I have it setup completely. We normally have approx 500 concurrent IMAP connections during the day. I am trying to convert from courier-imap to dovecot, but I have an issue with public namespace folders that are not writable by certain users. Please let me know if I need to provide more information or how I can help solve this issue. In courier-imap, we were able to take advantage of the maildir structure and standard unix users/groups to allow 'decsstaff' members to have full write access while 'decsall' members only have r/o unless also a member of 'decsstaff': -rw-rw-r-- 1 postlocal decsstaff 37597 May 5 23:37 /egr/mail/shared/decs/.support.In/cur/1178422658.M533373P54269.ice drwxrwxr-x 2 postlocal decsstaff 24576 May 5 23:37 /egr/mail/shared/decs/.support.In/cur drwxrws--- 6 postlocal decsall 4096 Apr 22 18:08 /egr/mail/shared/decs/.support.In drwxrwsr-x 34 postlocal wheel 4096 May 1 07:23 /egr/mail/shared/decs In short, upper directories are used to limit world access to just one group 'decsall', and once inside, r/o world access could be used as an anonymous substitution for 'decsall' and the group permissions would allow writing by 'decsstaff'. I don't know how courier stores and checks mail directory permissions, but it was aware when a user would not be able to delete or change messages and would return a READ-ONLY imap error when an attempt to change a message occurred (courier returns this in the IMAP response to update mail flags to reflect that the client wants to move to delete a mail). I understand that imap clients will usually only set status as read or deleted first and only try to delete an email for real on expunge, but it seems dovecot does not try to detect if a message is modifyable before it is too late for the user to find out. What happens is a user appears to be able to mark messages read, move them to another folder, or delete them (another way of moving to another folder eg. Trash). The user is clueless that the changes will not occur, and only finds out later when they reload the folder to find the messages back, or tries to expunge and get an unhelpful 'internal error' from dovecot (I can help trace this situation if it helps, but I think that stage should not be reached if things were working right). My configuration has the public folder control and indexes inside a user's home directory, so it will always be possible for dovecot to record message status changes, however I do not think dovecot should update the control and indexes to reflect an IMAP operation that is not allowed to complete due to restricted permissions on the actual email file. I suppose dovecot would have to check the permissions on mail message files. As a workaround, I wondered if I could apply dovecot ACLs to the public folders to reinforce my intentions of only allowing readonly access to certain users. If ACLs worked, I could even use it as a viable alternative to the unix permissions on the maildir contents and avoid using more unix groups than necessary. For most of our shared mailboxes, there will only be a few persons with write access, and up to a dozen people with read access, so even if I have to list every user seperately I could. However, I have not been able to get ACLs to do anything useful for me, and I hesitate to use the feature to work around might be a configuration error or bug. It is also more convenient for my migration procedure if I do not have to modify file permissions or structure during downtime to switch to dovecot. I am still not sure how dovecot ACLs affect the actual permissons of dovecot processes. I have mostly tried placing a dovecot-acl file in the maildir directory (alongside cur, tmp, new) and dovecot seems to be reading it according to the log, however I am not getting expected results from the ACLs. If the unix permissions allow a client to enter 'cur' and read mails, the mail client thinks it can delete or move them (until expunge) no matter even if I put acl entries with only 'r' permission. ACL questions (which might help others understand better as well): Is a 'user=foo rl' entry supposed to remove any possible write permissions in a maildir covered by such an ACL for user foo? If not, shouldn't it at least enforce restrictions on the available mail metadata which is more likely to be writable by the user? Is it even possible for an ACL to grant additional permissions above what is granted to a user by standard unix permissions? Does an empty ACL equate to the exact permissions granted by unix, and should an admin try to design an ACL to modify as little as possible, or should the admin try to use enough acl entries to form a complete description of the desired resulting permissions? I get the impression that broad ACLs should be listed first, and more specific ACLS listed below can add or remove ACL permissions applied earlier in the file as a 'last match' case. Should the acl plugin be able to print a summary of the resulting permissions in the log (with mail_debug=yes) when accessing a folder with an ACL? A snip from logs, I generated alot today and I'm not sure which portions would be most useful so I'll start with this: May 6 15:20:08 gribble dovecot: IMAP(mcdouga9): acl: initializing backend with data: vfile:/usr/local/etc/dovecot-acls May 6 15:20:08 gribble dovecot: IMAP(mcdouga9): acl: acl username = mcdouga9 May 6 15:20:08 gribble dovecot: IMAP(mcdouga9): acl: owner username = May 6 15:20:08 gribble dovecot: IMAP(mcdouga9): acl vfile: Global ACL directory: /usr/local/etc/dovecot-acls May 6 15:20:08 gribble dovecot: IMAP(mcdouga9): Namespace: type=public, prefix=#shared/decs/, sep=/, inbox=no, hidden=no, subscriptions=no May 6 15:20:08 gribble dovecot: IMAP(mcdouga9): maildir: data=/egr/mail/shared-dovecot2/decs:CONTROL=/home/mcdouga9/Maildir/dovecot/public/control/decs:INDEX=/home/mcdouga9/Maildir/dovecot/public/indexes/decs May 6 15:20:08 gribble dovecot: IMAP(mcdouga9): maildir: root=/egr/mail/shared-dovecot2/decs, index=/home/mcdouga9/Maildir/dovecot/public/indexes/decs, control=/home/mcdouga9/Maildir/dovecot/public/control/decs, inbox= May 6 15:20:08 gribble dovecot: IMAP(mcdouga9): opendir(/egr/mail/shared-dovecot2/decs/tmp) failed: Permission denied # dovecot --version 1.0.0 The reason I have so many public spaces below, is to match the existing directory structure we setup for courier, and also to help us keep the directory listing more tidy since otherwise we would end up with many folders all under one directory. Some of them have monthly rotation and it could get out of hand if we did not split them up. I understand it makes things more difficult for the global acl list at present, if I were to use it, but I may not have any conflicting folder names at this time. # dovecot -n # /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.egr.msu.edu.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.egr.msu.edu.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 0 mail_location: maildir:~/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mail_debug: yes mmap_disable: yes maildir_copy_with_hardlinks: yes mail_plugins: acl mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep namespace: type: public separator: / prefix: #shared/be/ location: maildir:/egr/mail/shared-dovecot2/be:CONTROL=%h/Maildir/dovecot/public/control/be:INDEX=%h/Maildir/dovecot/public/indexes/be namespace: type: public separator: / prefix: #shared/cee/ location: maildir:/egr/mail/shared-dovecot2/cee:CONTROL=%h/Maildir/dovecot/public/control/cee:INDEX=%h/Maildir/dovecot/public/indexes/cee namespace: type: public separator: / prefix: #shared/chems/ location: maildir:/egr/mail/shared-dovecot2/chems:CONTROL=%h/Maildir/dovecot/public/control/chems:INDEX=%h/Maildir/dovecot/public/indexes/chems namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared-dovecot2/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/public/indexes/decs namespace: type: public separator: / prefix: #shared/egr/ location: maildir:/egr/mail/shared-dovecot2/egr:CONTROL=%h/Maildir/dovecot/public/control/egr:INDEX=%h/Maildir/dovecot/public/indexes/egr namespace: type: public separator: / prefix: #shared/me/ location: maildir:/egr/mail/shared-dovecot2/me:CONTROL=%h/Maildir/dovecot/public/control/me:INDEX=%h/Maildir/dovecot/public/indexes/me namespace: type: public separator: / prefix: #shared/org/ location: maildir:/egr/mail/shared-dovecot2/org:CONTROL=%h/Maildir/dovecot/public/control/org:INDEX=%h/Maildir/dovecot/public/indexes/org namespace: type: public separator: / prefix: #shared/toptest/ location: maildir:/egr/mail/shared-dovecot2/toptest:CONTROL=%h/Maildir/dovecot/public/control/toptest:INDEX=%h/Maildir/dovecot/public/indexes/toptest namespace: type: public separator: / prefix: #shared/ugs/ location: maildir:/egr/mail/shared-dovecot2/ugs:CONTROL=%h/Maildir/dovecot/public/control/ugs:INDEX=%h/Maildir/dovecot/public/indexes/ugs namespace: type: public separator: / prefix: #shared/vprgs/ location: maildir:/egr/mail/shared-dovecot2/vprgs:CONTROL=%h/Maildir/dovecot/public/control/vprgs:INDEX=%h/Maildir/dovecot/public/indexes/vprgs namespace: type: private separator: / prefix: mail/ hidden: yes namespace: type: private separator: / location: maildir:~/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes inbox: yes auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth user: postfix group: postfix master: plugin: acl: vfile:/usr/local/etc/dovecot-acls Thanks.

6 13

[Dovecot] Log data on dovecot-auth socket
by Koen Vermeer 11 May '07

11 May '07

Hi, Is there any way I can monitor or log the data that is being received/sent on the dovecot-auth socket? I'm trying to debug some authentication stuff, and monitoring the conversation would be very useful. Thanks! Koen

2 2

[Dovecot] No authentication sockets found
by Stewart Dean 11 May '07

11 May '07

Thunderbird (2.0) is all of a sudden checking mail server capabilities and keep posting a popup saying: "Mail Server imap.bard.edu is not an IMAP4 mail server". To the best of my knowledge, neither Tbird nor DC have been updated, though the DC imap host was rebooted today 3 hours ago. Any info on what this is about? DC is currently serving out of a special port and in evaluation list....I can't imagine that this message would please the user community! Killing and restarting dovecot appears to have at least temporarily stopped this behavior. Also and perhaps related, I am seeing a syslog message: imap-login: No authentication sockets found child xxxxxx (login) returned error 89 using SSL encrypted IMAP service Background: DC 1.0, invoked from /etc/rc.local during boot. After stopping and restarting DC, things seem to work again. As per an old problem, I checked /var/run/dovecot, but it was empty with plenty of space. FWIW, the owner/group for login dir (/var/run/dovecot) is dovecot:dovecot, perm 755, while /var/run/dovecot/login is root:dovecot, 750. Host is running AIXV5.3 TL5 SP6 (latest level), DC is compiled with IBM's AIX for C, V6.0 dovecot -n output > # /usr/local/etc/dovecot.conf > listen: *:10143 > ssl_listen: *:10943 > disable_plaintext_auth: no > verbose_ssl: yes > login_dir: /var/run/dovecot/login > login_executable: /usr/local/libexec/dovecot/imap-login > verbose_proctitle: yes > first_valid_uid: 200 > mail_location: mbox:~/mail:INBOX=/var/spool/mail/%u:INDEX=/var/dcndx/%u > mail_debug: yes > mbox_dirty_syncs: no > mbox_lazy_writes: no > auth default: > verbose: yes > debug: yes > passdb: > driver: pam > userdb: > driver: passwd Do I need to set an O/S or SSL max for authentication sockets? If so, any info would be appreciated Thanks! -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

3 2

[Dovecot] locking questions
by Stewart Dean 11 May '07

11 May '07

I have Dovecot 1.0 in trial use by the IT staff, and have some locking questions Background, the mail server runs procmail, sendmail and NFS exports the user homedir and mailbox to a) a login shell host and b) a mailing list services host. It runs UWIMAP on the usual ports and dovecot on a arbitrary port number. Because of concern with NFS and file access contention. I have the following dovecot.conf file access parms set: a) mbox_dirty_syncs = no b) mbox_lazy_writes = no 1) I am seeing these errmsgs. Are they anything I should be concerned about, or are they just the inevitable result of lock contention for mailboxes and folders by procmail, imap/dovecot and NFS? > In syslog, I am seeing this: > > Apr 23 18:46:34 mercury mail:err|error dovecot: IMAP(sdean): stat(/var/spool/mail/sdean.lock) failed: No such file or directory > Apr 23 18:46:34 mercury mail:err|error dovecot: IMAP(sdean): file_lock_dotlock() failed with mbox file /var/spool/mail/sdean: No such file or directory > > While in maillog: > > Apr 23 18:46:34 mercury mail:err|error dovecot: IMAP(sdean): stat(/var/spool/mail/sdean.lock) failed: No such file or directory > Apr 23 18:46:34 mercury mail:err|error dovecot: IMAP(sdean): file_lock_dotlock() failed with mbox file /var/spool/mail/sdean: No such file or directory 2) I earlier asked, but don't think I got an answer to this question: > Everybody wants to lock differently > 1) procmail (delivering for sendmail), which seems to want to use > dotlocking, fcntl and lockf locking; for whatever reasons, the compile > time tests seem to disallow flock. > 2) UWIMAP which according to wiki.dovecot.org/Migration/UW uses: > mbox_read_locks = flock > mbox_write_locks = dotlock flock > 3) I want to run dovecot in the same environment as I switch over, for > which the locking strategy is supposed to be (according to > http://wiki.dovecot.org/MboxLocking) > mbox_read_locks = fcntl > mbox_write_locks = dotlock fcntl > > Since that same page in the wiki says. "*It's important that all > software that's reading or writing to mboxes use the same locking > settings.",* I had recompiled procmail so it only usedotlocking and > fcntl (thus removing lockf in addition to the disallowed flock)...but > now I bumped into the Migration/UW page and there looks to be a conflict. > > I want to be able to run dovecot (for testing) and UWIMAP (for all > users/production) at the same time (but never at the same time with > the same mailbox), but UWIMAP is doc'ed to use flock and Dovecot doesn't > It's conceivable that the different locking strategies of UWIMAP and > Dovecot don't matter because they would never be going after the same > mail files (right?) but there is procmail, which has to interact with > both. It appears to not use flock OK, even though the docs say it > should since UWIMAP expects it, and I've disabled lockf as well. But > will there be a conflict with procmail and dovecot's use of fcntl, > which UWIMAP doesn't use? > > Help. 3) I occasionally see "Disconnected for inactivity" message. How long before that happens? Is the interval configurable? Thanks to all. A brother sysadmin was stunned by performance gained in Dovecot. He has to keep postmaster messages for a period and filters them into a separate folder...over a long weekend this folder could get to 1500 messages and 150MB...with UWIMAP he could only delete approx 800 at a time and then had to wait 5-10 minutes for it to process. With dc, he marked off the whole folder and it was gone in seconds. -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

2 1