- dovecot - dovecot.org

[Dovecot] Using global und per-user sieve scripts at the same time
by Matthias Kellermann 11 Dec '07

11 Dec '07

Hi list, I'm trying to get a global sieve mailfilter with per-user sieve scripts to work. At the "lda"-section in the dovecot.conf file I have: global_script_path = /usr/local/etc/sieve/spam /usr/local/etc/sieve/spam: require "fileinto"; if header :contains ["Subject"] ["***Spam***"] { fileinto "Spam"; stop; } which will work fine delivering Spam Mails in a Spam dir for every user. If a user creates his own .dovecot.sieve file the global script will not work anymore. How can I enable global and local per-user sieve scripts at once? Is this possible? Do you know any workaround so all users will have the same (Spam-)sieve filters beside there own (global templates or so)? Matthias

5 11

[Dovecot] squat also not working in beta11
by Joe Wong 10 Dec '07

10 Dec '07

Hi Timo, Just tried squat in beta11. I also get the same error in the log: dovecot: Dec 10 14:21:17 Error: IMAP(joewong99:joew.outblaze.com): Corrupted squat uidlist file /mailfs/4/22/3/joewon g99:joew.outblaze.com@joew_outblaze_com/Maildir/dovecot.index.search.uids: Broken uidlists There is not change in my configuration file. # 1.1.beta11: /usr/local/etc/dovecot.conf log_path: /var/log/dovecot-1.1.log info_log_path: /var/log/dovecot-1.1.log protocols: imap ssl_disable: yes login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login login_user: nobody verbose_proctitle: yes mail_location: maildir:~/Maildir mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_drop_priv_before_exec: yes mail_plugins: fts fts_squat auth default: user: mdrop username_chars: abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@: username_translation: @: verbose: yes debug: yes debug_passwords: yes passdb: driver: sql args: /usr/local/etc/dovecot-sql.conf userdb: driver: sql args: /usr/local/etc/dovecot-sql.conf plugin: fts: squat --

2 5

Re: [Dovecot] New Mailbox Format Plugin Help
by Sacha Bernstein 10 Dec '07

10 Dec '07

On 12/8/07, Timo Sirainen <tss(a)iki.fi> wrote: > On Sat, 2007-12-08 at 10:11 -0500, Sacha Bernstein wrote: > > I would like to develop a new mailbox format for a very specialized > > application. Can anyone point me in the appropriate direction to get > > started writing a plugin? > > > > More detail for anyone interested: > > I need to store mail in a format that's quickly accessible and > > modifiable to an external process. This is for a very low-volume mail > > system (averaging less than 50 messages stored at any one time) that > > needs very good indexing. I need to create new mailboxes on the fly > > and perform fast searches across all the mailboxes. I intend to store > > all the messages in an SQL database. Any help getting started > > interfacing with dovecot (other than reading through _all_ the source > > code) would be greatly appreciated. > > v1.1 makes this a lot easier, so start with it instead of v1.0. You can > probably get started with my existing SQL backend code: > http://dovecot.org/list/dovecot/2007-November/026632.html > > The main idea is to implement APIs in lib-storage/mail-storage-private.h > and lib-storage/mailbox-list-private.h. Most of these calls can be > implemented by calling generic index_*() functions in > lib-storage/index/. Thanks, I'll get started with this. I'm sure I'll have more questions once I get further into it. If anyone's interested (especially any amateur radio operators that may be on this list), you can see what my plans are for the SQL storage backend for dovecot at w2sjb.org. Basically, the short version is a linux gateway to a radio-based email system for use in emergencies. Of course it has many other uses, but that's what I'm going to be using it for :-) Thanks, Sacha

1 0

Re: [Dovecot] Automatic Index Generation
by Maciej Poszywak 10 Dec '07

10 Dec '07

>> Timo Sirainen wrote: >> >>> I guess you mean logging in with POP3? There the slowdown comes from >>> getting all messages' virtual sizes. So your preindexer could probably >>> do this by running pop3 for all users. If you use only a single UID for >>> users, you could do something like: >>> >>> for user in `cat users`; do >>> export MAIL=maildir:/home/$user/Maildir >>> echo "quit" | /usr/local/libexec/dovecot/pop3 >>> done >>> >>> >>> >> vmu1@test:~$ env | grep MAIL >> MAIL=maildir:/var/mail/testuser >> vmu1@test:~$ echo "quit" | /usr/lib/dovecot/pop3 >> pop3(vmu1): Fatal: pop3_uidl_format setting is missing from config file >> >> I have pop3_uidl_format correctly defined in the config file. I've also >> tried to strace pop3, to find where it does look for it, but with no >> success. >> > > It gets it from environment: > > POP3_UIDL_FORMAT=... pop3 > > In general all settings are passed in environment using uppercased > names. > > Are you sure that all variables can be read from environment ? vmu1@mail:~$ env | grep -E "(MMAP|LOCK)" MMAP_DISABLED=yes MAIL_READ_MMAPED=no LOCK_METHOD=dotlock POP3_LOCK_SESSION=yes vmu1@mail:~$ echo "quit" | /usr/lib/dovecot/pop3 pop3(vmu1): Fatal: lock_method=dotlock and mmap_disable=no combination isn't supported. You don't _really_ want it anyway. vmu1@mail:~$ /usr/sbin/dovecot --version 1.0.rc15 -- Maciej Poszywak

2 1

[Dovecot] v1.1.beta11 released (Problems)
by Jerry Yeager 10 Dec '07

10 Dec '07

On Dec 9, 2007, at 1:48 PM, dovecot-request(a)dovecot.org wrote: > > > Message: 7 > Date: Sun, 09 Dec 2007 20:29:49 +0200 > From: Timo Sirainen <tss(a)iki.fi> > Subject: [Dovecot] v1.1.beta11 released > To: dovecot(a)dovecot.org > Message-ID: <1197224989.922.405.camel@hurina> > Content-Type: text/plain; charset="us-ascii" > > http://dovecot.org/releases/1.1/beta/dovecot-1.1.beta11.tar.gz > http://dovecot.org/releases/1.1/beta/dovecot-1.1.beta11.tar.gz.sig > > This one should be the last beta release before the first v1.1 release > candidate. > > I'll try to stay away from this list and Dovecot in general for the > next > 1,5 weeks. I've several exams coming up and I should have started > studying for them days ago already.. :) > stuff cut out > End of dovecot Digest, Vol 56, Issue 26 > *************************************** I ran into a problem moving from v 1.1 beta 10 to v 1.1 beta 11, the logs show: Dec 9 16:03:49 ns1 dovecot[331]: Dovecot v1.1.beta11 starting up Dec 9 16:03:50 ns1 dovecot[331]: imap-login: We couldn't drop root group privileges (wanted=150, gid=150, egid=0) Dec 9 16:03:50 ns1 dovecot[331]: Temporary failure in creating login processes, slowing down for now The last two messages then begin repeating until Dovecot is shut down. obviously, no one can log in. I tried changing the following # System user and group used to access mails. If you use multiple, userdb # can override these by returning uid or gid fields. You can use either numbers # or names. <doc/wiki/UserIds> #mail_uid = #mail_gid = to explicitly specifying # System user and group used to access mails. If you use multiple, userdb # can override these by returning uid or gid fields. You can use either numbers # or names. <doc/wiki/UserIds> mail_uid = Dovecot mail_gid = Dovecot As well as setting the setuid bit in login_executable = /usr/local/libexec/dovecot/imap-login to no avail. Dropping back to v 1.1 beta 10 fixed things for the moment. I did a quick scan of the example conf file and the wiki docs but could not find anything relevant. dovecot -n shows (essentially the same for both versions) # 1.1.beta11: /usr/local/etc/dovecot.conf base_dir: /var/run/dovecot/ protocols: imaps ssl_listen: *:993 ssl_cipher_list: ALL:!LOW:!SSLv2 verbose_ssl: yes login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login login_user: Dovecot login_greeting: Hello, Dovecot is ready. valid_chroot_dirs: /var/mail/vmail/ mail_extra_groups: mail mail_location: maildir:/var/spool/vmail/%d/%n:INDEX=/var/spool/vmail/ %d/%n:INBOX=/var/spool/vmail/%d/%n mail_debug: yes lock_method: flock auth default: mechanisms: plain login passdb: driver: passwd-file args: scheme=plain-md5 username_format=%u /etc/dovecot/passwd userdb: driver: static args: uid=virtual gid=virtual home=/var/spool/vmail/%d/%n socket: type: listen client: path: /var/spool/postfix/private/auth user: postfix group: postfix master: path: /usr/local/var/run/dovecot/auth-master user: virtual group: virtual This is running on OS X 10.5.1 ??? Jerry

2 2

[Dovecot] Dovceot 1.0.8 compile error after rewrite patch
by Lars Johansen 10 Dec '07

10 Dec '07

Added the rewrite patch for 1.0.8 without any problems, but when I tried building with: configure --prefix=/usr/local/dovecot, make gives this output: gcc -shared .libs/mail-log-plugin.o -Wl,-soname -Wl,lib20_mail_log_plugin.so -o .libs/lib20_mail_log_plugin.so ar cru .libs/lib20_mail_log_plugin.a mail-log-plugin.o ranlib .libs/lib20_mail_log_plugin.a creating lib20_mail_log_plugin.la (cd .libs && rm -f lib20_mail_log_plugin.la && ln -s ../lib20_mail_log_plugin.la lib20_mail_log_plugin.la) make[4]: Leaving directory `/usr/local/src/dovecot-1.0.8/src/plugins/mail-log' Making all in trash make[4]: Entering directory `/usr/local/src/dovecot-1.0.8/src/plugins/trash' if /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I../../.. -I../../../src/lib -I../../../src/lib-mail -I../../../src/lib-storage -I../../../src/plugins/quota -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -MT trash-plugin.lo -MD -MP -MF ".deps/trash-plugin.Tpo" -c -o trash-plugin.lo trash-plugin.c; \ then mv -f ".deps/trash-plugin.Tpo" ".deps/trash-plugin.Plo"; else rm -f ".deps/trash-plugin.Tpo"; exit 1; fi mkdir .libs gcc -DHAVE_CONFIG_H -I. -I. -I../../.. -I../../../src/lib -I../../../src/lib-mail -I../../../src/lib-storage -I../../../src/plugins/quota -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -MT trash-plugin.lo -MD -MP -MF .deps/trash-plugin.Tpo -c trash-plugin.c -fPIC -DPIC -o .libs/trash-plugin.o trash-plugin.c:110: warning: 'struct quota_root_transaction_context' declared inside parameter list trash-plugin.c:110: warning: its scope is only this definition or declaration, which is probably not what you want trash-plugin.c: In function 'trash_try_clean_mails': trash-plugin.c:134: error: dereferencing pointer to incomplete type trash-plugin.c:205: error: dereferencing pointer to incomplete type trash-plugin.c:205: error: dereferencing pointer to incomplete type trash-plugin.c:206: error: dereferencing pointer to incomplete type trash-plugin.c:207: error: dereferencing pointer to incomplete type trash-plugin.c:207: error: dereferencing pointer to incomplete type trash-plugin.c:208: error: dereferencing pointer to incomplete type trash-plugin.c: At top level: trash-plugin.c:214: warning: 'struct quota_root_transaction_context' declared inside parameter list trash-plugin.c: In function 'trash_quota_root_try_alloc': trash-plugin.c:216: error: dereferencing pointer to incomplete type trash-plugin.c:220: error: 'struct quota_backend_vfuncs' has no member named 'try_alloc' trash-plugin.c:238: warning: passing argument 1 of 'trash_try_clean_mails' from incompatible pointer type trash-plugin.c: At top level: trash-plugin.c:248: warning: 'struct quota_root_transaction_context' declared inside parameter list trash-plugin.c: In function 'trash_quota_root_try_alloc_bytes': trash-plugin.c:250: error: dereferencing pointer to incomplete type trash-plugin.c:254: error: 'struct quota_backend_vfuncs' has no member named 'try_alloc_bytes' trash-plugin.c:262: warning: passing argument 1 of 'trash_try_clean_mails' from incompatible pointer type trash-plugin.c: At top level: trash-plugin.c:272: warning: 'struct quota_root_transaction_context' declared inside parameter list trash-plugin.c: In function 'trash_quota_root_test_alloc_bytes': trash-plugin.c:274: error: dereferencing pointer to incomplete type trash-plugin.c:278: error: 'struct quota_backend_vfuncs' has no member named 'test_alloc_bytes' trash-plugin.c:286: warning: passing argument 1 of 'trash_try_clean_mails' from incompatible pointer type trash-plugin.c: In function 'trash_quota_root_created': trash-plugin.c:313: error: 'struct quota_root' has no member named 'v' trash-plugin.c:314: error: 'struct quota_root' has no member named 'v' trash-plugin.c:315: error: 'struct quota_root' has no member named 'v' trash-plugin.c:316: error: 'struct quota_root' has no member named 'v' trash-plugin.c:317: error: 'struct quota_root' has no member named 'v' make[4]: *** [trash-plugin.lo] Error 1 make[4]: Leaving directory `/usr/local/src/dovecot-1.0.8/src/plugins/trash' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/usr/local/src/dovecot-1.0.8/src/plugins' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/usr/local/src/dovecot-1.0.8/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/local/src/dovecot-1.0.8' make: *** [all] Error 2

2 1

[Dovecot] Automatic Index Generation
by Maciej Poszywak 10 Dec '07

10 Dec '07

Hello, I'm looking for an application/script/etc. which will allow me to recreate dovecot indexes by hand from shell. I'm preparing myself for migration to dovecot 1.0.rc15 (debian stable) and would like to create indexes before users start to login into new server to ease the load. My other option is to write a script which will log as each user and create indexes, however this seems a bit of painful way to do it. Anybody can point me in right direction? -- Maciej Poszywak

5 10

[Dovecot] dict: Leaked a t_pop() call in I/O handler
by Sergio Lopez 10 Dec '07

10 Dec '07

Hello, I've just installed another postfix+dovecot based mail service (I love dovecot! :-), and I found this message "dict: Leaked a t_pop() call in I/O handler (nil)" being logged each time I authenticate against it via POP3 or IMAP. Searching through the code, I've found this (this is Dovecot v1.0.8): ---- src/dict/dict-cache.c ---- void dict_cache_unref(struct dict_cache *cache, const char *uri, const char *username) { struct dict_entry *entry; t_push(); entry = hash_lookup(cache->dicts, t_strdup_printf("%s\t%s", username, uri)); i_assert(entry != NULL && entry->refcount > 0); if (--entry->refcount == 0) { hash_remove(cache->dicts, entry->user_uri); dict_deinit(&entry->dict); i_free(entry->user_uri); i_free(entry); } } ---- src/dict/dict-cache.c ---- Notice the "t_push()" call there. So calling t_pop() somewhere before the end of the function, makes dovecot work like a charm (again). P.D: I'm really surprised that nobody noticed this before. Am I the only man in the world who uses Dovecot storing quota information into a SQL Database? ;-) -- Sergio López - <sergio.lopez(a)nologin.es> Nologin Consulting S.L. - http://nologin.es

2 1

[Dovecot] Internal Error - Cannot select inbox
by Eric Beda 10 Dec '07

10 Dec '07

Hi, I am new to this list although i have been using dovecot for some time now, i've come accross this error when a user tries to log in * OK Dovecot ready. 1 login username password 1 OK Logged in. 1 select inbox 1 NO Internal error occurred. Refer to server log for more information. [2007-12-07 12:51:56] there are no errors reported in the log files related to that issue for both dovecot.log and maillog what i do receive in the log files is dovecot: Dec 07 12:55:46 Info: imap-login: Login: user=<username@domain>, method=PLAIN, rip=::ffff:127.0.0.1, lip=::ffff:127.0.0.1, secured dovecot: Dec 07 12:55:46 Info: IMAP(username@domain): Effective uid=500, gid=500 dovecot: Dec 07 12:55:46 Info: IMAP(aadu(a)uccmail.co.tz): mbox: data=/var/spool/mail/vmail/domain/username:INBOX=/var/spool/mail/vmail/domain/username/Inbox:INDEX=/var/spool/mail/vmail/domain/username:home=/var/spool/mail/vmail/domain/username/.imap/.imap dovecot: Dec 07 12:55:46 Info: IMAP(aadu(a)uccmail.co.tz): mbox: root=/var/spool/mail/vmail/domain/username, index=/var/spool/mail/vmail/domain/username, inbox=/var/spool/mail/vmail/domain/username/Inbox dovecot: Dec 07 12:55:46 Info: IMAP(username@domain): Disconnected i can't seem to pinpoint what is going wrong, some things to note i have existing users who can read email succesfully, its when i create a new user via postfixadmin, one should note that all directories are created and owned by vmail whose gid and uid is 500 and for the new users the directories and mboxes are created succesfully, but even if i copy over a working inbox of previous user to new user i still get an error so i don't think it is an mbox formatting issue another thing is that i can successfully check mail using mutt facility, even sending using mail clients (outlook express) is possible other configurations that might be helpful # dovecot --version 1.0.rc15 #cat /etc/group |grep vmail dovecot:x:97:vmail vmail:x:500:dovecot # /etc/dovecot.conf log_path: /var/log/dovecot.log protocols: pop3 pop3s imap imaps ssl_disable: yes login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login mail_location: mbox:/var/spool/mail/vmail/%d/%n:INBOX=/var/spool/mail/vmail/%d/%n/Inbox:INDEX=/var/spool/mail/vmail/%d/%n:home=/var/spool/mail/vmail/%d/%n/.imap/.imap mail_debug: yes maildir_copy_with_hardlinks: yes mbox_min_index_size: 200 mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 auth default: mechanisms: plain login passdb: driver: pam passdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf userdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master:

3 5

[Dovecot] Roadmap to future
by Timo Sirainen 10 Dec '07

10 Dec '07

v1.1.0 is finally getting closer, so it's time to start planning what happens after it. Below is a list of some of the larger features I'm planning on implementing. I'm not yet sure in which order, so I think it's time to ask again what features companies would be willing to pay for? Sponsoring Dovecot development gets you: - listed in Credits in www.dovecot.org - listed in AUTHORS file - you can tell me how you want to use the feature and I'll make sure that it's possible (within reasonable limits) - allows you to change the order in which features are implemented (more or less) As you can see below, some features are easier to implement when other features are implemented first. Several of them depend on v2.0 master rewrite, so I'm thinking that maybe it's time to finally finish that first. I know several companies have requested replication support, but since it kind of depends on several other features, it might be better to leave that for v2.1. But of course sponsoring development of those features that replication depends on gets you the replication support faster. :) I'm still trying to go to school (and this year I started taking CS classes as well as biotech classes..), so it might take a while for some features to get implemented. I'm anyway hoping for a v1.2 or v2.0 release before next summer. So, the list.. -------------- Implemented, but slightly buggy so not included in v1.1 (yet): - Mailbox list indexes - THREAD REFERENCES indexes - THREAD X-REFERENCES2 algorithm where threads are sorted by their latest message instead of the thread root message Could be implemented for v1.2: - Shared mailboxes and ACLs http://www.dovecot.org/list/dovecot/2007-April/021624.html - Virtual mailboxes: http://www.dovecot.org/list/dovecot/2007-May/022828.html - Depends on mailbox list indexes dbox plans, could be implemented for v1.2: - Support for single instance attachment storage - Global dbox storage so copying messages from mailbox to another involves only small metadata updates - Support some kind of hashed directories for storing message files, instead of storing everything in a single directory. - Finish support for storing multiple messages in a file v2.0 master/config rewrite -------------------------- It's mostly working already. One of the larger problems left is how to handle plugin settings. Currently Dovecot just passes everything in plugin {} section to mail processes without doing any checks. I think plugin settings should be checked just as well as the main settings. Also plugins should be able to create new sections instead of having to use plain key=value settings for everything. v2.0 currently runs a perl script on Dovecot's source tree to find settiongs from different places then then creates a single all-settings.c for config process. This obviously doesn't work for out-of-tree plugins, so there needs to be some other way for them to pass the allowed settings. There are two possibilities: a) Read them from separate files. This is a bit annoying, because it's not anymore possible to distribute simple plugin.c files and have them compiled into plugin.so files. b) Read them from the plugin itself. This would work by loading the plugin and then reading the variable containing the configuration. I'm a bit worried about security implications though, because the plugins could execute some code while they're loaded. But I guess a new unprivileged process could be created for doing this. Of course normally admins won't run any user-requested plugins anyway so this might not be a real issue anyway. :) Index file optimizations ------------------------ - Since these are not backwards compatible changes, the major version number should be increased whenever these get implemented. So I think these should be combined with v2.0 master/config rewrite. I'd rather not release v3.0 soon after v2.0 just for these.. - dovecot.index.log: Make it take less space! Get rid of the current "extension introduction" records that are large and written over and over again. Compress UID ranges using similar methods than Squat uses now. Try all kinds of other methods to reduce space usage. Write transaction boundaries, so the reader can process entire transactions at a time. This will make some things easier, such as avoiding NFS data cache flushes, replication support and also allows lockless writes using O_APPEND. - dovecot.index: Instead of keeping a single message record array, split it into two: A compressed list of message UIDs and array containing the rest of the data. This makes the file smaller and keeping UIDs separate allows better CPU cache (and maybe disk I/O) utilization for UID -> sequence lookups. At least this is the theory, I should benchmark this before committing to this design. :) There could also be a separate expunged UIDs list. UIDs existing in there would be removed from existing UIDs list, but it wouldn't affect calculating offsets to record data in the file. So while currently updating dovecot.index file after expunges requires recreating the whole file, with this method it would be possible to just write 4 bytes to the expunged UIDs list. The space reserved for expunged UIDs list would be pretty small though, so when it gets full the file would be recreated. - dovecot.index.cache: Better disk I/O and CPU cache utilization by keeping data close together in the file when it's commonly accessed toghether. For example message sizes and dates for all messages could be close to each others to optimize SORTing by message size/date. If clients use conflicting access patterns, the data could be either duplicated to optimize both patterns or just let it be slower for the other pattern. Deliver / LMTP server --------------------- Currently deliver parses dovecot.conf using its own parser. This has caused all kinds of problems. v2.0's master rewrite helps with this, because deliver can then just ask the configuration from config process the same way as other processes are. Another problem is that people who use multiple user IDs have had to make deliver setuid-root. I really hate this, but currently there's no better way. Better fix for this would be to use LMTP server instead. There could be either a separate LMTP client or deliver could support LMTP protocol as well. I'm not sure which one is better. How would LMTP server then get around the multiple user IDs issue? There are two possibilities: a) Run as root and temporarily set effective UID to the user whose mails we are currently delivering. This has the problem that if users have direct access to their mailboxes and index files, the mailbox handling code has to be bulletproof to avoid regular users gaining root privileges by doing special modifications to index files -- possibly at the same time as the mail is being delivered to exploit some race condition with mmaped index files.. b) Create a child process for each different UID and drop its privileges completely. If all users have different UIDs, this usually means forking a new process for each mail delivery, making the performance much worse than with a) method. It would anyway be possible to delay destroying the child processes in case a new message comes in for the same UID. This would probably mostly help only when UIDs are shared by multiple users, such as each domain having their own UID. I think I'll implement both and let admin choose which method to use. This same problem exists for several other things besides deliver/LMTP server. For example a) method is already implemented for expire plugin's expire-tool in v1.1. So this code needs to be somewhat generic so it won't have to be duplicated in multiple places. All of this pretty much requires the new v2.0 master rewrite. Adding these to v1.x would be ugly. Proxying -------- - These could be implemented to v1.2. - Log in normally (no proxying) if destination IP is the server itself. - Support for per-namespace proxying: namespace public { prefix = Public/ location = proxy:public.example.org } There are two choices for this: Dummy IMAP proxying or handling it with a mail storage backend. Dummy IMAP proxying would just let the remote server whose mailbox is currently selected handle most of the commands and their replies. I'm just afraid that this will become problematic in future. For example if the proxying server wants to send the client an event, such as "new message was added to this other mailbox", it would have to be sure that the remote server isn't currently in the middle of sending something. And the only way to be sure would be to parse its input, which in turn would make the proxying more complex. Then there would be problems if the servers support different IMAP extensions. And maybe some extensions will require knowing the state of multiple mailboxes, which will make handling them even more complex if the mailboxes are on different servers. Implementing this as a mail storage backend would mean that the proxying server parses all the IMAP commands internally and uses Dovecot's mail storage API to process them. The backend would then request data from the remote server when needed and cache it internally to avoid asking the same data over and over again. Besides caching, this would mean that there are no problems with extensions, since Dovecot handles the the exact same way as for other backends, requiring no additional code. To get better performance the storage backend would have to be smarter than most other Dovecot backends. For example while the current backends handle searches by reading through all the messages, the proxy backend should just send the search request to the remote server. The same for SORT. And for THREAD, which the current storage API doesn't handle.. And how would proxying server and remote server then communicate? Again two possibilities: Either a standard IMAP protocol, with possibly some optional Dovecot-extensions to improve performance, or a bandwidth-efficient protocol. The main advantage with IMAP protocol is that it could be used for proxying to other servers. Bandwidth-efficient protocol could be based on dovecot.index.log format, making it faster/easier to parse and generate. Once we're that far with proxying, we're close to having..: Replication ----------- The plans haven't changed much from: http://www.dovecot.org/list/dovecot/2007-May/022792.html The roadmap to complete replication support would be basically: 1. Implement proxying with mail storage API. 2. Implement asynchronous replication: - Replication master process collecting changes from other Dovecot processes via UNIX sockets. It would forward these changes to slaves. Note that this requires that only Dovecot is modifying mailboxes, mails must be delivered with Dovecot's deliver! - If some slaves are down, save the changes to files in case they come back up. In case they don't and we reach some specific max. log file size, keep also track of mailboxes that have changed and once the slaves do come back, make sure those mailboxes are synced. - If replication master is down and there have been changes to mailboxes.. Well, I haven't really thought about this. Maybe refuse to do any changes to mailboxes? Maybe have mail processes write to some "mailboxes changed" log file directly? Probably not that big of a deal since replication master just shouldn't be down. :) - Support for manual resync of all mailboxes, in case something was done outside Dovecot. - Replication slave processes would be reading incoming changes. I think this process should just make sure that the changes are read as fast as possible, so with sudden bursts of activity the slave would start saving the changes to a log instead of the master. - Replication mailbox writer processes would receive changes from the main slave process via UNIX sockets. They'd be responsible for actually saving the data to mailboxes. If multiple user IDs are used, this has the same problems as LMTP server. - Conflict resolution. If mailbox writer process notices that the UID already exists in the mailbox, both the old and the new message must be given new UIDs. This involves telling the replication master about the problem, so both servers can figure out together the new UIDs. I haven't thought this through yet. - Conflict resolution for message flags/keywords. The protocol would normally send flag changes incrementally, such as "add flag1, remove flag2". Besides that there could be a 8bit checksom of all the resulting flags/keywords. If the checksum doesn't match, request all the current flags. It would be of course possible to send the current flags always instead of incremental updates, but this would waste space when there are lots of keywords, and multi-master replication will work better when the updates are incremental. 3. Synchronous replication: - When saving messages, don't report success until at least one replication slave has acknowledged that it has also saved the message. Requires adding bidirectional communication with the replication master process. The message could sent to slaves in smaller parts, so if the message is large the wait at the end would basically consist of sending "finished saving UID 1234, please ack". - Synchronous expunges also? Maybe optional. The main reason for this is that it wouldn't be nice to break IMAP state by having expunged messages come back. 4. Multi-master replication: - Almost everything is already in place. The main problem here is IMAP UID allocation. They must be allocated incrementally, so each mailbox must have a globally increasing UID counter. The usage goes like: 1. Send the entire message body using a global unique ID to all servers. No need to wait for their acknowledgement. 2. Increase the global IMAP UID counter for this mailbox. 3. Send global UID => IMAP UID message to all servers. 4. Wait for one of them to reply with "OK". Because messages sent by different servers can end up being visible in different times, the servers must wait that they've seen all the previous UIDs before notifying client of a new UID. If the sending server crashed between steps 2 and 3, this never happens. So the replication processes must be able to handle that, and also handle requesting missing UIDs in case they had lost connections to other servers. - I just had the idea of global UID counters instead of global locks, so I haven't yet thought how the counters would be best implemented. It should be easier to implement than global locks though. - Non-incremental changes are somewhat problematic and could actually require global locks. The base IMAP protocol supports replacing flags, so if there is no locking and conflicting flag replacement commands are sent to two servers, which one of those commands should be effective? Timestamps would probably be enough for flag changes, but it may not be enough for some IMAP extensions (like CONDSTORE?). - If global locks are required, they would work by passing the lock from one server to another when requested. So as long as only a single server is modifying the mailbox, it would hold the global lock and there would be no need to keep requesting it for each operation. 5. Automation: - Automatically move/replicate users between servers when needed. This will probably be implemented separately long after 4. The replication processes would be ugly to implement for v1.x master, so this pretty much depends on v2.0 master rewrite.

4 6