- dovecot - dovecot.org

Re: [Dovecot] How do I increase the fd limit on OS X?
by Jerry Yeager 08 May '08

08 May '08

On May 8, 2008, at 1:04 AM, dovecot-request(a)dovecot.org wrote: > > > Message: 6 > Date: Wed, 7 May 2008 13:17:44 -0700 > From: Rob Frohne <rob.frohne(a)wallawalla.edu> > Subject: [Dovecot] How do I increase the fd limit on OS X? > To: Dovecot Mailing List <dovecot(a)dovecot.org> > Message-ID: <1210191465.19769.5.camel(a)frohro-d600.wwc.edu> > Content-Type: text/plain > > Hi All, > > I just upgraded to the new 1.1rc5 from 1.0 that I had been using and > it > advised me to increase the file descriptor limit. I'm not sure how to > do this. There is a command built into tcsh that allows me to do this > called limit, but sudo limit 4224 doesn't work, and usually dovecot is > started from the rc.local file if I recall right, and I'm not sure how > to set this up from there. Any advice. > > Thanks, > > Rob > -- > Rob Frohne <rob.frohne(a)wallawalla.edu> > Walla Walla University > > > Try this your_prompt_$ sudo sysctl -w kern.maxfiles=65536 your_prompt_$ sudo sysctl -w kern.maxfilesperproc=32768 You will be asked for your admin password. This will allow you to test things and see if it works better. The changes will revert to the system pre-defines when you re-boot or restart. IIRC, there is a master setting (unchangeable without recompiling OS X) that will not allow you to go over 65536, but I may be mistaken about that. For a permanent change, add the following lines to the /etc/sysctl.conf file: or the /etc/sysctl-macosxserver.conf file for Mac OS X Server You may have to create the file, especially if you are the only user. (Make sure that only root has access to this file!!!!!!) kern.maxfiles=65536 # System-wide limit kern.maxfilesperproc=32768 # Per-process limit This Is ONLY For OS X 10.3 or later. For earlier versions you need to use /etc/rc which was so long ago that I do not recall all of the commands (sorry). HTH, Jerry

1 0

[Dovecot] How do I increase the fd limit on OS X?
by Rob Frohne 08 May '08

08 May '08

Hi All, I just upgraded to the new 1.1rc5 from 1.0 that I had been using and it advised me to increase the file descriptor limit. I'm not sure how to do this. There is a command built into tcsh that allows me to do this called limit, but sudo limit 4224 doesn't work, and usually dovecot is started from the rc.local file if I recall right, and I'm not sure how to set this up from there. Any advice. Thanks, Rob -- Rob Frohne <rob.frohne(a)wallawalla.edu> Walla Walla University

2 1

Re: [Dovecot] thunderbird sane config (OT)
by Moser 08 May '08

08 May '08

Charles Marcus schrieb: > On 5/7/2008, rosenfield.albert(a)gmail.com (rosenfield.albert(a)gmail.com) > wrote: >> (Background: I'm currently experiencing that Thunderbird does not work >> against Postfix/Dovecot because the wizard interface for setting up an >> account does not accept TLS-only servers. Users just sit there with a >> broken Thunderbird, not knowing what to do :-/.) you can do it. > It would be nice if you had the ability to access the Advanced Options > when setting up a new account in TBird, but currently you don't... Not true! simple way: Just change the default config TB create your account from with the settings you need. You can find the file under your program dir i.e. C:\Programme\Mozilla Thunderbird\defaults\pref Compare the defaults to your working set of setting and change them accordingly. You can activate TLS or CRAM-MD5 and so an. You can also predefine LDAP servers etc. more elaborated way: Mozilla Mission Contol: -> http://developer.mozilla.org/en/docs/MCD,_Mission_Control_Desktop_AKA_AutoC… Hope it helps! Hans

2 1

[Dovecot] thunderbird sane config (OT)
by rosenfield.albert＠gmail.com 08 May '08

08 May '08

Does anyone have any experience putting together a sane config for Thunderbird in an automated fashion? Automated meaning "simple to distribute to users". -- (Background: I'm currently experiencing that Thunderbird does not work against Postfix/Dovecot because the wizard interface for setting up an account does not accept TLS-only servers. Users just sit there with a broken Thunderbird, not knowing what to do :-/.) (PS: Also, giving everybody a GMail-like threaded interface per default would be cool.) (PPS: If anything good comes up, I plan to put this in fx the Dovecot wiki.)

5 7

[Dovecot] pop/imap session
by Tomasz 08 May '08

08 May '08

Hello, Can dovecot log pop/imap session, and notify(by sending mail) admin in case of errors? Something like notify_classes in postfix. Regards Tomasz

4 5

[Dovecot] [bug] bit of a clearer error message desired - Can't load CA file... : Success
by Daniel Black 07 May '08

07 May '08

Not the clearest of error messages. A successful cannot load. May 7 21:05:29 10.10.10.213 dovecot: child 21500 (login) returned error 89 May 7 21:05:29 10.10.10.213 dovecot: child 21501 (login) returned error 89 May 7 21:05:29 10.10.10.213 dovecot: child 21502 (login) returned error 89 May 7 21:05:29 10.10.10.213 dovecot: child 21503 (login) returned error 89 May 7 21:05:29 10.10.10.213 dovecot: child 21505 (login) returned error 89 May 7 21:05:29 10.10.10.213 dovecot: pop3-login: Can't load CA file /etc/dovecot/all.der: Success May 7 21:05:29 10.10.10.213 dovecot: imap-login: Can't load CA file /etc/dovecot/all.der: Success /etc/dovecot/all.der was generated by concatinating the following (all in der format) http://www.cacert.org/certs/root.der http://www.cacert.org/certs/class3.der http://crl.cacert.org/revoke.crl http://crl.cacert.org/class3-revoke.crl # dovecot --version 1.0.10 reiserfs filesystem # uname -a Linux mail.cacert.org 2.6.22-vs2.2.0.7-gentoo #1 SMP Mon May 5 20:21:30 EST 2008 x86_64 GNU/Linux # dovecot -n # 1.0.10: /etc/dovecot/dovecot.conf protocols: imaps pop3s imap pop3 ssl_ca_file: /etc/dovecot/all.der ssl_cert_file: /etc/ssl/certs/ssl-cert-community-cacert.pem ssl_key_file: /etc/ssl/private/ssl-cert-community-cacert.key ssl_verify_client_cert: yes verbose_ssl: yes login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_greeting: You want mail? Well hurry up. first_valid_uid: 5000 mail_location: maildir:~/Maildir mail_read_mmaped: yes mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 pop3_uidl_format: %08Xu%08Xv auth default: mechanisms: plain login verbose: yes debug: yes debug_passwords: yes ssl_require_client_cert: yes ssl_username_from_cert: yes passdb: driver: pam args: session=yes mail userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix -- Daniel Black -- Proudly a Gentoo Linux User. Gnu-PG/PGP signed and encrypted email preferred http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x76677097 GPG Signature D934 5397 A84A 6366 9687 9EB2 861A 4ABA 7667 7097

2 1

[Dovecot] Clarification wanted on mail_access_groups vs. mail_privileged_group
by Mark Sapiro 07 May '08

07 May '08

I have just upgraded to dovecot 1.0.13. All the documentation I can find in the example .conf file and the NEWS and ChangeLog files seems to say that the mail_privileged_group = mail setting is all I should need to make dovecot use group mail to create dotlock files. My understanding from what I read is that mail_privileged_group is used to set the group used while creating dotlock files in (in my case) /var/spool/mail, and mail_access_groups is used to set the group used to actually access the mailbox. In my case, /var/spool/mail is group 'mail and group writable/searchable drwxrwxr-x 2 root mail 4096 May 7 08:35 /var/spool/mail and the individual mailboxes are owned by the respective users. So it seems that I would need "mail_privileged_group = mail" do be able to create dotlock files in /var/spool/mail, and that I would not need any mail_access_groups as the users can access their own mailboxes. But this doesn't work. With "mail_privileged_group = mail" and mail_access_groups unset, I get May 6 12:48:54 sbh16 dovecot: POP3(xxx): file_lock_dotlock() failed with mbox file /var/spool/mail/xxx: Permission denied Yet with "mail_access_groups = mail" and mail_privileged_group unset, it works with no problem. Is my understanding wrong? Is the documentation wrong? Is the implementation wrong? Or is there just something wierd in my case (possibly pop3_lock_session = yes)? # dovecot -n # 1.0.13: /usr/local/etc/dovecot.conf protocols: pop3 pop3s ssl_cert_file: /etc/postfix/grizz-cert.pem ssl_key_file: /etc/postfix/grizz-key.pem login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/pop3-login mail_access_groups: mail mail_location: /var/spool/mail/%u mbox_write_locks: fcntl dotlock mail_executable: /usr/local/libexec/dovecot/pop3 mail_plugin_dir: /usr/local/lib/dovecot/pop3 pop3_lock_session: yes pop3_uidl_format: %08Xu%08Xv auth default: mechanisms: plain apop login worker_max_count: 5 passdb: driver: passwd-file args: /usr/local/etc/dovecot.passwd passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 2

[Dovecot] Error in dovecot-auth logging
by Antonio Casado Rodríguez 07 May '08

07 May '08

Hi, i have this in /var/log/messages May 7 09:29:59 blade25 dovecot-auth: pam_ldap: ldap_result Timed out May 7 09:29:59 blade25 dovecot-auth: pam_ldap: ldap_result Can't contact LDAP server but i defined log_path and info_log_path in dovecot.conf "dovecot -n not show different values for "protocol lda"? Regards # dovecot -n # 1.0.10: /etc/dovecot.conf log_path: /Correo/log/dovecot/dovecot.log info_log_path: /Correo/log/dovecot/dovecot.info protocols: imaps pop3 managesieve ssl_cert_file: /etc/pki/dovecot/certs/mycert.crt ssl_key_file: /etc/pki/dovecot/private/mycert.key login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_executable(managesieve): /usr/libexec/dovecot/managesieve-login first_valid_uid: 200 mail_location: maildir:%h/.. mail_executable(default): /usr/local/sbin/post-login-imap-dovecot mail_executable(imap): /usr/local/sbin/post-login-imap-dovecot mail_executable(pop3): /usr/local/sbin/post-login-pop3-dovecot mail_executable(managesieve): /usr/libexec/dovecot/managesieve mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3 mail_plugin_dir(managesieve): /usr/lib64/dovecot/managesieve namespace: type: private separator: / location: maildir:%h/.. inbox: yes auth default: passdb: driver: pam userdb: driver: passwd plugin: quota: maildir:storage=1048576

2 1

[Dovecot] dovecot pop3 proxy with AUTH PLAIN
by Luis Barrueco 07 May '08

07 May '08

Hi, I tried as hard as I could, but I couldn't get this working. So here I come begging for help :) I need to configure dovecot to proxy some users' incoming pop3 connections to another server running the Zimbra Collaboration Suite, trying to log into zimbra via a master password. The problem I found is that zimbra only allows me to log in as a master user with the "AUTH PLAIN" mechanism, but it seems that dovecot uses the "USER.... PASS" authentication when proxying. Am I missing something? Has anyone ever tried to do the proxy+master password to zimbra? I'll post my configurations down here. I really appreciate your help- ----------------------------- dovecot version: 1.0.13 ----------------------------- "dovecot -n" output: # 1.0.13: /usr/local/etc/dovecot.conf protocols: imap pop3 ssl_disable: yes disable_plaintext_auth: no login_dir: /usr/local/var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login mail_privileged_group: mail mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv auth default: passdb: driver: pam passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: passwd ----------------------------- /usr/local/etc/dovecot-sql.conf: driver = mysql connect = host=172.16.0.252 dbname=dovecot user=usuario_dovecot password=miclave password_query = SELECT password, 'secret_zimbra' as pass, host, destuser, 'Y' AS proxy FROM proxy WHERE user = '%u' ----------------------------- sudo tcpdump -w output_tcpdump -i eth0 dst host 172.16.0.20 and dst port 110: .........USER mnunez master ......... .........PASS secret_zimbra ----------------------------- I hope this information was helpful! Cheers.

2 1

[Dovecot] Wish: change default log_timestamp
by Steffen Kaiser 07 May '08

07 May '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I'd suggest to change the default value of log_timestamp to something with complete date and time. Currently the string is missing the year. Maybe, %F (ISO date) would fit best. Bye, - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFIHz1uVJMDrex4hCIRAqIiAJsEBSsnhIZg2nGnuzPmJbiSf7VjrwCdHPTX T31joYCErSgOt5OAXa/11OM= =6cC1 -----END PGP SIGNATURE-----

2 1

[Dovecot] FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919
by Adam McDougall 07 May '08

07 May '08

This happened a while back, I was running 1.1rc3 up until today so I have no idea if it would have an effect. Sorry if its something that has been fixed. The error below was repeated many many times, I deleted the index so the user would stop getting disconnected (although I didn't hear any complaints). I have not seen this happen repeatedly in this manner to any other user yet. Apr 15 11:26:48 boomhauer dovecot: IMAP(username): FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919 Apr 15 11:26:48 boomhauer dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Broken virtual size for mail UID 5003 Apr 15 11:26:48 boomhauer dovecot: IMAP(username): Disconnected: Disconnected bytes=139/145074 Apr 15 11:27:08 boomhauer dovecot: IMAP(username): FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919 Apr 15 11:27:08 boomhauer dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Broken virtual size for mail UID 5003 Apr 15 11:27:08 boomhauer dovecot: IMAP(username): Disconnected: Disconnected bytes=139/145074 Apr 15 11:28:50 boomhauer dovecot: IMAP(username): FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919 Apr 15 11:28:50 boomhauer dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Broken virtual size for mail UID 5003 Apr 15 11:28:50 boomhauer dovecot: IMAP(username): Disconnected: Disconnected bytes=139/145074 Apr 15 11:38:02 boomhauer dovecot: IMAP(username): FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919 Apr 15 11:38:02 boomhauer dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Broken virtual size for mail UID 5003 Apr 15 11:38:02 boomhauer dovecot: IMAP(username): Disconnected: Disconnected bytes=139/145074 Apr 15 11:38:52 boomhauer dovecot: IMAP(username): FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919 Apr 15 11:38:52 boomhauer dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Broken virtual size for mail UID 5003 Apr 15 11:38:52 boomhauer dovecot: IMAP(username): Disconnected: Disconnected bytes=139/145074 >From dovecot-uidlist: 5003 W38919 S38270 :1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 1208208981.7303_0.hill:2,S # ls -l .snapshot/*/1208208981.7303_0.hill* -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.0/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.1/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.10/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.11/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.12/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.13/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.14/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.15/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.16/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.17/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.18/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.19/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.2/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.20/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.21/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.22/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.3/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.4/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.5/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.6/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.7/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.8/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.9/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/nightly.0/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/nightly.1/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/nightly.2/1208208981.7303_0.hill:2,S -rw------- 1 username faculty 5236 Apr 14 17:36 .snapshot/nightly.3/1208208981.7303_0.hill:2,S # mv dovecot.index.cache dovecot.index.cache.broken

2 1

[Dovecot] dovecot + exim + verify recipient / reject bad recipient at SMTP time
by Peter Hudec 07 May '08

07 May '08

hello, I'm quite new to dovecot. My instalation is working good Debian Etch - exim: 4.63-17 - dovecot: 1.0.rc15 I'm using virtual user setup, with LDA delivery method. There is no database backend at this time. Is there any way to configure exim to reject non-existing user /asking dovecot lda/ at SMTP time. The standard exim configuration in ACL section do not work: --- cut --- require message = e-mail address is not valid ($local_part@$domain / $recipient_verify_failure) domains = +virtual_domains verify = recipient --- cut --- Please, have any work working example ? This setup work good witch cyrus-imap and LMTP. best regards Peter Hudec

2 1

[Dovecot] Manual thread breaking from Thunderbird
by Christian Jaeger 07 May '08

07 May '08

Hello I'm looking for a way to manually break threads apart when independent threads are mixed up, mostly by users which aren't aware that they should not be replying to existing threads when starting a new one. Since Thunderbird afaik does the threading itself, it would require a solution that actually changes the In-Reply-To header (and maybe also that other (non-standard?) header which is being sent by the M$ clients). For me modifying the actual emails on disk would be fine, i.e. renaming the In-Reply-To header to Old-In-Reply-To and such. (I'd actually *prefer* a solution that modifies the server side store, as opposed to one which keeps that info on the client side only like mutt seems to be doing.) Is there a way to do this already? Seems like a task for a dovecot plugin--or for the xexec plugin and an external perl script or some such? For triggering the renaming, a hacky solution could be to misuse labels (e.g. marking a message starred and "Later" and "Important" at the same time would trigger the modification and remove the labels). When implementing such a thing: maybe also offer a way to modify subjects (for those cases where people are using short meaningless and recurring subjects--to both help breaking up the ensuing threading as well as giving them a useful replacement), maybe as an IMAP extension or such (to be used by Thunderbird plugins (?), or by web apps). (I can code in C and Perl, but don't generally have much time.) Christian.

6 21

[Dovecot] Problem finding inbox and pam_authenticate failed problem
by Chris Beech 07 May '08

07 May '08

Testing installation of Dovecot. Log is showing two errors. Any ideas how to fix this thing??? dovecot: May 07 05:19:06 Error: auth-worker(default): pam(Yoda,127.0.0.1): pam_authenticate() failed: User not known to the underlying authentication module dovecot: May 07 05:21:48 Error: IMAP(Yoda): open() failed with mbox file /var/mail/Yoda: Permission denied Here's the dovecot.conf file # 1.1.rc4: /usr/local/etc/dovecot.conf base_dir: /var/run/dovecot/ log_path: /var/log/dovecot.log info_log_path: /var/log/dovecot-info.log protocols: imap imaps pop3 pop3s listen(default): *:143 listen(imap): *:143 listen(pop3): *:10100 ssl_disable: yes disable_plaintext_auth: no login_dir: /var/run/dovecot//login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login verbose_proctitle: yes first_valid_uid: 1000 last_valid_uid: 5000 mail_privileged_group: mail mail_location: mbox:/var/empty:INBOX=/var/mail/%u:INDEX=MEMORY mail_debug: yes mbox_write_locks: fcntl mbox_very_dirty_syncs: yes mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 auth default: debug: yes debug_passwords: yes passdb: driver: pam passdb: driver: passwd passdb: driver: shadow passdb: driver: passwd-file args: /usr/local/etc/dovecot.passdb userdb: driver: passwd userdb: driver: passwd-file args: /usr/local/etc/dovecot.passdb socket: type: listen master: path: /var/run/dovecot/auth-master mode: 384 user: dovecot group: mail ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

1 0

[Dovecot] Question about Maildir automatic cleanup.
by José Ildefonso Camargo Tolosa 07 May '08

07 May '08

Hi! I want to automatically delete old messages from one user's Maildir, and I was thinking on running something like this: find /home/user/Maildir/cur/ /home/user/Maildir/new/ -daystart -mtime +15 -delete but I'm not sure if by deleting messages "manually" I will break the dovecot.index* files. What do you think? Thanks! Ildefonso. ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

3 2

[Dovecot] 1.07 to 1.0.13 mail_extra_groups
by Mark Sapiro 07 May '08

07 May '08

I have just upgraded from 1.0.7 to 1.0.13. I read the comments in the example .conf file and it seemed what I needed to enable dotlock access to mailboxes in /var/spool/mail (writable by 'mail' group) was "mail_privileged_group = mail" so I removed the "mail_extra_groups = mail" that I had in 1.0.7 and added "mail_privileged_group = mail", but I got errors like the following: May 6 12:48:54 sbh16 dovecot: POP3(xxx): file_lock_dotlock() failed with mbox file /var/spool/mail/xxx: Permission denied May 6 12:48:54 sbh16 dovecot: POP3(xxx): Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2008-05-06 12:48:54] May 6 12:48:54 sbh16 dovecot: POP3(xxx): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 May 6 12:48:54 sbh16 dovecot: pop3-login: Login: user=<xxx>, method=PLAIN, rip=72.52.113.36, lip=72.52.113.36, secured I then put "mail_extra_groups = mail" back in the configuration. This seems to have the effect of setting both "mail_privileged_group = mail" and "mail_access_groups = mail" as well as issuing "Warning: mail_extra_groups setting was often used insecurely so it is now deprecated, use mail_access_groups or mail_privileged_group instead" when dovecot is started. So my question is what do I need. I thought it would be just "mail_privileged_group = mail", but that doesn't work. Do I need both "mail_privileged_group = mail" and "mail_access_groups = mail" or just "mail_access_groups = mail". Also, are the comments describing these in the example .conf file correct, or are they reversed. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

1 0

[Dovecot] dovecot shows inbox of different user wher logged from the same host twice
by Anselmo Rosa herlen 06 May '08

06 May '08

I have squirrenmail 1.4.13 configured on FC8 with sendmail,procmail as delivery agent, dovecot(imap) ,Using MAILDIR, enabled secure_login plugin, and using password/shadow and system user . When I login through squirrelmail every thing works fine . But when I login again from the same host as a different user with new browser session, the mailbox of the new user displays correctly but the first user session displays the inbox of the second user . ie if I have two https sessions from a single machine and login as two different users the inbox of the later user is displayed in both the sessions. I checked the logs of dovecot and the following is noticed dovecot: May 06 11:12:25 Info: imap-login: Login: user=<ath>, method=PLAIN, rip=::ffff:127.0.0.1, lip=::ffff:127.0.0.1, secured dovecot: May 06 11:12:25 Info: IMAP(ath): Disconnected: Logged out there are lots of messages for a single login . Also the USER shown above is the user of the second session and is displayed when i try to access the first htts session dovecot -n (reports the following) # 1.0.5: /etc/dovecot.conf log_path: /var/log/dovecot protocols: imap imaps ssl_disable: yes login_dir: /var/run/dovecot/login login_executable: /usr/libexec/dovecot/imap-login mail_location: maildir:~/Maildir auth default: passdb: driver: pam userdb: driver: passwd kindly help Best Jokes, Best Friends, Best Food and more. Go to http://in.promos.yahoo.com/groups/bestofyahoo/

2 1

[Dovecot] deliver custom maildir creation
by Steve Roemen 06 May '08

06 May '08

/ I'm testing a new setup of postfix using deliver to put the messages into the user's maildir, and am curious if there is a way to make deliver create custom maildirs when it delivers to a valid user without a maildir? Right now it only creates the inbox, I'd like it to create the .sent/ .Trash/ and .SPAM dirs. Is this possible? Thanks -------------------------------------- / /Steve Roemen// /

5 8

[Dovecot] Problem with quota.
by john.wall＠cky.se 06 May '08

06 May '08

Hi, Im having problem with quota with 1.0.rc17-1ubuntu2.3. Problem is that dovecot wont bounce when quota are exceeded. I have set my quota limit to 2MB and my test-box is 3MB. here is my config: dovecot.conf: #ssl_cert_file = /etc/pki/dovecot/certs/myserver.example.com.crt #ssl_key_file = /etc/pki/dovecot/private/myserver.example.com.key #ssl_ca_file = /etc/pki/dovecot/certs/ca-bundle.crt mail_location = maildir:/home/vmail/%d/%u first_valid_uid = 5000 last_valid_uid = 5000 maildir_copy_with_hardlinks = yes disable_plaintext_auth = no protocols = imap imaps pop3 pop3s protocol imap { mail_plugins = quota imap_quota imap_client_workarounds = outlook-idle delay-newmail } protocol pop3 { mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } protocol lda { postmaster_address = postmaster(a)example.com mail_plugins = quota log_path = /var/log/dovecot-deliver.log info_log_path = /var/log/dovecot-deliver.log } auth default { # Having "login" also as a mechanism make sure outlook can use the auth smtpd as well # http://wiki.dovecot.org/Authentication/Mechanisms mechanisms = plain login passdb sql { args = /etc/dovecot/sql.conf } userdb sql { args = /etc/dovecot/sql.conf } userdb static { args = uid=5000 gid=5000 home=/home/vmail/%d/%n allow_all_users=yes } userdb prefetch { } user = nobody socket listen { master { path = /var/run/dovecot/auth-master mode = 0660 user = vmail group = vmail } client { path = /var/spool/postfix/private/auth mode = 0660 user = postfix group = postfix } } } dict { } plugin { quota = maildir:storage=1024:messages=1000 acl = vfile:/etc/dovecot/acls trash = /etc/dovecot/trash.conf } sql.conf: driver = mysql connect = host=localhost dbname=mdbv3 user=xxxx password=xxxx user_query = SELECT concat('/home/vmail/', maildir) as home, concat('maildir:/home/vmail/', maildir) as mail, 5000 AS uid, 5000 AS gid, concat('maildir:storage=', quota) AS quota FROM mailbox WHERE username = '%u' AND active = '1' password_query = SELECT username as user, password, concat('/home/vmail/', maildir) as userdb_home, concat('maildir:/home/vmail/', maildir) as userdb_mail, 5000 as userdb_uid, 5000 as userdb_gid FROM mailbox WHERE username = '%u' AND active = '1' Is this file even nessesary ? (postfix stuff) mysql_virtual_mailbox_limit_maps.cf (not showing login inf etc): query = SELECT quota FROM mailbox WHERE username='%s' AND active = '1'

2 1

[Dovecot] [Solved] Re: Quota not showing correctly after conversion (mb2md.pl)
by k bah 06 May '08

06 May '08

On Fri, 2008-04-25 at 09:56 +0800, k bah wrote: >> I have a problem, when a user has mails that exceed the limit in mb's >> (I set just size limit, no messages limit), neither >> "getquotaroot" (via telnet to port 143), neither webmail software >> (horde) show the quota usage correctly (I'm not talking about new >> mail, talking about those messages copied, when they exceed the size >> limit). Just after trying to delete messages it is shown (gives me an >> error message, quota exceeded, since Trash is not excluded from quota >> limits, for now, I'll exclude it later). This server has some issues: >> many old users have lots of mails, in thousands (1000-15000). Some >> have 50mb to 150mb of mails. The migration is part of the cleanup >> process. > That's quite a confusing paragraph. Do you just mean that "quota is > counted wrong for the old converted messages"? If the user received new mail on the new system, say, around 200kb, and then he/she logs in, messages from the old system are copied. If he had 100Mb of mails (old system), and quota is 20Mb, only the 200kb are shown. Dovecot was not restarted. The problem was what you pointed out, maildirsize had to be deleted, then everything seems to be ok - getquotaroot indicates 100Mb + 200kb. > How badly wrong is the quota? Show me some example values of what they > are and what they're supposed to be? How do you calculate what the quota > is supposed to be? User has +100Mb, quota 20Mb, quota indicates 0 or just the new mail count (messages received on the new system). As you said, just delete maildirsize. >> Is it possible that the migration order (creating dirs in that way) >> is causing the quota usage to be shown wrong? > If all this is before the imap process is started, I don't see any > problems. >> Is it possible that the amount of messages and the size can be the >> cause? > No. >> How can I manually tell dovecot to recalculate the quota, after >> copying messages (when a user logs in, and didn't receive new mails on >> the new mail system, but had messages big enough to exceed the quota >> limit) from the old system? > Deleting maildirsize file recalculates the quota. That was the problem! >> Should I run any commands to rebuild indexes or any operation to keep >> information integrity about the mails/mailboxes to dovecot, after >> copying the converted messages? > No. Ok = -- Powered by Outblaze

1 0

[Dovecot] imap fs quota (rpc) won't work?
by Adam McDougall 06 May '08

06 May '08

In the past I dabbled with the imap quota plugin with the fs backend because I wanted to report usage to my users (not limit them). At the time, the quota plugin would make dovecot crash when trying to write to a folder (I can bring up this report if needed). However, in a later beta of dovecot 1.1 I tried quota again but I cannot get it to report any results. I've been wanting to use fs:noenforcing but also tried just quota = fs to simplify. When I right click a folder stored in my homedir, thunderbird tells me 'the server does not support quotas' in the folder properties. What else can I do? Thanks! IModule loaded: /usr/local/lib/dovecot/imap/lib10_quota_plugin.so IModule loaded: /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota add storage dir = /home/mcdouga9/Maildir Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota block device = cypher:/vol/mail/home Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota mount point = /home gribble# quota -v mcdouga9 | grep home /home 10766672 20480000 20480000 963778 4294967295 4294967295 # 1.1.beta14: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: acl fts fts_squat quota imap_quota mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/public/indexes/de list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: quota: fs acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat

1 1

Re: [Dovecot] 1.1rc1: assertion failed: (output->offset > uid_list[0])
by Adam McDougall 06 May '08

06 May '08

Timo Sirainen wrote: > On Fri, 2008-03-07 at 00:17 -0500, Adam McDougall wrote: > >> $3 = {offset = 40896, stream_errno = 70, last_failed_errno = 70, overflow >> = 0, closed = 1, >> real_stream = 0x678500} >> > > Ah, 70 = Stale NFS handle in FreeBSD. Linux used something else. Squat > still needs some NFS fixes. > > Were these 'NFS fixes' done? I just put rc5 in production at my site an hour or two ago so I have just begun gathering log data. I may not see anything interesting for days but I will review them daily. (apologies if this gets double posted, I was sending mail and got disconnected)

1 0

[Dovecot] Couldn't init INBOX: Can't sync mailbox: Messages keep getting expunged
by David Jonas 06 May '08

06 May '08

Upon upgrading from 1.0.2 to 1.1rc4 I see this error for many of our users: Getting size of message UID=1 failed Couldn't init INBOX: Can't sync mailbox: Messages keep getting expunged Logging in with IMAP I would see a bunch of messages with no subject or time and blank bodies (usine horde/imp). Removing dovecot.index.log, dovecot.index.log.2, or dovecot-uidlist fixes the problem for the account (I chose dovecot.index.log). I imagine that forces a rebuild of the indexes. Removing other files in the directory didn't seem to make a difference, including dovecot.index or some arbitrary non-dovecot file. Is there a more conventional way around this problem?

3 9

[Dovecot] A previously unreported lsub/list discrepancy in 1.1rc5 and earlier
by Adam McDougall 06 May '08

06 May '08

Not sure about 1.0, I don't run it anymore :) But a few users discovered for a small issue that only affects a very small portion of my userbase where instructional correction will suffice. I guess I am reporting it in the interest of getting it out there, and I can live with it if not fixed but it might be an issue for some people. I'm running through my list of issues. Basically, if a user has a valid namespace prefix set such as mail/ (used to be required on old mail server but discouraged on new mail client setups with dovecot), lsub/list do not show public namespaces that the user has permission to via acl. If the user removes the namespace prefix, they can 'see' the public folders and subscribe. Additionally if the user subscribes to a public folder then puts mail/ back in the prefix, the folders are no longer listed. Not sure if public folders should be affected by the requested prefix in this manner, since they are not in the user's account. With prefix set: 3 lsub "" "mail/*" 4 list "" "mail/%" 5 list "" "mail/%/%" * OK [RAWLOG TIMESTAMP] 2008-05-05 21:07:16 * LSUB () "/" "mail/INBOX" * LSUB () "/" "mail/Sent Items" * LSUB () "/" "mail/Drafts" * LSUB () "/" "mail/Junk E-mail" * LSUB () "/" "mail/Sent" * LSUB () "/" "mail/Trash" 3 OK Lsub completed. * LIST (\HasNoChildren) "/" "mail/Drafts" * LIST (\HasNoChildren) "/" "mail/Sent Items" * LIST (\HasNoChildren) "/" "mail/Junk E-mail" * LIST (\HasNoChildren) "/" "mail/Sent" * LIST (\HasNoChildren) "/" "mail/Trash" 4 OK List completed. 5 OK List completed. Without prefix set: 3 lsub "" "*" 4 list "" "%" 5 list "" "%/%" * OK [RAWLOG TIMESTAMP] 2008-05-05 21:07:49 * LSUB () "/" "INBOX" * LSUB () "/" "Sent Items" * LSUB () "/" "Drafts" * LSUB () "/" "Junk E-mail" * LSUB () "/" "Sent" * LSUB () "/" "Trash" 3 OK Lsub completed. * LIST (\HasNoChildren) "/" "Drafts" * LIST (\HasNoChildren) "/" "Sent Items" * LIST (\HasNoChildren) "/" "Junk E-mail" * LIST (\HasNoChildren) "/" "Sent" * LIST (\HasNoChildren) "/" "Trash" * LIST (\HasNoChildren) "/" "INBOX" 4 OK List completed. * LIST (\Noselect \HasChildren) "/" "#shared/be" * LIST (\Noselect \HasChildren) "/" "#shared/me" * LIST (\Noselect \HasChildren) "/" "#shared/cee" 5 OK List completed.

2 1

[Dovecot] Replication milestone 1
by Timo Sirainen 06 May '08

06 May '08

5 milestones planned currently. I'll start with the always-on multi-master replication, because that's the most difficult one to get working correctly and efficiently. So if during its implementation I find some design problems, less code needs to be fixed. Milestone 0 will hopefully be done within a month. This includes reading and replying to the rest of the mails on this list and getting a v1.1.0 release out. :) There will most likely also be a v1.2.0 release this summer that has some new features on top of v1.1. Several companies have asked for replication within the past year. At least two are willing to contribute some money into its development this summer, but having more wouldn't hurt of course. :) Paying also guarantees (within reasonable limits) that I'll listen to what kind of a installation you're planning and make sure that the replication will work in that kind of a setup. So, below's the milestone 1 fully described with a couple of FIXMEs left. The rest of the milestones are described only briefly. 0. Prerequisites: - Change v1.2 CONDSTORE implementation to allocate modseqs differently. Currently log file sequence + offset determines the modseq. This can't work in a cluster since their log files are different. Instead add a new "UIDs [..] have modseq X" transaction log record that gets updated when needed. UID list can be 0 to update higest-modseq. To avoid adding modseq for all changes, we can be optimistic and assume that usually multiple servers aren't doing changes at the same time. So the next transaction after modseq=5 record will have modseq=6 and the following one modseq=7, etc. - Update v2.0 framework to include v1.2 changes and make it work. Base the replication code on top of v2.0. - Shared mailboxes making all users' all mailboxes available for read/write within the same (replication) process. 1. Incremental replication for existing mailboxes. IMAP, POP3 and deliver use a replication plugin, which connect to a replication process via UNIX socket (or TCP optionally for chroots). The plugin sends the SAVE, EXPUNGE and STORE changes to the replication process, which forwards them to other servers. The other servers receive the commands and store them to local filesystem. The sending and the receiving sides have quite distinct jobs, so they should probably be implemented as separate processes. They could still use the same TCP socket though, replication sending process writes to the socket and replication receiving process reads the socket. The processes still need to share some state though. Most importantly keeping track of the mailbox master servers. This is probably the most cleanly implemented by having a separate master tracking process which the sending/receiving processes talk to via UNIX socket. The rest of the data sharing is about forwarding command replies (SAVE, STORE with CONDSTORE flag) to replication sender, so it can pass them on to the originating mail process. This can be done easily using an extra pipe/socket between the processes. Examples use: - C1 = Client 1 - C2 = Client 2 - S = Replication sending process - R = Remote replication receiving process - Small letter before commands = IMAP-like command tag - * before commands = tag for a command that doesn't expect a reply 1.1. Master tracking process Master tracking process is connected to sending and receiving replication processes. LOCK: (mail process -> replication sender -> master tracker) - User name - Mailbox name Reply: - success LOCK is sent by replication plugin from mail processes when it wants to assign UIDs for new messages. Replication sender proxies this command to master tracker. If server is already the master, the command returns success immediately. Otherwise tracker looks up from its cache who the current master is and requests it using the REQUEST command. Once GIVE command is received, reply success to the mail process. FIXME: handle timeouts (and master disconnections) somehow? UNLOCK: (mail process -> replication sender -> master tracker) - User name - Mailbox name No reply. UNLOCK is sent by mail processes after they've finished assigning UIDs and sending replication SAVEs. If this was the last lock, pending TAKE requests for this mailbox are handled. GIVE: (replication receiver -> master tracker) - User name - Mailbox name No reply. Notification that this server is now the master for the specified mailbox. If there are any pending LOCK requests for this mailbox, they're replied to in here. TAKE ([mail process ->] replication receiver -> master tracker): - User name - Mailbox name - Server ID No reply. Request for giving master state for the specified server. When this mailbox contains no locks, the TAKE command is forwarded to replication sending process which passes it to the requesting server. While waiting for existing locks to be releases, handling new LOCK commands for the mailbox must be delayed to avoid lock starvation. When forwarding TAKE command, the destination server ID is cached as being the mailbox's current master. If we're not the current master for this mailbox, forward the request using REQUEST command. This also happens if there are multiple pending TAKE requests. The first one gets to become the master and the rest of them are handled as REQUEST commands. REQUEST (master tracker -> replication sender): - User name - Mailbox name - Server ID of the current assumed master - Server ID who wants to be the new master (e.g. ourself) Request master status to be moved to the specified server. If server ID isn't ours, cache it as being the mailbox's current master. If we don't have any idea who the master might be currently, use the root server ID. ROOT-SET (replication receiver -> master tracker): - Server ID This command causes all cached master states to be dropped. Any new LOCK requests are delayed. Once all mailboxes are unlocked, drop being a master from all mailboxes and send ROOT-CLEAR command to replication sender, which passes on the information that we've cleared the master states. The given server ID is set to be the new root. ROOT-CLEAR (replication receiver -> master tracker): - Server ID Given server ID has notified that it has cleared its master states. If we're not the current root, this command is forwarded to it. If we're the root and this was the last pending ROOT-CLEAR, start handling the pending REQUEST and LOCK commands. FIXME: How is initial root selection done? How is it decided who's the new one when previous one dies? 1.2. Replication sending 1.2.1. Saving (For now we ignore the possibility of saving as non-master.) When a mail process wants to give UIDs for newly saved messages, it first requests for master lock for the mailbox from the master tracker process. Once UIDs have been assigned for the new messages, the messages are sent to replication sender using SAVE commands. After the last one is sent, the master lock is dropped. Note that this lock duration should be as small as possible, so in Dovecot's code this means doing both locking and unlocking in mailbox_transaction_commit(). If multiple processes are sending SAVEs at the same time, the replication sender may receive the UIDs in wrong order. It must reorder them before forwarding the SAVEs to other servers. Since all master LOCK/UNLOCK commands pass through the replication sender, it can figure out when it has seen all the SAVEs based on them. For example: C1-C3 -> S: a LOCK [1] S -> C1-C3: a OK [Client 1 sends the first mail] C1 -> S: b SAVE uid=5 .. [SAVE uid=5 depends on dropping [1] locks (currently clients 1,2,3)] C1 -> S: * UNLOCK [1] (left 2) [Client 1 starts sending more mails in a separate transaction, lock counter is incremented because of the previous SAVE] C1 -> S: d LOCK [2] (remember lock[2].prev_uid=5) S -> C1: d OK [Client 2 sends its mail] C2 -> S: b SAVE uid=9 .. [SAVE uid=9 depends on dropping [1-2] locks (currently clients 1,2,3)] C2 -> S: * UNLOCK [1] (left 1) [Client 3 sends its mail] C3 -> S: b SAVE uid=3 .. [SAVE uid=3 depends on dropping [1] lock (currently client 3). It doesn't depend on [2] lock, because uid < lock[2].prev_uid (3 < 5)] C3 -> S: * UNLOCK [1] (left 0) [SAVEs depending on [1] locks are ordered and forwarded.] S -> R: x SAVE uid=3 .. S -> R: y SAVE uid=5 .. [Client 1 finishes its second transaction] C1 -> S: f SAVE uid=7 [SAVE uid=7 depends on dropping [2] lock (currently client 1)] C1 -> S: * UNLOCK [2] (left 0) [SAVEs depending on [2] locks are ordered and forwarded] S -> R: x SAVE uid=7 .. S -> R: y SAVE uid=9 .. The idea is that locks have a counter which is incremented if there was a SAVE command sent after the previous LOCK command. The previous SAVE's UID (prev_uid) is also stored to the lock. When the first SAVE after LOCK arrives, it's assigned a dependency on all existing locks which have uid > prev_uid. The lock counter order doesn't guarantee an ascending prev_uid order. (An actual lock counter probably isn't even needed, prev_uid should be enough as the lock ID. Lock counter simplifies the above example though.) All subsequent SAVEs from the client before the next UNLOCK will have the same dependency as the first SAVE. This is because by that time the transaction has been committed and all messages within the transaction have been given UIDs. So in fact it wouldn't really matter in which order the SAVEs are even sent to the replication sender. Although sending them in order allows an extra optimization: If only a single client is holding a lock, the SAVEs can be immediately forwarded as they're being received. After all lock dependencies have been dropped, it's guaranteed that all new SAVEs will have higher UIDs, so the SAVEs can be ordered and forwarded. A further optimization could use some kind of weak locks that are dropped after the client's first SAVE command. After the weak lock count has dropped to zero, the messages can started to be sent. For example assume two clients having the same lock counter. Both clients would be sending 10000 messages within the same transaction. Waiting for and buffering all of them would waste a lot of memory. Instead it would be possible to read the first SAVE from the first client and then block until the first SAVE is read from the second client. The SAVE with a smaller UID can be forwarded to remote servers. The next message is then read from that connection, and this is continued until all messages have been processed. A simpler solution would have been to order and forward SAVEs only when lock counter dropped to zero, but under very high load it's (at least theoretically) possible that the counter never drops to zero. In synchronous setups (recommended whenever possible) the SAVE commands use a tag (not "*"), which requests an ack reply from the remote. After replication sender has sent SAVE to a remote server, the remote server eventually replies back. This reply is forwarded from replication receiver to sender, which in turn forwards it to the mail process. After mail process has finished sending all SAVEs, it waits until it has received at least one ack for each SAVEd message. If the replication sender sent the SAVE to multiple servers, mail process may receive multiple acks for each message. It should just ignore the ones it doesn't care about. 1.2.2. Modseq updating If all mailbox modifications were done only while the server was the mailbox master, there wouldn't be any problems with modseq updates, because only one server could modify the mailbox at a time. This scales badly though, so it's preferrable to move the master only when absolutely required (for SAVEs) and handle modseqs another ways. Modseqs are incremented by each mailbox modification. All modification commands contain a modseq parameter. When applying the modification command make sure the local modseq is at least as high as the modification command's modseq. If the local modseq is higher, broadcast the new modseq to all other servers using MODSEQ command. When receiving a MODSEQ command, update the modseq only if its current value is lower than the received one. Remember that modseqs can also be updated for expunged messages. All this means that eventually everyone's modseq will be the highest assigned modseq for the change in the cluster. It doesn't matter if temporarily some clients saw lower modseqs. The worst that can happen is that some extra FLAG/MODSEQ updates are sent to the client. Although there is a problem if a client can jump from a server where it had higher modseqs to a server that hadn't yet received the incremented modseqs and performed a sync using the higher modseq. If the updated modseqs come after the sync, they're lost from the client. This could be avoided by issuing a cluster-wide sync wait before such syncing commands are run. 1.2.3. Expunging EXPUNGE commands can be replicated without waiting for anything. There are problems with messages getting expunged while COPY is being handled, but this will be resolved later when COPY is implemented. 1.2.4. Flag updates When STORE is received by a master, it checks if it has higher modseqs in messages than the modseq in STORE parameter. Then it applies all the changes and forwards the changes to other servers using possibly updated modseqs. For messages that had higher initial modseqs their current flags are sent back to the originating server to fix a potential desync. If STORE is initiated by the master, there are no flag desynchronization possibilities. The correctness of applying conflicting changes could be improved by including a "highest modseq of messages before this STORE" parameter. The applying server (master) could then find the flags from that point in time and apply changes on top of it. And finally on the top would be applied newer changes (that were already once applied). There could still be conflicting changes, but the result would be better for updates that had been delayed for a long time. Finding these flag changes and old flag values isn't simple though, so this won't be at least in initial implementation. STORE UNCHANGEDSINCE (CONDSTORE extension) is initially implemented by grabbing a master lock. If there are simultaneous STOREs in other servers, they'll temporarily have lower modseqs and different message flags. Eventually when the STOREs reach the master, they will be applied on top of the CONDSTORE modification. So even if normal STOREs were sent slightly earlier than CONDSTORE, they'd eventually be applied on top of the CONDSTORE change. This makes the STORE UNCHANGEDSINCE work correctly (even if with a slight delay), and our multi-master replication should be fully compatible with the CONDSTORE spec. 1.3. Replication receiving Replication receives 3 different types of data from remote sender: 1. Replies to commands this server's replication sender sent 2. Message texts (as part of SAVE) 3. Metadata updates (SAVEs, STORE, EXPUNGE, etc.) Command replies are handled by directly forwarding them to the replication sender process via a pipe/socket. 1.3.1. Receiving SAVEs The message text part of the SAVE is stored in a file. The file format depends on the server's configured default data format. For now we consider only maildir format where the data is written as-is to the file. This file is later hard linked to the destination maildir directory. If multiple partitions are used, the file should be written to the correct partition. The initial code could write the file to the destination mailbox's tmp/ directory to make sure the partition is correct. There may be some permission problems with this approach though, a separate replication drop directory would be nicer. The rest of the SAVE is processed similarly to other metadata updates. The path to the written file is added as new metadata to the SAVE command. 1.3.2. Metadata updates The primary replication receiver process's job is to read data from the network as fast as possible. Processing the actual updates can take a while, so the work is distributed to multiple worker processes. As discussed above, the primary replication receiver process still writes message texts to files. The workers could do this as well, but it would mean that the text needs to be sent via IPC to the workers, which adds extra overhead. If the message writing really becomes the bottleneck, this change could be done. The replication receiver tries to pass the same user's updates to the same workers so they can benefit from keeping mailboxes open longer. Each worker could be limited to e.g. 100 open mailboxes. The worker handles the SAVE/EXPUNGE/STORE commands using the normal mailbox handling functions. After the changes were successfully committed, commands that had tags are replied as having succeeded. If a change failed, a failure reply is sent instead for it. Replication receiver passes this reply to the replication sender (not to be confused by remote command replies which are also forwarded), which in turn passes it to the originating server. ------ 2. Mailbox synchronization for existing mailboxes. - As described in the replication protocol. - IMAP UID conflict resolution. 3. Mailbox list synchronization. - Track creates, deletes, renames. - Add unique global mailbox ID which is preserved across renames and allows replication to perform renaming (and symlinking?) 4. More features 4.1. Global UIDs - Save GUIDs to messages (dbox: metadata, maildir: dovecot-uidlist) - Tracking GUIDs globally - Implement COPY command - Implement SAVEs sending only global UIDs and FETCHing unknown ones 4.2. Support for untrusted user-run replication processes - Make SAVE and work without being a master - Make CONDSTORE's STORE UNCHANGEDSINCE work without being a master 4.3. Transactions - Wrap changes into transactions just like they're in the originating Dovecot's transaction (e.g. TX-BEGIN, TX-END commands) - On receiving side a transaction is handled in a single Dovecot transaction by a single worker. 4.4. Cluster-wide sync - Broadcast a ping packet and wait for a reply from all servers on the cluster. This ensures that all changes before the ping packet was sent have been received. - Then ping replication receiver process and have it ping worker processes to wait until all received changes have been committed. - CHECK command could do it. - CONDSTORE/QRESYNC commands that take modseq parameters could maybe do it? Or would it be too slow? It would at least increase reliability. We could probably also figure out some rules when the sync is needed and when not. 5. dbox support - Message text can't just be written to a file and renamed. At minimum some dbox metadata needs to be written to it. - Files can't be shared between mailboxes by hard linking like they can be with maildir

1 1