- dovecot - dovecot.org

Re: [Dovecot] Question about Dovecot and NFS
by Tony Kay 02 Feb '06

02 Feb '06

On Thu, 02 Feb 2006 11:48:02 +0200, Timo Sirainen <tss(a)iki.fi> wrote: > -- SNIP -- > As long as both hosts keep using dotlocking, Dovecot should be able to > handle the above without any corruption problems. > > > I did some testing w/dovecot 0.99 earlier in this same config, > > Forget those results and try with 1.0beta instead :) > I ran tests on 1.0b2. I could _not_ make dovecot misbehave on NFS w/two physical servers accessing the same mailbox. All of the "undesirable" things that happened (i.e. really poorly written clients suffering strange views of the mailbox due to their own stupidity) were well within IMAP spec., and are non-destructive/lossy. Kudos Timo! Thanks for writing some really excellent code. -- Anthony Kay University Computing Center (541) 346-1719 GPG Fingerprint: B0DB D46A 60AF FAE7 A94A 5075 0CB4 4D88 9F4F 7F09 The camera makes everyone a tourist in other people's reality, and eventually in one's own. Susan Sontag

1 0

[Dovecot] SRC.RPM for dovecot (cvs-2006-01-30) with lda
by Sergey Ivanov 02 Feb '06

02 Feb '06

Hi, there is dovecot.src.rpm for ALTLinux, (dovecot beta2 with sieve local delivery agent) uploaded to http://ftp.altlinux.org/pub/distributions/ALTLinux/Daedalus/SRPMS.daedalus/… -- Sergey Ivanov

1 0

Re: [Dovecot] Question about Dovecot and NFS
by Tony Kay 02 Feb '06

02 Feb '06

On Wed, 01 Feb 2006 11:45:16 +0200, Timo Sirainen <tss(a)iki.fi> wrote: > On Tue, 2006-01-31 at 22:42 -0800, Tony Kay wrote: > > How does dovecot handle this kind of conflict. I.e. how would one dovecot IMAP on server A detect that dovecot on server B had rewritten something like flags on the mailbox they are both accessing? I'm sure it doesn't leave the dot lock sitting around, since that would block mail delivery. > > It just checks if mbox's mtime has changed. If it has, it checks if > there are new mails or if it needs to do some other synchronization > (mbox_dirty_syncs / mbox_very_dirty_syncs causes it to delay it as long > as possible). > > UW-IMAP also checks if mtime has changed, but instead of trying to > figure out what changed it just disconnects the client with an > "unexpected mbox change" error. UW uses a "kiss of death" signal for collaborative locking...the mtime changes are all assumed to be appends as far as I know. The problem (which I can reproduce) comes with a situation like this: - IMAP host A and B access INBOX for the same user from different physical servers which share the inbox via NFS - Neither sees the other (because UW imap assumes lingering opens come from c-client apps, and it uses signals/flock/tmp to sync them, which are not available when A and B are on separate hosts). - Delete and expunge a message on B - 'A' can still read the expunged message. - Mark two messages for delete on A, including the one previously expunged by B - New message delivered to inbox (possibly by yet another host) - Expunge on A - New message is lost My theory is that the two IMAPs are both holding a fd open, and this causes one of the copies to become stale (in the sense it no longer has a filesystem name). The new mail is delivered to the non-stale box, and then the expunge on 'A' causes the stale open file to be "rewritten" over the top of the non-stale one. Result: mail loss. We also see things like message flags that magically reset themselves, etc. I did some testing w/dovecot 0.99 earlier in this same config, and it seems like it has some issues. For example a delete/expunge on IMAP A was "sort-of" seen by the B (fetch complained that the UIDs had changed, and would not work until a NOOP was done...this is technically OK), but B magically restored the deleted message with a new UID (which is not so good, since a client that caches stuff may now have a strange view of the world...I need to examine the set of "update" notices that went along with these events to better understand the implications). This also worries me because the "magic" reappearance indicates a copy from a stale fd, which could exhibit the same mail loss I produced w/UW IMAP. I am planning on updating to your latest source tree tomorrow (I am out of time today), and see if I can make it "lose" new mail. Opinions??? Thanks! Tony

2 1

[Dovecot] Can't send mail through new server
by Anne Wilson 02 Feb '06

02 Feb '06

I thought I had just about got the old server to the point of being switched off. I have been reading from the dovecot server for a couple of days, and this evening I switched the 'sending' to the new server as well - but I couldn't get anything out. I got: Error while uploading message Could not upload the message dated Wed Feb 1 21:2:56 2006 from Anne Wilson <cannewilson(a)tiscali.co.uk> with subject "xxx" on the server. The destination folder was sent-mail, which has the URL . The error message from the server communication is here: APPEND failed: Can't open append mailbox : Read-only file system ---------- I also got messages like: Error while retrieving messages from the server. Unable to open folder sent-mail. The server replied: SELECT failed: Mailbox open failed, aborted: Read-only file system. --------------- I like to save the sent-mail and trash on the server. ~/Maildir has all the folders including sent-mail and trash. I have the same UID on both boxes. What has gone wrong? Anne -- Registered Linux User No.293302 (http://counter.li.org/)

2 2

[Dovecot] Can't send mail from new server
by Anne Wilson 02 Feb '06

02 Feb '06

I thought I had just about got the old server to the point of being switched off. I have been reading from the dovecot server for a couple of days, and this evening I switched the 'sending' to the new server as well - but I couldn't get anything out. I got: Error while uploading message Could not upload the message dated Wed Feb 1 21:2:56 2006 from Anne Wilson <cannewilson(a)tiscali.co.uk> with subject "xxx" on the server. The destination folder was sent-mail, which has the URL . The error message from the server communication is here: APPEND failed: Can't open append mailbox : Read-only file system ---------- I also got messages like: Error while retrieving messages from the server. Unable to open folder sent-mail. The server replied: SELECT failed: Mailbox open failed, aborted: Read-only file system. --------------- I like to save the sent-mail and trash on the server. ~/Maildir has all the folders including sent-mail and trash. I have the same UID on both boxes. What has gone wrong? Anne -- Registered Linux User No.293302 (http://counter.li.org/)

1 0

[Dovecot] Can't send from the new server
by Anne Wilson 02 Feb '06

02 Feb '06

I thought I had just about got the old server to the point of being switched off. I have been reading from the dovecot server for a couple of days, and this evening I switched the 'sending' to the new server as well - but I couldn't get anything out. I got: Error while uploading message Could not upload the message dated Wed Feb 1 21:2:56 2006 from Anne Wilson <cannewilson(a)tiscali.co.uk> with subject "xxx" on the server. The destination folder was sent-mail, which has the URL . The error message from the server communication is here: APPEND failed: Can't open append mailbox : Read-only file system ---------- I also got messages like: Error while retrieving messages from the server. Unable to open folder sent-mail. The server replied: SELECT failed: Mailbox open failed, aborted: Read-only file system. --------------- I like to save the sent-mail and trash on the server. ~/Maildir has all the folders including sent-mail and trash. I have the same UID on both boxes. What has gone wrong? Anne -- Registered Linux User No.293302 (http://counter.li.org/)

1 0

[Dovecot] extremely slow delete/move operations?
by Gregory K. Ruiz-Ade 01 Feb '06

01 Feb '06

I've been working to migrate several hundered users on to our new mail servers, which run dovecot 0.99.11 (RHEL4). The problem I'm seeing is that deleting or moving a message seems to take a very long time. It's not uncommon for Thundirbird (1.0.7) to have delays of 5-10 seconds to delete a single message from my Inbox (<500k) and place it in my Trash folder (~100MB right now). The systems are Dell PowerEdge 1850 (dual 3.6GHz Xeon) with two 300GB SCSI drives in a hardware mirror. Filesystems are ext3 on LVM. Inboxes are on /var/spool/mail (/export/mail, symlinked into /var). User mail folders are in their home directories (/export/homes1/<user>, automounted to /home/<user>; some home directories are NFS automounted from a NetApp). Both the mail spool and home directories are available over NFS for compatibility reasons with the former setup. What's really bugging me is that I've used the same software (RHEL4 and dovecot 0.9x.x) at my last workplace, and the performance was outstanding; it beat UWIMAPd hands-down, especially with large mailboxes. Here, the opposite seems to be true. The primary difference seems to be that we're using NFS here, but I wasn't using NFS at my last job. I'm getting a lot of grief over the delays when users delete mail or move mail from their inbox to one of their mail folders. Enough so that they're now resisting moving to the new server. Where do I look to find out what's causing the slowness? What can I do to make it faster? Mailbox formats are mbox. Using these options in /etc/dovecot.conf: ----- protocols = imaps pop3s imap_listen = [::] pop3_listen = [::] ssl_cert_file = /usr/share/ssl/certs/csefast.ucsd.edu_cert.pem ssl_key_file = /usr/share/ssl/private/csefast.ucsd.edu_key.pem login_dir = /var/run/dovecot-login login = imap login = pop3 verbose_proctitle = yes first_valid_uid = 100 default_mail_env = mbox:~/:INBOX=/var/mail/%u mail_full_filesystem_access = yes mbox_locks = dotlock fcntl auth = default auth_mechanisms = plain auth_userdb = passwd auth_passdb = pam auth_user = root ----- Thanks in advance, Gregory -- Gregory K. Ruiz-Ade Sr. Systems Administrator Computer Science and Engineering University of California, San Diego Office: EBU3b 1216 Phone: (858) 822-2625 E-mail: gkra(a)ucsd.edu

9 10

Re: [Dovecot] Question about Dovecot and NFS
by Timo Sirainen 01 Feb '06

01 Feb '06

On Tue, 2006-01-31 at 22:42 -0800, Tony Kay wrote: > How does dovecot handle this kind of conflict. I.e. how would one dovecot IMAP on server A detect that dovecot on server B had rewritten something like flags on the mailbox they are both accessing? I'm sure it doesn't leave the dot lock sitting around, since that would block mail delivery. It just checks if mbox's mtime has changed. If it has, it checks if there are new mails or if it needs to do some other synchronization (mbox_dirty_syncs / mbox_very_dirty_syncs causes it to delay it as long as possible). UW-IMAP also checks if mtime has changed, but instead of trying to figure out what changed it just disconnects the client with an "unexpected mbox change" error.

1 0

[Dovecot] Immutable Maildir (or: "I know drag&drop")
by Michael Renner 31 Jan '06

31 Jan '06

Hi, I'm 'constantly' (every odd month) faced with users happily dragging around or (un)subscribing folders in our shared IMAP account. Is there a "clean" way to prevent users from doing that when having a Maildir? Unix permissions probably won't cut the slack since setting any Maildir (sub)directory to readonly (immutable or chmod u-w ) will probably cause dovecot to hit a wall at high speed sooner or later (lock files, rotated cache/log files) Got any other ideas? best regards, Michael Renner

2 1

[Dovecot] Question about Dovecot and NFS
by Tony Kay 31 Jan '06

31 Jan '06

Hi, We are using mbox and UW IMAP, and are having issues because UW IMAP does not support multiple client access when you use NFS for the mail files. I've done some research (see the comments after my 2 questions below), and most of the other major IMAP servers simply don't support it (courier is about the only one that has a positive comment). How much testing have you done w/NFS-mounted mail in mbox format? Have you found a reliable way to handle NFS mbox files? The UW IMAP docs say that multiple client access to mail spools that are NFS mounted is not safe, and the source has comments like this about NFS mail spools: /* Make fcntl() locking of NFS files be a no-op the way it is with flock() * on BSD. This is because the rpc.statd/rpc.lockd daemons don't work very * well and cause cluster-wide hangs if you exercise them at all. The * result of this is that you lose the ability to detect shared mail_open() * on NFS-mounted files. If you are wise, you'll use IMAP instead of NFS * for mail files. * * Sun alleges that it doesn't matter, because they say they have fixed all * the rpc.statd/rpc.lockd bugs. This is absolutely not true; huge amounts * of user and support time have been wasted in cluster-wide hangs. */ Cyrus' FAQ says this about NFS mail spools: (http://asg.web.cmu.edu/cyrus/imapd/install-FAQ.html#nfs) NFS (don't) Q: Weird things are happening. The mailspool is mounted over NFS, and... A: Don't mount the mail spool over NFS. Due to locking problems with NFS, this doesn't work (and for various other reasons, probably isn't worth the bother; IMAP appears to be I/O bound, not CPU bound, and scaling in this way is not a good idea). More specifically, if you use NFS and it looks to work, you may have locking problems in the future which will result in silently lost mail. If you use NFS and things go berzerk, it's because mmap(2) apparently has different semantics on local disk than it has on NFS. You can get the above behavior (silently lost mail) if you work at it. Courier is the only one that seems friendly (excerpt from their web site): Courier-IMAP is popular on Qmail/Exim/Postfix sites that are configured to use maildirs. The primary advantage of maildirs is that multiple applications can access the same Maildir simultaneously without requiring any kind of locking whatsoever. Maildir is a faster and more efficient way to store mail. It works particularly well over NFS, which has a long history of locking-related woes. -- Anthony Kay University Computing Center (541) 346-1719 GPG Fingerprint: B0DB D46A 60AF FAE7 A94A 5075 0CB4 4D88 9F4F 7F09 When they discover the center of the universe, a lot of people will be disappointed to discover they are not it. Bernard Bailey

3 2