[Dovecot] extremely slow delete/move operations?
I've been working to migrate several hundered users on to our new mail servers, which run dovecot 0.99.11 (RHEL4). The problem I'm seeing is that deleting or moving a message seems to take a very long time. It's not uncommon for Thundirbird (1.0.7) to have delays of 5-10 seconds to delete a single message from my Inbox (<500k) and place it in my Trash folder (~100MB right now).
The systems are Dell PowerEdge 1850 (dual 3.6GHz Xeon) with two 300GB SCSI drives in a hardware mirror. Filesystems are ext3 on LVM. Inboxes are on /var/spool/mail (/export/mail, symlinked into /var). User mail folders are in their home directories (/export/homes1/<user>, automounted to /home/<user>; some home directories are NFS automounted from a NetApp). Both the mail spool and home directories are available over NFS for compatibility reasons with the former setup.
What's really bugging me is that I've used the same software (RHEL4 and dovecot 0.9x.x) at my last workplace, and the performance was outstanding; it beat UWIMAPd hands-down, especially with large mailboxes. Here, the opposite seems to be true. The primary difference seems to be that we're using NFS here, but I wasn't using NFS at my last job.
I'm getting a lot of grief over the delays when users delete mail or move mail from their inbox to one of their mail folders. Enough so that they're now resisting moving to the new server.
Where do I look to find out what's causing the slowness? What can I do to make it faster?
Mailbox formats are mbox.
Using these options in /etc/dovecot.conf:
protocols = imaps pop3s imap_listen = [::] pop3_listen = [::] ssl_cert_file = /usr/share/ssl/certs/csefast.ucsd.edu_cert.pem ssl_key_file = /usr/share/ssl/private/csefast.ucsd.edu_key.pem login_dir = /var/run/dovecot-login login = imap login = pop3 verbose_proctitle = yes first_valid_uid = 100 default_mail_env = mbox:~/:INBOX=/var/mail/%u mail_full_filesystem_access = yes mbox_locks = dotlock fcntl auth = default auth_mechanisms = plain auth_userdb = passwd auth_passdb = pam auth_user = root
Thanks in advance,
Gregory
-- Gregory K. Ruiz-Ade Sr. Systems Administrator Computer Science and Engineering University of California, San Diego Office: EBU3b 1216 Phone: (858) 822-2625 E-mail: gkra@ucsd.edu
Gregory K. Ruiz-Ade <gkra <at> ucsd.edu> writes:
I've been working to migrate several hundered users on to our new mail servers, which run dovecot 0.99.11 (RHEL4). The problem I'm seeing is that deleting or moving a message seems to take a very long time. It's not uncommon for Thundirbird (1.0.7) to have delays of 5-10 seconds to delete a single message from my Inbox (<500k) and place it in my Trash folder (~100MB right now).
The size of the directory as a whole doesn't matter, the amount of files can cause problems (with certain filesystems).
What's really bugging me is that I've used the same software (RHEL4 and dovecot 0.9x.x) at my last workplace, and the performance was outstanding; it beat UWIMAPd hands-down, especially with large mailboxes. Here, the opposite seems to be true. The primary difference seems to be that we're using NFS here, but I wasn't using NFS at my last job.
NFS can be a real performance killer, but to be sure I'd strace (with -tt) a dovecot process and check where it spends most of it's time.
best regards, Michael Renner
On Monday 30 Jan 2006 18:21, Michael Renner wrote:
NFS can be a real performance killer, but to be sure I'd strace (with -tt) a dovecot process and check where it spends most of it's time.
Especially NFS with blocking writes on large files.... Done badly, with the wrong file system I've seen servers disks trash continuously, and IO down at something resembling paper tape (well I exaggerate slightly).
Check the NFS time for copying larges files, compare to say FTP, NFS should be slower than FTP, but only says 20% slower....
Gregory K. Ruiz-Ade wrote:
The primary difference seems to be that we're using NFS here
<snip> 8<
Mailbox formats are mbox.
Those are the two most important pieces of information in your message (IMHO). mbox is always slower than maildir when the filesize exceeds the operating memory for the server (i.e. it cannot be mmapped). NFS will always be slower than "local" access, even on NetApp's supposed 'high-performance' NFS. The combination is deadly, since every message stored in Trash requires copying the entire mbox file and appending the new message, then deleting the original and renaming the new file in its place.
Then there is the additional performance drag caused by Dovecot rewriting the index, as well as Thunderbird (which you should really upgrade to 1.5) keeping its local index up to date. You can confirm this by emptying your trash folder and seeing if the performance changes (which I am confident it will).
There are three things you can do to improve performance:
- Upgrade to Dovecot 1.0 stable (or even Beta2);
- Switch to Maildir (instead of mbox);
- Install a SAN (instead of the NFS mounted NAS).
The last is the only one that costs you hard capital; the other two only involve your time, which as we all know, doesn't count. ;-)
HTH
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
John Peacock wrote:
Those are the two most important pieces of information in your message (IMHO). mbox is always slower than maildir when the filesize exceeds the operating memory for the server (i.e. it cannot be mmapped).
Well, this is something that I cannot change. We're stuck with mbox for now. The last time I used dovecot (at my previous job), dovecot out-performed UWIMAPd on the same mbox mailboxes, so why it's the other way around is beyond me...
NFS will always be slower than "local" access, even on NetApp's supposed 'high-performance' NFS. The combination is deadly, since every message stored in Trash requires copying the entire mbox file and appending the new message, then deleting the original and renaming the new file in its place.
The thing is, these mail servers _are_ the local storage for the mail spools and home directories. There is no network involved when dovecot is accessing the INBOX or saved mail folders. Yes, they're "NFS" mounted, but this is RHEL4 (linux 2.6.x), and when an NFS share is mounted from the local host, it's done as a bind mount, which means local disk access.
Then there is the additional performance drag caused by Dovecot rewriting the index, as well as Thunderbird (which you should really upgrade to 1.5) keeping its local index up to date. You can confirm this by emptying your trash folder and seeing if the performance changes (which I am confident it will).
Should I then wonder about turning off Dovecot's indexing? If that's a "performance drag"...
Upgrading Thunderbird may not be so simple, either. These same clients were having much faster response times on a Sun E450 running Solaris 2.8 and UWIMAPd with the same logical setup (NFS mail spool and home directories mounted from the local system). Our Windows users can be more easily upgraded to later clients right now than our Linux/Unix users
There are three things you can do to improve performance:
- Upgrade to Dovecot 1.0 stable (or even Beta2);
I wasn't aware there was a 1.0 stable yet.
- Switch to Maildir (instead of mbox);
Not an option.
- Install a SAN (instead of the NFS mounted NAS).
As much as I may wish for it, it's not an option either. And, I point out again, that the mail servers have the necessary filesystems _locally_ mounted, by automount, via bind mounts:
gruizade@csefast(pts/22):~ 21 > mount | grep gruizade /export/homes1/gruizade on /home/gruizade type none (rw,bind)
I will work on testing 1.0 beta2 for now...
Thanks again,
Gregory
-- Gregory K. Ruiz-Ade Sr. Systems Administrator Computer Science and Engineering University of California, San Diego Office: EBU3b 1216 Phone: (858) 822-2625 E-mail: gkra@ucsd.edu
Gregory K. Ruiz-Ade wrote:
Well, this is something that I cannot change. We're stuck with mbox for now. The last time I used dovecot (at my previous job), dovecot out-performed UWIMAPd on the same mbox mailboxes, so why it's the other way around is beyond me...
Same hardware? Same file sizes? Same software rarely translates to equivalent performance unless the other parameters are also constant. NFS access in general and mbox files in particular degrade with filesize rather rapidly in my experience. I'm not saying maildir is perfect; hugh numbers of files in a single directory will eventually show up inefficiencies in the filesystem caching of directory entries.
NFS will always be slower than "local" access, even on NetApp's supposed 'high-performance' NFS. The combination is deadly, since every message stored in Trash requires copying the entire mbox file and appending the new message, then deleting the original and renaming the new file in its place.
The thing is, these mail servers _are_ the local storage for the mail spools and home directories. There is no network involved when dovecot is accessing the INBOX or saved mail folders. Yes, they're "NFS" mounted, but this is RHEL4 (linux 2.6.x), and when an NFS share is mounted from the local host, it's done as a bind mount, which means local disk access.
I misunderstood and thought that the user folders were on the NetApp (though I went back and re-read your original posting and you were clear that only some were). But, NFS mounts, even if they are local, AFAIK still means NFS filelocking and the associated performance drag. You might check out this page:
http://nfs.sourceforge.net/#section_band see whether your NFS options are tuned for performance (in particular what the setting of subtree_check is).
Upgrading Thunderbird may not be so simple, either. These same clients were having much faster response times on a Sun E450 running Solaris 2.8 and UWIMAPd with the same logical setup (NFS mail spool and home directories mounted from the local system). Our Windows users can be more easily upgraded to later clients right now than our Linux/Unix users
Don't discount this option until you try it; the TBird 1.5 upgrade is very painless (in my experience) and takes the existing profile unchanged. That's up to you, though. I was just pointing out that there are any number of places where the performance can drag.
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
On Mon, 2006-01-30 at 11:12 -0800, Gregory K. Ruiz-Ade wrote:
Those are the two most important pieces of information in your message The thing is, these mail servers _are_ the local storage for the mail spools and home directories. There is no network involved when dovecot is accessing the INBOX or saved mail folders. Yes, they're "NFS" mounted, but
John Peacock wrote: this is RHEL4 (linux 2.6.x), and when an NFS share is mounted from the local host, it's done as a bind mount, which means local disk access.
Depends on the nfs implementation you are using. But the main drawback here is certainly that you add the entire NFS protocol overhead for local operations.
Why not mount the shares with the bind option directly?
Then there is the additional performance drag caused by Dovecot rewriting the index, as well as Thunderbird (which you should really upgrade to 1.5) keeping its local index up to date. You can confirm this by emptying your trash folder and seeing if the performance changes (which I am confident it will).
Should I then wonder about turning off Dovecot's indexing? If that's a "performance drag"...
Upgrading Thunderbird may not be so simple, either. These same clients were
yet it might be a good test case. I personally experienced a dramatic increase in speed with my >700M mbox ...
regards
Udo Rader
-- B e s t S o l u t i o n . a t EDV Systemhaus GmbH
udo rader technischer leiter/CTO mobile ++43 660 5263642
eduard-bodem-gasse 8/3 A-6020 innsbruck fax ++43 512 935833 http://www.bestsolution.at phone ++43 512 935834
On Mon, 2006-01-30 at 13:24 -0500, John Peacock wrote:
Gregory K. Ruiz-Ade wrote:
The primary difference seems to be that we're using NFS here
<snip> 8<
Mailbox formats are mbox.
Those are the two most important pieces of information in your message (IMHO). mbox is always slower than maildir when the filesize exceeds the operating memory for the server (i.e. it cannot be mmapped). NFS will always be slower than "local" access, even on NetApp's supposed 'high-performance' NFS. The combination is deadly, since every message stored in Trash requires copying the entire mbox file and appending the new message, then deleting the original and renaming the new file in its place.
Dovecot doesn't copy the entire mbox file though, it just appends the new mail to it.
Anyway, I'd say the problem here has something to do with Dovecot being v0.99.x. Also is the trash mailbox accessed by something else than Dovecot? If that is done, 1.0beta with mbox_very_dirty_syncs=yes helps.
For what it's worth I noticed that delete/move got a lot slower with Beta1 as compared to Alpha5.
Gregory K. Ruiz-Ade wrote:
mailboxes. Here, the opposite seems to be true. The primary difference seems to be that we're using NFS here, but I wasn't using NFS at my last job.
- put the indexes in a local directory (or use INDEX=memory, but that's not as good)
- upgrade to 1.0.beta2
- migrate to maildir when possible
On Mon, Jan 30, 2006 at 09:44:30AM -0800, Gregory K. Ruiz-Ade wrote:
my Inbox (<500k) and place it in my Trash folder (~100MB right now). ... The primary difference seems to be that we're using NFS here, but I wasn't using NFS at my last job. ... Mailbox formats are mbox.
Big mboxes on NFS, brrrrrr. Off the cuff, I'd say that your problem is right there. Anyway, it's a recipe for problems. Why not maildirs? Preferably with a filesystem that handles large directories well.
HTH
participants (9)
-
Gregory K. Ruiz-Ade
-
Jakob Hirsch
-
John Peacock
-
Lorens
-
Marc Perkel
-
Michael Renner
-
Simon Waters
-
Timo Sirainen
-
Udo Rader