[Dovecot] avoiding getting maildir cur/ folder contents?
Hi,
On my server with many mails in one folder and slow IO, getting the mail listing in the cur/ folder is becoming a serious bottleneck. Dovecot is the only thing accessing the mail folders, so would there be any way to avoid getting a listing all the time?
johannes
On 18.7.2007, at 19.03, Johannes Berg wrote:
On my server with many mails in one folder and slow IO, getting the
mail listing in the cur/ folder is becoming a serious bottleneck.
Dovecot is the only thing accessing the mail folders, so would there be any
way to avoid getting a listing all the time?
Well, there are two things:
Remove all dirty states from the code, so if Dovecot changes cur/
it doesn't later try to re-sync the maildir just in case someone else
had changed it. I actually thought about doing this and adding a
maildir_dirty_syncs setting, but then I thought if people want
performance they'll use dbox. :) (Or you could even remove timestamp
comparing completely to make sure it never reads cur/)If you want to avoid readdir() completely, then dovecot-uidlist
file needs to be rewritten each time a flag is changed so it would
always contain up-to-date filenames. That would make it do more writes.
On Wed, 2007-07-18 at 23:45 +0300, Timo Sirainen wrote:
- Remove all dirty states from the code, so if Dovecot changes cur/
it doesn't later try to re-sync the maildir just in case someone else
had changed it. I actually thought about doing this and adding a
maildir_dirty_syncs setting, but then I thought if people want
performance they'll use dbox. :)
I guess I should just use dbox :) Is it compatible with maildir delivery as you had initially planned?
johannes
On 19.7.2007, at 21.00, Johannes Berg wrote:
On Wed, 2007-07-18 at 23:45 +0300, Timo Sirainen wrote:
- Remove all dirty states from the code, so if Dovecot changes cur/ it doesn't later try to re-sync the maildir just in case someone else had changed it. I actually thought about doing this and adding a maildir_dirty_syncs setting, but then I thought if people want performance they'll use dbox. :)
I guess I should just use dbox :)
Don't do that before I've had a chance to rewrite the whole thing in
the next few months.
Is it compatible with maildir delivery as you had initially planned?
Not currently, and I wasn't planning on making it. It would kind of
remove the high-performance idea :)
On Thu, 2007-07-19 at 23:34 +0300, Timo Sirainen wrote:
Don't do that before I've had a chance to rewrite the whole thing in
the next few months.
Heh ok.
Is it compatible with maildir delivery as you had initially planned?
Not currently, and I wasn't planning on making it. It would kind of
remove the high-performance idea :)
But it'd make delivery so much easier since I really depend on maildrop :) Time for some maildrop hacking to get it to run deliver, I suppose.
johannes
On Wednesday, July 18 at 06:03 PM, quoth Johannes Berg:
On my server with many mails in one folder and slow IO, getting the mail listing in the cur/ folder is becoming a serious bottleneck. Dovecot is the only thing accessing the mail folders, so would there be any way to avoid getting a listing all the time?
I don't suppose improving your IO is an option? I find turning on dir_hashing in ext3, and mounting my maildirs with noatime, both significantly improve my performance without requiring me to buy new hardware.
~Kyle
The fact that we live at the bottom of a deep gravity well, on the surface of a gas-covered planet going around a nuclear fireball 90 million miles away, and think this to be normal, is obviously some indication of how skewed our perspective tends to be... -- Douglas Adams
On Wed, 2007-07-18 at 15:37 -0600, Kyle Wheeler wrote:
I don't suppose improving your IO is an option? I find turning on dir_hashing in ext3, and mounting my maildirs with noatime, both significantly improve my performance without requiring me to buy new hardware.
I already have both of those options turned on, the problem is that it's running in Xen and I can't get off that right now.
johannes
On Thursday, July 19 at 12:11 PM, quoth Johannes Berg:
On Wed, 2007-07-18 at 15:37 -0600, Kyle Wheeler wrote:
I don't suppose improving your IO is an option? I find turning on dir_hashing in ext3, and mounting my maildirs with noatime, both significantly improve my performance without requiring me to buy new hardware.
I already have both of those options turned on, the problem is that it's running in Xen and I can't get off that right now.
Hmmm, so how many mails in one directory are we talking about?
If Xen is the only thing you can't get out of, have you considered using an alternate filesystem? For example, I hear ReiserFS does very well with Maildir (though I haven't tried it on anything big).
~Kyle
Men think epilepsy divine, merely because they do not understand it. But if they called everything divine which they do not understand, why, there would be no end of divine things. -- Hippocrates
On Thu, 2007-07-19 at 07:50 -0600, Kyle Wheeler wrote:
Hmmm, so how many mails in one directory are we talking about?
~35k on biggest worst folders. The thing is that dovecot seems to not just read the contents once but for each select or something.
If Xen is the only thing you can't get out of, have you considered using an alternate filesystem? For example, I hear ReiserFS does very well with Maildir (though I haven't tried it on anything big).
I don't trust reiserfs at all, it has eaten my data repeatedly. It would also be quite hard to go away from ext3 anyway :)
johannes
participants (3)
-
Johannes Berg
-
Kyle Wheeler
-
Timo Sirainen