[Dovecot] mail_fsync=never doesn't work?
Hello,
I've tried to set the above in Dovecot 2.0.3, but according to ktrace (FreeBSD) it still fsync()s a lot (pop3 processes for example). Is this switch useful at all?
On Mon, 2010-09-27 at 10:09 +0200, Attila Nagy wrote:
I've tried to set the above in Dovecot 2.0.3, but according to ktrace (FreeBSD) it still fsync()s a lot (pop3 processes for example). Is this switch useful at all?
I did a quick test and didn't see any fsyncs. Can you find out what file it's fsyncing? doveconf -n output could also be useful.
On 09/27/10 15:48, Timo Sirainen wrote:
On Mon, 2010-09-27 at 10:09 +0200, Attila Nagy wrote:
I've tried to set the above in Dovecot 2.0.3, but according to ktrace (FreeBSD) it still fsync()s a lot (pop3 processes for example). Is this switch useful at all? I did a quick test and didn't see any fsyncs. Can you find out what file it's fsyncing? doveconf -n output could also be useful. doveconf -n: # 2.0.4: /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 8.1-STABLE amd64 auth_cache_size = 104857600 auth_cache_ttl = 86400 s disable_plaintext_auth = no mail_fsync = never mail_gid = 999 mail_location = maildir:~/Maildir mail_plugins = $mail_plugins quota [...]
Should I set mail_fsync for only pop and imap instances? Currently I don't use Dovecot's lmtp or deliver, but will, so there I can't allow disabling fsync. I don't like setting this to never, but after a migration from courier, the machine couldn't handle the load with fsync, so this is a must.
The files:
69524 pop3 1285595867.816137 CALL
open(0x801004f08,O_RDWR|O_CREAT|O_EXCL,S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH)
69524 pop3 1285595867.816153 NAMI
"/home/hm01/16/12/user1/Maildir/temp.hm01.69524.579c9846c8ec2b26"
69524 pop3 1285595867.816238 RET open 9
[...]
69524 pop3 1285595867.816596 CALL fstat(0x9,0x7fffffffe4a0)
69524 pop3 1285595867.816608 STRU struct stat {dev=2188960719,
ino=263529548, mode=-rw------- , nlink=1, uid=999, gid=999, rdev=0,
atime=1285595867.815852103, stime=1285595867.815852103,
ctime=1285595867.815852103, birthtime=1285595867.815852103, size=0,
blksize=131072, blocks=1, flags=0x0 }
69524 pop3 1285595867.816620 RET fstat 0
[...]
69524 pop3 1285595867.822932 CALL fstat(0x9,0x7fffffffe570)
69524 pop3 1285595867.822955 STRU struct stat {dev=2188960719,
ino=263529548, mode=-rw------- , nlink=1, uid=999, gid=999, rdev=0,
atime=1285595867.815852103, stime=1285595867.815852103,
ctime=1285595867.815852103, birthtime=1285595867.815852103, size=0,
blksize=131072, blocks=1, flags=0x0 }
69524 pop3 1285595867.822974 RET fstat 0
[...]
69524 pop3 1285595867.823065 CALL write(0x9,0x8016d1200,0x5)
69524 pop3 1285595867.823116 GIO fd 9 wrote 5 bytes
"Trash"
69524 pop3 1285595867.823134 RET write 5
69524 pop3 1285595867.823148 CALL write(0x9,0x8006f5be3,0x1)
69524 pop3 1285595867.823173 GIO fd 9 wrote 1 byte
"
"
69524 pop3 1285595867.823186 RET write 1
69524 pop3 1285595867.823198 CALL write(0x9,0x8016d1206,0x6)
69524 pop3 1285595867.823220 GIO fd 9 wrote 6 bytes
"Drafts"
[...]
69524 pop3 1285595867.829999 CALL fsync(0x9)
69524 pop3 1285595868.119406 RET fsync 0
[...]
69524 pop3 1285595868.129605 CALL close(0x9)
69524 pop3 1285595868.129674 RET close 0
BTW, just curious, what is the rationale of the two fstats so close together? I've searched for some more examples and each of them were like this one. It seems that Dovecot copies the subscriptions file to this temporary file and then fsyncs it. And the question also arises: why? And more specifically: why in a pop3 server?
Thanks,
On Mon, 2010-09-27 at 16:31 +0200, Attila Nagy wrote:
doveconf -n: mail_plugins = $mail_plugins quota
One annoying problem about v2.0's doveconf -n output is that it hides when using multiple var=$var lines.. Did you have more plugins than just quota enabled?
Should I set mail_fsync for only pop and imap instances? Currently I don't use Dovecot's lmtp or deliver, but will, so there I can't allow disabling fsync. I don't like setting this to never, but after a migration from courier, the machine couldn't handle the load with fsync, so this is a must.
Yeah, either set it to only protocol imap/pop3 {} or alternatively put mail_fsync=no to protocol lda/lmtp {}.
BTW, just curious, what is the rationale of the two fstats so close together?
It's because of how the code works internally. It would be too much trouble to avoid the second call and would make the code uglier. fstat() is cheap anyway.
I've searched for some more examples and each of them were like this one. It seems that Dovecot copies the subscriptions file to this temporary file and then fsyncs it. And the question also arises: why?
Fixed: http://hg.dovecot.org/dovecot-2.0/rev/4959db811d29
Also http://hg.dovecot.org/dovecot-2.0/rev/432208994270 avoids multiple write() syscalls. Although it still does writes to temp file even when it's not necessary, but that's again annoyingly too much trouble to prevent..
And more specifically: why in a pop3 server?
Do you have autocreate plugin enabled with autosubscribe plugin settings? You didn't show your full doveconf output so I have to guess..
On 09/27/2010 04:52 PM, Timo Sirainen wrote:
On Mon, 2010-09-27 at 16:31 +0200, Attila Nagy wrote:
doveconf -n: mail_plugins = $mail_plugins quota One annoying problem about v2.0's doveconf -n output is that it hides when using multiple var=$var lines.. Did you have more plugins than just quota enabled? Yes: ./20-lmtp.conf: mail_plugins = $mail_plugins mail_log notify ./10-mail.conf:mail_plugins = $mail_plugins quota ./20-pop3.conf: mail_plugins = $mail_plugins autocreate ./20-imap.conf: mail_plugins = $mail_plugins imap_quota autocreate
Should I set mail_fsync for only pop and imap instances? Currently I don't use Dovecot's lmtp or deliver, but will, so there I can't allow disabling fsync. I don't like setting this to never, but after a migration from courier, the machine couldn't handle the load with fsync, so this is a must. Yeah, either set it to only protocol imap/pop3 {} or alternatively put mail_fsync=no to protocol lda/lmtp {}. That's what I want to do, when I'll start to use LMTP. BTW, just curious, what is the rationale of the two fstats so close together? It's because of how the code works internally. It would be too much trouble to avoid the second call and would make the code uglier. fstat() is cheap anyway. Yes, 42 or 24 us according to the trace. I've searched for some more examples and each of them were like this one. It seems that Dovecot copies the subscriptions file to this temporary file and then fsyncs it. And the question also arises: why? Fixed: http://hg.dovecot.org/dovecot-2.0/rev/4959db811d29 Thanks! But shouldn't "Avoid fsyncing subscriptions file when it doesn't change or if mail_fsync=never." be "Avoid re-writing subscriptions file when it doesn't change and fsyncing it if mail_fsync=never."? I mean (I haven't read the code context) if you know that it does not change, why write?
Also http://hg.dovecot.org/dovecot-2.0/rev/432208994270 avoids multiple write() syscalls. Although it still does writes to temp file even when it's not necessary, but that's again annoyingly too much trouble to prevent.. Hmm. It may be trouble, but inspecting a user's pop3 login (no fsync now) starting with the first appearance of subscription.lock, ending with the last I see more weird stuff: 62900 pop3 1285613658.763692 NAMI
"/home/hm01/user2/Maildir/subscriptions.lock" 62900 pop3 1285613658.763777 RET lstat -1 errno 2 No such file or directory [...] 62900 pop3 1285613658.783839 CALL unlink(0x8010b0a40) 62900 pop3 1285613658.783865 NAMI
"/home/hm01/user2/Maildir/subscriptions.lock" 62900 pop3 1285613658.784045 RET unlink 0
I see four(!) rewrites of the subscriptions file between the two. And also there is 20 ms in between, during that nothing happened, just creating a temporary file, linking it to subscriptions.lock, reading subscriptions, writing its conents to the temporary file, then deleting both the temporary file and subscriptions.lock and this is four times. (BTW, this is 20ms when virtually nobody uses the server, during the peak I think it will be more) Do you see the same, or is this just my config? Even one unnecessary read&write (which makes a lot more file system operations, so stresses the kernel in case of a lot of logins) can be significant with a thousand of logins per second, not talking about four. :)
And more specifically: why in a pop3 server? Do you have autocreate plugin enabled with autosubscribe plugin settings? You didn't show your full doveconf output so I have to guess..
Sorry. Yes. Here's the full config: # 2.0.4: /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 8.1-STABLE amd64 auth_cache_size = 104857600 auth_cache_ttl = 86400 s disable_plaintext_auth = no mail_fsync = never mail_gid = 999 mail_location = maildir:~/Maildir mail_plugins = $mail_plugins quota mail_uid = 999 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date passdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { autocreate = INBOX.Trash autocreate2 = INBOX.Drafts autocreate3 = INBOX.Sent autocreate4 = INBOX.Spam autosubscribe = INBOX.Trash autosubscribe2 = INBOX.Drafts autosubscribe3 = INBOX.Sent autosubscribe4 = INBOX.Spam mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change save mailbox_create mail_log_fields = uid box msgid size flags vsize from subject quota = maildir:User quota } protocols = pop3 imap lmtp service auth { unix_listener auth-userdb { mode = 0600 user = qmailldap } } service imap-login { process_min_avail = 16 service_count = 0 } service imap { client_limit = 8 process_min_avail = 16 service_count = 0 } service lmtp { inet_listener lmtp { port = 24 } user = qmailldap } service pop3-login { process_min_avail = 16 service_count = 0 } service pop3 { client_limit = 8 process_min_avail = 16 service_count = 0 } ssl = no userdb { driver = prefetch } userdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol lmtp { mail_plugins = $mail_plugins mail_log notify } protocol imap { mail_max_userip_connections = 1024 mail_plugins = $mail_plugins imap_quota autocreate } protocol pop3 { mail_max_userip_connections = 1024 mail_plugins = $mail_plugins autocreate }
Thanks for the high quality support.
On 27.9.2010, at 20.13, Attila Nagy wrote:
I've searched for some more examples and each of them were like this one. It seems that Dovecot copies the subscriptions file to this temporary file and then fsyncs it. And the question also arises: why? Fixed: http://hg.dovecot.org/dovecot-2.0/rev/4959db811d29 Thanks! But shouldn't "Avoid fsyncing subscriptions file when it doesn't change or if mail_fsync=never." be "Avoid re-writing subscriptions file when it doesn't change and fsyncing it if mail_fsync=never."? I mean (I haven't read the code context) if you know that it does not change, why write?
Ah, but I don't know :) The possibilities are:
a) Open a temp file and start writing new subscriptions to it. If we notice that the subscription is there already, abort and delete the temp file. This is how it works now.
b) Read the subscriptions to find out if it exists there, and if it doesn't then re-read it and start writing. Seems a bit wasteful.
c) Same as b) but read it into memory to avoid re-reading (possibly up to n kB to avoid wasting too much memory).
And since normally clients don't issue SUBSCRIBE commands unnecessarily, b) and c) seem like a lot of extra trouble. The only problem here is autocreate plugin which sucks in any case. Better to just fix that in v2.1.
Also http://hg.dovecot.org/dovecot-2.0/rev/432208994270 avoids multiple write() syscalls. Although it still does writes to temp file even when it's not necessary, but that's again annoyingly too much trouble to prevent.. Hmm. It may be trouble, but inspecting a user's pop3 login
Just disable autocreate plugin for pop3:
protocol !pop3 { mail_plugins = $mail_plugins autocreate }
(no fsync now) starting with the first appearance of subscription.lock, ending with the last I see more weird stuff: 62900 pop3 1285613658.763692 NAMI "/home/hm01/user2/Maildir/subscriptions.lock" 62900 pop3 1285613658.763777 RET lstat -1 errno 2 No such file or directory [...] 62900 pop3 1285613658.783839 CALL unlink(0x8010b0a40) 62900 pop3 1285613658.783865 NAMI "/home/hm01/user2/Maildir/subscriptions.lock" 62900 pop3 1285613658.784045 RET unlink 0
I see four(!) rewrites of the subscriptions file between the two.
That's because you have 4 autosubscribe entries. The current code doesn't allow doing multiple subscription changes at once. But anyway, again, the problem here is the autocreate plugin. I hated even adding it to included plugins, but too many people just wanted it.
(v2.1 way of handling it would be to have kind of virtual mailbox names that are always listed, but actually physically created only when it's opened the first time. That way you could also change them and if user has never actually accessed a removed mailbox it would automatically also disappear.)
On 09/27/10 23:26, Timo Sirainen wrote:
Thanks! But shouldn't "Avoid fsyncing subscriptions file when it doesn't change or if mail_fsync=never." be "Avoid re-writing subscriptions file when it doesn't change and fsyncing it if mail_fsync=never."? I mean (I haven't read the code context) if you know that it does not change, why write? Ah, but I don't know :) The possibilities are:
a) Open a temp file and start writing new subscriptions to it. If we notice that the subscription is there already, abort and delete the temp file. This is how it works now. I see, thanks. That's because you have 4 autosubscribe entries. The current code doesn't allow doing multiple subscription changes at once. But anyway, again, the problem here is the autocreate plugin. I hated even adding it to included plugins, but too many people just wanted it. Sadly, I'm one of them. Currently during the provisioning it's not guaranteed that a user will have a mailbox in the file system, so autocreating it during pop logins is a bless. :) But I will keep this in mind and will remove it as soon as I can. (v2.1 way of handling it would be to have kind of virtual mailbox names that are always listed, but actually physically created only when it's opened the first time. That way you could also change them and if user has never actually accessed a removed mailbox it would automatically also disappear.) Sounds good.
On Tue, 2010-09-28 at 08:29 +0200, Attila Nagy wrote:
Sadly, I'm one of them. Currently during the provisioning it's not guaranteed that a user will have a mailbox in the file system, so autocreating it during pop logins is a bless. :)
But POP3 doesn't use anything except INBOX, so why do you need to autocreate them there? I'd think it's useful only with IMAP.
On 09/28/2010 05:09 PM, Timo Sirainen wrote:
On Tue, 2010-09-28 at 08:29 +0200, Attila Nagy wrote:
Sadly, I'm one of them. Currently during the provisioning it's not guaranteed that a user will have a mailbox in the file system, so autocreating it during pop logins is a bless. :) But POP3 doesn't use anything except INBOX, so why do you need to autocreate them there? I'd think it's useful only with IMAP.
You mean the four folders, or the inbox itself? For the inbox itself: if there is no such directory, the login will fail and the user will look for errors. For the other folders: we have a crappy webmail, which messes directly with the maildir, and that also doesn't autocreate the mailboxes, so if the user only logs in via pop and webmail, and I only create inbox, he will miss trash, drafts and sent...
Weird, I know. :)
On Tue, 2010-09-28 at 19:03 +0200, Attila Nagy wrote:
On 09/28/2010 05:09 PM, Timo Sirainen wrote:
On Tue, 2010-09-28 at 08:29 +0200, Attila Nagy wrote:
Sadly, I'm one of them. Currently during the provisioning it's not guaranteed that a user will have a mailbox in the file system, so autocreating it during pop logins is a bless. :) But POP3 doesn't use anything except INBOX, so why do you need to autocreate them there? I'd think it's useful only with IMAP.
You mean the four folders, or the inbox itself? For the inbox itself: if there is no such directory, the login will fail and the user will look for errors.
INBOX is always auto-created. In earlier versions there were a couple of bugs that caused it to fail though.
For the other folders: we have a crappy webmail, which messes directly with the maildir, and that also doesn't autocreate the mailboxes, so if the user only logs in via pop and webmail, and I only create inbox, he will miss trash, drafts and sent...
Weird, I know. :)
Oh well. That also means then that my future autocreate plugin won't work with your webmail, because it doesn't physically create the mailboxes.
On 2010-09-28 1:06 PM, Timo Sirainen wrote:
Oh well. That also means then that my future autocreate plugin won't work with your webmail, because it doesn't physically create the mailboxes.
This should be handled by the user management system anyway.
Just auto-create the required folders when the user is created. Scripting such a process should be relatively easy, even for someone like me (pitiful scripting skills)...
--
Best regards,
Charles
On 09/28/2010 07:06 PM, Timo Sirainen wrote:
On Tue, 2010-09-28 at 19:03 +0200, Attila Nagy wrote:
On 09/28/2010 05:09 PM, Timo Sirainen wrote:
On Tue, 2010-09-28 at 08:29 +0200, Attila Nagy wrote:
Sadly, I'm one of them. Currently during the provisioning it's not guaranteed that a user will have a mailbox in the file system, so autocreating it during pop logins is a bless. :) But POP3 doesn't use anything except INBOX, so why do you need to autocreate them there? I'd think it's useful only with IMAP.
You mean the four folders, or the inbox itself? For the inbox itself: if there is no such directory, the login will fail and the user will look for errors. INBOX is always auto-created. In earlier versions there were a couple of bugs that caused it to fail though. I didn't know that, but see later... For the other folders: we have a crappy webmail, which messes directly with the maildir, and that also doesn't autocreate the mailboxes, so if the user only logs in via pop and webmail, and I only create inbox, he will miss trash, drafts and sent...
Weird, I know. :) Oh well. That also means then that my future autocreate plugin won't work with your webmail, because it doesn't physically create the mailboxes.
I hope we can switch to an IMAP based webmail until Dovecot 2.1 is released. :) And I would be more than happy to ditch maildir too for a distributed, replicated, highly available storage.
participants (3)
-
Attila Nagy
-
Charles Marcus
-
Timo Sirainen