[Dovecot] mail_fsync=never doesn't work?

[Attila Nagy]

  On 09/27/2010 04:52 PM, Timo Sirainen wrote:
> On Mon, 2010-09-27 at 16:31 +0200, Attila Nagy wrote:
>> doveconf -n:
>> mail_plugins = $mail_plugins quota
> One annoying problem about v2.0's doveconf -n output is that it hides
> when using multiple var=$var lines.. Did you have more plugins than just
> quota enabled?
Yes:
./20-lmtp.conf:  mail_plugins = $mail_plugins mail_log notify
./10-mail.conf:mail_plugins = $mail_plugins quota
./20-pop3.conf:  mail_plugins = $mail_plugins autocreate
./20-imap.conf:  mail_plugins = $mail_plugins imap_quota autocreate

>> Should I set mail_fsync for only pop and imap instances? Currently I
>> don't use Dovecot's lmtp or deliver, but will, so there I can't allow
>> disabling fsync. I don't like setting this to never, but after a
>> migration from courier, the machine couldn't handle the load with fsync,
>> so this is a must.
> Yeah, either set it to only protocol imap/pop3 {} or alternatively put
> mail_fsync=no to protocol lda/lmtp {}.
That's what I want to do, when I'll start to use LMTP.
>> BTW, just curious, what is the rationale of the two fstats so close
>> together?
> It's because of how the code works internally. It would be too much
> trouble to avoid the second call and would make the code uglier. fstat()
> is cheap anyway.
Yes, 42 or 24 us according to the trace.
>> I've searched for some more examples and each of them were like this
>> one. It seems that Dovecot copies the subscriptions file to this
>> temporary file and then fsyncs it. And the question also arises: why?
> Fixed: http://hg.dovecot.org/dovecot-2.0/rev/4959db811d29
Thanks! But shouldn't "Avoid fsyncing subscriptions file when it doesn't 
change or if mail_fsync=never." be "Avoid re-writing subscriptions file 
when it doesn't change and fsyncing it if mail_fsync=never."?
I mean (I haven't read the code context) if you know that it does not 
change, why write?

> Also http://hg.dovecot.org/dovecot-2.0/rev/432208994270 avoids multiple
> write() syscalls. Although it still does writes to temp file even when
> it's not necessary, but that's again annoyingly too much trouble to
> prevent..
Hmm. It may be trouble, but inspecting a user's pop3 login (no fsync 
now) starting with the first appearance of subscription.lock, ending 
with the last I see more weird stuff:
  62900 pop3     1285613658.763692 NAMI  
"/home/hm01/user2/Maildir/subscriptions.lock"
  62900 pop3     1285613658.763777 RET   lstat -1 errno 2 No such file 
or directory
[...]
  62900 pop3     1285613658.783839 CALL  unlink(0x8010b0a40)
  62900 pop3     1285613658.783865 NAMI  
"/home/hm01/user2/Maildir/subscriptions.lock"
  62900 pop3     1285613658.784045 RET   unlink 0

I see four(!) rewrites of the subscriptions file between the two. And 
also there is 20 ms in between, during that nothing happened, just 
creating a temporary file, linking it to subscriptions.lock, reading 
subscriptions, writing its conents to the temporary file, then deleting 
both the temporary file and subscriptions.lock and this is four times. 
(BTW, this is 20ms when virtually nobody uses the server, during the 
peak I think it will be more)
Do you see the same, or is this just my config?
Even one unnecessary read&write (which makes a lot more file system 
operations, so stresses the kernel in case of a lot of logins) can be 
significant with a thousand of logins per second, not talking about four. :)

>> And more specifically: why in a pop3 server?
> Do you have autocreate plugin enabled with autosubscribe plugin
> settings? You didn't show your full doveconf output so I have to guess..
>
Sorry. Yes. Here's the full config:
# 2.0.4: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 8.1-STABLE amd64
auth_cache_size = 104857600
auth_cache_ttl = 86400 s
disable_plaintext_auth = no
mail_fsync = never
mail_gid = 999
mail_location = maildir:~/Maildir
mail_plugins = $mail_plugins quota
mail_uid = 999
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date
passdb {
   args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
   driver = ldap
}
plugin {
   autocreate = INBOX.Trash
   autocreate2 = INBOX.Drafts
   autocreate3 = INBOX.Sent
   autocreate4 = INBOX.Spam
   autosubscribe = INBOX.Trash
   autosubscribe2 = INBOX.Drafts
   autosubscribe3 = INBOX.Sent
   autosubscribe4 = INBOX.Spam
   mail_log_events = delete undelete expunge copy mailbox_delete 
mailbox_rename flag_change save mailbox_create
   mail_log_fields = uid box msgid size flags vsize from subject
   quota = maildir:User quota
}
protocols = pop3 imap lmtp
service auth {
   unix_listener auth-userdb {
     mode = 0600
     user = qmailldap
   }
}
service imap-login {
   process_min_avail = 16
   service_count = 0
}
service imap {
   client_limit = 8
   process_min_avail = 16
   service_count = 0
}
service lmtp {
   inet_listener lmtp {
     port = 24
   }
   user = qmailldap
}
service pop3-login {
   process_min_avail = 16
   service_count = 0
}
service pop3 {
   client_limit = 8
   process_min_avail = 16
   service_count = 0
}
ssl = no
userdb {
   driver = prefetch
}
userdb {
   args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
   driver = ldap
}
protocol lmtp {
   mail_plugins = $mail_plugins mail_log notify
}
protocol imap {
   mail_max_userip_connections = 1024
   mail_plugins = $mail_plugins imap_quota autocreate
}
protocol pop3 {
   mail_max_userip_connections = 1024
   mail_plugins = $mail_plugins autocreate
}


Thanks for the high quality support.