NFS and performances
Paul Kudla (SCOM.CA Internet Services Inc.)
paul at scom.ca
Wed Feb 15 14:56:10 UTC 2023
Good morning
I apologize in advance as you probably don't want to here this.
I have a replicated system and tried to use NFS to a file share server
with dedicated gigabit links etc and my second replicated system.
I have 300+ accounts and many have 20+ gig of data over 600+ folders,
your setup seems larger.
I spent 2 months trying to make this work reliably with nothing working out.
that being said (and this IS NOT a dovecot thing) NFS simply will not
work reliably especially in the environment that you seem to be using
I went to local SDRAM drives on the second server and have had zero
issues since.
NFS tweaks can be done and dovecot does try to support this but Linux
flavors (i use FreeBSD) all seem to handle NFS slightly differently thus
leading to the issues of timeouts, data not so much being dropped but
delayed between the NFS mount points.
NFS inherently on most systems runs a 30 second cache and file locking
for the mailboxes can usually is an issue.
Just easier to use hdd's on any local server.
NFS is good for tar backups etc though.
Happy Wednesday !!!
Thanks - paul
Paul Kudla
Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3
Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email paul at scom.ca
On 2023-02-15 9:25 a.m., tomate aceite wrote:
> Hello, i have some question about NFS, dovecot director, and imap settings.
>
> I was reading all dovecot documentation ad mail lists, but some aspect
> are not clear to me.
>
> I am looking for performance / tunning my infra to work in a more
> efficient way because we experiences some issues some days ago.
>
> This is my infra:
>
> I got an infra with 2 dovecot-directors and 3 imap backend.
> I got all the emails stored in a common NFS share filer to all the imap
> nodes. ( Index are locally stored in each imap node.)
>
>
> My NFS mount options:
>
> (0)#: nfsstat -m
>
> /data/mail from myipaddress:/export/mail/maildirs
> Flags:
> rw,nosuid,noexec,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.205,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.205
>
>
>
> *Questions*: ( https://wiki.dovecot.org/PerformanceTuning
> <https://wiki.dovecot.org/PerformanceTuning> ,
> https://doc.dovecot.org/configuration_manual/nfs/
> <https://doc.dovecot.org/configuration_manual/nfs/> >> i am following
> these steps )
>
> 1) Is my NFS correct setup with the mount options well optimized ? Not
> sure if someone is using the same flags that me or got a better
> recomendation to used.
>
> 2) Set *mmap_disable = yes ??? * >> This must be set to yes if you
> store indexes to shared filesystems. In my case i got them locally in
> each imap node not in NFS share folder.
>
> I got setup mmap_disable = no , is this correct? I think no is the
> correct option here with indexes locally.
>
> because i can read here:
>
> https://wiki1.dovecot.org/NFS <https://wiki1.dovecot.org/NFS>
>
>
> >> High performance NFS setup with indexes on local disk (see below
> for benefits):
>
> mmap_disable = no
>
>
>
> 3) Set*mail_fsync = always ???*
>
> Documentation: https://wiki.dovecot.org/PerformanceTuning
> <https://wiki.dovecot.org/PerformanceTuning>
>
> always
>
> Use fsync after all disk writes.
>
> Recommended for NFS to make sure there aren’t any delayed write()s.
>
>
> 3.a) where i can setup this option *mail_fsync = always , *because i
> run doveconf -n in director, and imap nodes, and they are not showing
> nothing.
>
> 3.b) *In which node ? *Do i need to add the setting in dovecot.conf in
> *director node or in imap node or in both ?* Not sure if this is the
> correct way:
>
> This is an attemp of setup, not sure if is correct?
>
> 0)#: doveconf -n
> # 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.13 (cdd19fe3)
> # OS: Linux 5.10.0-20-amd64 x86_64 Debian 11.6
>
> mail_debug = yes
> mail_fsync = always
> mail_max_userip_connections = 20
> mail_nfs_storage = yes
> mail_plugins = " notify mail_log"
> mail_privileged_group = mail
>
>
>
> protocol lmtp {
> mail_fsync = always
> mail_plugins = " notify mail_log sieve mail_lua push_notification
> push_notification_lua"
> plugin {
> ...
> }
>
>
>
> 4) Do not set *mail_nfs_index *or *mail_nfs_storage* (i.e. keep them as
> no) ?????
>
> First option make sense but the second one not.
>
> https://doc.dovecot.org/settings/core/#core_setting-mail_nfs_storage
> <https://doc.dovecot.org/settings/core/#core_setting-mail_nfs_storage>
>
> mail_nfs_storage
>
> Default: no
>
> Values: Boolean
>
> Flush NFS caches whenever it is necessary to do so.
>
>
> This setting should only be enabled if you are using multiple
> servers on NFS.
>
>
> So should be possible to enable this option *mail_nfs_storage = yes ?????*
>
>
>
>
> 4) I got this setting in *dovecot-sql.conf ( director ) *
>
> driver = mysql
> connect = host=myserver.X dbname=maildb user=dovecot
> password=XXXXXXXXXXXXXXXX
> default_pass_scheme = SHA
> password_query = select login as user, crypt_sha1 as password, home
> as userdb_home, uid AS userdb_uid, gid AS userdb_gid,
> concat(maildir,':INDEX=/data/indexes/',login) as userdb_mail from
> mailbox left join aliases on aliases.systemid = aliasid where login
> = '%u' and inactive = 0;
> user_query = select home, maildir as mail, uid, gid from mailbox
> left join aliases on aliases.systemid = aliasid where login = '%u'
> and inactive = 0;
> iterate_query = select distinct login as user from mailbox;
>
>
>
> I would like to implement * ITERINDEX* and probably *VOLATILEDIR *but
> not sure if this is the correct change or if i need to change something
> into the database.
>
> from:
>
> password_query = select login as user, crypt_sha1 as password, home
> as userdb_home, uid AS userdb_uid, gid AS userdb_gid,
> concat(maildir,':INDEX=/data/indexes/',login) as userdb_mail from
> mailbox left join aliases on aliases.systemid = aliasid where login
> = '%u' and inactive = 0;
>
>
> to:
>
> password_query = select login as user, crypt_sha1 as password, home
> as userdb_home, uid AS userdb_uid, gid AS userdb_gid,
> concat(maildir,':INDEX=/data/indexes/',login,':ITERINDEX',':VOLATILEDIR=/tmp/%2.256Nu/%u',login) as userdb_mail from mailbox left join aliases on aliases.systemid = aliasid where login = '%u' and inactive = 0;
>
>
>
> But not sure if is working correctly, *i checked debuging* in my imap
> node in this way:
>
> auth_verbose = yes
> auth_verbose_passwords = no
> auth_debug = yes
> auth_debug_passwords = yes
> mail_debug = yes
> verbose_ssl = yes
>
>
> This is the output log:
>
>
> Feb 15 09:32:53 my-imap-server dovecot: auth: Debug:
> sql(manolo1 at mydomain,10.X.X.109,<eWEh7bj09N4KAjxt>): cache hit:
> {SHA}8376922a27e83b9eadcdec3596a70bf6c4db5730#011user=manolo1 at mydomain#011userdb_home=/data/mail/tmydomain/users/manolo1#011userdb_uid=700#011userdb_gid=700#011userdb_mail=/data/mail/mydomain/users/manolo1/Maildir/:INDEX=/data/indexes/manolo1 at mydomain:ITERINDEX:VOLATILEDIR=/tmp/3a/manolo1 at mydomain*
> *
>
> *Not sure if is correct the output or should i need to change something
> into the database or this change only afect locally?*
>
>
> 5) *mailbox_list_index = yes ???*
>
> Mailbox list indexes can be used to optimize IMAP STATUS commands.
> They are
> also required for IMAP NOTIFY extension to be enabled.
>
> *
> It is this option necesary to add? in order to work with ITERINDEX
> option ?* https://www.mail-archive.com/dovecot@dovecot.org/msg75035.html
> <https://www.mail-archive.com/dovecot@dovecot.org/msg75035.html> ( i
> see your answer to this post )
>
> 6) *What is the best way to do a maintenance in a backend?*
>
> The last time i flushed all the user in a director from a backend node
> to do a maintenance, after the maintenance done i rebalanced the
> director cluster, but the next day
> when user started tp connect we got high peak I/O , etc with a lot of
> problems, because new users in this backend node were re-indexes o
> creating new indexes.
> *Are there any way to do in a safer way? *Without loosing the hashed for
> the directors nodes.
>
> Thanks in advance!!!!!!!
>
>
>
> --
> This message has been scanned for viruses and
> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
> believed to be clean.
More information about the dovecot
mailing list