[Dovecot] dovecot and nfs readdir vs readdirplus operations
Hello,
We are having performance problems trying to migrate our pop/imap servers to a new version. Our old servers are 4 debian lenny with 5GB of RAM running of XenServer VMs with kernel 2.6.32-4-amd64 and dovecot 1.1.16. New servers are 4 ubuntu 12.04 with dovecot 2.1.5 running on vmware vm with 6 cores and 16GB of RAM and kernel 3.2.0-24-generic. On both server we are using nfs 3 with same configuration (regardless of internal kernel differences, but we have no customized any of them, we are using vanilla kernels with default configurations).
The problem we have is that new servers have performance problems. Even when have a small part of our total users (about 25%) directed to the new farm, performance is very poor, even useless.
Looking for NFS problems, we have found a lot of differences in nfs operations. For example, this is the nfsstat of one of a new servers at this moment:
myotis21:~# nfsstat Client rpc stats: calls retrans authrefrsh 414528349 885 37
Client nfs v3: null getattr setattr lookup access readlink 0 0% 95673837 23% 3961938 0% 89586364 21% 110097351 26% 2930961 0% read write create mkdir symlink mknod
20009850 4% 6065319 1% 3757720 0% 1557 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 6378134 1% 281 0% 2602358 0% 555097 0% 53126619 12% 15615402 3% fsstat fsinfo pathconf commit 113256 0% 26152 0% 0 0% 4026151 0%
and this is the same on one of the new ones:amateo_adm@myotis31:~$ nfsstat Server rpc stats: calls badcalls badclnt badauth xdrcall 0 0 0 0 0
Client rpc stats: calls retrans authrefrsh 178040318 675 178040800
Client nfs v3: null getattr setattr lookup access readlink 0 0% 24350345 13% 5045924 2% 10939469 6% 30185146 16% 142865 0% read write create mkdir symlink mknod
8818016 4% 6058614 3% 2877653 1% 420 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 2842562 1% 69 0% 2961239 1% 634038 0% 0 0% 82921863 46% fsstat fsinfo pathconf commit 70861 0% 18754 0% 9377 0% 152702 0%
Although nfs configuration is the same, there are a lot of differences on readdir vs readdirplus nfs operations. In fact, in the old one we have 12% readdir operations and 3% of readdirplus. And in the new one we have 46% of readdirplus and no readdir operations.
Although readdirplus is supposed to be an optimization in nfs3, in situations when you have big directories and uses just a few entries of these directories it could be worse. So we think if this could be the problem (or one of them).
Any idea of this difference? And if this difference could be significative?PS: I have attached doveconf -n of the new server.
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
On 4.7.2012, at 21.49, Angel L. Mateo wrote:
Although nfs configuration is the same, there are a lot of differences on readdir vs readdirplus nfs operations. In fact, in the old one we have 12% readdir operations and 3% of readdirplus. And in the new one we have 46% of readdirplus and no readdir operations.
I'm not entirely sure, but I think it's the kernel that decides if readdir or readdirplus is used and Dovecot can't affect that decision. (Unless maybe kernel does some heuristics.)
PS: I have attached doveconf -n of the new server.
At least this reduces performance:
mail_nfs_storage = yes
Also maildir_very_dirty_syncs=yes improves performance by reducing readdirs. It's safe to use as long as only Dovecot is reading the Maildir.
El 04/07/12 23:55, Timo Sirainen escribió:
Also maildir_very_dirty_syncs=yes improves performance by reducing readdirs. It's safe to use as long as only Dovecot is reading the Maildir.
Is it safe to use it although a user could have open simultaneous sessions in different servers?
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
On 5.7.2012, at 8.44, Angel L. Mateo wrote:
El 04/07/12 23:55, Timo Sirainen escribió:
Also maildir_very_dirty_syncs=yes improves performance by reducing readdirs. It's safe to use as long as only Dovecot is reading the Maildir.
Is it safe to use it although a user could have open simultaneous sessions in different servers?
Each server has local indexes? Then it's not very optimal.. but it doesn't break anything badly, Dovecot just might not immediately notice when another server has done changes.
El 05/07/12 07:49, Timo Sirainen escribió:
On 5.7.2012, at 8.44, Angel L. Mateo wrote:
El 04/07/12 23:55, Timo Sirainen escribió:
Also maildir_very_dirty_syncs=yes improves performance by reducing readdirs. It's safe to use as long as only Dovecot is reading the Maildir.
Is it safe to use it although a user could have open simultaneous sessions in different servers?
Each server has local indexes? Then it's not very optimal.. but it doesn't break anything badly, Dovecot just might not immediately notice when another server has done changes.
Yes, indexes are local.-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
El 04/07/12 23:55, Timo Sirainen escribió:
On 4.7.2012, at 21.49, Angel L. Mateo wrote:
Although nfs configuration is the same, there are a lot of differences on readdir vs readdirplus nfs operations. In fact, in the old one we have 12% readdir operations and 3% of readdirplus. And in the new one we have 46% of readdirplus and no readdir operations.
I'm not entirely sure, but I think it's the kernel that decides if readdir or readdirplus is used and Dovecot can't affect that decision. (Unless maybe kernel does some heuristics.)
I think you are right. I guess that the differnce between my old system (who uses readdir and readdirplus) and the new one (which just uses readdirplus) is because of changes on the kernel. For some threads I have read (for example, http://www.spinics.net/lists/linux-nfs/msg19957.html) it seems that from some version in kernel source (2.6.36 if I remmeber well) some nfs improvements related with readdir and readdirplus (that makes nfs client only uses readdirplus when it was better) and recents kernels always use readdirplus.
Concluding... in my systems, with users with hugh mail folders (thousands of mails) in maildir format, disabling rdirplus with mount options (mount option nordirplus) increases performance (maybe we could do more precise test and this could be documented in dovecot's nfs documentation)
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
On 5.7.2012, at 15.55, Angel L. Mateo wrote:
I think you are right. I guess that the differnce between my old system (who uses readdir and readdirplus) and the new one (which just uses readdirplus) is because of changes on the kernel. For some threads I have read (for example, http://www.spinics.net/lists/linux-nfs/msg19957.html) it seems that from some version in kernel source (2.6.36 if I remmeber well) some nfs improvements related with readdir and readdirplus (that makes nfs client only uses readdirplus when it was better) and recents kernels always use readdirplus.
Concluding... in my systems, with users with hugh mail folders (thousands of mails) in maildir format, disabling rdirplus with mount options (mount option nordirplus) increases performance (maybe we could do more precise test and this could be documented in dovecot's nfs documentation)
If readdirplus requires filesystem to do stat()s on each file, then that's definitely bad and not something Dovecot needs. The only use is when listing files in ~/Maildir/ and determining which ones are directories and which ones are files. But even that defaults to assuming that everything beginning with "." is a directory if the readdir() call didn't specify it, and in those systems where this isn't true you can set maildir_stat_dirs=yes setting.
this is small folders. From my practice average are 10000-20000 mails. Huge are over 150000.
did tests with artifically created million mail folder.
except i needed to rise vsize limit of dovecot process in config , and except first index creation it works smooth with dovecot taking little CPU and (after other programs generated some VM pressure) very little RAM, with minimal pagein.
roundcube webmail worked smooth including search, thunderbird was extremely slow, imap itself wasn't generating considerable load.
On 5.7.2012, at 15.55, Angel L. Mateo wrote:
Concluding... in my systems, with users with hugh mail folders (thousands of mails) in maildir format, disabling rdirplus with mount options (mount option nordirplus) increases performance (maybe we could do more precise test and this could be documented in dovecot's nfs documentation)
BTW. What NFS server are you using? The problem is mainly on the server side, so would be useful to know which NFS server implementations (if any?) can handle readdirplus without such a huge performance loss. Mainly I'm wondering about NetApp, since that's what most large Dovecot NFS installations use.
Timo,
For Netapp NFS mailstorage, what options can use for mount point nfs? for better perfomance with huge maildir folders? (most of my customers use pop3 protocol instead imap) Any kernel customize? All of my mta servers is Centos 6
Regards
El 05-07-2012 10:46, Timo Sirainen escribió:
On 5.7.2012, at 15.55, Angel L. Mateo wrote:
Concluding... in my systems, with users with hugh mail folders (thousands of mails) in maildir format, disabling rdirplus with mount options (mount option nordirplus) increases performance (maybe we could do more precise test and this could be documented in dovecot's nfs documentation) BTW. What NFS server are you using? The problem is mainly on the server side, so would be useful to know which NFS server implementations (if any?) can handle readdirplus without such a huge performance loss. Mainly I'm wondering about NetApp, since that's what most large Dovecot NFS installations use.
-- Mauricio López Riffo Red Hat Certified Engineer 804006455319519 Administrador de Servicios Internet Área Ingeniería Gtd Internet S.A. http://www.grupogtd.com/ Moneda 920, Oficina 602 - Fono : +562 4139742
I don't think there are NFS special options on NetApp filers but you can use ethernet jumbo frames to boost network access between your server and the filer.
You also can use iSCSI between server and filer but it will waste a lot of disk space on the Ontap volume.
I use NFS between a FreeBSD server and a 3210 netapp I use MBOX format , inboxes are living on the local raid array of the server and IMAP folders are located on a NetApp dedicated volume NFS mounted.
The netapp filler has an amazing capacity to support very high load due to special cache hardware (nvram) we have only one 3210 that support all users's data + softwares of our users (2000) it never failed on slowdown accesses.
Le 05/07/2012 17:11, Mauricio López Riffo a écrit :
Timo,
For Netapp NFS mailstorage, what options can use for mount pointnfs? for better perfomance with huge maildir folders? (most of my customers use pop3 protocol instead imap) Any kernel customize? All of my mta servers is Centos 6
Regards
El 05-07-2012 10:46, Timo Sirainen escribió:
On 5.7.2012, at 15.55, Angel L. Mateo wrote:
Concluding... in my systems, with users with hugh mail folders(thousands of mails) in maildir format, disabling rdirplus with mount options (mount option nordirplus) increases performance (maybe we could do more precise test and this could be documented in dovecot's nfs documentation)
BTW. What NFS server are you using? The problem is mainly on the server side, so would be useful to know which NFS server implementations (if any?) can handle readdirplus without such a huge performance loss. Mainly I'm wondering about NetApp, since that's what most large Dovecot NFS installations use.
El 05/07/12 16:46, Timo Sirainen escribió:
On 5.7.2012, at 15.55, Angel L. Mateo wrote:
Concluding... in my systems, with users with hugh mail folders (thousands of mails) in maildir format, disabling rdirplus with mount options (mount option nordirplus) increases performance (maybe we could do more precise test and this could be documented in dovecot's nfs documentation)
BTW. What NFS server are you using? The problem is mainly on the server side, so would be useful to know which NFS server implementations (if any?) can handle readdirplus without such a huge performance loss. Mainly I'm wondering about NetApp, since that's what most large Dovecot NFS installations use.
We are using a celerra from EMC, conected to an EMC SAN.-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337
participants (5)
-
Angel L. Mateo
-
Frank Bonnet
-
Mauricio López Riffo
-
Timo Sirainen
-
Wojciech Puchar