[Dovecot] getdents() with 4KB buffer - seems slow (Maildir, large inbox)
Hi,
I have a very large inbox (~146K mails) in Maildir format and dovecot seems to spend a lot of time rescanning the directory, especially when the server is loaded. I'm not sure whether this is triggered by Thunderbird or done regularly, but it takes longer when the server is loaded, so sometimes it seems that it is scanning continuously. Since it takes around 2000 getdents64() syscalls to scan my inbox, I'm wondering whether the buffer size is a bit small with 4KB and might benefit from being configurable:
... getdents64(8, /* 73 entries */, 4096) = 4088 getdents64(8, /* 73 entries */, 4096) = 4088 getdents64(8, /* 73 entries */, 4096) = 4088 getdents64(8, /* 73 entries */, 4096) = 4088 getdents64(8, /* 73 entries */, 4096) = 4088 ...
(strace output of an imap process belonging to me)
The behaviour is identical on 1.0.15 and 1.2.13 (I just upgraded).
Regards, Marinos
PS. I know that the main reason for slowness is my large inbox and the server being loaded (disk I/O), but still, so many syscalls just to rescan the inbox seems wasteful.
# dovecot -n # 1.2.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-6-xen-686 i686 Debian 4.0 log_timestamp: %Y-%m-%d %H:%M:%S ssl_listen: * ssl_cert_file: XXXXX (censored) ssl_key_file: XXXXX login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login login_process_size: 32 login_processes_count: 8 verbose_proctitle: yes mail_privileged_group: mail mail_location: maildir:~/Maildir mbox_write_locks: fcntl dotlock mail_process_size: 2048 imap_client_workarounds: outlook-idle auth default: passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix
On Mon, 2010-08-30 at 17:38 +0200, Marinos Yannikos wrote:
I have a very large inbox (~146K mails) in Maildir format and dovecot seems to spend a lot of time rescanning the directory, especially when the server is loaded. I'm not sure whether this is triggered by Thunderbird or done regularly, but it takes longer when the server is loaded, so sometimes it seems that it is scanning continuously.
You could see if maildir_very_dirty_syncs=yes is helpful.
Since it takes around 2000 getdents64() syscalls to scan my inbox, I'm wondering whether the buffer size is a bit small with 4KB and might benefit from being configurable:
getdents64() is an internal kernel function and shouldn't be called directly (Dovecot uses readdir(), which is implemented with getdents64()). Also I doubt the buffer size is configurable and it probably wouldn't even help if a larger one was used, 2000 isn't yet that many syscalls.
participants (2)
-
Marinos Yannikos
-
Timo Sirainen