[Dovecot] Dovecot extremely slow!
Please help,
Dovecot is running extremely slow for the last couple of weeks and it seems to be getting worse (or my patience running short).
I attach the 10-master configuration and the log file after running strace according to: http://wiki.dovecot.org/Debugging/ProcessTracing
I can click on an email and wait for a minute or more before receiving a connection dropped or no error at all. I use many clients (thunderbird, windows 8 mail, maildroid for android, squirrelmail) and they all have similar behavior. It happens both in the inbox and on imap subfolders. Sometimes it helps changing subfolders back and forth.
I have many imap folders organized in up to 3 levels of subfolders and use postfix for delivery.
Let me know any other info you require. Thanks!
Patricio--
Computerisms
Bob Miller
867-334-7117 / 867-633-3760
http://computerisms.ca
On Wed, 2013-09-25 at 16:15 -0600, LuKreme wrote:
On 25 Sep 2013, at 16:05 , Patricio Rojo <pato@oan.cl> wrote:
I attach the 10-master configuration
That’s not that useful.
doveconf -n is useful
As are the server logs, as opposed to the strace output...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 25 Sep 2013, Patricio Rojo wrote:
I can click on an email and wait for a minute or more before receiving a connection dropped or no error at all. I use many clients (thunderbird, windows 8 mail, maildroid for android, squirrelmail) and they all have similar behavior. It happens both in the inbox and on imap subfolders. Sometimes it helps changing subfolders back and forth.
I have many imap folders organized in up to 3 levels of subfolders and use postfix for delivery.
What about I/O load on the server? Something in kernel log? Do you use FTS? Do you get many messages at once?
Then, as Lukreme and Bob already said, provide doveconf -n and check out Dovecot logs, for "Error" and "Warning".
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iQEVAwUBUkP1Ol3r2wJMiz2NAQJ+ewf/ZcGWyYnAx50iZZ8/jkO9c9BU5WmRmMA3 AaBx8fM8IrSXnWtCUY+WcaIvn2wl9MnCFQn2Onigqv52wwdUXppuBKBqKlPKRl0b MF8MkqUh1hrM8gIqBNHNiMWhGJXKcMRF5+fk2JtgFtDzew5x2bvsd+g1WlAf5cPo 8W5gsEP8wfpYxNgsnMW4yzokJdoXUa9laqUKgOqArtXVsbRE/sJ5Kh8c71tj+YY3 J4G5aenCxunjjs6caJbKN4YuvLptI2vSw2WhAc2c5WnVtXvRsTARsAlsQAJo+kLU +aDTbaW1ChldCHzUkRSBEEH5cU8ij3yD2p0TRaYMdakeNxaf8MdYfw== =zrAV -----END PGP SIGNATURE-----
On 9/25/2013 5:05 PM, Patricio Rojo wrote:
Please help,
Dovecot is running extremely slow for the last couple of weeks and it seems to be getting worse (or my patience running short).
Progressive degradation of mail server performance, whether an IMAP mailbox server, or an MTA, is almost always caused by the storage subsystem, usually due to filesystem free space fragmentation.
If you have a parity RAID array, have lost a disk and are running in degraded mode, this can also cause large IO latency, slowing Dovecot.
Another common cause is heavy swap usage. If you have a runaway process or one with a memory leak, this will eat up physical RAM, causing heavy swap usage. If swap resides on the same spindles as your mailboxes, this will degrade Dovecot performance.
If your box is hosted at a colo facility, or is in fact a VPS, it's always possible a network problem or a clogged shared segment at the provider is causing packet loss, which can also cause the client delay behavior you have described. If this server resides behind consumer ADSL there could be a problem with your DSL provider's network.
In other words, if you didn't change the Dovecot configuration on the day the performance first dropped, or very shortly before, then the performance problem has nothing to do with Dovecot. And this is almost always the case with performance degradation. The source of the problem lie outside Dovecot, again, usually in the storage stack. Start your troubleshooting there.
-- Stan
Thanks all for the quick and knowledgeable replies!
More details on my system: the imap folders, but the former is found on local disk on /var/mail.
- Debian 7.1 server hosting many daemons which do not show any slow behavior at all (apache, postfix, nfs, autofs, ssh, ...), nor is it slow to run any application for test (no resource intensive application is run routinely on this machine due to its low 4Gb RAM, in any case)
- /home partition nfs mounted from a remote firewalled QNAP NAS server (TS-869U-RP), which also serves other machines (RAID-5 setup with currently no bad disks). When logging in as user in any of those machines including the dovecot server, I notice no delay (remember that dovecot hangs for 60 or more seconds). Also, the inbox hangs as often as
- user authentification using ldap with a daemon hosted on a different server than dovecot's (and firewalled from the outside)
- the logs files give no warnings or errors other than the typical failed connection attempts from chinese or so hackers. I do however, find the following lines in mail.log every once in a while:
Sep 26 11:02:20 wasabi dovecot: imap(pato): Disconnected: Disconnected in IDLE in=8017978 out=490892
Sep 26 11:02:21 wasabi dovecot: imap-login: Login: user=<pato>, method=PLAIN, rip=24.58.62.118, lip=146.83.9.56, mpid=3964, TLS, session\
=<lcGR0UnnugAYOj52>
Sep 26 11:03:23 wasabi dovecot: imap-login: Disconnected (no auth attempts in 1 secs): user=<>, rip=24.58.62.118, lip=146.83.9.56, TLS, \
session=<uOJE1UnnxQAYOj52>
Sep 26 11:03:26 wasabi dovecot: imap-login: Login: user=<pato>, method=PLAIN, rip=24.58.62.118, lip=146.83.9.56, mpid=3973, TLS, session\
=<PCFr1UnnxgAYOj52>
Sep 26 11:05:00 wasabi dovecot: imap(pato): Disconnected: Disconnected in IDLE in=1205 out=28366
note how it receives a 'user=<>' from the same ip it received a valid user a minute ago (and this is the timescale of my problem). though.
- When the problem started I did a lot of rather simultaneous changes to my system (change the hardware of my dovecot's host, moved the ldap daemon from the dovecot machine to a firewalled machine, installed the QNAP NAS, updated CA certificate ...), so it is hard to pinpoint the cause of this. Every other daemon is working as good as it was before,
- 'doveconf -n' output is below.
Thank you very much!!
PatricioPS: Please warn me if any of the information I have given can be used to exploit my system. I have tried to be very careful with this
# 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.1 mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } protocols = " imap" service auth { inet_listener { port = 12345 } unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } ssl_cert = </etc/dovecot/wasabi.imap.crt ssl_key = </etc/dovecot/private/wasabi.imap.nopwd.key userdb { driver = passwd }
hi,
Sep 26 11:03:23 wasabi dovecot: imap-login: Disconnected (no auth attempts in 1 secs): user=<>, rip=24.58.62.118, lip=146.83.9.56, TLS, \
session=<uOJE1UnnxQAYOj52>
Sep 26 11:03:26 wasabi dovecot: imap-login: Login: user=<pato>, method=PLAIN, rip=24.58.62.118, lip=146.83.9.56, mpid=3973, TLS, session\
=<PCFr1UnnxgAYOj52>
try enabling the debug settings in your dovecot.conf, maybe you can get more info:
#auth_debug = yes #auth_debug_passwords = yes #mail_debug = yes
You also mention that your auth server is on a separate machine, and 60 seconds seems a lot like a timeout threshold, maybe you are having intermittent problems there. Maybe if you could tail the dovecot and the ldap logs simultaneously then repeat your test, you would see a discrepancy on the auth server when the dovecot logs show user=<>
ssl_cert = </etc/dovecot/wasabi.imap.crt ssl_key = </etc/dovecot/private/wasabi.imap.nopwd.key
Hmm... a low-level guess: maybe you need to speicify your CA here? I don't *think* that would explain your slowness, but I suppose there could be a timeout looking for it...
userdb { driver = passwd }
On 9/26/2013 7:47 AM, Patricio Rojo wrote:
- /home partition nfs mounted from a remote firewalled QNAP NAS server (TS-869U-RP), which also serves other machines (RAID-5 setup with currently no bad disks).
I assume this NAS properly implements various locking services? Dovecot, like most mail MUA + MTAs, makes use of various filesystem locking primitives to maintain conherence in a multi-user access scenario. If QNAP's stack doesn't implement proper NFS locking, this is probably a cause of these odd lags.
You can probably add a "nolock" to your /etc/fstab to resolve it, but you risk mailbox corruption.
You mentioned it was firewalled... are you allowing the lockd port through to the QNAP from the Dovecot machine that's mounting it? NFS2 + 3 implement locking via communication with a "lock manager" that listens on port 4045, if I recall.
=R=
On 9/26/2013 9:47 AM, Patricio Rojo wrote:
You failed to mention every client device you've tested is connecting to the server from over 5000 miles away, across continents and an ocean, with your packets traversing multiple national and political boundaries.
rip=24.58.62.118, lip=146.83.9.56
cpe-24-58-62-118.twcny.res.rr.com not found: 2(SERVFAIL) Time Warner Cable, New York Red Universitaria Nacional Santiago, CL Observatorio Astronomico Nacional
Have you performed extensive packet tracing to eliminate the network paths as the source of the problem? From here...
~$ telnet 146.83.9.56 993 Trying 146.83.9.56... ^C
~$ telnet 146.83.9.56 143 Trying 146.83.9.56... ^C
~$ telnet 146.83.9.56 25 Trying 146.83.9.56... ^C
~$ telnet 146.83.9.56 587 Trying 146.83.9.56... ^C
Given connections to the Dovecot host are apparently firewalled, either holes have been punched for 24.58.62.118, or you're going through a VPN tunnel. I'd guess your problems are network or firewall related, not Dovecot related.
-- Stan
participants (6)
-
Bob Miller
-
LuKreme
-
Patricio Rojo
-
Robin
-
Stan Hoeppner
-
Steffen Kaiser