Mail slowed down to a crawl...
Been moving along just fine for a couple years now, then in the last two days, email has slowed to a crawl.
Retrieving email via IMAP is very slow, progress bar in the mail client shows that it is downloading messages almost constantly, like it never closes the connection, or it’s getting the messages very slowly. The server that dovecot/postfix is on has also bogged down to a crawl.
The postfix queue is sitting with hundreds of queued up good customer email.
It’s almost acting like dovecot is not getting the messages from postfix, or they’re being delayed for some reason.
How to approach fixing this?
Jeff J.
Den 15.11.2018 21:00, skrev StarionTech (IMAP):
Been moving along just fine for a couple years now, then in the last two days, email has slowed to a crawl.
Retrieving email via IMAP is very slow, progress bar in the mail client shows that it is downloading messages almost constantly, like it never closes the connection, or it’s getting the messages very slowly. The server that dovecot/postfix is on has also bogged down to a crawl.
The postfix queue is sitting with hundreds of queued up good customer email.
It’s almost acting like dovecot is not getting the messages from postfix, or they’re being delayed for some reason.
How to approach fixing this?
Check server logs for errors. Both dovecot and postfix struggling suggest some common underlying cause.
Do measurements on Cpu, disk, cpu-wait. Which process is hogging the server ? Spam-ass acting up? Drives filling up?
Gut instinct says one or more of your drives are failing, or you have file or file-system corruption, so if you are not running smartd, first priority is installing smartd/smartmontools and see what falls out.
Last time I had this issue it turned out GFS2 was the culprit.
http://www.postfix.org/QSHAPE_README.html
Regards Durga Prasad +919849111010
-----Original Message----- From: dovecot [mailto:dovecot-bounces@dovecot.org] On Behalf Of Håkon Alstadheim Sent: 16 November 2018 04:05 To: dovecot@dovecot.org Subject: Re: Mail slowed down to a crawl...
Den 15.11.2018 21:00, skrev StarionTech (IMAP):
Been moving along just fine for a couple years now, then in the last two days, email has slowed to a crawl.
Retrieving email via IMAP is very slow, progress bar in the mail client shows that it is downloading messages almost constantly, like it never closes the connection, or it’s getting the messages very slowly. The server that dovecot/postfix is on has also bogged down to a crawl.
The postfix queue is sitting with hundreds of queued up good customer email.
It’s almost acting like dovecot is not getting the messages from postfix, or they’re being delayed for some reason.
How to approach fixing this?
Check server logs for errors. Both dovecot and postfix struggling suggest some common underlying cause.
Do measurements on Cpu, disk, cpu-wait. Which process is hogging the server ? Spam-ass acting up? Drives filling up?
Gut instinct says one or more of your drives are failing, or you have file or file-system corruption, so if you are not running smartd, first priority is installing smartd/smartmontools and see what falls out.
This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
Yes, I thought all of those things too. Spent a good part of yesterday eliminating them as the cause.
top and iotop do not show any issues. No spam filtering on this box (it’s a separate server/software). Drive is 1/3 full. No smart errors.
The queue problem (and I don’t know why this isn’t handled better) turned out to be a couple of corrupted messages in the queue.
Had about 400 messages in the postfix queue, as soon as I deleted the two corrupt messages, all 400 went out in almost 30 seconds. I found those two messages from log entries, something about qmgr couldn’t delete or move the message, permission denied. The messages had no attachments and looked to be typical mail message traffic. I didn’t open them up though to see what was wrong.
The postfix queue problem is fixed, but message retrieval is still extremely slow, so I’m still thinking there’s a dovecot issue. No error messages from dovecot or postfix that I can see.
I thought I read something about storing mail in maildir vs. some other format, or even something about authentication errors causing slow dovecot transactions.
Jeff J.
On Nov 15, 2018, at 4:35 PM, Håkon Alstadheim <hakon@alstadheim.priv.no> wrote:
Den 15.11.2018 21:00, skrev StarionTech (IMAP):
Been moving along just fine for a couple years now, then in the last two days, email has slowed to a crawl.
Retrieving email via IMAP is very slow, progress bar in the mail client shows that it is downloading messages almost constantly, like it never closes the connection, or it’s getting the messages very slowly. The server that dovecot/postfix is on has also bogged down to a crawl.
The postfix queue is sitting with hundreds of queued up good customer email.
It’s almost acting like dovecot is not getting the messages from postfix, or they’re being delayed for some reason.
How to approach fixing this?
Check server logs for errors. Both dovecot and postfix struggling suggest some common underlying cause.
Do measurements on Cpu, disk, cpu-wait. Which process is hogging the server ? Spam-ass acting up? Drives filling up?
Gut instinct says one or more of your drives are failing, or you have file or file-system corruption, so if you are not running smartd, first priority is installing smartd/smartmontools and see what falls out.
If postfix complained about can't access/delete/move a file, a likely culprit is filesystem damage, which can cause a while raft of problems with seemingly unrelated error messages. Other suspects are flaky hardware, bad RAM, or whatever else can cause the OS to lie to postfix about file access. These same things can mess up dovecot access too.
Unmount the drive and fsck it a few times. Power cycle / reboot the system.
-- Noel Jones
On 11/16/2018 10:05 AM, StarionTech (IMAP) wrote:
Yes, I thought all of those things too. Spent a good part of yesterday eliminating them as the cause.
top and iotop do not show any issues. No spam filtering on this box (it’s a separate server/software). Drive is 1/3 full. No smart errors.
The queue problem (and I don’t know why this isn’t handled better) turned out to be a couple of corrupted messages in the queue.
Had about 400 messages in the postfix queue, as soon as I deleted the two corrupt messages, all 400 went out in almost 30 seconds. I found those two messages from log entries, something about qmgr couldn’t delete or move the message, permission denied. The messages had no attachments and looked to be typical mail message traffic. I didn’t open them up though to see what was wrong.
The postfix queue problem is fixed, but message retrieval is still extremely slow, so I’m still thinking there’s a dovecot issue. No error messages from dovecot or postfix that I can see.
I thought I read something about storing mail in maildir vs. some other format, or even something about authentication errors causing slow dovecot transactions.
Jeff J.
On Nov 15, 2018, at 4:35 PM, Håkon Alstadheim <hakon@alstadheim.priv.no> wrote:
Den 15.11.2018 21:00, skrev StarionTech (IMAP):
Been moving along just fine for a couple years now, then in the last two days, email has slowed to a crawl.
Retrieving email via IMAP is very slow, progress bar in the mail client shows that it is downloading messages almost constantly, like it never closes the connection, or it’s getting the messages very slowly. The server that dovecot/postfix is on has also bogged down to a crawl.
The postfix queue is sitting with hundreds of queued up good customer email.
It’s almost acting like dovecot is not getting the messages from postfix, or they’re being delayed for some reason.
How to approach fixing this? Check server logs for errors. Both dovecot and postfix struggling suggest some common underlying cause.
Do measurements on Cpu, disk, cpu-wait. Which process is hogging the server ? Spam-ass acting up? Drives filling up?
Gut instinct says one or more of your drives are failing, or you have file or file-system corruption, so if you are not running smartd, first priority is installing smartd/smartmontools and see what falls out.
participants (4)
-
DurgaPrasad - DatasoftComnet
-
Håkon Alstadheim
-
Noel
-
StarionTech (IMAP)