[Dovecot] Problems with the pop3 part of dovecot
Hello,
I originally just typed up a 5 paragraph essay on my problems but just deleted it all.
basically I need to find out where to start looking to find out what is the problem with my dovecot pop3 setup. I switched from courier imap/pop3 to dovecot imap/pop3. I have had super high loads (40+ and overnight one server went to 511) and its been a nightmare fighting with this new setup for the last two weeks.
I did manage to narrow the problem down to it being with the pop3 part of dovecot. I am not sure why the load keeps going so high. The first week I attributed it to users having to redownload mail and reindex (we were aware of the UIDL problem), but two weeks later one of my machines overnight went to 511. I have changed so many config options to try to make it work that I have become overwhelmed and feel like I have tried everything to get the right settings.
I am running four 64bit machines with amd 9850 cpus, 4 gigs ram, the mailstore is nfs mounted, the nfs server is using xfs. Debian lenny/sid.
I dont know what to do, I am almost debating going back to courier pop3 and going through the week of support calls of customers downloading mail again but that is a last ditch effort. I can understand if i just need to wait it out while things are indexed for dovecot and stuff, but machines falling over and loads of 40 to 500 just for dovecot seems not right.
so, where do I start?
On Sun, 2008-10-26 at 09:57 -0700, Brandon Lamb wrote:
basically I need to find out where to start looking to find out what is the problem with my dovecot pop3 setup. I switched from courier imap/pop3 to dovecot imap/pop3. I have had super high loads (40+ and overnight one server went to 511) and its been a nightmare fighting with this new setup for the last two weeks.
I did manage to narrow the problem down to it being with the pop3 part of dovecot. I am not sure why the load keeps going so high.
Usually this is because Dovecot wants to calculate the messages' virtual sizes. See maildir performance in http://wiki.dovecot.org/POP3Server. There were also some issues with this where Dovecot didn't always add the virtual size to dovecot-uidlist, but they were fixed in v1.1.5.
So the initial high loads are expected if your file names don't contain ,W=size in them, but after that it should slow down. Of course if your new messages don't contain ,W= in them either, the load is still higher than it could be (but then again, Courier does the same thing so it shouldn't be higher than with Courier).
One possibility would be to just make Dovecot violate POP3 spec and return file sizes as message sizes. Some servers like qmail-pop3d do that and they apparently work well enough.. This could be done by changing mail_get_virtual_size to mail_get_physical_size in src/pop3/client.c
On Sun, Oct 26, 2008 at 10:15 AM, Timo Sirainen tss@iki.fi wrote:
On Sun, 2008-10-26 at 09:57 -0700, Brandon Lamb wrote:
basically I need to find out where to start looking to find out what is the problem with my dovecot pop3 setup. I switched from courier imap/pop3 to dovecot imap/pop3. I have had super high loads (40+ and overnight one server went to 511) and its been a nightmare fighting with this new setup for the last two weeks.
I did manage to narrow the problem down to it being with the pop3 part of dovecot. I am not sure why the load keeps going so high.
Usually this is because Dovecot wants to calculate the messages' virtual sizes. See maildir performance in http://wiki.dovecot.org/POP3Server. There were also some issues with this where Dovecot didn't always add the virtual size to dovecot-uidlist, but they were fixed in v1.1.5.
So the initial high loads are expected if your file names don't contain ,W=size in them, but after that it should slow down. Of course if your new messages don't contain ,W= in them either, the load is still higher than it could be (but then again, Courier does the same thing so it shouldn't be higher than with Courier).
One possibility would be to just make Dovecot violate POP3 spec and return file sizes as message sizes. Some servers like qmail-pop3d do that and they apparently work well enough.. This could be done by changing mail_get_virtual_size to mail_get_physical_size in src/pop3/client.c
Im willing to try this, I really dont want to go back to courier, dovecot just seems like a way cleaner setup. I found the line to edit so I will recompile and give that a try.
I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else. I also found a couple posts saying to use INDEX=MEMORY for the pop3 part, but I didnt understand if that actually DISABLED pop3 using indexes or if it really stored something in memory, and if so what happens on a reboot, so I am confused on if i should tell the pop3 part to use the same /dovecot-index that I specified for imap.
I also dont know if I made so many changes that I didnt give any of them time to work themselves out. We were just in panic mode trying to put out fires hehe.
On Sun, 2008-10-26 at 10:21 -0700, Brandon Lamb wrote:
I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else. I also found a couple posts saying to use INDEX=MEMORY for the pop3 part, but I didnt understand if that actually DISABLED pop3 using indexes or if it really stored something in memory, and if so what happens on a reboot, so I am confused on if i should tell the pop3 part to use the same /dovecot-index that I specified for imap.
v1.1 handles index files with POP3 a lot better than v1.0. I don't think it's a good idea to disable them.
This reminds me, I should look into updating the courier-dovecot-migrate.pl to copy the virtual sizes and POP3 UIDLs from Courier's files to dovecot-uidlist now that v1.1 supports them both.
On Sun, Oct 26, 2008 at 10:25 AM, Timo Sirainen tss@iki.fi wrote:
On Sun, 2008-10-26 at 10:21 -0700, Brandon Lamb wrote:
I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else. I also found a couple posts saying to use INDEX=MEMORY for the pop3 part, but I didnt understand if that actually DISABLED pop3 using indexes or if it really stored something in memory, and if so what happens on a reboot, so I am confused on if i should tell the pop3 part to use the same /dovecot-index that I specified for imap.
v1.1 handles index files with POP3 a lot better than v1.0. I don't think it's a good idea to disable them.
This reminds me, I should look into updating the courier-dovecot-migrate.pl to copy the virtual sizes and POP3 UIDLs from Courier's files to dovecot-uidlist now that v1.1 supports them both.
So using 1.1.5 would you recommend that I have INDEX=MEMORY or INDEX=/dovecot-index (which is nfs stored dir) for pop3?
On Sun, 2008-10-26 at 10:30 -0700, Brandon Lamb wrote:
On Sun, Oct 26, 2008 at 10:25 AM, Timo Sirainen tss@iki.fi wrote:
On Sun, 2008-10-26 at 10:21 -0700, Brandon Lamb wrote:
I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else. I also found a couple posts saying to use INDEX=MEMORY for the pop3 part, but I didnt understand if that actually DISABLED pop3 using indexes or if it really stored something in memory, and if so what happens on a reboot, so I am confused on if i should tell the pop3 part to use the same /dovecot-index that I specified for imap.
v1.1 handles index files with POP3 a lot better than v1.0. I don't think it's a good idea to disable them.
This reminds me, I should look into updating the courier-dovecot-migrate.pl to copy the virtual sizes and POP3 UIDLs from Courier's files to dovecot-uidlist now that v1.1 supports them both.
So using 1.1.5 would you recommend that I have INDEX=MEMORY or INDEX=/dovecot-index (which is nfs stored dir) for pop3?
INDEX=/dovecot-index
Timo Sirainen ha scritto:
On Sun, 2008-10-26 at 10:30 -0700, Brandon Lamb wrote:
On Sun, Oct 26, 2008 at 10:25 AM, Timo Sirainen tss@iki.fi wrote:
On Sun, 2008-10-26 at 10:21 -0700, Brandon Lamb wrote:
I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else. I also found a couple posts saying to use INDEX=MEMORY for the pop3 part, but I didnt understand if that actually DISABLED pop3 using indexes or if it really stored something in memory, and if so what happens on a reboot, so I am confused on if i should tell the pop3 part to use the same /dovecot-index that I specified for imap.
v1.1 handles index files with POP3 a lot better than v1.0. I don't think it's a good idea to disable them.
This reminds me, I should look into updating the courier-dovecot-migrate.pl to copy the virtual sizes and POP3 UIDLs from Courier's files to dovecot-uidlist now that v1.1 supports them both.
So using 1.1.5 would you recommend that I have INDEX=MEMORY or INDEX=/dovecot-index (which is nfs stored dir) for pop3?
INDEX=/dovecot-index
What happened if an external program (webmail..) change the inbox directly? Is there a problem with pop3 server or dovecot recreate the index?
-- Roberto Tagliaferri Responsabile Progettazione & Produzione TosNet s.r.l. - Internet Service Provider r.tagliaferri@tosnet.it www.tosnet.it
On Oct 27, 2008, at 11:59 AM, Roberto Tagliaferri - Tosnet srl wrote:
What happened if an external program (webmail..) change the inbox
directly? Is there a problem with pop3 server or dovecot recreate the index?
Dovecot updates the indexes automatically ("rebuild" is a wrong term).
Brandon,
I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else.
You can configure Exim to write ,W=vsize even without use_crlf. Configuring
{,S=$message_size,W=${eval:$message_size + $message_linecount + 3}}within maildir_tag setting of exim works especially well for me.
Heiko
Heiko Schlichting Freie Universität Berlin heiko@FU-Berlin.DE Zentraleinrichtung für Datenverarbeitung (ZEDAT) Telefon +49 30 838-54327 Fabeckstraße 32 Telefax +49 30 838454327 14195 Berlin
On Sunday 26 of October 2008, Heiko Schlichting wrote:
Brandon,
I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else.
You can configure Exim to write ,W=vsize even without use_crlf. Configuring
{,S=$message_size,W=${eval:$message_size + $message_linecount + 3}}within maildir_tag setting of exim works especially well for me.
Hm, does it count correctly?
I have mails delivered via exim with this rule:
new/1225046769.H367740P9241.mbox2.agnat.pl,S=1377,W=1413
cat new/1225046769.H367740P9241.mbox2.agnat.pl,S=1377,W=1413 | wc -l 37
so 1377 + 37 + 3 = 1417 which doesn't match 1413.
Looking with khexview shows that lines are terminated by LF.
Heiko
-- Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/
On Sun, 2008-10-26 at 20:56 +0200, Arkadiusz Miskiewicz wrote:
On Sunday 26 of October 2008, Heiko Schlichting wrote:
Brandon,
I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else.
You can configure Exim to write ,W=vsize even without use_crlf. Configuring
{,S=$message_size,W=${eval:$message_size + $message_linecount + 3}}within maildir_tag setting of exim works especially well for me.
Hm, does it count correctly?
I'm also wondering that.
I have mails delivered via exim with this rule:
new/1225046769.H367740P9241.mbox2.agnat.pl,S=1377,W=1413
cat new/1225046769.H367740P9241.mbox2.agnat.pl,S=1377,W=1413 | wc -l 37
so 1377 + 37 + 3 = 1417 which doesn't match 1413.
But this doesn't seem correct either. File size + linecount should be the same as W. Is the file size really 1377 bytes?
For example a file that Dovecot has created:
% wc -cl 1224418671.M66880P13668.hurina,W=2087 56 2031 1224418671.M66880P13668.hurina,W=2087 % echo 2031+56|bc 2087
On Sunday 26 of October 2008, Timo Sirainen wrote:
On Sun, 2008-10-26 at 20:56 +0200, Arkadiusz Miskiewicz wrote:
On Sunday 26 of October 2008, Heiko Schlichting wrote:
Brandon,
I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else.
You can configure Exim to write ,W=vsize even without use_crlf. Configuring
{,S=$message_size,W=${eval:$message_size + $message_linecount + 3}}within maildir_tag setting of exim works especially well for me.
Hm, does it count correctly?
I'm also wondering that.
I have mails delivered via exim with this rule:
new/1225046769.H367740P9241.mbox2,S=1377,W=1413
cat new/1225046769.H367740P9241.mbox2,S=1377,W=1413 | wc -l 37
so 1377 + 37 + 3 = 1417 which doesn't match 1413.
But this doesn't seem correct either. File size + linecount should be the same as W. Is the file size really 1377 bytes?
It is:
-rw-rw---- 1 mail mail 1377 2008-10-26 19:46 new/1225046769.H367740P9241.mbox2,S=1377,W=1413
Also adding 3 is weird. Exim documentation says "The blank line that separates the message header from the body is not counted." so it's only 1 line, not 3.
For example a file that Dovecot has created:
% wc -cl 1224418671.M66880P13668.hurina,W=2087 56 2031 1224418671.M66880P13668.hurina,W=2087 % echo 2031+56|bc 2087
-- Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/
On Sunday 26 of October 2008, Arkadiusz Miskiewicz wrote:
Also adding 3 is weird. Exim documentation says "The blank line that separates the message header from the body is not counted." so it's only 1 line, not 3.
Ok, it also says that $message_linecount can be incremented when exim adds some headers. That's probably what +3 was trying to do (1+ some 2 headers?).
Unfortunately it looks that W=${eval:$message_size + $message_linecount + 3} is broken rule for exim :/
-- Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/
On Sunday 26 of October 2008, Heiko Schlichting wrote:
Brandon,
I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else.
You can configure Exim to write ,W=vsize even without use_crlf. Configuring
{,S=$message_size,W=${eval:$message_size + $message_linecount + 3}}within maildir_tag setting of exim works especially well for me.
Hm, does it count correctly?
Yes, but it might be necessary to adjust the constant (3) for a different environment. The 3 assumes that the options return_path_add and delivery_date_add are both set. Together with the separator line between header and body, which is not counted by exim, the constant has to the 3.
If someone adds or remove headers conditionally during delivery things become more complicated.
Heiko
Heiko Schlichting Freie Universität Berlin heiko@FU-Berlin.DE Zentraleinrichtung für Datenverarbeitung (ZEDAT) Telefon +49 30 838-54327 Fabeckstraße 32 Telefax +49 30 838454327 14195 Berlin
participants (6)
-
Arkadiusz Miskiewicz
-
Brandon Lamb
-
Heiko Schlichting
-
Heiko Schlichting
-
Roberto Tagliaferri - Tosnet srl
-
Timo Sirainen