On Sun, Oct 26, 2008 at 10:15 AM, Timo Sirainen <tss@iki.fi> wrote:

On Sun, 2008-10-26 at 09:57 -0700, Brandon Lamb wrote:

...

basically I need to find out where to start looking to find out what is the problem with my dovecot pop3 setup. I switched from courier imap/pop3 to dovecot imap/pop3. I have had super high loads (40+ and overnight one server went to 511) and its been a nightmare fighting with this new setup for the last two weeks.

I did manage to narrow the problem down to it being with the pop3 part of dovecot. I am not sure why the load keeps going so high.

Usually this is because Dovecot wants to calculate the messages' virtual sizes. See maildir performance in http://wiki.dovecot.org/POP3Server. There were also some issues with this where Dovecot didn't always add the virtual size to dovecot-uidlist, but they were fixed in v1.1.5.

So the initial high loads are expected if your file names don't contain ,W=size in them, but after that it should slow down. Of course if your new messages don't contain ,W= in them either, the load is still higher than it could be (but then again, Courier does the same thing so it shouldn't be higher than with Courier).

One possibility would be to just make Dovecot violate POP3 spec and return file sizes as message sizes. Some servers like qmail-pop3d do that and they apparently work well enough.. This could be done by changing mail_get_virtual_size to mail_get_physical_size in src/pop3/client.c

Im willing to try this, I really dont want to go back to courier, dovecot just seems like a way cleaner setup. I found the line to edit so I will recompile and give that a try.

I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else. I also found a couple posts saying to use INDEX=MEMORY for the pop3 part, but I didnt understand if that actually DISABLED pop3 using indexes or if it really stored something in memory, and if so what happens on a reboot, so I am confused on if i should tell the pop3 part to use the same /dovecot-index that I specified for imap.

I also dont know if I made so many changes that I didnt give any of them time to work themselves out. We were just in panic mode trying to put out fires hehe.

On Sun, Oct 26, 2008 at 10:25 AM, Timo Sirainen <tss@iki.fi> wrote:

On Sun, 2008-10-26 at 10:21 -0700, Brandon Lamb wrote:

...

I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else. I also found a couple posts saying to use INDEX=MEMORY for the pop3 part, but I didnt understand if that actually DISABLED pop3 using indexes or if it really stored something in memory, and if so what happens on a reboot, so I am confused on if i should tell the pop3 part to use the same /dovecot-index that I specified for imap.

v1.1 handles index files with POP3 a lot better than v1.0. I don't think it's a good idea to disable them.

This reminds me, I should look into updating the courier-dovecot-migrate.pl to copy the virtual sizes and POP3 UIDLs from Courier's files to dovecot-uidlist now that v1.1 supports them both.

So using 1.1.5 would you recommend that I have INDEX=MEMORY or INDEX=/dovecot-index (which is nfs stored dir) for pop3?

Timo Sirainen ha scritto:

On Sun, 2008-10-26 at 10:30 -0700, Brandon Lamb wrote:

...

On Sun, Oct 26, 2008 at 10:25 AM, Timo Sirainen <tss@iki.fi> wrote:

...

On Sun, 2008-10-26 at 10:21 -0700, Brandon Lamb wrote:

...

I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else. I also found a couple posts saying to use INDEX=MEMORY for the pop3 part, but I didnt understand if that actually DISABLED pop3 using indexes or if it really stored something in memory, and if so what happens on a reboot, so I am confused on if i should tell the pop3 part to use the same /dovecot-index that I specified for imap.

v1.1 handles index files with POP3 a lot better than v1.0. I don't think it's a good idea to disable them.

This reminds me, I should look into updating the courier-dovecot-migrate.pl to copy the virtual sizes and POP3 UIDLs from Courier's files to dovecot-uidlist now that v1.1 supports them both.

So using 1.1.5 would you recommend that I have INDEX=MEMORY or INDEX=/dovecot-index (which is nfs stored dir) for pop3?

INDEX=/dovecot-index

What happened if an external program (webmail..) change the inbox directly? Is there a problem with pop3 server or dovecot recreate the index?

-- Roberto Tagliaferri Responsabile Progettazione & Produzione TosNet s.r.l. - Internet Service Provider r.tagliaferri@tosnet.it www.tosnet.it

Brandon,

I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else.

You can configure Exim to write ,W=vsize even without use_crlf. Configuring

{,S=$message_size,W=${eval:$message_size + $message_linecount + 3}}

within maildir_tag setting of exim works especially well for me.

Heiko

Heiko Schlichting Freie Universität Berlin heiko@FU-Berlin.DE Zentraleinrichtung für Datenverarbeitung (ZEDAT) Telefon +49 30 838-54327 Fabeckstraße 32 Telefax +49 30 838454327 14195 Berlin

On Sun, 2008-10-26 at 20:56 +0200, Arkadiusz Miskiewicz wrote:

On Sunday 26 of October 2008, Heiko Schlichting wrote:

...

Brandon,

...

I did modify exim to add the W=size but it also required use_crlf and that seemed to break something else.

You can configure Exim to write ,W=vsize even without use_crlf. Configuring

{,S=$message_size,W=${eval:$message_size + $message_linecount + 3}}

within maildir_tag setting of exim works especially well for me.

Hm, does it count correctly?

I'm also wondering that.

I have mails delivered via exim with this rule:

new/1225046769.H367740P9241.mbox2.agnat.pl,S=1377,W=1413

cat new/1225046769.H367740P9241.mbox2.agnat.pl,S=1377,W=1413 | wc -l 37

so 1377 + 37 + 3 = 1417 which doesn't match 1413.

But this doesn't seem correct either. File size + linecount should be the same as W. Is the file size really 1377 bytes?

For example a file that Dovecot has created:

% wc -cl 1224418671.M66880P13668.hurina,W=2087 56 2031 1224418671.M66880P13668.hurina,W=2087 % echo 2031+56|bc 2087

On Sunday 26 of October 2008, Arkadiusz Miskiewicz wrote:

Also adding 3 is weird. Exim documentation says "The blank line that separates the message header from the body is not counted." so it's only 1 line, not 3.

Ok, it also says that $message_linecount can be incremented when exim adds some headers. That's probably what +3 was trying to do (1+ some 2 headers?).

Unfortunately it looks that W=${eval:$message_size + $message_linecount + 3} is broken rule for exim :/

-- Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/