multiple messages per second to a single mailbox

[Gregory Finch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
 
On 2015-08-14 7:52 AM, Chad M Stewart wrote:
>
> The problem happened again this morning.  Removing fsync calls helped,
but I'm not sure about leaving that enabled long term.
>
> I still believe the problem is multiple dovecot processes trying to
write to a single folder at the same time.  (If I could run dtrace I
might be able to cobble together a script to prove it.)
>
> I tried writing a sieve script to direct the messages to a set of
folders, but I'm not able to make the logic work. I was thinking
something like: generate random # ($N) between say 1-10, then file
message into folder mail$N.  But I didn't find a method to do that
within sieve.  
>
> My next thought was to try parsing the message-id header.  If the
first character is [0-5] then fileinto mail1, etc.  Then I could go so
far as having 36 subfolders which the messages could be written too. 
This mailbox only keeps messages for a rolling 1d window.  Right now for
example it has 260,186 messages in the INBOX.
>
> The sieve script I tried (with only about 4 hours of sleep) was
>
> require ["fileinto","regex"];
>
> if header :regex "message-id" "^1" {
>    fileinto "mail1";
> } else {
>    keep;
> }
>
>
> If anyone has some suggestions on how I might spread the messages out
over multiple folders I'd like to hear your thoughts.  Again the servers
are configured using maildir, so each folder should have its own index
and thus file locking contention should be less, at least so goes the
theory in my head.
>
>
> Thank you,
> Chad
>
With that many messages, it may be the index updates that are slowing
things down. Depending on your needs or how you use this folder (it
sounds automated to me), you may not need the indexes on disk at all. If
you set the indexes to RAM, it may help.

mail_location = maildir:~/Maildir:INDEX=MEMORY

as per http://wiki2.dovecot.org/MailLocation#Index_files

If indexes in RAM isn't an option, maybe just put them on a separate
spindle / SSD would help.

Please note I'm shooting from the hip with this advice, I don't have a
system that pushes dovecot that hard.

Greg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
 
iQIcBAEBCAAGBQJV0e/oAAoJECD7Htp+IT9eUXsQALgtSocaTUXWjMzp1oIEZTsT
IOBzc25q4LZtNWeQpa/czmc9TBQfmqcRzXspyqu/dTxdesJ05WL3PiD/c66Bsmb8
0652a+6l7rLgMroIgsYCQZNBaoU+7FgIakWQlyHA/Ods7DTFNNRvAVNhZ6e6NcvR
pkIM5EtLaHzc7318ahR2mC4tRHbmlSxZT/fnWuCu185VyuG6GiHum0piyplEIRqw
ngbHkMROgD/OJTKScngRP73umLznxOfReVrM50tIlgpONY+ngTLvD6rnkbLO1t1Y
KdVAc49iEQPTRuRVDDS+kf2haUx9Rh8Zz0R4VRdoiBC9bcRmeuQDNU4mg4fBCgQF
JlkIkuYBvZe/y7TcxJ5rJXNtEn6g4Uew7mIt+sMnGx7Ro55kl0X61Qg/IWu+F8m5
aO9Y7jMjptsjm/04YWlaeCNUPolgUQrFLDjiO5uycEZh1NReNZEH5AFM0cYWMhBK
PWzDRXwv4XEypNA+y+uDLdN3mX/KcWMUv90ipJT5PdrLPjvgjTbyU/Wr391H01Kf
85ShLVvBsV2NGvXY0ZRAxuF5Ndq04TNrRYDvVgzb4XlCH++l2WeZJ2CZ8b1KL5AC
BydVyp5tWoP5TIzelG6NcMcqNjD05rtmcH2QCT1BLMksY9+7YkbU6IX+W5Dc5sOX
krjSPNAsLOFJJCxBIZ/S
=AYpS
-----END PGP SIGNATURE-----