-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-08-14 7:52 AM, Chad M Stewart wrote:
The problem happened again this morning. Removing fsync calls helped,
but I'm not sure about leaving that enabled long term.
I still believe the problem is multiple dovecot processes trying to
write to a single folder at the same time. (If I could run dtrace I might be able to cobble together a script to prove it.)
I tried writing a sieve script to direct the messages to a set of
folders, but I'm not able to make the logic work. I was thinking something like: generate random # ($N) between say 1-10, then file message into folder mail$N. But I didn't find a method to do that within sieve.
My next thought was to try parsing the message-id header. If the
first character is [0-5] then fileinto mail1, etc. Then I could go so far as having 36 subfolders which the messages could be written too. This mailbox only keeps messages for a rolling 1d window. Right now for example it has 260,186 messages in the INBOX.
The sieve script I tried (with only about 4 hours of sleep) was
require ["fileinto","regex"];
if header :regex "message-id" "^1" { fileinto "mail1"; } else { keep; }
If anyone has some suggestions on how I might spread the messages out
over multiple folders I'd like to hear your thoughts. Again the servers are configured using maildir, so each folder should have its own index and thus file locking contention should be less, at least so goes the theory in my head.
Thank you, Chad
With that many messages, it may be the index updates that are slowing things down. Depending on your needs or how you use this folder (it sounds automated to me), you may not need the indexes on disk at all. If you set the indexes to RAM, it may help.
mail_location = maildir:~/Maildir:INDEX=MEMORY
as per http://wiki2.dovecot.org/MailLocation#Index_files
If indexes in RAM isn't an option, maybe just put them on a separate spindle / SSD would help.
Please note I'm shooting from the hip with this advice, I don't have a system that pushes dovecot that hard.
Greg -----BEGIN PGP SIGNATURE----- Version: GnuPG v2
iQIcBAEBCAAGBQJV0e/oAAoJECD7Htp+IT9eUXsQALgtSocaTUXWjMzp1oIEZTsT IOBzc25q4LZtNWeQpa/czmc9TBQfmqcRzXspyqu/dTxdesJ05WL3PiD/c66Bsmb8 0652a+6l7rLgMroIgsYCQZNBaoU+7FgIakWQlyHA/Ods7DTFNNRvAVNhZ6e6NcvR pkIM5EtLaHzc7318ahR2mC4tRHbmlSxZT/fnWuCu185VyuG6GiHum0piyplEIRqw ngbHkMROgD/OJTKScngRP73umLznxOfReVrM50tIlgpONY+ngTLvD6rnkbLO1t1Y KdVAc49iEQPTRuRVDDS+kf2haUx9Rh8Zz0R4VRdoiBC9bcRmeuQDNU4mg4fBCgQF JlkIkuYBvZe/y7TcxJ5rJXNtEn6g4Uew7mIt+sMnGx7Ro55kl0X61Qg/IWu+F8m5 aO9Y7jMjptsjm/04YWlaeCNUPolgUQrFLDjiO5uycEZh1NReNZEH5AFM0cYWMhBK PWzDRXwv4XEypNA+y+uDLdN3mX/KcWMUv90ipJT5PdrLPjvgjTbyU/Wr391H01Kf 85ShLVvBsV2NGvXY0ZRAxuF5Ndq04TNrRYDvVgzb4XlCH++l2WeZJ2CZ8b1KL5AC BydVyp5tWoP5TIzelG6NcMcqNjD05rtmcH2QCT1BLMksY9+7YkbU6IX+W5Dc5sOX krjSPNAsLOFJJCxBIZ/S =AYpS -----END PGP SIGNATURE-----