We run around 5 dovecot (debian etch 1.0.rc15) POP/IMAP 'nodes' using the LVS load balancer and an NFS based SAN. it works pretty well. I love the robustness of load balancing POP/IMAP. We do a reasonable amount of throughput through these especially at peak times pushing our SAN to around 1.5k IOP/s
We currently have two issues with this setup. One of which is NFS index corruption issues we get due to NFS/dovecot locking. Basically the UUID list or a .index gets corrupt. This causes a full re-indexing of the mailbox / broken mailbox until i delete the indexes. In the UUID lists case the symptom tends to effect use who use POP rather than IMAP and insist on keeping messages on the server. Because it's corrupt it gets rebuilt one way or the other and the users email client proceeds to redownload the entire mailbox again until he remarks them to be saved. This tends to annoy the user a lot. After a bit of testing we do however expect this to be fixed by version 1.1. However if anyone has any comments on this I would certainly be interested.
The other issue is a little tricky to describe or even log effectively. Occasionally a node basically receives more connections than it is able to handle. The connection count goes through the roof for that node. It will go up way beyond 150 dovecot authentication threads and 100 or so active POP/IMAP dovecot threads. The IO_wait/cpu and memory usage also tends to be spiking at this point. The server gets to the tipping point where it can no longer serve it's POP/IMAP requests fast enough compared to the number of connections it's getting. I'd be fine with this but this creates some less than desirable symptoms:
We obviously reach the auth thread cap eventually so any new auth requests basically get refused by the server. To the user this is a seen as an outlook/mail client re-auth pop-up request. This annoys them. Ideally if the server stops accepting auth requests it should fall off our load balancer until it can consistently accept them again. Since the LVS detects a node fail by whether the tcp port is still open this doesn't happen, since dovecot keeps the port open. This is obviously more an LVS issue and not for this mailing list I expect unless anyone has any config tweaking tips here?
Now here's my real gripe. Dovecot does not handle running out resources very gracefully at all in our setup. It does start killing threads after a while. I get multiple *"dovecot: child 17989 (login) killed with signal 9". *I'm not exactly sure what's going on here because after this all I can see is the machine totally out of memory and the kernel starts killing absolutely everything. All services are killed (including ssh etc..) and I plug a monitor into the server and find the last few lines of the console listing init and other rather important things having just been killed. At this point it is a case of power cycling the server and all is back to normal again.
I imagine there's not a huge amount of people using dovecot in this way. But anyone got any recommendations here? I really like using dovecot in this setup it handles it pretty well and the redundancy and functionality options it provides have been invaluable.
Neil