Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.01.2012 21:54, Timo Sirainen wrote:
We have bought new hardware for this project too, so we might not be able to help out massively on that front... we do have NFS operations monitored though so we should at least be able to compare that metric since the underlying storage operating system is the same. All NetApp hardware runs their Data ONTAP operating system, so the metrics are assured to be the same :)
How about this... are there any tools available (that you know of) to capture real live customer POP3/IMAP traffic and replay it against a separate system? That might be a feasible option for doing a like-for-like comparison in our environment? We could probably get something in place to simulate the load if we can do something like that...
That might be the one thing (unfortunately) which prevents us from going with the dbox format. I understand the same issue can actually occur on Dovecot Maildir as well, but because Maildir works without these index files, we were willing to just go with it. I will raise it again, but there has been a lot of push back about introducing a single point of failure, even though this is a perceived one.
The biggest challenge I have at the moment if I try to sell the dbox
format is providing some kind of data on the expected gains from this.
If it's only a 10% reduction in NFS operations for the typical user,
then it's probably not worth our while.
No, we're using the nginx proxy layer to ensure that if a user in Sydney (for example) tries to access a Perth mailbox, their connection is redirected (by nginx) to the Perth POP/IMAP servers. Postfix configuration is handling the same thing on the LMTP side.
The requirement here is for all users to have the same settings regardless of location, but still be able to locate the email servers and data close to the customer.
fts-solr. I've been using Lucene/Solr interchangeably when discussing this project with my peers :)
Our existing systems run with about 21K concurrent IMAP connections at any one point in time, not counting Webmail POP3 runs at about 3600 concurrent connections, but since those are not long lived it's not particularly indicative of customer numbers. Vague recollection is something like 25% IMAP, 55-60% POP3, rest < 20% Webmail. I'd have to go back and check the breakdown again.
Yep, I'm considering that. The flip side is that it's actually going to be difficult for us to change mail format once we've migrated into this system, but we have an opportunity for (literally) a month long testing phase beginning in Feb/March which will let us test as many possibilities as we can.
On Wed, 2012-01-18 at 22:36 +0800, Lee Standen wrote:
I've thought about that too before, but with IMAP traffic it doesn't work very well. Even if the storages were 100% synchronized at startup, the session states could easily become desynced. For example if client does a NOOP at the same time when two mails are being delivered to the mailbox, serverA might show only one of them while serverB would show two of them because it was executed a tiny bit later. All of the client's future commands could then be affected by this desync.
(OK, I wrote the above thinking about a real-time system where you could redirect the client's traffic to two systems, but basically same problems exist for offline replays too. Although it would be easier to fix the replays to handle this.)
Are you planning on also redirecting POP3/IMAP connections to somewhat randomly to the different servers? I really don't recommend that, even with Maildir.. Some of the errors will be user visible, even if no actual data loss happens. Users may get disconnected, and sometimes might have to clean their client's cache.
What is a single point of failure there?
The mailbox format switching can be done one user at a time with zero downtime with dsync.
<snip>
I'm in the middle of working on a Maildir->mdbox migration as well, and likewise, over NFS (all Netapps but moving to Sun), and likewise with split LDA and IMAP/POP servers (and both of those served out of pools). I was hoping doing things like setting "mail_nfs_index = yes" and "mmap_disable = yes" and "mail_fsync = always/optimized" would mitigate most of the risks of index corruption, as well as probably turning indexing off on the LDA side of things--i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed.
On 18.1.2012, at 19.54, Mark Moseley wrote:
They help, but aren't 100% effective and they also make the performance worse.
as well as probably turning indexing off on the LDA side of things
You can't turn off indexing with dbox.
What's the problem with director-based architecture?
On Wed, Jan 18, 2012 at 07:58:31PM +0200, Timo Sirainen wrote:
It hasn't been working reliably for lmtp in v2.0. To quote yourself:
----8<----8<----8<-----8<-----8<-----8<----8<-----8<----8<----8<--
I think the way I originally planned LMTP proxying to work is simply too
complex to work reliably, perhaps even if the code was bug-free. So
instead of reading+writing DATA at the same time, this patch changes the
DATA to be first read into memory or temp file, and then from there read
and sent to the LMTP backends:
http://hg.dovecot.org/dovecot-2.1/raw-rev/51d87deb5c26
----8<----8<----8<-----8<-----8<-----8<----8<-----8<----8<----8<--
unfortunately I haven't tested that patch, so I have no idea if it fixed the issues or not...
-jf
On 18.1.2012, at 20.51, Jan-Frode Myklebust wrote:
Yes, besides that :)
I'm not sure if that patch is useful or not. The important patch to fix it is http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c
On Wed, Jan 18, 2012 at 09:03:18PM +0200, Timo Sirainen wrote:
Besides that it's great!
So with that oneliner on our directors, you expect lmtp proxying trough director to be better than lmtp to rr-dns towards backend servers? If so, I guess we should give it another try.
-jf
On Wed, Jan 18, 2012 at 09:03:18PM +0200, Timo Sirainen wrote:
I now implemented this patch on our directors, and pointed postfix at them. No problem seen so far, but I'm still a bit uncertain about the LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS. I know we're experienceing quite large delays when fsync'ing (slow IMAP APPEND). Do you think increasing LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS is a sensible workaround if we start seeing lmtp_proxy_output_timeout problems again ?
-jf
On Mon, Feb 06, 2012 at 10:29:03PM +0200, Timo Sirainen wrote:
Hopefully not.. maybe just me being confused by the error message about "lmtp_proxy_output_timeout". After adding http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c on friday, we haven't seen any problems so it looks like this problem is solved.
But it doesn't seem unthinkable that ext3 users might see more than 60s for fsyncs... "Some stalls on the order of minutes have been reported" ref: https://lwn.net/Articles/328363/
I think even if you increase Dovecot's timeout you'll soon reach your MTA's LMTP timeout.
My MTA's default is 10 minutes..
http://www.postfix.org/postconf.5.html#lmtp_data_done_timeout
-jf
On Mon, Feb 06, 2012 at 10:01:03PM +0100, Jan-Frode Myklebust wrote:
Crap, saw 6 "message might be sent more than once" messages from postfix yesterday, all at the time of this crash on the director postfix/lmtp was talking with:
Feb 6 16:13:10 loadbalancer2 dovecot: lmtp(6601): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy->data_input->eof)
Feb 6 16:13:10 loadbalancer2 dovecot: lmtp(6601): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0 [0x2ab6f193d680] -> /usr/lib64/dovecot/libdovecot.so.0 [0x2ab6f193d6d6] -> /usr/lib64/dovecot/libdovecot.so.0 [0x2ab6f193cb93] -> dovecot/lmtp [0x406d75] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xcd) [0x2ab6f194859d] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x68) [0x2ab6f1949558] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x2d) [0x2ab6f194820d] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x2ab6f1936a83] -> dovecot/lmtp(main+0x144) [0x403fa4] -> /lib64/libc.so.6(__libc_start_main+0xf4) [0x35f8a1d994] -> dovecot/lmtp [0x403da9]
Feb 6 16:13:10 loadbalancer2 dovecot: master: Error: service(lmtp): child 6601 killed with signal 6 (core dumps disabled)
Should I try increasing LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS, or do you have any other ideas for what might be causing it ?
-jf
On 7.2.2012, at 10.25, Jan-Frode Myklebust wrote:
The backend server didn't reply within LMTP_PROXY_DEFAULT_TIMEOUT_MSECS (30 secs). It still shouldn't have crashed of course, and that crash is already fixed in v2.1 (in the LMTP simplification change). Anyway, you can fix this without recompiling by returning e.g. "proxy_timeout=60" passdb extra field for 60 secs timeout.
On Thu, Feb 09, 2012 at 01:48:09AM +0200, Timo Sirainen wrote:
It's actually 60 sec in v2.0
http://hg.dovecot.org/dovecot-2.0/file/750db4b4c7d3/src/lmtp/lmtp-proxy.c#l13
It still shouldn't have crashed of course, and that crash is already fixed in v2.1 (in the LMTP simplification change).
Do you think we should rather run v2.1-rc* on our dovecot directors (for IMAP, POP3 and LMTP), even if we keep the backend servers on v2.0 ?
Anyway, you can fix this without recompiling by returning e.g. "proxy_timeout=60" passdb extra field for 60 secs timeout.
Thanks, well consider that option if it crashes too often... Have only seen this problem once for the last week.
-jf
On 9.2.2012, at 14.56, Jan-Frode Myklebust wrote:
- LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS is not LMTP_PROXY_DEFAULT_TIMEOUT_MSECS
Yes, I've done a lot of improvements to proxying and error handling/logging in v2.1. Also I'm planning on finishing my email backlog soon and making the last v2.1-rc before renaming it to v2.1.0.
On Wed, Jan 18, 2012 at 9:58 AM, Timo Sirainen <tss@iki.fi> wrote:
In testing, it seemed very much like the benefits of reducing IOPS by up to a couple orders of magnitude outweighed having to use those settings. Both in scripted testing and just using a mail UI, with the NFS-ish settings, I didn't notice any lag and doing things like checking a good-sized mailbox were at least as quick as Maildir. And I'm hoping that reducing IOPS across the entire set of NFS servers will compound the benefits quite a bit.
Ah, too bad. I was hoping I could get away with the LDA not updating the index but just dropping the message into storage/m.# but it'd still be seen on the IMAP/POP side--but hadn't tested that. Guess that's not the case.
Nothing, per se. It's just that migrating to mdbox *and* to a director architecture is quite a bit more added complexity than simply migrating to mdbox alone.
Hopefully, I'm not hijacking this thread. This seems pretty pertinent as well to the OP.
participants (4)
-
Jan-Frode Myklebust
-
Lee Standen
-
Mark Moseley
-
Timo Sirainen