[Dovecot] Large subjects increase memory-usage and enlarge index-files
Several times we already had the problems, that accounts with more the 1.3 or 1.7 billion e-mails in one folder run out-of-memory, even if vsize_limit of 750 MB is set.
In this case, the lmtpd-process haven't been able to allocate more memory to read/write/update the index-files and crashed (and the index-files become corrupted at the end.)
[Please -- don't discuss about the need of INBOXes with 1.7 million (unread) e-mails (don't discuss that with ME. Personally, I agree, that there's NO need for that...).]
But: We also noticed accounts with ~ 300.000 e-Mails running out of memory in the same situations. This happends, if the subject is very large (subject or some other header attributes).
And: We've been able to reproduce out-of-memory-Problems with just 13.000 e-mails with VERY long subjects (e.g.: network monitoring status informations), even with a vsize_limit of 750 MB (which is already very much).
13.000 e-mails isn't very much. And it's easy to inject several thousands of prepared e-mails.
Having many mails for accounts with huge (and broken) index-files slows down the delivery rate VERY much and increases the need for memory and cpu resources and I/O very much.
So: This could be used for a very easy to do denial-of-service attac against Dovecot-based mailservers.
I don't have a clear solution for that, Dovecot needs the subject information in its index files. But it looks like, it isn't a good idea to put the whole subject into the index. Maybe it's better/necessary to use just the first 50-70 characters for that and to keep the rest away from the index?
I think I would prefer that even if that means, that accessing those folders with "special" e-mails will become slower because Dovecot has to get those informations directly from the e-mail.
This performance issue is just a problem for the user.
But crashing lmtpd-processes and lowering the delivery rate is a *real* problem for the whole IMAP-cluster.
Peer
-- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-42 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
On Sat, 06 Oct 2012 23:32:56 +0200, Peer Heinlein said:
Several times we already had the problems, that accounts with more the 1.3 or 1.7 billion e-mails in one folder run out-of-memory, even if vsize_limit of 750 MB is set.
In this case, the lmtpd-process haven't been able to allocate more memory to read/write/update the index-files and crashed (and the index-files become corrupted at the end.)
[Please -- don't discuss about the need of INBOXes with 1.7 million (unread) e-mails (don't discuss that with ME. Personally, I agree, that there's NO need for that...).]
But: We also noticed accounts with ~ 300.000 e-Mails running out of memory in the same situations. This happends, if the subject is very large (subject or some other header attributes).
And: We've been able to reproduce out-of-memory-Problems with just 13.000 e-mails with VERY long subjects (e.g.: network monitoring status informations), even with a vsize_limit of 750 MB (which is already very much).
13.000 e-mails isn't very much. And it's easy to inject several thousands of prepared e-mails.
Having many mails for accounts with huge (and broken) index-files slows down the delivery rate VERY much and increases the need for memory and cpu resources and I/O very much.
So: This could be used for a very easy to do denial-of-service attac against Dovecot-based mailservers.
I don't have a clear solution for that, Dovecot needs the subject information in its index files. But it looks like, it isn't a good idea to put the whole subject into the index. Maybe it's better/necessary to use just the first 50-70 characters for that and to keep the rest away from the index?
I think I would prefer that even if that means, that accessing those folders with "special" e-mails will become slower because Dovecot has to get those informations directly from the e-mail.
This performance issue is just a problem for the user.
But crashing lmtpd-processes and lowering the delivery rate is a *real* problem for the whole IMAP-cluster.
Peer
While the real solution is being decided, can I avoid this possible DOS attack by using procmail to /dev/null anything with more than a 256 byte subject, before it ever gets to Dovecot IMAP?
Thanks
SteveT
Steve Litt * http://www.troubleshooters.com/ * http://twitter.com/stevelitt Troubleshooting Training * Human Performance
On 7.10.2012, at 0.32, Peer Heinlein wrote:
Several times we already had the problems, that accounts with more the 1.3 or 1.7 billion e-mails in one folder run out-of-memory, even if vsize_limit of 750 MB is set.
In this case, the lmtpd-process haven't been able to allocate more memory to read/write/update the index-files and crashed (and the index-files become corrupted at the end.)
I don't think dovecot.index file is much of a problem. With 1M mails it usually only takes something like 8-32 MB of memory depending on what mailbox format is used. dovecot.index.log file doesn't depend on the mailbox size at all. The main problem is dovecot.index.cache file.
I've thought about the cache file problems earlier also, but it's a bit difficult to figure out the best solution for it. And since nobody had actually complained about it, I hadn't really done anything about it. Also I hadn't previously thought of LMTP/LDA processes crashing because of it, that's a bigger problem than IMAP process crashing. Although I think you're getting a lot more of "mmap(dovecot.index.cache) failed: Out of memory" errors than crashes for large mailboxes?
So, subproblems related to this:
Filling out dovecot.index.cache too easily. A rather simple possibility that would catch all the possible ways would be to limit the max. size of a single message's cache entry to X kilobytes (64?). If it becomes larger, it's simply not written to the cache file.
Filling out memory too easily. If a long header is wanted to be cached or used for other purposes (e.g. Message-ID), it's still fully read into memory. Add some reasonable limit to max. length of a single header. Can't be too small, because some headers are legitimately pretty long (DKIM and such). Maybe something like 10kB would be safe enough for everyone?
If existing dovecot.index.cache is larger than X MB, shrink it first below X. Shrinking could begin with trying to do it the nice way of removing only unneeded data, but if that fails it could forcibly just remove some old messages. The X would have to be related to the process's VSZ limit.
Dovecot currently doesn't close index files immediately when mailbox is closed, because it's thinking that IMAP clients might reopen the index soon anyway. Max 3 indexes can be kept open, so 3x already different very large indexes can be too much. I'm not sure if this is actually useful at all. Maybe I should disable it for LMTP, or maybe just remove it completely.
The 3. part is what I like changing the least. An alternative solution would be to just not map the entire cache file into memory all at once. The code was actually originally designed to do just that, but munmap()ing + mmap()ing again wasn't very efficient. But for LMTP there's really no need to map the whole file. All it really wants is to read a couple of header records and then append to the file. Maybe it could use an alternative code path that would simply do that instead of mmap()ing anything. It wouldn't solve it for IMAP though.
I don't have a clear solution for that, Dovecot needs the subject information in its index files. But it looks like, it isn't a good idea to put the whole subject into the index. Maybe it's better/necessary to use just the first 50-70 characters for that and to keep the rest away from the index?
50-70 is way too little. The cached subject gets sent to the IMAP client. I think 200 bytes would be minimum and 1000 would be something I could probably even hardcode. But anyway, subject isn't the only way to trigger this and 1000 bytes is too low for some headers.
On Mon, 2012-10-08 at 03:11 +0300, Timo Sirainen wrote:
The 3. part is what I like changing the least. An alternative solution would be to just not map the entire cache file into memory all at once. The code was actually originally designed to do just that, but munmap()ing + mmap()ing again wasn't very efficient. But for LMTP there's really no need to map the whole file. All it really wants is to read a couple of header records and then append to the file. Maybe it could use an alternative code path that would simply do that instead of mmap()ing anything. It wouldn't solve it for IMAP though.
Attached patch changes LMTP/LDA to not mmap() the target mailbox's cache file. I did a few quick tests and it seems to work. I'll probably commit it to 2.1 hg after some more tests.
On 10/7/2012 7:11 PM, Timo Sirainen wrote:
I don't think dovecot.index file is much of a problem. With 1M mails it usually only takes something like 8-32 MB of memory depending on what mailbox format is used. dovecot.index.log file doesn't depend on the mailbox size at all. The main problem is dovecot.index.cache file.
I've thought about the cache file problems earlier also, but it's a bit difficult to figure out the best solution for it. And since nobody had actually complained about it, I hadn't really done anything about it. Also I hadn't previously thought of LMTP/LDA processes crashing because of it, that's a bigger problem than IMAP process crashing. Although I think you're getting a lot more of "mmap(dovecot.index.cache) failed: Out of memory" errors than crashes for large mailboxes?
So, subproblems related to this:
Filling out dovecot.index.cache too easily. A rather simple possibility that would catch all the possible ways would be to limit the max. size of a single message's cache entry to X kilobytes (64?). If it becomes larger, it's simply not written to the cache file.
Filling out memory too easily. If a long header is wanted to be cached or used for other purposes (e.g. Message-ID), it's still fully read into memory. Add some reasonable limit to max. length of a single header. Can't be too small, because some headers are legitimately pretty long (DKIM and such). Maybe something like 10kB would be safe enough for everyone?
If existing dovecot.index.cache is larger than X MB, shrink it first below X. Shrinking could begin with trying to do it the nice way of removing only unneeded data, but if that fails it could forcibly just remove some old messages. The X would have to be related to the process's VSZ limit.
Dovecot currently doesn't close index files immediately when mailbox is closed, because it's thinking that IMAP clients might reopen the index soon anyway. Max 3 indexes can be kept open, so 3x already different very large indexes can be too much. I'm not sure if this is actually useful at all. Maybe I should disable it for LMTP, or maybe just remove it completely.
The 3. part is what I like changing the least. An alternative solution would be to just not map the entire cache file into memory all at once. The code was actually originally designed to do just that, but munmap()ing + mmap()ing again wasn't very efficient. But for LMTP there's really no need to map the whole file. All it really wants is to read a couple of header records and then append to the file. Maybe it could use an alternative code path that would simply do that instead of mmap()ing anything. It wouldn't solve it for IMAP though.
50-70 is way too little. The cached subject gets sent to the IMAP client. I think 200 bytes would be minimum and 1000 would be something I could probably even hardcode. But anyway, subject isn't the only way to trigger this and 1000 bytes is too low for some headers.
Nearly all mail servers have two resources of interest here in great excess: CPU cycles, and cache/RAM b/w, due to multicore CPUs and 2-4 memory channels per socket. The two bottlenecks are IO bandwidth/latency, and, for many, RAM capacity. So let's take advantage of both the strengths and weaknesses of our hardware to possibly address the above issue.
What happens if we insert a subroutine to compress/decompress each field in the cache array files individually, in real time? You should still be able to mmap the files. The individual array fields and total cache file sizes would be much smaller on disk and in memory. Any cache file contents mapped to memory, that aren't currently being used, are stored compressed in memory, directly addressing the problem in this thread. When a field is needed we decompress it on the fly after reading it from memory. This should be very fast as the fields are relatively small. When it's written out we compress on the fly. With each field stored compressed on disk, not only is file size decreased, but more importantly, each read/write moves more data per physical IO. So not only are increasing storage capacity, we're also decreasing IOPS.
It would be preferable to do this de/compression in kernel rather than user space, but I don't think that's a real option. However, libz and libbz2 are pretty fast and small, and the code easily fits in CPU cache. Combined with the massive L1/L2/L3 and RAM b/w of modern systems, execution in user space should still be very fast, and not noticeably degrade performance.
I'm not a programmer, so I have no idea if this is even plausible, or possible. But if it is, it seems worth exploring, as it would seem to benefit Dovecot performance in multiples areas, and possibly solve this, and other current/future memory capacity and/or performance related problems.
-- Stan
participants (4)
-
Peer Heinlein
-
Stan Hoeppner
-
Steve Litt
-
Timo Sirainen