[Dovecot] imap memory footprint rather large
Dear list,
I am experimenting with a new mail handling setup and it involves a single IMAP folder with just under 70'000 messages. When OfflineIMAP connects to the server, the imap process starts to eat up a lot of memory:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15607 madduck 35 19 283m 244m 239m D 16.9 49.3 0:09.96 imap
On the contrary, when "online" client, such as Thunderbird connect, memory usage is around 10m, which is entirely acceptable.
The way offlineimap reads may is by FETCHing metadata, then APPENDing new local mail, SEARCHing for the UIDs of each uploaded mail, and finally FETCHing new remote mail.
Memory use seems to be O(n) in the size of the folder. On the folder with 70k messages, dovecot seems to allocate 280m of memory, which it then fills to about 70% during the metadata FETCH, and then keeps growing while APPEND/SEARCHing the new local messages.
The 70k mailbox is just short of 600Mb in size on disk. Dovecot uses 280Mb to serve it. Is it possible that dovecot is reading too much into memory, or over-optimising?
Can I somehow tweak this to lower the memory footprint?
Cheers,
also sprach martin f krafft madduck@madduck.net [2007.08.13.2259 +0200]:
Memory use seems to be O(n) in the size of the folder. On the folder with 70k messages, dovecot seems to allocate 280m of memory, which
I just saw in the logs:
mmap() failed with index cache file /home/madduck/.maildir/.store/dovecot.index.cache: Cannot allocate memory
and looking at the file, it's in fact 280m in size.
Does dovecot need to read/mmap the entire file? Is there a way to vacuum/reduce/optimise the cache?
On Mon, 2007-08-13 at 23:24 +0200, martin f krafft wrote:
also sprach martin f krafft madduck@madduck.net [2007.08.13.2259 +0200]:
Memory use seems to be O(n) in the size of the folder. On the folder with 70k messages, dovecot seems to allocate 280m of memory, which
I just saw in the logs:
mmap() failed with index cache file /home/madduck/.maildir/.store/dovecot.index.cache: Cannot allocate memory
and looking at the file, it's in fact 280m in size.
Does dovecot need to read/mmap the entire file?
mmaping the whole file doesn't have problems, as long as it fits into memory. By default mail_process_size=256 meaning it's limited to 256MB. This is just to make sure that if there are memory leaks Dovecot doesn't eat all the memory. So you could grow the value or even set it to 0 to disable it, and it would work just fine.
Is there a way to vacuum/reduce/optimise the cache?
You can always delete it, but if your client wants the same information all over again it gets grown to the same size. Probably it doesn't after the initial mailbox load. Dovecot should also drop unused fields from it after a week or so, but currently this isn't done.
also sprach Timo Sirainen tss@iki.fi [2007.08.13.2324 +0100]:
Is there a way to vacuum/reduce/optimise the cache?
You can always delete it, but if your client wants the same information all over again it gets grown to the same size. Probably it doesn't after the initial mailbox load. Dovecot should also drop unused fields from it after a week or so, but currently this isn't done.
Any news on that front?
On Mon, 2008-05-12 at 18:00 +0100, martin f krafft wrote:
also sprach Timo Sirainen tss@iki.fi [2007.08.13.2324 +0100]:
Is there a way to vacuum/reduce/optimise the cache?
You can always delete it, but if your client wants the same information all over again it gets grown to the same size. Probably it doesn't after the initial mailbox load. Dovecot should also drop unused fields from it after a week or so, but currently this isn't done.
Any news on that front?
v1.1 drops fields that aren't accessed after 30 days.
also sprach Timo Sirainen tss@iki.fi [2008.05.12.1813 +0100]:
v1.1 drops fields that aren't accessed after 30 days.
And that interval is hardcoded or configurable?
Also, do you have an ETA on the 1.1 release? As you may know, we're freezing Debian stable in August or September and it would be good to get 1.1 in with enough time for testing beforehand.
On Mon, 2008-05-12 at 18:19 +0100, martin f krafft wrote:
also sprach Timo Sirainen tss@iki.fi [2008.05.12.1813 +0100]:
v1.1 drops fields that aren't accessed after 30 days.
And that interval is hardcoded or configurable?
Hardcoded in src/lib-index/mail-cache-private.h:
/* Drop fields that haven't been accessed for n seconds */ #define MAIL_CACHE_FIELD_DROP_SECS (3600*24*30)
There are several other optimization related defines. I don't think it's really worth making configurable.
Also, do you have an ETA on the 1.1 release? As you may know, we're freezing Debian stable in August or September and it would be good to get 1.1 in with enough time for testing beforehand.
It depends on how many bug reports I still receive.. There are a few bug reports I haven't yet had time to look at. Anyway I think it'll be released in June.
On Mon, 2007-08-13 at 22:59 +0200, martin f krafft wrote:
The way offlineimap reads may is by FETCHing metadata, then APPENDing new local mail, SEARCHing for the UIDs of each uploaded mail, and finally FETCHing new remote mail.
What exactly do you mean by FETCHing metadata? Something like ENVELOPE or BODYSTRUCTURE? And this is fetched for all messages instead of just new ones? That could easily explain why cache is so large.
also sprach Timo Sirainen tss@iki.fi [2007.08.14.0028 +0200]:
What exactly do you mean by FETCHing metadata? Something like ENVELOPE or BODYSTRUCTURE? And this is fetched for all messages instead of just new ones? That could easily explain why cache is so large.
The code is:
response = imapobj.fetch('1:%d' % maxmsgid, '(FLAGS UID INTERNALDATE)')[1]
meaning that it obtains (FLAGS UID INTERNALDATE) for all messages in a folder every time.
It needs to do this to be able to synchronise flags. But does it mean that the server has to keep it all in memory? I am not sure...
On Tue, 2007-08-14 at 08:13 +0200, martin f krafft wrote:
also sprach Timo Sirainen tss@iki.fi [2007.08.14.0028 +0200]:
What exactly do you mean by FETCHing metadata? Something like ENVELOPE or BODYSTRUCTURE? And this is fetched for all messages instead of just new ones? That could easily explain why cache is so large.
The code is:
response = imapobj.fetch('1:%d' % maxmsgid, '(FLAGS UID INTERNALDATE)')[1]
meaning that it obtains (FLAGS UID INTERNALDATE) for all messages in a folder every time.
FLAGS and UID are stored in dovecot.index file, and caching only INTERNALDATE would take maybe 16 bytes/message.
So I guess most of the data in your dovecot.index.cache file came from some initial FETCH ENVELOPE/BODYSTRUCTURE/etc. for all messages. If you delete it, it won't probably get as large anymore.
I'm not sure if there's anything I can do on Dovecot's side to make this work better. This shouldn't be a problem except for large mailboxes that are accessed with Dovecot for the first time. There the possibilities are to cache wanted data immediately so that it can be accessed fast the next time, or not cache it at all the first time and if it's needed again doing the whole thing all over again.
participants (2)
-
martin f krafft
-
Timo Sirainen