[Dovecot] dovecot.index.cache 1/3rd of INBOX size
I've noticed that my dovecot.index.cache file has got very large (24MB, which is more than 1/3 of my INBOX size). I've tried deleting (well, renaming) the dovecot.index.* files, and re-opening my INBOX, and the new dovecot.index.cache is only 90KB so far.
I'm using dovecot-1.0-stable and mbox format. My mail_cache_fields and mail_never_cache_fields are set to the default.
Looking back in the list archives, I found a message from Timo that suggested the cache entries are created for each message when the indexes are created, rather than when each message is accessed, so it ought not to grow? What happens when a message is deleted?
I "strings/grep" ed through the dovecot.index.cache file and it looks like it's not caching any headers it shouldn't (like Received, X-Spam-XYZ etc.), but they seem to occur multiple times for each message.
None of my other dovecot testers seem to have huge cache files (yet!).
Anybody else seen this?
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
On Tuesday 14 June 2005 15:35, Chris Wakelin wrote:
I've noticed that my dovecot.index.cache file has got very large (24MB, which is more than 1/3 of my INBOX size). I've tried deleting (well, renaming) the dovecot.index.* files, and re-opening my INBOX, and the new dovecot.index.cache is only 90KB so far.
I'm using dovecot-1.0-stable and mbox format. My mail_cache_fields and mail_never_cache_fields are set to the default.
Looking back in the list archives, I found a message from Timo that suggested the cache entries are created for each message when the indexes are created, rather than when each message is accessed, so it ought not to grow? What happens when a message is deleted?
I "strings/grep" ed through the dovecot.index.cache file and it looks like it's not caching any headers it shouldn't (like Received, X-Spam-XYZ etc.), but they seem to occur multiple times for each message.
None of my other dovecot testers seem to have huge cache files (yet!).
Anybody else seen this?
A sample from some mailboxes here. Dovecot 0.99.14 and maildirs.
Username Mailbox Size Index Size Messages A 41M 5M 7467 B 115M 24M 12221 C 303M 40M 63067 D 1.7G 8.5M 10459 E 505M 2M 1193 F 109M 25M 30594 G 763M 1.6M 2723
Best Wishes, Chris
-- Dominic GoodforBusiness.co.uk I.T. Services for SMEs in the UK.
On 14.6.2005, at 17:35, Chris Wakelin wrote:
I've noticed that my dovecot.index.cache file has got very large (24MB, which is more than 1/3 of my INBOX size). I've tried deleting (well, renaming) the dovecot.index.* files, and re-opening my INBOX, and the new dovecot.index.cache is only 90KB so far.
Fixed in CVS. The mails weren't being marked as deleted in cache file because of a simple mistake ("!=" -> "=="). So after the change the files still won't get compressed immediately, but it'll work in future.
Timo Sirainen wrote:
On 14.6.2005, at 17:35, Chris Wakelin wrote:
I've noticed that my dovecot.index.cache file has got very large (24MB, which is more than 1/3 of my INBOX size). I've tried deleting (well, renaming) the dovecot.index.* files, and re-opening my INBOX, and the new dovecot.index.cache is only 90KB so far.
Fixed in CVS. The mails weren't being marked as deleted in cache file because of a simple mistake ("!=" -> "=="). So after the change the files still won't get compressed immediately, but it'll work in future.
Thanks! I've applied the fix (I think!) to dovecot-stable-20050617 and am now using it. My dovecot.index.cache is 5MB or so at the moment, and still seems to be growing, even when I delete lots of old messages. Presumably there needs to be 20% "deleted space" before anything will happen.
Does it ever rebuild the cache from scratch in normal operation? What happens if a non-dovecot process deletes messages, do they ever get removed from the cache?
Lastly, do you have a tool or utility to analyse the cache files so I can see how many messages/headers and how much "deleted space" there is? "strings" isn't too reliable an analysis method ;)
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
On Fri, 2005-06-17 at 12:41 +0100, Chris Wakelin wrote:
Fixed in CVS. The mails weren't being marked as deleted in cache file because of a simple mistake ("!=" -> "=="). So after the change the files still won't get compressed immediately, but it'll work in future.
Thanks! I've applied the fix (I think!) to dovecot-stable-20050617 and am now using it.
I should probably backport that too..
My dovecot.index.cache is 5MB or so at the moment, and still seems to be growing, even when I delete lots of old messages. Presumably there needs to be 20% "deleted space" before anything will happen.
Right. so you'll need to grow and delete 1MB more :)
Does it ever rebuild the cache from scratch in normal operation? What happens if a non-dovecot process deletes messages, do they ever get removed from the cache?
The external process doesn't delete the mail from dovecot.index file, so Dovecot notices that it got deleted and deletes it from index and from cache.
Lastly, do you have a tool or utility to analyse the cache files so I can see how many messages/headers and how much "deleted space" there is? "strings" isn't too reliable an analysis method ;)
Not really.. Wouldn't be too difficult to implement using mail-index.h and mail-cache.h APIs though. Any volunteers? ;)
The cache file anyway heavily relies on index file. Without that there's really know way of knowing what parts of the file belong to which mails (or deleted mails).
participants (3)
-
Chris Wakelin
-
Dominic Marks
-
Timo Sirainen