Disk space usage with mdbox
Hello,
I am using dovecot with mdbox+sis and I notice an ever increasing disk space usage since I converted mailboxes from Maildir to mdboxes. I have checked with a user and while it actually uses only 65K, his mdbox files on disk are more than 6G. The backup of his mailbox is just 64K. Any suggestions ?
Below are some relevant commands:
[root@mail ~]# doveadm quota get -u test Quota name Type Value Limit % User quota STORAGE 10135 31457280 0 User quota MESSAGE 186
0
[root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm purge -u test [root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm backup -u test mdbox:/tmp/MDBOX_TEMP/ [root@mail ~]# du -k --max-depth=1 /tmp/MDBOX_TEMP/ 16 /tmp/MDBOX_TEMP/mailboxes 65540 /tmp/MDBOX_TEMP/storage 65568 /tmp/MDBOX_TEMP/
Did you "purged" the deleted mails for this user? On mdbox you must run "doveadm purge -u $USER" to whipe out any as deleted marked mails etc.
Details: http://wiki2.dovecot.org/Tools/Doveadm/Purge
I use a nightly cronjob wor this.
Regards Urban
Am 01.04.2015 um 23:26 schrieb Alexandros Soumplis:
Hello,
I am using dovecot with mdbox+sis and I notice an ever increasing disk space usage since I converted mailboxes from Maildir to mdboxes. I have checked with a user and while it actually uses only 65K, his mdbox files on disk are more than 6G. The backup of his mailbox is just 64K. Any suggestions ?
Below are some relevant commands:
[root@mail ~]# doveadm quota get -u test Quota name Type Value Limit % User quota STORAGE 10135 31457280 0 User quota MESSAGE 186 - 0
[root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm purge -u test [root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm backup -u test mdbox:/tmp/MDBOX_TEMP/ [root@mail ~]# du -k --max-depth=1 /tmp/MDBOX_TEMP/ 16 /tmp/MDBOX_TEMP/mailboxes 65540 /tmp/MDBOX_TEMP/storage 65568 /tmp/MDBOX_TEMP/
Yes I have done it, it is the third command I have run, without any change.
a.
On 02/04/2015 09:53 πμ, Urban Loesch wrote:
Did you "purged" the deleted mails for this user? On mdbox you must run "doveadm purge -u $USER" to whipe out any as deleted marked mails etc.
Details: http://wiki2.dovecot.org/Tools/Doveadm/Purge
I use a nightly cronjob wor this.
Regards Urban
Am 01.04.2015 um 23:26 schrieb Alexandros Soumplis:
Hello,
I am using dovecot with mdbox+sis and I notice an ever increasing disk space usage since I converted mailboxes from Maildir to mdboxes. I have checked with a user and while it actually uses only 65K, his mdbox files on disk are more than 6G. The backup of his mailbox is just 64K. Any suggestions ?
Below are some relevant commands:
[root@mail ~]# doveadm quota get -u test Quota name Type Value Limit % User quota STORAGE 10135 31457280 0 User quota MESSAGE 186 - 0
[root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm purge -u test [root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm backup -u test mdbox:/tmp/MDBOX_TEMP/ [root@mail ~]# du -k --max-depth=1 /tmp/MDBOX_TEMP/ 16 /tmp/MDBOX_TEMP/mailboxes 65540 /tmp/MDBOX_TEMP/storage 65568 /tmp/MDBOX_TEMP/
-- Alexandros Soumplis Systems Engineer (M.Sc, RHCE, VCP, SCSA)
National Documentation Center / N.H.R.F. Information Systems Department 48, Vas. Constantinou Av. 116 35 Athens, Greece phone: +30 210 7273984 fax: +30 210 7252223 email: soumplis@ekt.gr http://www.ekt.gr
Nobody any suggestions ?
On 02/04/15 10:09, Alexandros Soumplis wrote:
Yes I have done it, it is the third command I have run, without any change.
a.
On 02/04/2015 09:53 πμ, Urban Loesch wrote:
Did you "purged" the deleted mails for this user? On mdbox you must run "doveadm purge -u $USER" to whipe out any as deleted marked mails etc.
Details: http://wiki2.dovecot.org/Tools/Doveadm/Purge
I use a nightly cronjob wor this.
Regards Urban
Am 01.04.2015 um 23:26 schrieb Alexandros Soumplis:
Hello,
I am using dovecot with mdbox+sis and I notice an ever increasing disk space usage since I converted mailboxes from Maildir to mdboxes. I have checked with a user and while it actually uses only 65K, his mdbox files on disk are more than 6G. The backup of his mailbox is just 64K. Any suggestions ?
Below are some relevant commands:
[root@mail ~]# doveadm quota get -u test Quota name Type Value Limit % User quota STORAGE 10135 31457280 0 User quota MESSAGE 186
0[root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm purge -u test [root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm backup -u test mdbox:/tmp/MDBOX_TEMP/ [root@mail ~]# du -k --max-depth=1 /tmp/MDBOX_TEMP/ 16 /tmp/MDBOX_TEMP/mailboxes 65540 /tmp/MDBOX_TEMP/storage 65568 /tmp/MDBOX_TEMP/
Shot in the dark here...haven't tried it myself so no promises. There's probably a much better way to do this but sometimes a little "brute force" helps.
Theoretical steps:
- Stop all mail processes - both SMTP and IMAP.
- Use "doveadm sync" to backup the user mailbox, e.g., doveadm backup -u user mdbox:/tmp/usermail
- Verify /tmp/usermail looks reasonable
- Move the user's mail store somewhere else, e.g. /tmp/useroldmail
- Move /tmp/usermail to the user's mailstore
- Verify file permissions are set correctly in the mailstore
- Start the mail processes...and test.
-- Daniel
On 4/3/2015 1:04 PM, Alexandros Soumplis wrote:
Nobody any suggestions ?
On 02/04/15 10:09, Alexandros Soumplis wrote:
Yes I have done it, it is the third command I have run, without any change.
a.
On 02/04/2015 09:53 πμ, Urban Loesch wrote:
Did you "purged" the deleted mails for this user? On mdbox you must run "doveadm purge -u $USER" to whipe out any as deleted marked mails etc.
Details: http://wiki2.dovecot.org/Tools/Doveadm/Purge
I use a nightly cronjob wor this.
Regards Urban
Am 01.04.2015 um 23:26 schrieb Alexandros Soumplis:
Hello,
I am using dovecot with mdbox+sis and I notice an ever increasing disk space usage since I converted mailboxes from Maildir to mdboxes. I have checked with a user and while it actually uses only 65K, his mdbox files on disk are more than 6G. The backup of his mailbox is just 64K. Any suggestions ?
Below are some relevant commands:
[root@mail ~]# doveadm quota get -u test Quota name Type Value Limit % User quota STORAGE 10135 31457280 0 User quota MESSAGE 186
0[root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm purge -u test [root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm backup -u test mdbox:/tmp/MDBOX_TEMP/ [root@mail ~]# du -k --max-depth=1 /tmp/MDBOX_TEMP/ 16 /tmp/MDBOX_TEMP/mailboxes 65540 /tmp/MDBOX_TEMP/storage 65568 /tmp/MDBOX_TEMP/
Daniel thank you for the advice, I have done this exact procedure to clean up some space and I reduced the used space by more than 60% (huge gain!). Yet it is still a mystery why this happened first place and more important if it will occur again, time will tell. I have kept one mailbox out of this procedure (the "live" size is 8,8G, the backup size is less than 1GB) to do some testing or debugging if possible.
Another question I am not able to answer is if this procedure cleans up potentialy orphaned attachments as I store them seperately for SiS to work.
a.
On 04/04/15 02:40, Daniel Miller wrote:
Shot in the dark here...haven't tried it myself so no promises. There's probably a much better way to do this but sometimes a little "brute force" helps.
Theoretical steps:
- Stop all mail processes - both SMTP and IMAP.
- Use "doveadm sync" to backup the user mailbox, e.g., doveadm backup -u user mdbox:/tmp/usermail
- Verify /tmp/usermail looks reasonable
- Move the user's mail store somewhere else, e.g. /tmp/useroldmail
- Move /tmp/usermail to the user's mailstore
- Verify file permissions are set correctly in the mailstore
- Start the mail processes...and test.
-- Daniel
On 4/3/2015 1:04 PM, Alexandros Soumplis wrote:
Nobody any suggestions ?
On 02/04/15 10:09, Alexandros Soumplis wrote:
Yes I have done it, it is the third command I have run, without any change.
a.
On 02/04/2015 09:53 πμ, Urban Loesch wrote:
Did you "purged" the deleted mails for this user? On mdbox you must run "doveadm purge -u $USER" to whipe out any as deleted marked mails etc.
Details: http://wiki2.dovecot.org/Tools/Doveadm/Purge
I use a nightly cronjob wor this.
Regards Urban
Am 01.04.2015 um 23:26 schrieb Alexandros Soumplis:
Hello,
I am using dovecot with mdbox+sis and I notice an ever increasing disk space usage since I converted mailboxes from Maildir to mdboxes. I have checked with a user and while it actually uses only 65K, his mdbox files on disk are more than 6G. The backup of his mailbox is just 64K. Any suggestions ?
Below are some relevant commands:
[root@mail ~]# doveadm quota get -u test Quota name Type Value Limit % User quota STORAGE 10135 31457280 0 User quota MESSAGE 186 - 0
[root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm purge -u test [root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm backup -u test mdbox:/tmp/MDBOX_TEMP/ [root@mail ~]# du -k --max-depth=1 /tmp/MDBOX_TEMP/ 16 /tmp/MDBOX_TEMP/mailboxes 65540 /tmp/MDBOX_TEMP/storage 65568 /tmp/MDBOX_TEMP/
Why things happen is one of the great mysteries and joys of being a sysadmin...
As for cleaning up SIS, I believe the "doveadm purge" command takes care of it. I have a daily cronjob which includes:
#!/bin/sh # Permanently expunge old Trash doveadm expunge -A mailbox Trash savedbefore 2w # Optimize mdbox storage doveadm purge -A # Update fts indexes doveadm fts rescan -A
-- Daniel
On 4/5/2015 1:14 PM, Alexandros Soumplis wrote:
Daniel thank you for the advice, I have done this exact procedure to clean up some space and I reduced the used space by more than 60% (huge gain!). Yet it is still a mystery why this happened first place and more important if it will occur again, time will tell. I have kept one mailbox out of this procedure (the "live" size is 8,8G, the backup size is less than 1GB) to do some testing or debugging if possible.
Another question I am not able to answer is if this procedure cleans up potentialy orphaned attachments as I store them seperately for SiS to work.
a.
On 04/04/15 02:40, Daniel Miller wrote:
Shot in the dark here...haven't tried it myself so no promises. There's probably a much better way to do this but sometimes a little "brute force" helps.
Theoretical steps:
- Stop all mail processes - both SMTP and IMAP.
- Use "doveadm sync" to backup the user mailbox, e.g., doveadm backup -u user mdbox:/tmp/usermail
- Verify /tmp/usermail looks reasonable
- Move the user's mail store somewhere else, e.g. /tmp/useroldmail
- Move /tmp/usermail to the user's mailstore
- Verify file permissions are set correctly in the mailstore
- Start the mail processes...and test.
-- Daniel
On 4/3/2015 1:04 PM, Alexandros Soumplis wrote:
Nobody any suggestions ?
On 02/04/15 10:09, Alexandros Soumplis wrote:
Yes I have done it, it is the third command I have run, without any change.
a.
On 02/04/2015 09:53 πμ, Urban Loesch wrote:
Did you "purged" the deleted mails for this user? On mdbox you must run "doveadm purge -u $USER" to whipe out any as deleted marked mails etc.
Details: http://wiki2.dovecot.org/Tools/Doveadm/Purge
I use a nightly cronjob wor this.
Regards Urban
Am 01.04.2015 um 23:26 schrieb Alexandros Soumplis:
Hello,
I am using dovecot with mdbox+sis and I notice an ever increasing disk space usage since I converted mailboxes from Maildir to mdboxes. I have checked with a user and while it actually uses only 65K, his mdbox files on disk are more than 6G. The backup of his mailbox is just 64K. Any suggestions ?
Below are some relevant commands:
[root@mail ~]# doveadm quota get -u test Quota name Type Value Limit % User quota STORAGE 10135 31457280 0 User quota MESSAGE 186 - 0
[root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm purge -u test [root@mail ~]# du -k --max-depth=1 /mdboxes/test/ 220 /mdboxes/test/mailboxes 6029348 /mdboxes/test/storage 6029592 /mdboxes/test/
[root@mail ~]# doveadm backup -u test mdbox:/tmp/MDBOX_TEMP/ [root@mail ~]# du -k --max-depth=1 /tmp/MDBOX_TEMP/ 16 /tmp/MDBOX_TEMP/mailboxes 65540 /tmp/MDBOX_TEMP/storage 65568 /tmp/MDBOX_TEMP/
On 10 Apr 2015, at 21:55, Daniel Miller <dmiller@amfes.com> wrote:
Why things happen is one of the great mysteries and joys of being a sysadmin...
As for cleaning up SIS, I believe the "doveadm purge" command takes care of it. I have a daily cronjob which includes:
#!/bin/sh # Permanently expunge old Trash doveadm expunge -A mailbox Trash savedbefore 2w # Optimize mdbox storage doveadm purge -A # Update fts indexes doveadm fts rescan -A
doveadm index updates the indexes. doveadm fts rescan makes sure that after something has broken it makes sure to update all the mails to the index after indexing is done - so it by itself doesn't even add any missing mails to the index.(And with fts-lucene it's actually looking if there are any mails that were supposed to have been deleted already - with Solr it should do that too but the code isn't written yet. So I don't think that needs to be run nightly (if ever automatically).
So with "fts_autoindex=yes" enabled in plugins, is there anything I should be running regularly, other than the expunge/purge, to ensure Solr indexes are updated? I have a daily Solr optimize, and Solr autocommit enabled.
I have an hourly "doveadm index -A *" command - is that unneeded?
-- Daniel
On 4/12/2015 6:53 AM, Timo Sirainen wrote:
On 10 Apr 2015, at 21:55, Daniel Miller <dmiller@amfes.com> wrote:
Why things happen is one of the great mysteries and joys of being a sysadmin...
As for cleaning up SIS, I believe the "doveadm purge" command takes care of it. I have a daily cronjob which includes:
#!/bin/sh # Permanently expunge old Trash doveadm expunge -A mailbox Trash savedbefore 2w # Optimize mdbox storage doveadm purge -A # Update fts indexes doveadm fts rescan -A doveadm index updates the indexes. doveadm fts rescan makes sure that after something has broken it makes sure to update all the mails to the index after indexing is done - so it by itself doesn't even add any missing mails to the index.(And with fts-lucene it's actually looking if there are any mails that were supposed to have been deleted already - with Solr it should do that too but the code isn't written yet. So I don't think that needs to be run nightly (if ever automatically).
On 14 Apr 2015, at 19:11, Daniel Miller <dmiller@amfes.com> wrote:
So with "fts_autoindex=yes" enabled in plugins, is there anything I should be running regularly, other than the expunge/purge, to ensure Solr indexes are updated? I have a daily Solr optimize, and Solr autocommit enabled.
Should be enough.
I have an hourly "doveadm index -A *" command - is that unneeded?
Shouldn't be needed.
-- Daniel
On 4/12/2015 6:53 AM, Timo Sirainen wrote:
On 10 Apr 2015, at 21:55, Daniel Miller <dmiller@amfes.com> wrote:
Why things happen is one of the great mysteries and joys of being a sysadmin...
As for cleaning up SIS, I believe the "doveadm purge" command takes care of it. I have a daily cronjob which includes:
#!/bin/sh # Permanently expunge old Trash doveadm expunge -A mailbox Trash savedbefore 2w # Optimize mdbox storage doveadm purge -A # Update fts indexes doveadm fts rescan -A doveadm index updates the indexes. doveadm fts rescan makes sure that after something has broken it makes sure to update all the mails to the index after indexing is done - so it by itself doesn't even add any missing mails to the index.(And with fts-lucene it's actually looking if there are any mails that were supposed to have been deleted already - with Solr it should do that too but the code isn't written yet. So I don't think that needs to be run nightly (if ever automatically).
participants (4)
-
Alexandros Soumplis
-
Daniel Miller
-
Timo Sirainen
-
Urban Loesch