[Dovecot] size of "m.XXX" in mdbox
Hi,
How to tune the size of each "m.xxx" file in mdbox format , in order to optimize the ratio size/number of files ? (I can see for instance more than 100 000 files in one directory due to very large mailbox size)
If such parameter exists, what is the typical value ?
Thanks
Joan
On 6.1.2011, at 0.07, Joan Moreau wrote:
How to tune the size of each "m.xxx" file in mdbox format , in order to optimize the ratio size/number of files ? (I can see for instance more than 100 000 files in one directory due to very large mailbox size)
mdbox_rotate_size setting sets the max size of one mdbox file. There's really no ratio setting.. It would be difficult since as more messages get added more files would have to be merged, which would generate extra disk I/O, and typically people care about disk I/O rather than the inode count.
If such parameter exists, what is the typical value ?
Default is 2 MB files.
so, why after a dsync from sdbox to mdbox, I have 100 000's of files of about 20k ? (and not 2MB !)
-rw-r--r-- 1 mailusers mail 21K 2011-01-03 02:47 m.1 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 02:47 m.10 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 02:47 m.100 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 02:51 m.1000 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10000 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100000 -rw-r--r-- 1 mailusers mail 174K 2011-01-03 09:04 m.100001 -rw-r--r-- 1 mailusers mail 175K 2011-01-03 09:04 m.100002 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100003 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100004 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100005 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 09:04 m.100006 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100007 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:04 m.100008 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100009 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10001 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:04 m.100010 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100011 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100012 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 09:05 m.100013 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:05 m.100014 -rw-r--r-- 1 mailusers mail 173K 2011-01-03 09:05 m.100015 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100016 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:05 m.100017 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100018 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100019 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10002 -rw-r--r-- 1 mailusers mail 177K 2011-01-03 09:05 m.100020 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:05 m.100021 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100022 -rw-r--r-- 1 mailusers mail 52K 2011-01-03 09:05 m.100023 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100024 -rw-r--r-- 1 mailusers mail 177K 2011-01-03 09:05 m.100025 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:05 m.100026
On Thu, 6 Jan 2011 00:15:30 +0200, Timo Sirainen wrote:
On 6.1.2011, at 0.07, Joan Moreau wrote:
How to tune the size of each "m.xxx" file in mdbox format , in order to optimize the ratio size/number of files ? (I can see for instance more than 100 000 files in one directory due to very large mailbox size) mdbox_rotate_size setting sets the max size of one mdbox file. There's really no ratio setting.. It would be difficult since as more messages get added more files would have to be merged, which would generate extra disk I/O, and typically people care about disk I/O rather than the inode count.
If such parameter exists, what is the typical value ? Default is 2 MB files.
What does "doveconf | grep mdbox" say? This isn't normal..
On 6.1.2011, at 0.33, Joan Moreau wrote:
so, why after a dsync from sdbox to mdbox, I have 100 000's of files of about 20k ? (and not 2MB !)
-rw-r--r-- 1 mailusers mail 21K 2011-01-03 02:47 m.1 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 02:47 m.10 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 02:47 m.100 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 02:51 m.1000 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10000 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100000 -rw-r--r-- 1 mailusers mail 174K 2011-01-03 09:04 m.100001 -rw-r--r-- 1 mailusers mail 175K 2011-01-03 09:04 m.100002 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100003 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100004 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100005 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 09:04 m.100006 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100007 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:04 m.100008 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100009 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10001 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:04 m.100010 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100011 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100012 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 09:05 m.100013 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:05 m.100014 -rw-r--r-- 1 mailusers mail 173K 2011-01-03 09:05 m.100015 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100016 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:05 m.100017 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100018 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100019 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10002 -rw-r--r-- 1 mailusers mail 177K 2011-01-03 09:05 m.100020 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:05 m.100021 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100022 -rw-r--r-- 1 mailusers mail 52K 2011-01-03 09:05 m.100023 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100024 -rw-r--r-- 1 mailusers mail 177K 2011-01-03 09:05 m.100025 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:05 m.100026
On Thu, 6 Jan 2011 00:15:30 +0200, Timo Sirainen wrote:
On 6.1.2011, at 0.07, Joan Moreau wrote:
How to tune the size of each "m.xxx" file in mdbox format , in order to optimize the ratio size/number of files ? (I can see for instance more than 100 000 files in one directory due to very large mailbox size) mdbox_rotate_size setting sets the max size of one mdbox file. There's really no ratio setting.. It would be difficult since as more messages get added more files would have to be merged, which would generate extra disk I/O, and typically people care about disk I/O rather than the inode count.
If such parameter exists, what is the typical value ? Default is 2 MB files.
# doveconf | grep mdbox mdbox_preallocate_space = no mdbox_rotate_interval = 0 mdbox_rotate_size = 2 M
On Thu, 6 Jan 2011 00:40:44 +0200, Timo Sirainen wrote:
What does "doveconf | grep mdbox" say? This isn't normal..
On 6.1.2011, at 0.33, Joan Moreau wrote:
so, why after a dsync from sdbox to mdbox, I have 100 000's of files of about 20k ? (and not 2MB !) -rw-r--r-- 1 mailusers mail 21K 2011-01-03 02:47 m.1 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 02:47 m.10 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 02:47 m.100 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 02:51 m.1000 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10000 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100000 -rw-r--r-- 1 mailusers mail 174K 2011-01-03 09:04 m.100001 -rw-r--r-- 1 mailusers mail 175K 2011-01-03 09:04 m.100002 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100003 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100004 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100005 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 09:04 m.100006 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100007 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:04 m.100008 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100009 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10001 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:04 m.100010 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100011 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100012 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 09:05 m.100013 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:05 m.100014 -rw-r--r-- 1 mailusers mail 173K 2011-01-03 09:05 m.100015 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100016 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:05 m.100017 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100018 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100019 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10002 -rw-r--r-- 1 mailusers mail 177K 2011-01-03 09:05 m.100020 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:05 m.100021 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100022 -rw-r--r-- 1 mailusers mail 52K 2011-01-03 09:05 m.100023 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100024 -rw-r--r-- 1 mailusers mail 177K 2011-01-03 09:05 m.100025 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:05 m.100026 On Thu, 6 Jan 2011 00:15:30 +0200, Timo Sirainen wrote:
On 6.1.2011, at 0.07, Joan Moreau wrote:
How to tune the size of each "m.xxx" file in mdbox format , in order to optimize the ratio size/number of files ? (I can see for instance more than 100 000 files in one directory due to very large mailbox size) mdbox_rotate_size setting sets the max size of one mdbox file. There's really no ratio setting.. It would be difficult since as more messages get added more files would have to be merged, which would generate extra disk I/O, and typically people care about disk I/O rather than the inode count.
If such parameter exists, what is the typical value ? Default is 2 MB files.
That is weird then.. Have you looked inside these files? Does each one have only a single mail or are there a couple? What Dovecot version? These files were created by dsync I guess? What about the last few files that (I assume) are created by dovecot-lda, are they any bigger?
On 6.1.2011, at 0.47, Joan Moreau wrote:
# doveconf | grep mdbox mdbox_preallocate_space = no mdbox_rotate_interval = 0 mdbox_rotate_size = 2 M
On Thu, 6 Jan 2011 00:40:44 +0200, Timo Sirainen wrote:
What does "doveconf | grep mdbox" say? This isn't normal..
On 6.1.2011, at 0.33, Joan Moreau wrote:
so, why after a dsync from sdbox to mdbox, I have 100 000's of files of about 20k ? (and not 2MB !) -rw-r--r-- 1 mailusers mail 21K 2011-01-03 02:47 m.1 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 02:47 m.10 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 02:47 m.100 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 02:51 m.1000 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10000 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100000 -rw-r--r-- 1 mailusers mail 174K 2011-01-03 09:04 m.100001 -rw-r--r-- 1 mailusers mail 175K 2011-01-03 09:04 m.100002 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100003 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100004 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100005 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 09:04 m.100006 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100007 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:04 m.100008 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100009 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10001 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:04 m.100010 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100011 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100012 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 09:05 m.100013 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:05 m.100014 -rw-r--r-- 1 mailusers mail 173K 2011-01-03 09:05 m.100015 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100016 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:05 m.100017 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100018 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100019 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10002 -rw-r--r-- 1 mailusers mail 177K 2011-01-03 09:05 m.100020 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:05 m.100021 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100022 -rw-r--r-- 1 mailusers mail 52K 2011-01-03 09:05 m.100023 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100024 -rw-r--r-- 1 mailusers mail 177K 2011-01-03 09:05 m.100025 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:05 m.100026 On Thu, 6 Jan 2011 00:15:30 +0200, Timo Sirainen wrote:
On 6.1.2011, at 0.07, Joan Moreau wrote:
How to tune the size of each "m.xxx" file in mdbox format , in order to optimize the ratio size/number of files ? (I can see for instance more than 100 000 files in one directory due to very large mailbox size) mdbox_rotate_size setting sets the max size of one mdbox file. There's really no ratio setting.. It would be difficult since as more messages get added more files would have to be merged, which would generate extra disk I/O, and typically people care about disk I/O rather than the inode count.
If such parameter exists, what is the typical value ? Default is 2 MB files.
It looks like each of them has one or more messages.
Looks like if the msg is too big, (i.e. above the 20k or so), there is only one message in the m.xxx
regarding lda, my "mdbox" folders are just a mirror via dsync, no lda incoming in the mdbox folders
On Thu, 6 Jan 2011 00:51:00 +0200, Timo Sirainen wrote:
That is weird then.. Have you looked inside these files? Does each one have only a single mail or are there a couple? What Dovecot version? These files were created by dsync I guess? What about the last few files that (I assume) are created by dovecot-lda, are they any bigger?
On 6.1.2011, at 0.47, Joan Moreau wrote:
# doveconf | grep mdbox mdbox_preallocate_space = no mdbox_rotate_interval = 0 mdbox_rotate_size = 2 M On Thu, 6 Jan 2011 00:40:44 +0200, Timo Sirainen wrote:
What does "doveconf | grep mdbox" say? This isn't normal.. On 6.1.2011, at 0.33, Joan Moreau wrote:
so, why after a dsync from sdbox to mdbox, I have 100 000's
of files of about 20k ? (and not 2MB !) -rw-r--r-- 1 mailusers mail 21K 2011-01-03 02:47 m.1 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 02:47 m.10 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 02:47 m.100 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 02:51 m.1000 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10000 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100000 -rw-r--r-- 1 mailusers mail 174K 2011-01-03 09:04 m.100001 -rw-r--r-- 1 mailusers mail 175K 2011-01-03 09:04 m.100002 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100003 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100004 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100005 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 09:04 m.100006 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100007 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:04 m.100008 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100009 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10001 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:04 m.100010 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:04 m.100011 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100012 -rw-r--r-- 1 mailusers mail 17K 2011-01-03 09:05 m.100013 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:05 m.100014 -rw-r--r-- 1 mailusers mail 173K 2011-01-03 09:05 m.100015 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100016 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:05 m.100017 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100018 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100019 -rw-r--r-- 1 mailusers mail 14K 2011-01-03 03:35 m.10002 -rw-r--r-- 1 mailusers mail 177K 2011-01-03 09:05 m.100020 -rw-r--r-- 1 mailusers mail 20K 2011-01-03 09:05 m.100021 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100022 -rw-r--r-- 1 mailusers mail 52K 2011-01-03 09:05 m.100023 -rw-r--r-- 1 mailusers mail 18K 2011-01-03 09:05 m.100024 -rw-r--r-- 1 mailusers mail 177K 2011-01-03 09:05 m.100025 -rw-r--r-- 1 mailusers mail 19K 2011-01-03 09:05 m.100026 On Thu, 6 Jan 2011 00:15:30 +0200, Timo Sirainen wrote:
On 6.1.2011, at 0.07, Joan Moreau wrote:
How
mdbox_rotate_size setting sets the max size of one mdbox file. There's really no ratio setting.. It would be difficult since as more messages get added more files would have to be merged, which would generate extra disk I/O, and typically people care about disk I/O rather
If such parameter exists, what is
to tune the size of each "m.xxx" file in mdbox format , in order to optimize the ratio size/number of files ? (I can see for instance more than 100 000 files in one directory due to very large mailbox size) than the inode count. the typical value ?
Default is 2 MB files.
On 05/01/2011 22:47, Joan Moreau wrote:
# doveconf | grep mdbox mdbox_preallocate_space = no mdbox_rotate_interval = 0 mdbox_rotate_size = 2 M
Is the space between the '2' and the 'M' significant?
Only in my configuration file it says "2M" with no space.
I was wondering if the space might make it stop parsing the value and make it interpret as 2 bytes?
Bill
participants (4)
-
Charles Marcus
-
Joan Moreau
-
Timo Sirainen
-
William Blunn