[Dovecot] Maildir messages
Hi all,
Hopefully, I'm sending this to the right place...
Can someone guide me on changing the Maildir messages files
back to the single file format I used to have instead of the
dovecot.index /cur /tmp /new format my newer server currently
has. I'm running dovecot v1.2.9 postfix v2.7 on Ubuntu v10.4
Thanks,
Nick. . . . .
MU/Webcasting-- _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ . . . . . . . . Please join my Tech MeetUp group that meets in the NYC metro area Webcasting and Video Streaming Network: http://TheWebcasting.net . . . . . . . . _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/
Am 12.07.2012 08:12, schrieb Nick@5sg.eu:
Hi all, Hopefully, I'm sending this to the right place... Can someone guide me on changing the Maildir messages files back to the single file format I used to have instead of the dovecot.index /cur /tmp /new format my newer server currently has. I'm running dovecot v1.2.9 postfix v2.7 on Ubuntu v10.4 Thanks, Nick. . . . . MU/Webcasting-- _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ . . . . . . . . Please join my Tech MeetUp group that meets in the NYC metro area Webcasting and Video Streaming Network: http://TheWebcasting.net . . . . . . . . _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/
perhaps look here
http://wiki2.dovecot.org/MailLocation
-- Best Regards MfG Robert Schetterer
Hopefully, I'm sending this to the right place... Can someone guide me on changing the Maildir messages filesback to the single file format I used to have instead of the dovecot.index /cur /tmp /new format my newer server currently has. I'm running dovecot v1.2.9 postfix v2.7 on Ubuntu v10.4
i don't really understand why you want to do it, single file mailboxes are plain inefficient.
maybe - just as in article below
http://www.bishnet.net/tim/blog/tag/maildir-to-mbox/
you like archive in such format
On 7/12/2012 5:32 AM, Wojciech Puchar wrote:
i don't really understand why you want to do it, single file mailboxes are plain inefficient.
mbox is more efficient than maildir for many common operations, specifically from a filesystem IO perspective:
- new mail delivery
- searching a mailbox folder
- mass deletion of emails from one mailbox folder
maildir is more IO efficient when reading and deleting individual emails.
mbox puts the load on the mail server application and on memory. maildir puts the load on the IO subsysetem. Which is precisely why Timo created the mdbox mail storage format, attempting to get the best of both worlds.
-- Stan
specifically from a filesystem IO perspective:
- new mail delivery
not much difference.
- searching a mailbox folder
if you do linear search then yes
- mass deletion of emails from one mailbox folder
3 is not true.
maildir is more IO efficient when reading and deleting individual emails.
and making backups. deleting from maildir means just delete a file, not shuffle data.
Everyone needs to make backups while it is unlinkely that anyone like every backup to be effectively full backup.
mbox puts the load on the mail server application and on memory.
and on I/O too - often quite a bit
maildir puts the load on the IO subsysetem. Which is precisely why Timo created the mdbox mail storage format, attempting to get the best of both worlds. And this is great idea and actually works :)
mbox may make sense for archive storage. you create archive folder once and never modify anything
On 7/13/2012 4:09 AM, Wojciech Puchar wrote:
specifically from a filesystem IO perspective:
- new mail delivery
not much difference.
maildir requires 3 (or is it 4?) metadata operations and a file write op
mbox requires a single file append operation.
- searching a mailbox folder
if you do linear search then yes
I'm referring to full text body search. In which case every single file in a maildir directory must be opened and searched in succession, serially, as Dovecot doesn't create a search thread for each maildir file and perform them in parallel across multiple cores.
With mbox, you open a single file and search it. CPU/RAM bandwidth is many orders of magnitude faster and cheaper than disk IOPS. Thus mbox is faster at full text search than any other mailbox storage format, period. Full text indexes can help greatly with both formats, but often the indexes are old, and a full search is necessary, making mbox much faster.
- mass deletion of emails from one mailbox folder
3 is not true.
It most certainly is true. You need to read up on how email deletes are performed on mbox files, or mdbox files for that matter.
maildir is more IO efficient when reading and deleting individual emails.
and making backups.
Wrong again. Streaming a single file to D2D or tape is much faster than random reading hundreds or thousands of maildir files.
deleting from maildir means just delete a file, not shuffle data.
For a single email delete operation maildir is faster, as it requires a single metadata IO. When deleting many emails, say hundreds to thousands, as in deleting a very large folder, mbox is *much* faster. This is because CPU/mem are many orders of magnitude faster than disk, and deleting hundreds or thousands of maildir files requires hundreds or thousands of random metadata IOPS to the filesystem directory.
When Qmail hit the scene with maildir format, everyone loved it. That is, until their mailbox counts skyrocketed, and their systems slow to a crawl because their disk arrays simply couldn't keep up with all the IOPS.
Everyone needs to make backups while it is unlinkely that anyone like every backup to be effectively full backup.
See: rdiff-backup, et al
mbox puts the load on the mail server application and on memory.
and on I/O too - often quite a bit
Sure, if it's a busy server. But the IOPS load will always be much less than maildir given the same workload.
maildir puts the load on the IO subsysetem. Which is precisely why Timo created the mdbox mail storage format, attempting to get the best of both worlds.
And this is great idea and actually works :)
Yep.
mbox may make sense for archive storage. you create archive folder once and never modify anything
Many of us still use mbox for IMAP and POP user accounts, and it still works great. And many maildir converts switched back to mbox when the storage hardware required to satisfy their ever increasing maildir IOPS load began draining their entire IT budgets.
mbox is a pretty smart email storage format especially given its age. It can do more with lesser storage hardware. Many simply don't give it the credit it deserves.
-- Stan
On 2012-07-13 8:07 AM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
mbox is a pretty smart email storage format especially given its age. It can do more with lesser storage hardware. Many simply don't give it the credit it deserves.
The ideal is a hybrid - maildir for new mail, mbox (or mdbox) for longer term storage...
--
Best regards,
Charles
Am 13.07.2012 14:19, schrieb Charles Marcus:
On 2012-07-13 8:07 AM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
mbox is a pretty smart email storage format especially given its age. It can do more with lesser storage hardware. Many simply don't give it the credit it deserves.
The ideal is a hybrid - maildir for new mail, mbox (or mdbox) for longer term storage...
may I ask how you do archivate maildirs? With mbox I used mhonarc but with maildir I do not really have a soluton yet.
Andreas
the credit it deserves.
The ideal is a hybrid - maildir for new mail, mbox (or mdbox) for longer term storage...
True. Not did this as never got I/O problems because of mail.
For larger installations in may be adventageous to use mbox based archives on RAID5 (this would be mostly write-once) and RAID1 for rest.
not much difference.
maildir requires 3 (or is it 4?) metadata operations and a file write op
your remarks are mostly true, except concentrates too much on uncommon situations and uncommon strange case of probably tens of thousands of mail accounts that must be very limited.
From my practice mail accounts are at least 1GB large, often 10GB. more often than not other user data is on the same disk.
i've never had more than 50 users on single disk set (which is always 2 disks in mirror), And I/O was never ever a problem except first index build etc. in dovecot 2 you can do this in controller way using doveadm.
Mail I/O doesn't make that much part of I/O for me, still i found a bit speedup by moving dovecot indexes to SSD if it exist.
while disks get larger user data amount get larger too.
Today tape backups are hardly used because it is plain uneconomic today.
most do backups by copying differences to other disk, preferably in other place.
Adventages of pure Maildir format outweights everything else as you have one file -> one mail.
mdbox is something in between.
On 7/13/2012 7:28 AM, Wojciech Puchar wrote:
not much difference.
maildir requires 3 (or is it 4?) metadata operations and a file write op
your remarks are mostly true, except concentrates too much on uncommon situations and uncommon strange case of probably tens of thousands of mail accounts that must be very limited.
There are many folks on this list who ran out of IOPS due to maildir at one time or another. Nowhere near tens of thousands of accounts were not required. To solve the problem, they either purchased more spindles, added RAID cache, or converted to mbox, or mdbox, depending on when they ran into the problem (mdbox is very young).
From my practice mail accounts are at least 1GB large, often 10GB. more often than not other user data is on the same disk.
Typical SMB setup it seems.
i've never had more than 50 users on single disk set (which is always 2 disks in mirror), And I/O was never ever a problem except first index build etc. in dovecot 2 you can do this in controller way using doveadm.
Mail I/O doesn't make that much part of I/O for me, still i found a bit speedup by moving dovecot indexes to SSD if it exist.
With only 50 users it's hard to saturate one disk with a mail workload.
while disks get larger user data amount get larger too.
Today tape backups are hardly used because it is plain uneconomic today.
Tape is far more cost effective than a GbE MAN link to a hot site.
most do backups by copying differences to other disk, preferably in other place.
Again, those links are usually more costly than off site tape rotation services, and the backup window much larger.
Adventages of pure Maildir format outweights everything else as you have one file -> one mail.
While that is maildir's greatest strength, it is also it's greatest weakness.
-- Stan
mail accounts that must be very limited.
There are many folks on this list who ran out of IOPS due to maildir at one time or another. Nowhere near tens of thousands of accounts were not required. To solve the problem, they either purchased more
did you actually monitored WHAT part of job generates I/O and how?
Can you post details. On just one simple
- how many mailboxes per disk head
- what filesystem used
- what OS
- how data is laid out.
spindles, added RAID cache, or converted to mbox, or mdbox, depending on when they ran into the problem (mdbox is very young).
From my practice mail accounts are at least 1GB large, often 10GB. more often than not other user data is on the same disk.
Typical SMB setup it seems.
True. More - i always try to mix services. No separate "file servers", "mail servers", "whatever" servers but all on same. On large install divide by groups of people, not services.
This way i really cannot get to high IOPS.
Mail I/O doesn't make that much part of I/O for me, still i found a bit speedup by moving dovecot indexes to SSD if it exist.
With only 50 users it's hard to saturate one disk with a mail workload.
True! That is my point.
With user taking few GBs of mail space (few tens not uncommon), as much or more "documents" or other data, you just will never have more than 50 of then on one disk.
More users=more disks, same (small) amount of I/O per disk head.
while disks get larger user data amount get larger too.
Today tape backups are hardly used because it is plain uneconomic today.
Tape is far more cost effective than a GbE MAN link to a hot site.
I really cannot imagine a scale of operation that would require gigabit link to do backups over rsync that only moves changes!
It would be like 10000 users on like 100 disks.
With that site i would think of tape.
Of course - no I/O problem to backup such 100 disk setup to tape.
Again, those links are usually more costly than off site tape rotation services, and the backup window much larger.
my practice shows exactly opposite case.
If you would be right i would already use tapes or even pseudo-tapes== few cheapers hard drives connected over eSATA.
Adventages of pure Maildir format outweights everything else as you have one file -> one mail.
While that is maildir's greatest strength, it is also it's greatest weakness.
For my needs strength is great and weakness is small.
mdbox may help in your cases.
Seems like you do same ill service like 1000 (or more) mail accounts per hard disks.
My point was always "never delete old data, disks are cheap".
Am 13.07.2012 20:45, schrieb Wojciech Puchar:
Typical SMB setup it seems.
True. More - i always try to mix services. No separate "file servers", "mail servers", "whatever" servers but all on same. On large install divide by groups of people, not services.
This way i really cannot get to high IOPS
"cannot" is plain wrong, you are satisfied only by luck
Seems like you do same ill service like 1000 (or more) mail accounts per hard disks
there is nothing ill
http://h18000.www1.hp.com/products/quickspecs/13551_na/13551_na.pdf 1 Gb Ethernet / iSCSI -> 17200 Random IOPS with RAID10 1 Gb Ethernet / iSCSI -> 9400 Random IOPS with RAID5 1 Gb Ethernet / iSCSI -> 5600 Random IOPS with RAID6
and here ware are at a point where your local disks are outperformed because they give you typically 100-300 IOPS per disk which you refused over a complete thread believing you localSATA is the best thing whichs exists
and yes, the dedicated storage-controllers and caches of a SAN storage are what is outperforming you at real load
participants (7)
-
anmeyer＠anup.de
-
Charles Marcus
-
Nick＠5sg.eu
-
Reindl Harald
-
Robert Schetterer
-
Stan Hoeppner
-
Wojciech Puchar