[Dovecot] mail spool filesystem
Hi!
I´m about to migrate a system whith 5000 accounts whith (~ 500GB) from "postfix/courier-imap/maildrop/mysql" to a new hardware whith "postfix/dovecot/dovecot/mysql".
I´ll make a separate partition (raid 1) for the mail spool (/var/spool/vmail) and want to now what type of filesystem to use on it to increase performance. I read that XFS is a good choice, but is not too reliable...
Any sugestions?
Thanks in advance,
--
_ Julio Cesar Covolato
0v0<julio@psi.com.br>/(_)\ F: 55-11-3129-3366 ^ ^ PSI INTERNET
On Wed, 17 Aug 2011 10:23:24 -0300, Julio Cesar Covolato wrote:
I´ll make a separate partition (raid 1) for the mail spool (/var/spool/vmail) and want to now what type of filesystem to use on it to increase performance. I read that XFS is a good choice, but is not too reliable...
XFS is reliable.
I recommend ext4 or xfs. Both are very good reliable filesystems.
You should rather worry about the mailboxformat. The performance difference is much greater than xfs or ext4.
Best regards,
Morten
I read that XFS is a good choice, but is not too reliable...
Are you using Maildir or MBOX?
In any case: XFS would be my last choice:
XFS is nice if you are working with large files (> 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files.
If performance is a problem (iostat output?): Mount it with 'noatime' or/and use a special device for the journal.
Regards, Adrian
Adrian Ulrich adrian@blinkenlights.ch wrote:
I read that XFS is a good choice, but is not too reliable...
Are you using Maildir or MBOX?
In any case: XFS would be my last choice:
XFS is nice if you are working with large files (> 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files.
I don't know, but my Courier server (Maildir++ format, 10k users, 12 million mails on server, some users with over 20k mails in one directory/folder) runs _way_ smoother with XFS. I also tested ext4, but this was no large gain over ext3 (with dir_index of course).
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 8/17/2011 9:42 AM, Adrian Ulrich wrote:
I read that XFS is a good choice, but is not too reliable...
Are you using Maildir or MBOX?
In any case: XFS would be my last choice:
XFS is nice if you are working with large files (> 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files.
XFS was designed for parallelism, whether with large files or small, though it has been optimized a bit more for large file throughput. In yet another attempt to dispel the XFS "small file problem" myth, XFS has never had a performance problem with "small" files. In the past XFS did have a performance problem with large metadata operations due to the way the delayed allocation had been designed. The perennial example of this was the horrible unlink performance when whacking a kernel tree with 'rm -rf'. It used to take forever, multiple tens of times slower than Reiser or EXT. This metadata bottleneck in the delayed allocation path was largely resolved by Dave Chinner's delayed logging patch which was experimental in 2.6.35 and is enabled by default in 2.6.39 and later. XFS metadata performance is now on par with that of EXT3/4.
Because of this, and XFS' use of allocation groups, today, for a busy IMAP server with lots of maildir mailboxen, one of the highest performance storage stack setups is the following:
- A dozen or more hardware or software RAID1 mirrors
- A linear concat over the mirrors
- XFS with 2*num_mirrors allocation groups, mounted with 'inode64'
- maildir mailboxes
This setup will give you significantly higher real IOPS than any striped array setup with any filesystem atop, for a couple of reasons:
No partial stripe width writes, and no unnecessary full stripe reads. All reads and writes match the page size and filesystem block size of 4KB.
In the example above, you have two AGs per mirror pair, 24 total AGs on 12 mirrors. The first two maildir directories will be created in AGs 1 and 2 on the first mirror. The second two in AGs 3 & 4 on the 2nd mirror pair, and so on. The 25th/26th directories will 'wrap' back to AGs 1 & 2 and the directory creation pattern will continue.
Because of its allocation group design XFS is the only filesystem that can accomplish this level of parallelism with a concatenated array and small email files. All others must rely on striped arrays, either RAID10 or 5/6. These come with the inefficiencies of writing/reading files as small as 2KB on a stripe ranging from 256KB-1MB or larger, depending on the number of disks in the array and the chosen stripe size. If you have a high write load, the Linux allocator will pack multiple files into a single stripe, but one rarely sees 100% efficiency here. Even at 100% on writes, at low read rates, you end up reading a lot of full 256KB-1MB stripes just to get a 2KB file, wasting bandwidth and filling up the buffer cache with unneeded data, not to mention any read cache on your hardware RAID controller or SAN head.
The only potential downside to this setup is the rare situation where your current logged in users all have their mailbox in the same AG or two AGs on the same spindle. I've yet to see this happen, though it is a theoretical possibility, though the probability is extremely low.
-- Stan
On Fri, Aug 19, 2011 at 03:48:00AM -0500, Stan Hoeppner wrote:
On 8/17/2011 9:42 AM, Adrian Ulrich wrote:
I read that XFS is a good choice, but is not too reliable...
Are you using Maildir or MBOX?
In any case: XFS would be my last choice:
XFS is nice if you are working with large files (> 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files.
XFS was designed for parallelism, whether with large files or small, ...
Anyone been using ZFS on FreeBSD for mail spool storage?
-- Kelsey Cummings - kgc@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Kelsey Cummings wrote:
On Fri, Aug 19, 2011 at 03:48:00AM -0500, Stan Hoeppner wrote:
On 8/17/2011 9:42 AM, Adrian Ulrich wrote:
I read that XFS is a good choice, but is not too reliable... Are you using Maildir or MBOX?
In any case: XFS would be my last choice:
XFS is nice if you are working with large files (> 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files. XFS was designed for parallelism, whether with large files or small, ...
Anyone been using ZFS on FreeBSD for mail spool storage?
I'm using ZFS on FreeBSD 8.2. But to be honest, this is a personal/private mail-system with limited mailboxes. I really like the snapshots in ZFS. Every night I make a snapshot which I send to another server for backup. Every hour I make a new snapshot from which I send the incremental to the backup-server. That way if the worst happens I will only loose 1 hour of mail.
ZFS was built for data integrity, not speed so if it's speed you are looking for this might not be the fastest but maybe fast enough.
N.
Quoting Nick Rosier nick+dovecot@bunbun.be:
Kelsey Cummings wrote:
On Fri, Aug 19, 2011 at 03:48:00AM -0500, Stan Hoeppner wrote:
On 8/17/2011 9:42 AM, Adrian Ulrich wrote:
I read that XFS is a good choice, but is not too reliable... Are you using Maildir or MBOX?
In any case: XFS would be my last choice:
XFS is nice if you are working with large files (> 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files. XFS was designed for parallelism, whether with large files or small, ...
Anyone been using ZFS on FreeBSD for mail spool storage?
I'm using ZFS on FreeBSD 8.2. But to be honest, this is a
personal/private mail-system with limited mailboxes. I really like
the snapshots in ZFS. Every night I make a snapshot which I send to
another server for backup. Every hour I make a new snapshot from
which I send the incremental to the backup-server. That way if the
worst happens I will only loose 1 hour of mail.ZFS was built for data integrity, not speed so if it's speed you are
looking for this might not be the fastest but maybe fast enough.
I just migrated to FreeBSD 8.2 ZFS (6 drive u320 raidz + 16GB mirrored
SSD ZIL) from OpenSolaris ZFS (10 drive sata raidz10). About 4k daily
users.
The new setup feels a little faster than the old, but honestly the old
ran just fine under normal load. One of the things that pushed the
migration was really more CPU for Thunderbird clients using
compression, which caused a noticeable delay.
Ditto on the snapshots.
Rick
On 8/17/11 7:42 AM, Adrian Ulrich wrote:
I read that XFS is a good choice, but is not too reliable...
Are you using Maildir or MBOX?
In any case: XFS would be my last choice:
XFS is nice if you are working with large files (> 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files.
I'd have to disagree. This is completely anecdotal, but I originally deployed ext3 on all of my mail servers (Dovecot maildir) and spools (Postfix) until they started exhibiting loading issues when busy. Reformatting into XFS resolved the problem with no other changes. I didn't have time to do any comparisons or gather statistics since it was an emergency situation and this was before ext4, but XFS has performed flawlessly for me.
~Seth
I'm testing out ZFS-fuse on my new install (talked about it on the other thread), no issues so far. The builtin deduplication and compression sure do help a lot, roughly 30% less storage space required so far.
They don't advertise it as exactly "production" quality, but I'm willing to try it out, we're doing regular backups. The mail system hasn't gone live yet though, so I'm a bit uneasy on the performance side of things under heavy load.
On Fri, Aug 19, 2011 at 14:49, Seth Mattinen sethm@rollernet.us wrote:
On 8/17/11 7:42 AM, Adrian Ulrich wrote:
I read that XFS is a good choice, but is not too reliable...
Are you using Maildir or MBOX?
In any case: XFS would be my last choice:
XFS is nice if you are working with large files (> 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files.
I'd have to disagree. This is completely anecdotal, but I originally deployed ext3 on all of my mail servers (Dovecot maildir) and spools (Postfix) until they started exhibiting loading issues when busy. Reformatting into XFS resolved the problem with no other changes. I didn't have time to do any comparisons or gather statistics since it was an emergency situation and this was before ext4, but XFS has performed flawlessly for me.
~Seth
On 08/19/2011 02:45 PM, Felipe Scarel wrote:
I'm testing out ZFS-fuse on my new install (talked about it on the other thread), no issues so far. The builtin deduplication and compression sure do help a lot, roughly 30% less storage space required so far.
They don't advertise it as exactly "production" quality, but I'm willing to try it out, we're doing regular backups. The mail system hasn't gone live yet though, so I'm a bit uneasy on the performance side of things under heavy load.
You are aware that there's a real in-kernel ZFS implementation under Linux now, right? See http://zfsonlinux.org/. I've done some very basic testing with it, and so far, it works. Going through FUSE is slower than pissing tar; this implementation won't have that problem.
FUSE is useful for many things. Performance-sensitive filesystems on production servers is oh-so-NOT one of them. ;)
-Dave-- Dave McGuire Port Charlotte, FL
I was not aware of that... I went with FUSE to test the deduplication feature of ZFS. I'll check out this link you've provided, many thanks Dave. :)
On Fri, Aug 19, 2011 at 15:48, Dave McGuire mcguire@neurotica.com wrote:
On 08/19/2011 02:45 PM, Felipe Scarel wrote:
I'm testing out ZFS-fuse on my new install (talked about it on the other thread), no issues so far. The builtin deduplication and compression sure do help a lot, roughly 30% less storage space required so far.
They don't advertise it as exactly "production" quality, but I'm willing to try it out, we're doing regular backups. The mail system hasn't gone live yet though, so I'm a bit uneasy on the performance side of things under heavy load.
You are aware that there's a real in-kernel ZFS implementation under Linux now, right? See http://zfsonlinux.org/. I've done some very basic testing with it, and so far, it works. Going through FUSE is slower than pissing tar; this implementation won't have that problem.
FUSE is useful for many things. Performance-sensitive filesystems on production servers is oh-so-NOT one of them. ;)
-Dave-- Dave McGuire Port Charlotte, FL
Good luck!
FYI, my mail spools are on ZFS filesystems under Solaris on UltraSPARC. It is lightning fast with 100+ dovecot imap processes pounding away. I've not yet enabled compression and done the copy/recopy dance, though.
-DaveOn 08/19/2011 02:57 PM, Felipe Scarel wrote:
I was not aware of that... I went with FUSE to test the deduplication feature of ZFS. I'll check out this link you've provided, many thanks Dave. :)
On Fri, Aug 19, 2011 at 15:48, Dave McGuiremcguire@neurotica.com wrote:
On 08/19/2011 02:45 PM, Felipe Scarel wrote:
I'm testing out ZFS-fuse on my new install (talked about it on the other thread), no issues so far. The builtin deduplication and compression sure do help a lot, roughly 30% less storage space required so far.
They don't advertise it as exactly "production" quality, but I'm willing to try it out, we're doing regular backups. The mail system hasn't gone live yet though, so I'm a bit uneasy on the performance side of things under heavy load.
You are aware that there's a real in-kernel ZFS implementation under Linux now, right? See http://zfsonlinux.org/. I've done some very basic testing with it, and so far, it works. Going through FUSE is slower than pissing tar; this implementation won't have that problem.
FUSE is useful for many things. Performance-sensitive filesystems on production servers is oh-so-NOT one of them. ;)
-Dave-- Dave McGuire Port Charlotte, FL
-- Dave McGuire Port Charlotte, FL
Thanks, I've read some of the FAQ and install instructions and it seems pretty straightforward... I wish I could use Solaris but we're virtualizing everything on our Dell blade through VMWare ESXi and it's somewhat of a "company policy" to use the template Debian that's maintained by the senior sysadmin.
About the compression, I've read some benchmarks/tests and the default lzjb algorithm seems to be a good cost/benefit for the usual applications. Without many reads to the filesystem, gzip compresses a whole lot better tho.
On Fri, Aug 19, 2011 at 16:01, Dave McGuire mcguire@neurotica.com wrote:
Good luck!
FYI, my mail spools are on ZFS filesystems under Solaris on UltraSPARC. It is lightning fast with 100+ dovecot imap processes pounding away. I've not yet enabled compression and done the copy/recopy dance, though.
-DaveOn 08/19/2011 02:57 PM, Felipe Scarel wrote:
I was not aware of that... I went with FUSE to test the deduplication feature of ZFS. I'll check out this link you've provided, many thanks Dave. :)
On Fri, Aug 19, 2011 at 15:48, Dave McGuiremcguire@neurotica.com wrote:
On 08/19/2011 02:45 PM, Felipe Scarel wrote:
I'm testing out ZFS-fuse on my new install (talked about it on the other
thread), no issues so far. The builtin deduplication and compression sure do help a lot, roughly 30% less storage space required so far.
They don't advertise it as exactly "production" quality, but I'm willing to try it out, we're doing regular backups. The mail system hasn't gone live yet though, so I'm a bit uneasy on the performance side of things under heavy load.
You are aware that there's a real in-kernel ZFS implementation under Linux now, right? See http://zfsonlinux.org/. I've done some very basic testing with it, and so far, it works. Going through FUSE is slower than pissing tar; this implementation won't have that problem.
FUSE is useful for many things. Performance-sensitive filesystems on production servers is oh-so-NOT one of them. ;)
-Dave-- Dave McGuire Port Charlotte, FL
-- Dave McGuire Port Charlotte, FL
On 08/19/2011 03:07 PM, Felipe Scarel wrote:
Thanks, I've read some of the FAQ and install instructions and it seems pretty straightforward... I wish I could use Solaris but we're virtualizing everything on our Dell blade through VMWare ESXi and it's somewhat of a "company policy" to use the template Debian that's maintained by the senior sysadmin.
Ahh, "company policies"...restricting innovation and hampering productivity and efficiencty for decades!
About the compression, I've read some benchmarks/tests and the default lzjb algorithm seems to be a good cost/benefit for the usual applications. Without many reads to the filesystem, gzip compresses a whole lot better tho.
I agree. I'm running a biggish Usenet news server in a similar configuration, but with compression enabled. I'm getting compression ratios of 1.26x with a ~12GB news spool, using gzip compression. I was expecting a bit more compression, but I'm certainly not complaining.
-Dave-- Dave McGuire Port Charlotte, FL
On 8/19/2011 10:49 AM, Seth Mattinen wrote:
On 8/17/11 7:42 AM, Adrian Ulrich wrote:
I read that XFS is a good choice, but is not too reliable... Are you using Maildir or MBOX?
In any case: XFS would be my last choice:
XFS is nice if you are working with large files (> 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files.
I'd have to disagree. This is completely anecdotal, but I originally deployed ext3 on all of my mail servers (Dovecot maildir) and spools (Postfix) until they started exhibiting loading issues when busy. Reformatting into XFS resolved the problem with no other changes. I didn't have time to do any comparisons or gather statistics since it was an emergency situation and this was before ext4, but XFS has performed flawlessly for me.
~Seth
+1. :)
-- Daniel
On 8/17/2011 8:23 AM, Julio Cesar Covolato wrote:
Hi!
I´m about to migrate a system whith 5000 accounts whith (~ 500GB) from "postfix/courier-imap/maildrop/mysql" to a new hardware whith "postfix/dovecot/dovecot/mysql".
I´ll make a separate partition (raid 1) for the mail spool (/var/spool/vmail) and want to now what type of filesystem to use on it to increase performance. I read that XFS is a good choice, but is not too reliable...
With only a single spindle of seek performance, which is what mirroring (RAID 1) gives you, about 150-300 seeks/second depending on which disks you use, the filesystem will not be a limiting factor, no matter which one you choose. The low IOPS of the disk will limit your performance. Thus, choose the filesystem you are most comfortable, and experienced, in managing.
All other factors being equal (proper fit, use, administration, etc) XFS is as reliable, if not more reliable, than any other Linux filesystem. I'm guessing that what you read related to a bug that was fixed in 2007, which previously could cause corruption in certain circumstances, mainly with many transactions in flight during a power failure, i.e. no UPS or a failed UPS.
-- Stan
On 8/17/2011 9:23 AM, Julio Cesar Covolato wrote:
Hi!
I´m about to migrate a system whith 5000 accounts whith (~ 500GB) from "postfix/courier-imap/maildrop/mysql" to a new hardware whith "postfix/dovecot/dovecot/mysql".
I´ll make a separate partition (raid 1) for the mail spool (/var/spool/vmail) and want to now what type of filesystem to use on it to increase performance. I read that XFS is a good choice, but is not too reliable...
We run ext3 and ext4. Individual mailboxes with a few hundred thousand messages in Maildir on top of ext3 ran fine (800k messages, 4GB mailbox was not unusual). Slowly migrating file systems over to ext4 as we have time (or rollout new hardware).
Frankly, for that big of a mail store, I'd go with RAID 1+0 over a minimum of 4 spindles for the storage of the mbox / Maildir files. If you have heavy usage, seek time might be your biggest enemy.
Keeping the postfix spools (/var/spool/postfix) on a separate set of disks (like the RAID 1 array that you use to run the operating system off of) helps.
participants (15)
-
Adrian Ulrich
-
Daniel L. Miller
-
Dave McGuire
-
Felipe Scarel
-
Johan Hendriks
-
Julio Cesar Covolato
-
Kelsey Cummings
-
Marcin Mirosław
-
Morten Stevens
-
Nick Rosier
-
Rick Romero
-
Seth Mattinen
-
Stan Hoeppner
-
Sven Hartge
-
Thomas Harold