Brent Clark wrote:

...

Daniel Watts wrote:

...

Could I possibly have some feedback on what the recommended filesystems are? I've heard of ReiserFS but was wondering what other options there are and how they compare.

Reiser has traditionally been a very good choice for maildir because it has infinite inodes, it is very fast on directories with large numbers of files, and it does sub allocation so small files take less space. And it's very fast. Maildir is the area where Reiser does best.

Thanks for this - I have heared many maildir admins laud Reiser. How is it for ongoing stability and reliability? I suppose with using any non-mainstream technology (ie ext stuff) the admins concern is that it is less well tested for bugs and corruption.

Eg i see many people saying xfs is great but who wouldn't think of having it put into production.

On Fri, 2006-05-05 at 17:24 +0100, Daniel Watts wrote:

...

Brent Clark wrote:

...

Daniel Watts wrote:

...

Could I possibly have some feedback on what the recommended filesystems are? I've heard of ReiserFS but was wondering what other options there are and how they compare.

Reiser has traditionally been a very good choice for maildir because it has infinite inodes, it is very fast on directories with large numbers of files, and it does sub allocation so small files take less space. And it's very fast. Maildir is the area where Reiser does best.

Thanks for this - I have heared many maildir admins laud Reiser. How is it for ongoing stability and reliability? I suppose with using any non-mainstream technology (ie ext stuff) the admins concern is that it is less well tested for bugs and corruption.

three or four years ago we had a major reiserfs corruption due to faulty memory modules (as I know now). The result was a completely unusable and unrepairable partition that could eventually, after two weeks or so, be repaired with the help of Hans Reiser himself. Even though I really appreciated direct involvement of the main developer, I will avoid it by any means.

Eg i see many people saying xfs is great but who wouldn't think of having it put into production.

XFS is one of the most mature filesystems around for UNIX systems. It has been developed by SGI for its IRIX OS for a _very_ long time. If POSIX compliance (ACLs!!), out of the box quota support, scalability (up to 9 million terabytes max. capacity, IIRC), sophisticated backup/restore facitilites (including snapshots and more), online file defragmentation and much more are of importance for you, then go for XFS ... well, and you certainly guessed it, we are using it for almost all of our servers :-)

Regards

Udo Rader

-- BestSolution.at EDV Systemhaus GmbH http://www.bestsolution.at

On Fri, 2006-05-05 at 17:23 -0500, Richard Laager wrote:

On Fri, 2006-05-05 at 22:39 +0200, Udo Rader wrote:

...

three or four years ago we had a major reiserfs corruption due to faulty memory modules (as I know now). The result was a completely unusable and unrepairable partition that could eventually, after two weeks or so, be repaired with the help of Hans Reiser himself. Even though I really appreciated direct involvement of the main developer, I will avoid it by any means.

Not that I really care what you use, but don't you think it's a bit illogical to avoid a filesystem based on problems you know were caused by faulty hardware? Just how is the filesystem supposed to avoid problems from bad RAM?

Yes, you are right, the cause for this incident was faulty memory and I don't blame reiserfs for failing due to this. But the effect was a unrepairable filesystem and that again was a problem with the repair tools available then. And that definetively left a bad after taste for me.

It may be pure coincidence, but I never experienced anything equally desastrous with any other FS (knocking on wood right now :-).

Udo Rader

-- BestSolution.at EDV Systemhaus GmbH http://www.bestsolution.at

Certainly, but my completely anecdotal experience is that I've seen
lots of people complain about bad memory causing corruption on their
reiser partitions, while far less people complain about the same
problem with other file systems. I'm not saying reiser is inherently
fragile, but it's a suspicious correlation.

If reiser *wasn't* fragile, I'd expect to see more people complaining
how bad memory corrupted their ext3/xfs/whatever partition.

On May 7, 2006, at 12:44 PM, Charles Marcus wrote:

...

Yes, you are right, the cause for this incident was faulty memory
and I don't blame reiserfs for failing due to this. But the effect was a unrepairable filesystem and that again was a problem with the repair tools available then.

Not necessarily... faulty memory could cause corruption that NO
file system repair tools could repair.

--

Best regards,

Charles

On Sun, 2006-05-07 at 15:44 -0400, Charles Marcus wrote:

...

Yes, you are right, the cause for this incident was faulty memory and I don't blame reiserfs for failing due to this. But the effect was a unrepairable filesystem and that again was a problem with the repair tools available then.

Not necessarily... faulty memory could cause corruption that NO file system repair tools could repair.

Hmm, what kind of corruption should that be? I was not talking about individual files being lost but an entire partition being inaccessible.

At least in my naive world this is something that should never happen at all (unless the storage media breaks). AFAIK, any modern filesystem keeps backups of mission critical data like for example superblocks, but please feel free to correct me, if I am missing something here.

But in order to become on-topic again, what I was trying to say was that the quality and availability of disaster recovery tools/procedures is of major importance when choosing a FS for any server and that is where reiserfs failed at least for my part.

Udo Rader

-- BestSolution.at EDV Systemhaus GmbH http://www.bestsolution.at

On Sun, May 07, 2006 at 11:56:01PM +0200, Udo Rader wrote:

On Sun, 2006-05-07 at 15:44 -0400, Charles Marcus wrote:

...

...

Yes, you are right, the cause for this incident was faulty memory and I don't blame reiserfs for failing due to this. But the effect was a unrepairable filesystem and that again was a problem with the repair tools available then.

Not necessarily... faulty memory could cause corruption that NO file system repair tools could repair.

Hmm, what kind of corruption should that be? I was not talking about individual files being lost but an entire partition being inaccessible.

At least in my naive world this is something that should never happen at all (unless the storage media breaks). AFAIK, any modern filesystem keeps backups of mission critical data like for example superblocks, but please feel free to correct me, if I am missing something here.

if you are really worried about silent filesystem corruption caused by bad hardware, I strongly recommend investigating Solaris' ZFS, which can detect and repair data corruption assuming you have some redundancy (ie. mirror or raidz). plus it's always consistent on disk (no fsck, ever). and you get ACLs, snapshots, online expansion, ... the list goes on.

and getting back on topic, ZFS works very well for Maildir :)

grant.

On Fri, May 05, 2006 at 17:24:29 +0100, Daniel Watts wrote:

...

Brent Clark wrote:

...

Daniel Watts wrote:

...

Could I possibly have some feedback on what the recommended filesystems are? I've heard of ReiserFS but was wondering what other options there are and how they compare.

Reiser has traditionally been a very good choice for maildir because it has infinite inodes, it is very fast on directories with large numbers of files, and it does sub allocation so small files take less space. And it's very fast. Maildir is the area where Reiser does best.

Thanks for this - I have heared many maildir admins laud Reiser. How is it for ongoing stability and reliability? I suppose with using any non-mainstream technology (ie ext stuff) the admins concern is that it is less well tested for bugs and corruption.

Hi, when i switched from mbox to Maildir half a year ago i wanted to put my "Mail-Partition" on a reiserfs. The migration failed 3 times due to reiserfs-corruption. I don't remember the exact error message, sorry. Admitted, the transition put a heavy load on the fs (~4GB of Mails), still it should have worked. I reformatted the partition with XFS and all problems went away :).

Before that i used and recommended ReiserFS, now i use XFS whenever possible and didn't have problems until now...

Eg i see many people saying xfs is great but who wouldn't think of having it put into production.

On Tue, 2006-05-09 at 16:32, Wouter Van Hemel wrote:

In real life on general purpose servers, the gains have been quite marginal, though. Filesystem change isn't a miracle cure for performance problems, obviously; if that's the problem, more disks to spread the transactions over make a much bigger difference I/O wise.

If you put a huge number of files in the same directory, the filesystem type can make a big difference in access time. Remember that before you can create a new file you must scan the current list first to see if that name already exists and the whole operation has to happen atomically with the directory locked. Filesystems that index the directories can help compared to a linear scan although there are some tradeoffs. Also some never shrink a directory when files are removed so you continue to scan all the empty slots.

Long ago I used a benchmark program called 'postmark' to test the speed of file creation/deletion operations that are typical in maildir environments. I haven't been able to find it recently although the last time I mentioned it someone said it was in the debian repositories and available via apt-get.

-- Les Mikesell lesmikesell@gmail.com

On Tue, 09 May 2006 16:49:33 -0500 Les Mikesell lesmikesell@gmail.com wrote:

On Tue, 2006-05-09 at 16:32, Wouter Van Hemel wrote:

...

In real life on general purpose servers, the gains have been quite marginal, though. Filesystem change isn't a miracle cure for performance problems, obviously; if that's the problem, more disks to spread the transactions over make a much bigger difference I/O wise.

If you put a huge number of files in the same directory, the filesystem type can make a big difference in access time. Remember that before you can create a new file you must scan the current list first to see if that name already exists and the whole operation has to happen atomically with the directory locked. Filesystems that index the directories can help compared to a linear scan although there are some tradeoffs. Also some never shrink a directory when files are removed so you continue to scan all the empty slots.

IIRC all typical filesystems for Linux (ext3, xfs, jfs, reiserfs) use directory indexing, usually by means of a b-tree.

It's important to note that these filesystems each have their own strengths, and performance will depend on many factors such as the size and number of files, parallellism, number and type of disks, fragmentation, i/o load, possibly even cpu load. Are we talking about a relaying mailserver or end-user storage? Do the users move or delete a lot of files? Do they rather use imap, or pop3? What other activities run on the machine? How do you see the reliability/performance trade-off?

In real life, things aren't as clean-cut as in most of those generic benchmarks, and people tend to attach too much importance to them and then usually get into silly flamewars. :)

Long ago I used a benchmark program called 'postmark' to test the speed of file creation/deletion operations that are typical in maildir environments. I haven't been able to find it recently although the last time I mentioned it someone said it was in the debian repositories and available via apt-get.

I seem to remember to have used it once too, also in a very vague past.

Now, I'm not sure how valid the results would be if, for instance, there's a webserver serving dynamic webmail pages in the same time...

In the past, I've spent (wasted) quite some time benchmarking things like FreeBSD vs Linux, Perl vs PHP, template systems, etc. Now I believe that people should just pick what they feel comfortable with, because the differences are often not that large and it's rarely worth their time and money.

(Though, that's often not what people want to hear. :) )

On Tue, 09 May 2006 19:48:19 -0500 Les Mikesell lesmikesell@gmail.com wrote:

On Tue, 2006-05-09 at 19:32, Wouter Van Hemel wrote:

...

IIRC all typical filesystems for Linux (ext3, xfs, jfs, reiserfs) use directory indexing, usually by means of a b-tree.

They are optional on ext3 and I don't think they are on by default.

That, I don't know either, but it would make more sense if they would be.

If the main use is a mail server with maildir storage the speed of creating/deleting files is going to the the main factor.

That depends on one's priorities.

I would go for reliability and recovery possibility. Those are different requirements, but as valid. Web content can be uploaded easily, but mailbox recovery is a messy affair. I've not had a machine with i/o being the limiting factor -- at least for imap mailbox storage.

I suppose one could shave off some milliseconds, but that hasn't been my priority for mailbox storage servers. And personally, I would first try to get the mailqueue (if any) and dovecot's indexes on another disk.

Apart from those considerations, I totally agree that it makes sense to look at filesystems that deal well with operations on small files.

...

In the past, I've spent (wasted) quite some time benchmarking things like FreeBSD vs Linux, Perl vs PHP, template systems, etc. Now I believe that people should just pick what they feel comfortable with, because the differences are often not that large and it's rarely worth their time and money.

Well, now you can usually afford to throw a few more gigs of RAM in and let buffering solve the problem. That used to be much more expensive.

True.

On Wed, May 10, 2006 at 02:32:40AM +0200, Wouter Van Hemel wrote:

It's important to note that these filesystems each have their own strengths, and performance will depend on many factors such as the size and number of files, parallellism, number and type of disks, fragmentation, i/o load, possibly even cpu load. Are we talking about a relaying mailserver or end-user storage? Do the users move or delete a lot of files? Do they rather use imap, or pop3? What other activities run on the machine? How do you see the reliability/performance trade-off?

In real life, things aren't as clean-cut as in most of those generic benchmarks, and people tend to attach too much importance to them and then usually get into silly flamewars. :)

Agreed, mostly. I feel that I need to point the only time ever where I have noted real-life difference between journaling file-systems during normal operation.

I use an rsync backup scheme that depends heavily on hard links (googling rsync backup hard link should give you details). A typical day's backup would consist of maybe a thousand files spread across a directory tree consisting of some ten thousand directories and some hundred thousand+ hard links to files shared between daily backups. I moved the storage from ext3 to reiser3. I noted that deleting the directory tree representing a day's backup went from insignificant minutes to tens of minutes, hundreds of minutes, sometimes up to a thousand-some minutes.

I found benchmarks that seemed to confirm this advantage of ext3 over reiser3, so I didn't try any tuning. I didn't feel strongly enough about it to move the server back to ext3, I just separated the deleting from the backing up so the the one did not delay the other.

Of course this use is probably not relevant to a server used for Maildir or mbox, but it's worth noting that while there can be differences in performance, the usage pattern has to be really extreme before you see any difference at the application layer.

...

Long ago I used a benchmark program called 'postmark' to test the speed of file creation/deletion operations that are typical in maildir environments. I haven't been able to find it recently although the last time I mentioned it someone said it was in the debian repositories and available via apt-get.

I seem to remember to have used it once too, also in a very vague past.

Well, I have a debian machine, so it's easy to check!

% postmark 1.51-3 File system benchmark from NetApp % % Benchmark that's based around small file operations similar to % those used on large mail servers and news servers. Has been % ported to NT so should be good for comparing OSs. % % http://www.netapp.com/tech_library/postmark.html

HTH.

IIRC all typical filesystems for Linux (ext3, xfs, jfs, reiserfs) use directory indexing, usually by means of a b-tree.

It's important to note that these filesystems each have their own strengths, and performance will depend on many factors such as the size and number of files, parallellism, number and type of disks, fragmentation, i/o load, possibly even cpu load. Are we talking about a relaying mailserver or end-user storage? Do the users move or delete a lot of files? Do they rather use imap, or pop3? What other activities run on the machine? How do you see the reliability/performance trade-off?

In real life, things aren't as clean-cut as in most of those generic benchmarks, and people tend to attach too much importance to them and then usually get into silly flamewars. :)

For the sake of this discussion I think it would be most relevant to assume a machine dedicated to Dovecot IMAP Maildir storage without any other services running on it. In terms of load type does it matter whether there are a lot of light usage users or just a few massively heavy usage users? Maximal mailbox reads and writes is probably the best abstract way to avoid the 'how many users / how many mail files?' problem.

Now I believe that people should just pick what they feel comfortable with, because the differences are often not that large and it's rarely worth their time and money.

(Though, that's often not what people want to hear. :) )

Actually that is my gut feeling too. However some people dont' really have a preference and so might as well spend a bit of time choosing the best system in terms of (theoretical) performance for their particular application.