[Dovecot] linux 2.4 vs 2.6 kernel
hi I read ( I think in dovecot wiki) that ext2 or ext3 fs on a 2.4 kernel can get slow due to the number of files in a directory. Does anyone know if kernel 2.6 has similar limitation? I'm not at that point now, but before the box goes live, I'm wondering if I should resolve that. Glenn
Actually - yes. Same problem. But the solution is to use the Reiser Filesystem. It doesn't have the ext3 problem.
vmstech wrote:
hi I read ( I think in dovecot wiki) that ext2 or ext3 fs on a 2.4 kernel can get slow due to the number of files in a directory. Does anyone know if kernel 2.6 has similar limitation? I'm not at that point now, but before the box goes live, I'm wondering if I should resolve that. Glenn
-- Marc Perkel - marc@perkel.com
Spam Filter: http://www.junkemailfilter.com My Blog: http://marc.perkel.com
Marc Perkel wrote:
Actually - yes. Same problem. But the solution is to use the Reiser Filesystem. It doesn't have the ext3 problem.
Now, I don't want to start a religious war, but...
Ext2 has the advantages that 1) it can fall back to ext2 in recovery cases, and 2) because of that, it has all the well-tested ext2 recovery tools available.
People I know who've used Reiser say it's wonderfuly fast, but if it corrupts, well... save your time, and go straight to restoring your backups.
Also, take a look at the tune2fs options. I understand -O dir_index will set it to use a b-tree index for faster handling of large directories.
-- Curtis Maloney cmaloney@cardgate.net
Curtis Maloney wrote:
Marc Perkel wrote:
Actually - yes. Same problem. But the solution is to use the Reiser Filesystem. It doesn't have the ext3 problem.
Now, I don't want to start a religious war, but...
Ext2 has the advantages that 1) it can fall back to ext2 in recovery cases, and 2) because of that, it has all the well-tested ext2 recovery tools available.
People I know who've used Reiser say it's wonderfuly fast, but if it corrupts, well... save your time, and go straight to restoring your backups.
Also, take a look at the tune2fs options. I understand -O dir_index will set it to use a b-tree index for faster handling of large directories.
-- Curtis Maloney cmaloney@cardgate.net
I've been using it for 5 years and it works great. It specifically eliminates the speed problems of Maildir where you have thousands of files in a single directory. And it has infinite inodes so you never run out of them. Reiser is ideal for email systems.
-- Marc Perkel - marc@perkel.com
Spam Filter: http://www.junkemailfilter.com My Blog: http://marc.perkel.com
I administer a mail system at work that runs ext3 with Maildir and we upgraded from a 2.4 kernel to a 2.6 kernel in May and did notice a speed improvement in mail store intensive tasks, like sorting. I do not know that the change is not due to an improved RAID array driver though.
On the other hand, my personal mail systems have always run reiserfs and all of my data intensive servers at work run reiserfs and I have not had a problem with any of them. I do insist on at least some form of RAID, preferably in hardware to minimalize the chances that I will fond out how good my FS's recovery tools are.
good luck, Adam
On Mon, 2005-07-18 at 18:33 -0700, Marc Perkel wrote:
Curtis Maloney wrote:
Marc Perkel wrote:
Actually - yes. Same problem. But the solution is to use the Reiser Filesystem. It doesn't have the ext3 problem.
Now, I don't want to start a religious war, but...
Ext2 has the advantages that 1) it can fall back to ext2 in recovery cases, and 2) because of that, it has all the well-tested ext2 recovery tools available.
People I know who've used Reiser say it's wonderfuly fast, but if it corrupts, well... save your time, and go straight to restoring your backups.
Also, take a look at the tune2fs options. I understand -O dir_index will set it to use a b-tree index for faster handling of large directories.
-- Curtis Maloney cmaloney@cardgate.net
I've been using it for 5 years and it works great. It specifically eliminates the speed problems of Maildir where you have thousands of files in a single directory. And it has infinite inodes so you never run out of them. Reiser is ideal for email systems.
-- Adam Todorski at@proyektx.net
Now, I don't want to start a religious war, but... cool a religious war...
A quick google shows that most of the discussion on these things took place years ago (2001-2003) & not a lot thats current, so its hard to know what still applies.
People I know who've used Reiser say it's wonderfuly fast, but if it
corrupts, well... save your time, and go straight to restoring your backups.
ReiserFS does seems just like the ducksnuts in terms of performance, but it seems to have accumulated some horror stories along the way: http://www.linuxsa.org.au/pipermail/linuxsa/2002-January/038035.html
Marcs being runing it for 5 years, no issues - any one else using it?
It also seems a non-trivial to convert an existing ext3 install. I don't have hands on access to the box ($75 for remote hands on), so resizing the partition, convertfs to new partition, rebooting, deleting old partion resising new partion all seems a bit daunting when I cant actually sit at the console JIC if I stuff up along the way.
If I'm going to go to that effort, I should probably contemplate xfs and jfs and make best choise first up - anyone got any experience with these where maildir is concerned?
Also, take a look at the tune2fs options. I understand -O dir_index
will set it to use a b-tree index for faster handling of large directories.
Check this out relating to this option: http://www.namesys.com/benchmarks.html I haven't fully digested but it seems that ext3 doesn't get close, + there are other caveats.
All this information is quite old, and I wonder whats changed since then, both for rieser and ext3, also [xj]fs, which were considered pre-natal when most of these were written.
So is anyone out there using ext3 with our without dir_index for significant maildir imap installs and just delighted with it?
Thanks all Glenn
On Mon, 2005-07-18 at 18:33 -0700, Marc Perkel wrote:
Curtis Maloney wrote:
Marc Perkel wrote:
Actually - yes. Same problem. But the solution is to use the Reiser Filesystem. It doesn't have the ext3 problem.
Now, I don't want to start a religious war, but...
Ext2 has the advantages that 1) it can fall back to ext2 in recovery cases, and 2) because of that, it has all the well-tested ext2 recovery tools available.
People I know who've used Reiser say it's wonderfuly fast, but if it corrupts, well... save your time, and go straight to restoring your backups.
-- Curtis Maloney cmaloney@cardgate.net
I've been using it for 5 years and it works great. It specifically eliminates the speed problems of Maildir where you have thousands of files in a single directory. And it has infinite inodes so you never run out of them. Reiser is ideal for email systems.
If you don't got to Reiser then you might as well stay with what you got. Reiser is the only file system that's fast when it comes to lots of fules in a single directory. That's it's strongest point.
But conversion isn't trivial but not too dificult. You just copy off all your files onto another drive. Reformat with reiser, the copy them all back.
Reiser is different because it's a file system on top of a database. That's what makes it so fast on maildir specifically. It's the best where you have a lot of small files.
On Mon, 2005-07-18 at 19:22 -0700, Marc Perkel wrote:
If you don't got to Reiser then you might as well stay with what you got. Reiser is the only file system that's fast when it comes to lots of fules in a single directory. That's it's strongest point.
Ofcourse, ext3 in 2.6 and xfs als have hashed/btree'd directories.
Mike.
On Tue, 19 Jul 2005, vmstech wrote:
People I know who've used Reiser say it's wonderfuly fast, but if it
corrupts, well... save your time, and go straight to restoring your backups. ReiserFS does seems just like the ducksnuts in terms of performance, but it seems to have accumulated some horror stories along the way: http://www.linuxsa.org.au/pipermail/linuxsa/2002-January/038035.html
Marcs being runing it for 5 years, no issues - any one else using it?
I've been using it for many years. At home, no issues.
We used to have lots of severe corruption issues at work on production servers with ext2/ext3 (and a few with xfs). We switched to reiserfs and no more issues. Not just a few servers either.
So i'd say quoting horror stories from 2002 definitely no longer applies. Its like quoting horror stories about kernel 2.4 (same time frame). And about as relevant :-)
-Dan
Dan Hollis wrote:
On Tue, 19 Jul 2005, vmstech wrote:
People I know who've used Reiser say it's wonderfuly fast, but if it
corrupts, well... save your time, and go straight to restoring your backups.
ReiserFS does seems just like the ducksnuts in terms of performance, but it seems to have accumulated some horror stories along the way: http://www.linuxsa.org.au/pipermail/linuxsa/2002-January/038035.html
Marcs being runing it for 5 years, no issues - any one else using it?
I've been using it for many years. At home, no issues.
We used to have lots of severe corruption issues at work on production servers with ext2/ext3 (and a few with xfs). We switched to reiserfs and no more issues. Not just a few servers either.
So i'd say quoting horror stories from 2002 definitely no longer applies. Its like quoting horror stories about kernel 2.4 (same time frame). And about as relevant :-)
-Dan
I've installed it on about 30 servers with no problems at all. I think there was one release of the kernel in 2002 thar had a reiser bug, but I never used that one.
If you were running a few big files I'd say use ext3. But when it comes to lots of little files like maildir you're looking at like a 10x speed increaes.
-- Marc Perkel - marc@perkel.com
Spam Filter: http://www.junkemailfilter.com My Blog: http://marc.perkel.com
vmstech vmstech@tpg.com.au writes:
ReiserFS does seems just like the ducksnuts in terms of performance, but it seems to have accumulated some horror stories along the way: http://www.linuxsa.org.au/pipermail/linuxsa/2002-January/038035.html
Marcs being runing it for 5 years, no issues - any one else using it?
I'm currently phasing reiserfs out everywhere. While I don't recall major accidents (should I write incidents? nevermind) since the later 3.6 versions in the later 2.4 kernels, reiserfs has its limitations. Tail merging, space efficiency, is slow to write, so you'll usually mount -o notail. reiserfs has a limited number of files per hash value that a file system can hold, and the earlier reiserfs versions became extremely slow in the face of hash collisions. The quality of the reiserfsprogs is IMHO inferior to e2fsprogs.
ext3 has learnt the dir_index feature (dirhash) a while ago, it's a rather smooth migration (unmount, tune2fs, e2fsck -f -D to create indexes, mount) - but it turns out I don't have directories that are *so* full it would really matter for me. It appears to work well though.
It also seems a non-trivial to convert an existing ext3 install.
There is no conversion. Backup, reformat, restore.
If I'm going to go to that effort, I should probably contemplate xfs and jfs and make best choise first up - anyone got any experience with these where maildir is concerned?
I haven't dared put my Maildirs into either of these.
I used XFS once for a CVS spool and after a crash ended up with zero blocks in the ,v files, which caused the cvs server to crash. I'm not sure if that has been fixed since. I've never seen this happen with ext3.
SUSE are somewhat discontinuing JFS support in their SUSE Linux 9.3 distribution for "technical reasons" that aren't detailed.
Note however that ext3fs supports data journalling, so you can have all the expensive scattered synchronous writes (and the Maildir format has lots of these) turned into linear synchronous writes, which speeds up things quite a bit.
-- Matthias Andree
Am Dienstag, 19. Juli 2005 03:28 schrieb Curtis Maloney:
People I know who've used Reiser say it's wonderfuly fast, but if it corrupts, well... save your time, and go straight to restoring your backups.
That's exactly my experience. I used ReiserFS 2 and 3 for many years and plan to switch some partitions currently running ext3 back to ReiserFS in the near future, but I'd advise anyone using it to have complete and current working backups.
Reiser is fast and saves space, but can cause havoc when the system crashes. The way Reiser works can cause a crash to corrupt files which where not even opened at the time the system went down.
I've lost 3 reiser partitions during the years IIRC and in these cases reiserfsck never helped but rather nailed the coffin, so to speak...
The reason I'm currently using ext3 is solely that my old computer became a bit unstable lately due to hardware flaws which made me abandon reiser 3 rather quickly ;) , and I wanted to test the new computer I had bought to make sure it will run stable enough to drive reiser partitions. It has passed this test in my eyes and I'll give reiser 3 another shot.
My old Am486 internet router and mail server flawlessly runs a reiser 3 partition since a few years without recreating the filesystem and serves IMAP maildir folders and news via leafnode from that partition. But that's no missin critical system compared to what you guys are doing. ;)
Greetings,
Gunter
-- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- http://aachen.uni-dsl.de/ - Der direkte Draht in's Hochschulnetz! + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- Datenblatt: Die angegebenen Werte können auch unter optimalen Bedingungen mittels angepaßter Testfälle sicher nicht übertroffen werden. (Frank Paulsen) -- http://www.iks-jena.de/mitarb/lutz/usenet/Fachbegriffe.der.Informatik.htm l#375 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+PGP-verschlüsselte Mails bevorzugt! +
Why would you even consider a filesystem that corrupts files randomly? Not trolling, its a serious question.
On Tue, 19 Jul 2005 20:35:30 +0200 Gunter Ohrner G.Ohrner@post.rwth-aachen.de wrote:
: Reiser is fast and saves space, but can cause havoc when the system : crashes. The way Reiser works can cause a crash to corrupt files : which where not even opened at the time the system went down. : : I've lost 3 reiser partitions during the years IIRC and in these : cases reiserfsck never helped but rather nailed the coffin, so to : speak...
Am Dienstag, 19. Juli 2005 20:46 schrieb Peter Hessler:
Gunter Ohrner G.Ohrner@post.rwth-aachen.de wrote: : Reiser is fast and saves space, but can cause havoc when the system : crashes. The way Reiser works can cause a crash to corrupt files : which where not even opened at the time the system went down.
: I've lost 3 reiser partitions during the years IIRC and in these : cases reiserfsck never helped but rather nailed the coffin, so to : speak...
Why would you even consider a filesystem that corrupts files randomly? Not trolling, its a serious question.
I had most problems that I had when I still used reiser 2 with Linux 2.2, which has long been obsoleted since reiser 3 was first included in the mainline kernels since 2.4.1 or something.
All problems occured during bad system crashes, situations which really should not happen and which would have caused damages with any file system. So Reiser does not "corrupt files randomly", however in my experience it's true that the damage caused by Reiserfs can exceed the damage done by ext2. So far I had no serious crashes with ext3, so I cannot say anything about this fs.
I just also see reiser's advantages over competing filesystems in some areas. Dovecots speed improved significantly when I switched my (clearly underpowered) mailserver machine from pure ext2 to reiserfs. It had also had a few crashes (mainly power outages) since I set it up and I have not had any problems with the system or it's reiserfs partition which carries the whole /var fs so far.
Greetings,
Gunter
-- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- http://aachen.uni-dsl.de/ - Der direkte Draht in's Hochschulnetz! + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The duke had a mind that ticked like a clock and, like a clock, it regularly went cuckoo. -- (Terry Pratchett, Wyrd Sisters) +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+PGP-verschlüsselte Mails bevorzugt! +
Gunter Ohrner G.Ohrner@post.rwth-aachen.de writes:
All problems occured during bad system crashes, situations which really should not happen
they DO happen.
-- Matthias Andree
We use Reiserfs 3 on over a thousand servers and I've only lost one. That one was caused by us unplugging it on purpose to test the filesystem. It made it through a couple tests, then got messed up and I managed to get it partially back. We tried ext3 (this is ~3 years ago though) and found it not very stable.
I can't comment on speed with a large number of files, as we use NFS for data storage, but I find that Reiser's pretty stable. Mostly we're using 2.4 kernels.
Todd
On Tue, 19 Jul 2005, Peter Hessler wrote:
Why would you even consider a filesystem that corrupts files randomly? Not trolling, its a serious question.
On Tue, 19 Jul 2005 20:35:30 +0200 Gunter Ohrner G.Ohrner@post.rwth-aachen.de wrote:
: Reiser is fast and saves space, but can cause havoc when the system : crashes. The way Reiser works can cause a crash to corrupt files : which where not even opened at the time the system went down. : : I've lost 3 reiser partitions during the years IIRC and in these : cases reiserfsck never helped but rather nailed the coffin, so to : speak...
I'm at the once bitten stage with Reiserfs, unless someone knows how to deal with bad blocks short of 'dd'-ing to another disk and back - the disk won't remap the bad sectors unless you try to write to them and its not trivial to find out how! Having said that, SuSE uses it as the default, so we've gone with that on our SuSE installs so far.
Anybody tried Veritas Filesystem on Solaris for IMAP (even better, Dovecot) access? Is it significantly better than Solaris UFS? I'd heard that UFS performs terribly for Maildir.
Chris
Todd Burroughs wrote:
We use Reiserfs 3 on over a thousand servers and I've only lost one. That one was caused by us unplugging it on purpose to test the filesystem. It made it through a couple tests, then got messed up and I managed to get it partially back. We tried ext3 (this is ~3 years ago though) and found it not very stable.
I can't comment on speed with a large number of files, as we use NFS for data storage, but I find that Reiser's pretty stable. Mostly we're using 2.4 kernels.
Todd
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Chris Wakelin c.d.wakelin@reading.ac.uk writes:
I'm at the once bitten stage with Reiserfs, unless someone knows how to deal with bad blocks short of 'dd'-ing to another disk and back - the disk won't remap the bad sectors unless you try to write to them and its not trivial to find out how!
badblocks in non-destructive read-write mode - but it can take many hours to complete depending on how many blocks are broken and how many retries the drive performs. RAID helps against bad blocks, at the expense of write speed. It doesn't help against kernel bugs (SuSE Linux 9.2 megaraid driver - avoid SUSE 9.2 on megaraid at all cost unless you know how to build a driver update! I haven't looked at 9.3 yet, 9.1 was fine AFAIR)
Having said that, SuSE uses it as the default, so we've gone with that on our SuSE installs so far.
SUSE have special patches in their kernels so reiserfs supports the ext3 "data=ordered" and "data=journal" semantics, the default/vanilla reiserfs operates in the same metadata-only journal mode as do XFS and JFS. SUSE also enable the write barrier stuff by default, whereas it defaults to off in vanilla kernels (leaving aside the fact it requires fairly recent kernels to work).
-- Matthias Andree
On Tue, 19 Jul 2005, Peter Hessler wrote:
Why would you even consider a filesystem that corrupts files randomly? Not trolling, its a serious question.
This was our experience with ext3. We switched to reiserfs. No more corruption. Not just a few servers either. Just a data point or two (or forty).
Our experience with XFS was pretty disturbing too, but that was many years ago. It might have gotten better since. Once bitten twice shy though.
-Dan
vmstech wrote:
hi I read ( I think in dovecot wiki) that ext2 or ext3 fs on a 2.4 kernel can get slow due to the number of files in a directory. Does anyone know if kernel 2.6 has similar limitation? I'm not at that point now, but before the box goes live, I'm wondering if I should resolve that. Glenn
i was trying to find out the same thing a few weeks back..
from what I understand, the 2.6 kernel has some significant optimisations to ext3... i think ext3 may now have B-trees to speed up access to directories with a lot of files/subdirectories? can anyone confirm this?
I was also looking at Reiser as an alternative... but as RH doesn't support it at all anymore, i didn't want to risk my precious maildirs :)
Michael
On Tue, 19 Jul 2005, vmstech wrote:
I read ( I think in dovecot wiki) that ext2 or ext3 fs on a 2.4 kernel can get slow due to the number of files in a directory. Does anyone know if kernel 2.6 has similar limitation? I'm not at that point now, but before the box goes live, I'm wondering if I should resolve that.
Yes, ext2/ext3 is subject to the performance degradation unless you use special patches.
You can use xfs, jfs or reiserfs(recommended) as they dont suffer from this issue.
-Dan
On 2005-07-19 11:14:03 +1000, vmstech wrote:
hi I read ( I think in dovecot wiki) that ext2 or ext3 fs on a 2.4 kernel can get slow due to the number of files in a directory. Does anyone know if kernel 2.6 has similar limitation? I'm not at that point now, but before the box goes live, I'm wondering if I should resolve that. Glenn
Quick answer:
ext3 with dir_index is fine. this link has some nice speed comparision: http://lwn.net/Articles/10904/
hope this helps
darix
p.s.: my personal preference is XFS.
Marcus Rueckert wrote:
On 2005-07-19 11:14:03 +1000, vmstech wrote:
Quick answer:
ext3 with dir_index is fine. this link has some nice speed comparision: http://lwn.net/Articles/10904/
hope this helps
darix
p.s.: my personal preference is XFS.
Wow - that's interesting. I didn't know there was a btree option for Ext3. I thought only Reiser had that. If this actually works then it should perform like Reiser.
Not trying to get into a religious war about file systems but when you are looking at 100,000 files in a single directory you need a database type index to get any performance. That's why I'm a reiser fan for this purpose. If Ext3 has that then that's exciting. Of course you have to do whatever it takes to make that work because Ext3 didn't used to be that way.
Does XFS have a btree search in it?
vmstech wrote:
Wow - that's interesting. I didn't know there was a btree option for Ext3. I thought only Reiser had that. I
is btree and htree the same thing? Sorry - way out of my depth here.
I doubt they are exactly the same but the concept is that a tree required to get speed is what they have in common. If you don't have a tree then you have to do a linear search. If you have 100,000 files in a directory it will take an average of 50,000 comparisons for every file you open. That can get really slow. In a tree system you'll probably have under 10 comparisons. That's why people prefer Reiser for Maildir where you have lots of files in one directory.
If you had a few big files and wanted speed then Ext3 is faster. If you have lots of little files in a single directory you want reiser.
-- Marc Perkel - marc@perkel.com
Spam Filter: http://www.junkemailfilter.com My Blog: http://marc.perkel.com
vmstech wrote:
hi I read ( I think in dovecot wiki) that ext2 or ext3 fs on a 2.4 kernel can get slow due to the number of files in a directory. Does anyone know if kernel 2.6 has similar limitation? I'm not at that point now, but before the box goes live, I'm wondering if I should resolve that. Glenn
Having quickly scanned this thread and seen many stories both ways and horrors from using anything on a disk, I thought I'd present one thing that should weight heavy for everyone, no matter what filesystems and drives you use, have used, lost data on or saved the day with:
Stick with what you already use, unless there is a really overwhelming reason not to. In my case I run a good few machines and only use ext3, and yes I know that I could squeeze out more speed by adopting reiserfs in places, or some other feature, but ask yourself this: admin-wise, what is the overhead in having to deal with one extra item to solve a very specific problem in one single spot?
In my case it would mean to have to learn the works for a different filesystem than I already use. ext3 with dir_index solves the problem 90% for us, and everything remains the same admin-wise. If someone at the PHB-level wants that extra performance, explain it nicely in terms of cost for extra knowledge, or a simple: buy better iron.
my two cents, Alexander
PS: to at least have mentioned dovecot in my reply: it works a bit too well, I haven't had a problem with it yet :D
participants (14)
-
Adam Todorski
-
Alexander Hoogerhuis
-
Chris Wakelin
-
Curtis Maloney
-
Dan Hollis
-
Gunter Ohrner
-
Marc Perkel
-
Marcus Rueckert
-
Matthias Andree
-
Michael Tibben
-
Miquel van Smoorenburg
-
Peter Hessler
-
Todd Burroughs
-
vmstech