Todd Vierling wrote:
It mostly has to do with the filesystem being used. The System V based extended filesystems (which most people know as "ext2" and "ext3") have the same history as the BSD filesystem ("ufs" or "ffs"), and neither is particularly efficient at handling a directory of more than 10k files or so. And it's not usually the lookup of a single file that's a problem; it's listing the directory (when doing index synchronization) that will be slow.
ext3 directories are hash databases, much faster than linear searches/listings.
Other options now exist that improve this quite a bit. On Linux, there's reiserfs; on BSD, there's lfs, and some enhancements to ffs as part of the "FFSv2" project.
Last I heard Reiser was dropped by RedHat. It's still an option on some Linux distros, e.g. CentOS.