OK, so it isn't really the kind of bug I first thought it would be. Instead it's allocating/leaking way too much memory.
I agree. I watched it closer again this time and it filled up all available memory within 30 seconds.
Can you test what exactly is the difference? INDEX=MEMORY or being in NFS? I don't see how either of those would make a difference here.
To me it's currently looking like Solaris iconv() is returning some weird results in some condition.
You're right, it doesn't look like an INDEX=MEMORY issue. Here's the breakdown:
Server1
mmap_disable = no mail_nfs_storage = no mail_nfs_index=no
Local Disk + Normal Index = No Panic Local Disk + Memory Index = No Panic
Server2
mmap_disable = yes mail_nfs_storage = yes mail_nfs_index=yes
NFS + Normal Index = Panic NFS + Memory Index = Panic
So at first I thought NFS is definitely effecting it. Here is a top from the server1 without NFS three minutes into a search through 6000 emails:
21014 dhalik 1 10 0 7920K 6352K cpu/0 2:46 46.21% imap
That looks much better.
So I tried the same thing on server2 with NFS and it filled up then panicked.
But just to be sure I moved my homedir to local disk temporarily, and you are totally right. The problem happened again! So now I'm stumped. It looks like it is neither NFS or INDEX, but an issue with Solaris as you suggested. It just seems odd that I can't reproduce it on one of the servers, yet the other two do it every single time. As far as I know the servers have the same libraries as well:
Panic Server:
bash-2.05$ uname -a SunOS 5.9 Generic_118558-39 sun4u sparc SUNW,Ultra-80 bash-2.05$ ldd /usr/local/libexec/dovecot/imap libiconv.so.2 => /usr/local/lib/libiconv.so.2 libdl.so.1 => /usr/lib/libdl.so.1 librt.so.1 => /usr/lib/librt.so.1 libnsl.so.1 => /usr/lib/libnsl.so.1 libsocket.so.1 => /usr/lib/libsocket.so.1 libsendfile.so.1 => /usr/lib/libsendfile.so.1 libc.so.1 => /usr/lib/libc.so.1 libaio.so.1 => /usr/lib/libaio.so.1 libmd5.so.1 => /usr/lib/libmd5.so.1 libmp.so.2 => /usr/lib/libmp.so.2 /usr/platform/SUNW,Ultra-80/lib/libc_psr.so.1 /usr/platform/SUNW,Ultra-80/lib/libmd5_psr.so.1
Non Panic Server:
bash-2.05$ uname -a SunOS 5.9 Generic_118558-39 sun4u sparc SUNW,Ultra-250 bash-2.05$ ldd /usr/local/libexec/dovecot/imap libiconv.so.2 => /usr/local/lib/libiconv.so.2 libdl.so.1 => /usr/lib/libdl.so.1 librt.so.1 => /usr/lib/librt.so.1 libnsl.so.1 => /usr/lib/libnsl.so.1 libsocket.so.1 => /usr/lib/libsocket.so.1 libsendfile.so.1 => /usr/lib/libsendfile.so.1 libc.so.1 => /usr/lib/libc.so.1 libaio.so.1 => /usr/lib/libaio.so.1 libmd5.so.1 => /usr/lib/libmd5.so.1 libmp.so.2 => /usr/lib/libmp.so.2 /usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1 /usr/platform/SUNW,Ultra-250/lib/libmd5_psr.so.1