[Dovecot] Dovecot index issues (Solaris/Thunderbird)
Folks -
My setup is that my mail is filed (by procmail) in ~/.mail/Maildir, an NFS filesystem. But I'm running Dovecot (1.0alpha5) on my local machine (as myself on port 2143), just to serve this mail to my browser (Thunderbird 1.0.2).
The NFS data is on a netapp which seems to be creating .nfs files LIKE CRAZY. I expect these are the remnants of lock files, so I'm trying to move my indexes onto local disk. To that end, I've created /var/tmp/mario/dovecot and put the following into my dovecot.conf:
default_mail_env = maildir:%h/.mail/Maildir:INDEX=/var/tmp/mario/dovecot
Dovecot happily makes index files, but then fails to serve some of my subscribed folders!
Thunderbird is less than helpful in telling me what's going wrong, but dtmail shows that dovecot is reporting "out of sync" and a couple other errors (UID out of sequence), then closing the connection (for those folders). This may happen only after Thunderbird has tried to open the folder that I see these errors with dtmail:
Dovecot ready. [UNSEEN 227] First unseen. [UIDNEXT 341] Predicted next UID
IMAP connection broken (server response) No-op dead stream [CLOSED] No-op dead stream
I've turned mmap off and set locking to fcntl.
As a last resort, I've turned indexes off default_mail_env = maildir:%h/.mail/Maildir:INDEX=MEMORY but this makes thunderbird reload all headers every time I open a folder, which takes a very long time.
Any suggestions? How do I debug this further?
MarioOn Mon, 2006-01-16 at 16:46 -0700, Mario Nigrovic-rvvk40 wrote:
default_mail_env = maildir:%h/.mail/Maildir:INDEX=/var/tmp/mario/dovecot
This looks correct.
Dovecot happily makes index files, but then fails to serve some of my subscribed folders!
Do you see them in the mailbox list, or don't you just see mails inside them?
Thunderbird is less than helpful in telling me what's going wrong, but dtmail shows that dovecot is reporting "out of sync" and a couple other errors (UID out of sequence), then closing the connection (for those folders). This may happen only after Thunderbird has tried to open the folder that I see these errors with dtmail:
Dovecot ready. [UNSEEN 227] First unseen. [UIDNEXT 341] Predicted next UID
IMAP connection broken (server response) No-op dead stream [CLOSED] No-op dead stream
Doesn't Dovecot's logs say anything? Since this looks like Dovecot closes the connection, and in that case it should have written something to logs..
I've turned mmap off and set locking to fcntl.
If indexes are in local disk, that's not needed.
As a last resort, I've turned indexes off default_mail_env = maildir:%h/.mail/Maildir:INDEX=MEMORY but this makes thunderbird reload all headers every time I open a folder, which takes a very long time.
Thunderbird should be caching the headers locally. If it reloads them completely, it means the message UIDs have changed. That shouldn't be happening.
What about ~/.mail/Maildir/dovecot-uidlist file? It should contain something like:
1 1127056150 5891 31 1059753523.2405.mbox:2,F 32 1059753524.2405.mbox:2,F
The second number in first line (1127056150) is the UIDVALIDITY. Does it keep changing? The rest of the lines contain <UID> <maildir file>. Do their UIDs keep changing?
Timo -
Thanks for helping me out on this!
Timo Sirainen wrote:
On Mon, 2006-01-16 at 16:46 -0700, Mario Nigrovic-rvvk40 wrote:
Dovecot happily makes index files, but then fails to serve some of my subscribed folders!
Do you see them in the mailbox list, or don't you just see mails inside them?
Thunderbird still shows all my folders, but won't show messages for some folders. There does seem to be some correlation with the number of messages in the folders (larger folders don't display contents). It just sits there displaying "opening folder" (or something like that) in the lower left status bar.
Thunderbird is less than helpful in telling me what's going wrong, but dtmail shows that dovecot is reporting "out of sync" and a couple other errors (UID out of sequence), then closing the connection (for those folders). This may happen only after Thunderbird has tried to open the folder that I see these errors with dtmail:
Dovecot ready. [UNSEEN 227] First unseen. [UIDNEXT 341] Predicted next UID
IMAP connection broken (server response) No-op dead stream [CLOSED] No-op dead stream
Doesn't Dovecot's logs say anything? Since this looks like Dovecot closes the connection, and in that case it should have written something to logs..
log_path's log file (/home/mario/dovecot/log) has this, which might be interesting:
dovecot: Jan 23 07:12:04 Warning: imap(rvvk40): Compressing cache file /var/tmp/mario/dovecot/.Freescale.ATIC.SAM/dovecot.index.cache dovecot: Jan 23 07:12:04 Warning: imap(rvvk40): Growing pool 'extensions' with: 512 dovecot: Jan 23 07:12:04 Warning: imap(rvvk40): Growing pool 'extensions' with: 512 dovecot: Jan 23 07:12:04 Error: imap(rvvk40): Sending log messages too fast, throttling.. dovecot: Jan 23 07:12:05 Warning: imap(rvvk40): Growing pool 'extensions' with: 512
(Folder .Freescale.ATIC.SAM is one which hangs on displaying contents.)
info_log_path's log file just has this sort of sequence repeated:
dovecot: Jan 23 07:27:04 Info: imap-login: Login: user=<rvvk40>, method=plain, rip=10.213.160.81, lip=172.16.26.151 dovecot: Jan 23 07:27:04 Info: imap(rvvk40): Effective uid=10019173, gid=21 dovecot: Jan 23 07:27:04 Info: imap(rvvk40): maildir: data=/home/mario/.mail/Maildir:INDEX=/var/tmp/mario/dovecot dovecot: Jan 23 07:27:04 Info: imap(rvvk40): maildir: root=/home/mario/.mail/Maildir, index=/var/tmp/mario/dovecot, control=, inbox=
As a last resort, I've turned indexes off default_mail_env = maildir:%h/.mail/Maildir:INDEX=MEMORY but this makes thunderbird reload all headers every time I open a folder, which takes a very long time.
Thunderbird should be caching the headers locally. If it reloads them completely, it means the message UIDs have changed. That shouldn't be happening.
What about ~/.mail/Maildir/dovecot-uidlist file? It should contain something like:
1 1127056150 5891 31 1059753523.2405.mbox:2,F 32 1059753524.2405.mbox:2,F
The second number in first line (1127056150) is the UIDVALIDITY. Does it keep changing? The rest of the lines contain <UID> <maildir file>. Do their UIDs keep changing?
Hmm. I don't see any such file! Can it be recreated, or maybe restored from backup?
Mario-- I don't need a name; my number's just fine. It's nobody else's -- just mine, all mine.
On Mon, 2006-01-23 at 07:42 -0700, Mario Nigrovic-rvvk40 wrote:
dovecot: Jan 23 07:12:04 Warning: imap(rvvk40): Compressing cache file /var/tmp/mario/dovecot/.Freescale.ATIC.SAM/dovecot.index.cache dovecot: Jan 23 07:12:04 Warning: imap(rvvk40): Growing pool 'extensions' with: 512 dovecot: Jan 23 07:12:04 Warning: imap(rvvk40): Growing pool 'extensions' with: 512 dovecot: Jan 23 07:12:04 Error: imap(rvvk40): Sending log messages too fast, throttling.. dovecot: Jan 23 07:12:05 Warning: imap(rvvk40): Growing pool 'extensions' with: 512
Don't compile with --enable-debug. It's not meant for normal users and these kind of problems can happen. The "throttling" line starts slowing down the imap process a lot.
What about ~/.mail/Maildir/dovecot-uidlist file? It should contain something like:
1 1127056150 5891 31 1059753523.2405.mbox:2,F 32 1059753524.2405.mbox:2,F
The second number in first line (1127056150) is the UIDVALIDITY. Does it keep changing? The rest of the lines contain <UID> <maildir file>. Do their UIDs keep changing?
Hmm. I don't see any such file! Can it be recreated, or maybe restored from backup?
Dovecot always creates it. If you're able to open the maildir, the first thing that happens is that dovecot-uidlist file gets created. Each mailbox has its own one of those..
Timo Sirainen wrote:
On Mon, 2006-01-23 at 07:42 -0700, Mario Nigrovic-rvvk40 wrote:
dovecot: Jan 23 07:12:04 Warning: imap(rvvk40): Compressing cache file /var/tmp/mario/dovecot/.Freescale.ATIC.SAM/dovecot.index.cache dovecot: Jan 23 07:12:04 Warning: imap(rvvk40): Growing pool 'extensions' with: 512 dovecot: Jan 23 07:12:04 Warning: imap(rvvk40): Growing pool 'extensions' with: 512 dovecot: Jan 23 07:12:04 Error: imap(rvvk40): Sending log messages too fast, throttling.. dovecot: Jan 23 07:12:05 Warning: imap(rvvk40): Growing pool 'extensions' with: 512
Don't compile with --enable-debug. It's not meant for normal users and these kind of problems can happen. The "throttling" line starts slowing down the imap process a lot.
Re-compiled without --enable-debug. Shorter log files, but still no messages about why the server is stopping.
What about ~/.mail/Maildir/dovecot-uidlist file? It should contain something like:
1 1127056150 5891 31 1059753523.2405.mbox:2,F 32 1059753524.2405.mbox:2,F
The second number in first line (1127056150) is the UIDVALIDITY. Does it keep changing? The rest of the lines contain <UID> <maildir file>. Do their UIDs keep changing?
Hmm. I don't see any such file! Can it be recreated, or maybe restored from backup?
Dovecot always creates it. If you're able to open the maildir, the first thing that happens is that dovecot-uidlist file gets created. Each mailbox has its own one of those..
I've done a thorough search of the while .mail/Maildir hierarchy, and there are no such files here! I also don't see any in the INDEX directory.
Mario-- I don't need a name; my number's just fine. It's nobody else's -- just mine, all mine.
On Mon, 2006-01-23 at 09:11 -0700, Mario Nigrovic-rvvk40 wrote:
What about ~/.mail/Maildir/dovecot-uidlist file? It should contain something like:
1 1127056150 5891 31 1059753523.2405.mbox:2,F 32 1059753524.2405.mbox:2,F
The second number in first line (1127056150) is the UIDVALIDITY. Does it keep changing? The rest of the lines contain <UID> <maildir file>. Do their UIDs keep changing?
Hmm. I don't see any such file! Can it be recreated, or maybe restored from backup?
Dovecot always creates it. If you're able to open the maildir, the first thing that happens is that dovecot-uidlist file gets created. Each mailbox has its own one of those..
I've done a thorough search of the while .mail/Maildir hierarchy, and there are no such files here! I also don't see any in the INDEX directory.
Well, the problem must be related to that then. The dovecot-uidlist files are created immediately after mailbox is opened. Dovecot only deletes them if their contents are corrupted, but then it also writes about that to logs (the log line contains dovecot-uidlist text).
If the dovecot-uidlist file doesn't exists and there isn't anything in logs about them, something's really wrong.
participants (2)
-
Mario Nigrovic-rvvk40
-
Timo Sirainen