[Dovecot] Dovecot-1.0.beta1: New mail sometimes fails to appear

Hi there...

Having updated to 1.0.beta1, I for a while thought a bad bug with IDLE handling that caused new mail in INBOX not to appear when using Mozilla Thunderbird has been solved. Although I can not reproduce one kind of problem anymore, after a short while it again appears that one does not always see one's new mail with Thunderbird.

I can not consistently reproduce the bug, but from some network captures it indeed appears that sometimes Thunderbird has (in one of its connections to the server) INBOX SELECTed and IDLE on it, new mail arrives, but 10 minutes afterwards it was still not sent an EXISTS message.

Initially the symptoms appeared to suggest that something really bad happened to the index files, but I tried INDEX=MEMORY and that did not help much. Thunderbird using multiple connections to the server does of course only make debugging more difficult. But it appears that without multiple connections, one has to constantly cycle around SELECTing to get notified on more than one mailbox, so that may justified.

I am a bit lost on this now, and would appreciate some ideas. Although there were several related bugs I noticed in Thunderbird 1.0, it appears that they are gone in 1.5, and yet now it really seems Dovecot is doing something wrong here.

Attached is the configuration file I use for the testing installation of Dovecot 1.0.beta1 (somebody tell me if you see something wrong there). The mailbox arrangement is as follows, right now (until we have a better way):

• INBOX in /var/mail/%u
• most mailboxes in ~/imap.maildir/... as maildirs.
• mboxes in ~/imap.mbox/... accessible as #imap.mbox/...
• the mbox:~/mail/ is for when namespaces are not used - now there are so it should not be relevant

Except for myself, and one test account I have, I have two guinea-pigs using it (who are well aware that it is still under development and that it will break once in a while) and reporting problems to me.

Vaguely (since I can't figure the exact reasons), the symptom can be described as follows: When using Mozilla Thunderbird to access mail, new mail arrives to INBOX (mbox:/var/mail/%u), but is sometimes not seen for a long while after it arrives. Sometimes the first (or first few) message arriving appears, but then no new messages appear, and sometimes everything works right and immediately new mail is seen. However whenever (according to Thunderbird) something changes in the mailbox, the mailbox is seen up-to-date as it really was at that time.

Thanks, any help appreciated, -- Tom

-- Tom Alsberg - hacker (being the best description fitting this space) Web page: http://www.cs.huji.ac.il/~alsbergt/ DISCLAIMER: The above message does not even necessarily represent what my fingers have typed on the keyboard, save anything further.

# Dovecot IMAP server configuration at CS.HUJI ############################################## # # $Id$

# GENERAL PARAMETERS: #####################

# Base directory where to store runtime data: base_dir = /var/run/dovecot

# Protocols we want to be serving: protocols = imap pop3

# IP or host address where to listen in for connections: listen = *

ssl_disable = yes

# LOGIN PROCESSES: ##################

# Directory for authentication sockets: #login_dir = /var/run/dovecot/login

# chroot login process to the login_dir: #login_chroot = yes

# User to use for the login process: login_user = nobody

# Maximum process size in megabytes: #login_process_size = 16

# Have each login be processed in its own process: #login_process_per_connection = yes

# Number of login processes to create: #login_processes_count = 3

# Maximum number of extra login processes to create: #login_max_processes_count = 128

# Maximum number of connections allowed in login state: #login_max_logging_users = 256

# Greeting message for clients: login_greeting = Dovecot mail access server at CS.HUJI ready.

# MAIL PROCESSES: #################

# Maximum number of running mail processes: #max_mail_processes = 1024

# Show more verbose process title (in ps): verbose_proctitle = yes

# Valid UID range for users: first_valid_uid = 10 #last_valid_uid = 0

# Valid GID range for users: #first_valid_gid = 1 #last_valid_gid = 0

# Extra groups for mail processes: #mail_extra_groups =

# ':' separated list of directories under which chrooting is allowed: #valid_chroot_dirs =

# Default chroot directory for mail processes: #mail_chroot =

# Default MAIL environment to use when it's not set: default_mail_env = mbox:~/mail/:INBOX=/var/mail/%u

# NAMESPACES: #############

namespace private { separator = / prefix = "" location = maildir:~/imap.maildir/ }

namespace private { separator = / prefix = "#imap.mbox/" location = mbox:~/imap.mbox/:INBOX=/var/mail/%u inbox = yes }

# MAIL STORE: #############

# Space-separated list of fields to initially save into cache file: #mail_cache_fields =

# Space-separated list of fields to never save into cache file: #mail_never_cache_fields =

# Like mailbox_check_interval, but used for IDLE command: #mailbox_idle_check_interval = 30

# Whether to allow full filesystem access to clients: #mail_full_filesystem_access = no

# Maximum allowed length for mail keyword name: #mail_max_keyword_length = 50

# Whether to save mails with CR+LF instead of plain LF: #mail_save_crlf = no

# Whether to use mmap to access mail files: #mail_read_mmaped = no

# Whether to disable mmap at all: mmap_disable = yes

# Whether to disable write to mmap: #mmap_no_write = no

# Locking method for index files: lock_method = dotlock

# Whether to list only directories in LIST command: maildir_stat_dirs = yes

# Whether to copy mail to other folders using hard links: #maildir_copy_with_hardlinks = no

# Locking methods to use for locking mbox: mbox_read_locks = dotlock mbox_write_locks = dotlock

# Maximum time in seconds to wait for lock (all of them): mbox_lock_timeout = 300

# Timeout to overwrite dotlock files if mailbox isn't modified: mbox_dotlock_change_timeout = 30

# Whether to read only appended data on change (unless corrupt): #mbox_dirty_syncs = yes

# Like mbox_dirty_syncs, but no full syncs with SELECT, EXAMINE, # EXPUNGE, or CHECK: #mbox_very_dirty_syncs = no

# Whether to delay writing mbox headers until doing a full write sync: #mbox_lazy_writes = yes

# umask to use for mail files and directories: umask = 0077

# Whether drop all privileges before exec of mail process: #mail_drop_priv_before_exec = no

# Maximum mail process size in megabytes: #mail_process_size = 256

# Log prefix for mail processes: mail_log_prefix = "%Us(%u): "

# IMAP SPECIFIC SETTINGS: #########################

protocol imap { # Login executable location: #login_executable = /usr/local/dovecot/libexec/dovecot/imap-login

# IMAP executable location:
#mail_executable	= /usr/local/dovecot/libexec/dovecot/imap

# Maximum IMAP command line length in bytes:
#imap_max_line_length	= 65536

# Suypport for dynamically loadable modules:
#mail_use_modules	= no
#mail_modules		= /usr/local/dovecot/lib/dovecot/imap

# Whether to send IMAP capabilities in greeting message:
login_greeting_capability	= yes

# Workarounds for various client bugs:
imap_client_workarounds	= outlook-idle netscape-eoh

}

# POP3 SPECIFIC SETTINGS: #########################

protocol pop3 { # Login executable location: #login_executable = /usr/local/dovecot/libexec/dovecot/pop3-login

# POP3 executable location:
#mail_executable	= /usr/local/dovecot/libexec/dovecot/pop3

# Whether to not update recent/seen flags in POP3 sessions:
#pop3_no_flag_updates	= no

# Whether to support deprecated LAST command:
#pop3_enable_last	= no

# POP3 UIDL format to use:
#pop3_uidl_format	= %08Xu%08Xv

# Support for dynamically loadable modules
#mail_user_modules	= no
#mail_modules		= /usr/local/dovecot/lib/dovecot/pop3

# Workarounds for various client bugs:
pop3_client_workarounds	= outlook-no-nuls oe-ns-eoh

}

# AUTHENTICATION PROCESSES: ###########################

# Authentication executable location: #auth_executable = /usr/local/dovecot/libexec/dovecot/dovecot-auth

# Maximum authentication process size in megabytes: #auth_process_size = 256

# Authentication cache size in kilobytes: #auth_cache_size = 0

# Time to live in seconds for cached data: #auth_cache_ttl = 3600

# Space separated list of realms for SASL authentication mechanisms: #auth_realms =

# Default realm/domain to use if none was specified: #auth_default_realm =

# List of allowed characters in username: #auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.-_@

# Username character translation before it's looked up in database: #auth_username_translation =

# Whether to disable plaintext authentication disable_plaintext_auth = no

# More verbose authentication logging: auth_verbose = yes

# Even more verbose authentication logging for debugging purposes: auth_debug = yes

auth default { # Space separated list of wanted authentication mechanisms mechanisms = plain

# Password database specification:
passdb pam {
}

# User database specification:
userdb passwd {
}

# User to use for the authentication process:
user	= root

# Directory where to chroot the process:
#chroot	=

# Number of authentication processes to create:
#count	= 1

# Whether to require a valid SSL client certificate for
# authentication:
#ssl_require_client_cert	= no

}