[Dovecot] Dovecot-1.0.beta1: New mail sometimes fails to appear

[Tom Alsberg]

Hi there...

Having updated to 1.0.beta1, I for a while thought a bad bug with IDLE 
handling that caused new mail in INBOX not to appear when using Mozilla 
Thunderbird has been solved.  Although I can not reproduce one kind of 
problem anymore, after a short while it again appears that one does not 
always see one's new mail with Thunderbird.

I can not consistently reproduce the bug, but from some network captures 
it indeed appears that sometimes Thunderbird has (in one of its 
connections to the server) INBOX SELECTed and IDLE on it, new mail 
arrives, but 10 minutes afterwards it was still not sent an EXISTS message.

Initially the symptoms appeared to suggest that something really bad 
happened to the index files, but I tried INDEX=MEMORY and that did not 
help much.
Thunderbird using multiple connections to the server does of course only 
make debugging more difficult.  But it appears that without multiple 
connections, one has to constantly cycle around SELECTing to get 
notified on more than one mailbox, so that may justified.

I am a bit lost on this now, and would appreciate some ideas.  Although 
there were several related bugs I noticed in Thunderbird 1.0, it appears 
that they are gone in 1.5, and yet now it really seems Dovecot is doing 
something wrong here.

Attached is the configuration file I use for the testing installation of 
Dovecot 1.0.beta1 (somebody tell me if you see something wrong there). 
The mailbox arrangement is as follows, right now (until we have a better 
way):

* INBOX in /var/mail/%u
* most mailboxes in ~/imap.maildir/... as maildirs.
* mboxes in ~/imap.mbox/... accessible as #imap.mbox/...
* the mbox:~/mail/ is for when namespaces are not used - now there are 
so it should not be relevant

Except for myself, and one test account I have, I have two guinea-pigs 
using it (who are well aware that it is still under development and that 
  it will break once in a while) and reporting problems to me.

Vaguely (since I can't figure the exact reasons), the symptom can be 
described as follows:  When using Mozilla Thunderbird to access mail, 
new mail arrives to INBOX (mbox:/var/mail/%u), but is sometimes not seen 
for a long while after it arrives.  Sometimes the first (or first few) 
message arriving appears, but then no new messages appear, and sometimes 
everything works right and immediately new mail is seen.  However 
whenever (according to Thunderbird) something changes in the mailbox, 
the mailbox is seen up-to-date as it really was at that time.

   Thanks, any help appreciated,
   -- Tom

-- 
   Tom Alsberg - hacker (being the best description fitting this space)
   Web page:	http://www.cs.huji.ac.il/~alsbergt/
DISCLAIMER:  The above message does not even necessarily represent what
my fingers have typed on the keyboard, save anything further.
-------------- next part --------------
# Dovecot IMAP server configuration at CS.HUJI
##############################################
#
# $Id$

# GENERAL PARAMETERS:
#####################

# Base directory where to store runtime data:
base_dir	= /var/run/dovecot

# Protocols we want to be serving:
protocols	= imap pop3

# IP or host address where to listen in for connections:
listen		= *

ssl_disable	= yes


# LOGIN PROCESSES:
##################

# Directory for authentication sockets:
#login_dir	= /var/run/dovecot/login

# chroot login process to the login_dir:
#login_chroot	= yes

# User to use for the login process:
login_user	= nobody

# Maximum process size in megabytes:
#login_process_size	= 16

# Have each login be processed in its own process:
#login_process_per_connection	= yes

# Number of login processes to create:
#login_processes_count	= 3

# Maximum number of extra login processes to create:
#login_max_processes_count	= 128

# Maximum number of connections allowed in login state:
#login_max_logging_users	= 256

# Greeting message for clients:
login_greeting	= Dovecot mail access server at CS.HUJI ready.


# MAIL PROCESSES:
#################

# Maximum number of running mail processes:
#max_mail_processes	= 1024

# Show more verbose process title (in ps):
verbose_proctitle	= yes

# Valid UID range for users:
first_valid_uid		= 10
#last_valid_uid		= 0

# Valid GID range for users:
#first_valid_gid	= 1
#last_valid_gid		= 0

# Extra groups for mail processes:
#mail_extra_groups	=

# ':' separated list of directories under which chrooting is allowed:
#valid_chroot_dirs	=

# Default chroot directory for mail processes:
#mail_chroot	=

# Default MAIL environment to use when it's not set:
default_mail_env	= mbox:~/mail/:INBOX=/var/mail/%u


# NAMESPACES:
#############

namespace	private {
	separator	= /
	prefix		= ""
	location	= maildir:~/imap.maildir/
}


namespace	private {
	separator	= /
	prefix		= "#imap.mbox/"
	location	= mbox:~/imap.mbox/:INBOX=/var/mail/%u
	inbox		= yes
}


# MAIL STORE:
#############

# Space-separated list of fields to initially save into cache file:
#mail_cache_fields	=

# Space-separated list of fields to never save into cache file:
#mail_never_cache_fields	=

# Like mailbox_check_interval, but used for IDLE command:
#mailbox_idle_check_interval	= 30

# Whether to allow full filesystem access to clients:
#mail_full_filesystem_access	= no

# Maximum allowed length for mail keyword name:
#mail_max_keyword_length	= 50

# Whether to save mails with CR+LF instead of plain LF:
#mail_save_crlf	= no

# Whether to use mmap to access mail files:
#mail_read_mmaped	= no

# Whether to disable mmap at all:
mmap_disable	= yes

# Whether to disable write to mmap:
#mmap_no_write	= no

# Locking method for index files:
lock_method	= dotlock

# Whether to list only directories in LIST command:
maildir_stat_dirs	= yes

# Whether to copy mail to other folders using hard links:
#maildir_copy_with_hardlinks	= no

# Locking methods to use for locking mbox:
mbox_read_locks		= dotlock
mbox_write_locks	= dotlock

# Maximum time in seconds to wait for lock (all of them):
mbox_lock_timeout	= 300

# Timeout to overwrite dotlock files if mailbox isn't modified:
mbox_dotlock_change_timeout	= 30

# Whether to read only appended data on change (unless corrupt):
#mbox_dirty_syncs	= yes

# Like mbox_dirty_syncs, but no full syncs with SELECT, EXAMINE,
# EXPUNGE, or CHECK:
#mbox_very_dirty_syncs	= no

# Whether to delay writing mbox headers until doing a full write sync:
#mbox_lazy_writes	= yes

# umask to use for mail files and directories:
umask			= 0077

# Whether drop all privileges before exec of mail process:
#mail_drop_priv_before_exec	= no

# Maximum mail process size in megabytes:
#mail_process_size	= 256

# Log prefix for mail processes:
mail_log_prefix		= "%Us(%u): "


# IMAP SPECIFIC SETTINGS:
#########################

protocol imap {
	# Login executable location:
	#login_executable	= /usr/local/dovecot/libexec/dovecot/imap-login

	# IMAP executable location:
	#mail_executable	= /usr/local/dovecot/libexec/dovecot/imap

	# Maximum IMAP command line length in bytes:
	#imap_max_line_length	= 65536

	# Suypport for dynamically loadable modules:
	#mail_use_modules	= no
	#mail_modules		= /usr/local/dovecot/lib/dovecot/imap

	# Whether to send IMAP capabilities in greeting message:
	login_greeting_capability	= yes

	# Workarounds for various client bugs:
	imap_client_workarounds	= outlook-idle netscape-eoh
}


# POP3 SPECIFIC SETTINGS:
#########################

protocol pop3 {
	# Login executable location:
	#login_executable	= /usr/local/dovecot/libexec/dovecot/pop3-login

	# POP3 executable location:
	#mail_executable	= /usr/local/dovecot/libexec/dovecot/pop3

	# Whether to not update recent/seen flags in POP3 sessions:
	#pop3_no_flag_updates	= no

	# Whether to support deprecated LAST command:
	#pop3_enable_last	= no

	# POP3 UIDL format to use:
	#pop3_uidl_format	= %08Xu%08Xv

	# Support for dynamically loadable modules
	#mail_user_modules	= no
	#mail_modules		= /usr/local/dovecot/lib/dovecot/pop3

	# Workarounds for various client bugs:
	pop3_client_workarounds	= outlook-no-nuls oe-ns-eoh
}


# AUTHENTICATION PROCESSES:
###########################

# Authentication executable location:
#auth_executable	= /usr/local/dovecot/libexec/dovecot/dovecot-auth

# Maximum authentication process size in megabytes:
#auth_process_size	= 256

# Authentication cache size in kilobytes:
#auth_cache_size	= 0

# Time to live in seconds for cached data:
#auth_cache_ttl		= 3600

# Space separated list of realms for SASL authentication mechanisms:
#auth_realms	=

# Default realm/domain to use if none was specified:
#auth_default_realm	=

# List of allowed characters in username:
#auth_username_chars	= abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.-_@

# Username character translation before it's looked up in database:
#auth_username_translation	=

# Whether to disable plaintext authentication
disable_plaintext_auth	= no

# More verbose authentication logging:
auth_verbose	= yes

# Even more verbose authentication logging for debugging purposes:
auth_debug	= yes

auth	default {
	# Space separated list of wanted authentication mechanisms
	mechanisms	= plain


	# Password database specification:
	passdb pam {
	}

	# User database specification:
	userdb passwd {
	}

	# User to use for the authentication process:
	user	= root

	# Directory where to chroot the process:
	#chroot	=

	# Number of authentication processes to create:
	#count	= 1

	# Whether to require a valid SSL client certificate for
	# authentication:
	#ssl_require_client_cert	= no
}