July 2005 - dovecot - dovecot.org

[Dovecot] 1.0-test78
by Timo Sirainen 22 Jul '05

22 Jul '05

http://dovecot.org/test/ Fixes: - Crashes in non-x86 64bit systems - Bugs in cache file header caching (you probably should delete all dovecot.index.cache files to be sure the bug won't haunt you in future) - FETCH ENVELOPE patch by Chris Wakelin (I'll try to figure out what to do with the list patch later) Known bugs left: - Thunderbird + maildir: moving lots of messages from mailbox to another may cause deadlock (lock timeouts in half a minute or so and fails the operation). - I guess I should try to look at the Thunderbird + POP3 + lots of messages -> DELE error too again.. - I've heard that 1.0-tests are a lot slower than 0.99.x or 1.0-stable when checking new mail (with kmail), ie. client asking STATUS of all mailboxes. I've looked at their strace outputs and the difference (5-10x slower) comes from 1.0-test waiting longer in poll() for client input, which doesn't really make any sense to me. Can anyone verify this or think of reasons why this could happen? Maybe I'm setting some socket option differently, although I don't think I've changed anything..

8 15

[Dovecot] 1.0-stable and nfs
by Cyril Feraudet 22 Jul '05

22 Jul '05

Hi all, I've a little pb with 1.0-stable on nfs share without lockd : with : - lock_method = dotlock - mmap_disable = yes I've : - dovecot: Jul 22 11:58:33 Error: IMAP(xxxxxxxx): open() failed with index file (null): Bad address with : - lock_method = dotlock - mmap_disable = no I've : - dovecot: Jul 22 11:59:47 Error: IMAP(xxxxxxxx): lock_method=dotlock and mmap_disable=no combination isn't supported. You don't _really_ want it anyway. Someone have an idea ? Thanks, Cyril Feraudet

2 3

[Dovecot] dovecot starts and exit immediately? - help needed
by James Sun 22 Jul '05

22 Jul '05

Since nobody answered my previous question on dovecot marking new mails as old mails, I decided to build a version of dovecot of my own to debug. I downloaded and compiled dovecot 1.0 (stable). After setting up the conf file, I ran dovecot but it exited immediate with status 0. As a result, remote client cannot connect and retrieve mails. There is no man page, and I only found two command line options, -F and -c. None of these options seem to matter. Any suggestions? I must have done something simple and wrong here. Jun

2 4

[Dovecot] Bug in 0.99.14 mbox handling - mail UIDs are renumbered without bumping UIDVALIDITY
by David McBride 22 Jul '05

22 Jul '05

Package: dovecot-imapd Version: 0.99.14-1 Severity: important Hi, I think I've identified a bug in Dovecot 0.99.14 as packaged and distributed by Debian Sarge. I believe the bug lies in the original Dovecot source, not with any modifications made by Debian. Specifically, this bug deals with Dovecot's handling of message unique identifiers (UIDs) for IMAP clients when using the mbox mailbox storage format. Some background for those not familiar with the details; (I had to look a lot of this stuff up; see also http://www.ietf.org/rfc/rfc2060.txt) --- 8< -------------------------------------------------------------- All IMAP servers are required to maintain unique identifiers for each email in an IMAP mailbox that are persistent between sessions. Dovecot stores the unique identifiers for each email in an mbox in an X-UID header. Clients which maintain a local cache of already-retrieved emails use an email's UID as a key; thus, if the UID of a particular email were to change on a server then the client's cached state would be invalid. To cope with the case that the UIDs need to be invalidated, whether because the original store has been lost or for some other reason, there is also the UIDVALIDITY number. This 32bit integer must be increased to some higher value to indicate that any previously cached email ids are now invalid. In Dovecot-managed mbox files, this number is stored in the X-IMAPbase header in the first message of that mbox and is typically the uid-invalidation time expressed in unix timestamp format (ie the number of seconds elapsed since the epoch.) --- 8< -------------------------------------------------------------- Okay, the actual bug: Dovecot routinely (and apparently unneccessarily) renumbers all of the UIDs in our mbox-backed mailboxes without updating the UIDVALIDITY count. This results in caching mail clients (such as Evolution and Thunderbird) presenting the incorrect email to the user when browsing their mail. Configuration: We are using Dovecot as a standalone daemon (ie without using inetd.) We have configured the default mail environment with the following string: default_mail_env = mbox:%h/Mail/:INBOX=%h/.email:INDEX=MEMORY ie each end-user's primary INBOX is an mbox, ~/.email, and other IMAP folders are also stored as mboxes in the directory ~/Mail/. Indexes are maintained in memory but are not stored on disk. I don't think any of the rest of the configuration is relavant, but I'm happy to detail anything else about the setup if needed. Reproducing the bug: I'm afraid I don't have a proper protocol dump demonstrating this bug, so I'm just going to describe all of the end-user steps to recreate the problem: 1) Connect to the Dovecot IMAP service with a caching mail client, such as Evolution or Thunderbird. Download a local copy of the IMAP server contents of a test mbox-backed mailbox to the local cache. For example, I have a test mbox with old root mail. Messages are numbered from 1 to 163. The most recent message has an X-UID: field of 163. 2) Delete one of the messages in the middle of the mailbox. For example, mark the second-oldest message in the mailbox, which has a UID of 2, for deletion. Expunge the mailbox. 3) The UIDs of the other messages at this point have not been altered. Disconnect from the IMAP server. (With Evolution, I actually had to restart the program to reproduce this bug.) 4) Reconnect to the IMAP service, and allow the client to re-sync. You will find that the X-UID of all of the messages after the one that was deleted have been decremented -- and, if you're using a caching client, all of the message *bodies* will appear to have been shifted down by one. Reading the headers in the mbox file directly, Dovecot has renumbered all of the UIDs from 1 to N but not altered the UIDVALIDITY field: dwm@kalimdor $ grep X-UID: Root X-UID: 1 X-UID: 2 [... all numbers between 2 and 161 follow ...] X-UID: 161 X-UID: 162 dwm@kalimdor $ grep X-IMAPbase: Root X-IMAPbase: 1063356421 162 1063356421 == Sept 2003, when the mailbox was first created. This is a bug. Either Dovecot should not renumber the UIDs of each message, or (if it really is necessary), it should increase the UIDVALIDITY field to some higher number to invalidate the caches stored by any remote clients. This bug wasn't observed when using previous versions of Dovecot (we were previously using 0.99.10.6 before this upgrade without noticing anything broken.) I'm hoping someone familiar with the source tree can identify and suggest a fix relatively quickly; failing that, I'll take a look around to see if I can work out where the problem is and implement a fix it myself. Cheers, David -- David McBride <dwm(a)tastycake.net>

1 0

[Dovecot] Changes made to imap descriptor handling in test78?
by Dominic Marks 22 Jul '05

22 Jul '05

Hello, What has changed from test77 to test78 in the way imap manages its descriptors? I upgraded to test78 this morning on a small site which I use as a testbed for testing various things. Running the same copy of my kqueue patch with test77 works quite well [*], with test78 imap has some problems. Here is some debugging output I recorded: Jul 14 12:55:27 egg dovecot: imap-login: Login: user=<dom>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, SSL Jul 14 12:55:30 egg dovecot: IMAP(dom): kqueue was given an unknown descriptor 1, -1: No such file or directory The second line points refers to io_loop_handler_remove, the tcp/ip socket (1) is having the read filter (-1) removed, suspending events from the socket, or at least, that is what should be happening. From kevent(2) for ENOENT: [ENOENT] The event could not be found to be modified or deleted. This suggests to me that we are incorrectly calling io_loop_handler_remove before we should. This error happens immediately after a client opens their mailbox. I have checked pop3 and it does not share the problem. I'm going to have a look at the CVS mail archive to see if I can work out what has changed. [*] Occasionally I was receiving ENOENT in test77 also in io_loop_handler_remove, but only occasionally, perhaps a few times a day for each client. This also didn't occur when opening the mailbox, but some time later. I'm still looking for the trigger for that. Cheers, -- Dominic Marks

2 1

[Dovecot] dovecot pop3s delivers new messages as old (unread) messages ...
by James Sun 22 Jul '05

22 Jul '05

I just set up dovecot on fedora core 3 with only pop3s protocol enabled. On my client machine, I use fetchmail to get the mail and then go through procmail and eventually ready by mutt. Everything seems to work fine, except that all new emails are marked old unread emails (i.e., marked with 'O' flag instead of 'N'). This is annoying because some open source lists are so large that I really rely on the "N" flag to read my emails. I tested the same client setting with gmail and my isp (comcast) pop3 accounts. They work fine - new mails are marked as "N". This makes me suspect dovecot is not set up correctly or something on the pop3 server machine is not right. Anybody has an idea? Thanks in advance. Jun Here are more detailed info: -bash-3.00# rpm -qa | grep dovecot dovecot-0.99.13-3.FC3 /etc/dovecot.conf: ---------------------------- protocols = pop3s imap_listen = [::] pop3_listen = [::] imaps_listen = [::] pop3s_listen = [::] ssl_cert_file = /usr/share/ssl/certs/dovecot.pem ssl_key_file = /usr/share/ssl/private/dovecot.pem login_dir = /var/run/dovecot-login login = imap login = pop3 mbox_locks = fcntl auth = default auth_mechanisms = plain auth_userdb = passwd auth_passdb = pam auth_user = root auth_verbose = no .fetchmailrc on client machine -------------------------------------- poll mypop.server.com protocol pop3 user "jsun" password "xxxx" is jsun here ssl

2 3

[Dovecot] Double Emails
by Matt Juszczak 22 Jul '05

22 Jul '05

Hi all, I originally posted this on postfix's mailing list but then realized that this COULD be a pop3 server problem since that downloads and deletes mail. This has only been happening since I upgraded from .99 to 1.0-7 ... Our users have been reporting double messages in their inbox, as well as emails that they deleted a long time ago reappearing. The only thing in my maillog is a bunch of: mbox sync: UID inserted in the middle of mailbox /home/sdf/mail/Level3 (6293 > 3) Thats happening for IMAP folder and /var/mail/* mbox folders alike. I can't seem to figure out the problem, but we have at least 50 customers calling in today alone to complain about double messages. More are probably experiencing it and just not calling. Any ideas? Thanks! -Matt

3 5

Re: [Dovecot] Update notifier
by Sebastjan Trepca 22 Jul '05

22 Jul '05

Great! Yes, of course, I just didn't knew that plugins are supported. Are there any documents on this subject? How do you add a plugin to dovecot? Just compile it and copy it to dovecot's lib/imap (like in example) ? Thanks, Sebastjan On 7/22/05, Timo Sirainen <tss(a)iki.fi> wrote: > On Fri, 2005-07-22 at 12:08 +0200, Sebastjan Trepca wrote: > > I'm trying to add a code in dovecot that will notify my services when > > a message is updated/added/moved/copied. I checked dovecot code a bit > > and mail-storage.c seems to be a nice place for tracking but I'm not > > sure if I can track all changes there. > > There are no update/move commands in IMAP, so you'll just need to hook > into saving and copying. Normally copying also goes through save unless > you're using maildir + maildir_copy_with_hardlinks=yes. > > See quota plugin for example how to hook into saving and copying: > http://dovecot.org/patches/1.0/quota.tar.gz > > Probably better to do it as a plugin than directly modify the sources, > so you don't have to keep patching it for each new release. > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.0 (GNU/Linux) > > iD8DBQBC4PdEyUhSUUBViskRApj9AJoCzEM8iNRNaEIcHdGxZDKoSbatFQCfaTMc > 62w52fPQr0uE8wb1A8y9kUY= > =9gtQ > -----END PGP SIGNATURE----- > > >

2 1

[Dovecot] Another minor IMAP LIST issue
by Chris Wakelin 22 Jul '05

22 Jul '05

I've just noticed that Dovecot 1.0-stable and 1.0-test78 don't include mbox folders with names beginning with "." in the IMAP LIST output. These are often used to store "hidden" folders for storing things like IMAP client configuration (e.g. Pine 4.x, IMHO, Prayer). Usually the user shouldn't be able to see these, but there are occassions when they might. It seems to me to be better to let the client decide. As I understand it, the IMAP RFC suggests that if you can "SELECT folderpath" a folder, you should also be able to list it with 'LIST "" "folderpath"' I've written a patch against 1.0-stable (it also works with test78) to add an extra namespace option "showdotfiles" (default "no"), to include folders beginning with "." (but excluding ".imap" and "." and ".." for obvious reasons!). It's used as follows :- # default namespace namespace private { separator = "/" prefix = inbox = yes showdotfiles = yes } # for backwards compatibility with UW-IMAP: namespace private { separator = "/" prefix = "mail/" hidden = yes showdotfiles = yes } ... Let me know if you find it useful! Best Wishes, Chris -- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin(a)reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094 diff -ruN dovecot-1.0-stable.orig/src/imap/cmd-list.c dovecot-1.0-stable/src/imap/cmd-list.c --- dovecot-1.0-stable.orig/src/imap/cmd-list.c Fri Mar 18 20:00:44 2005 +++ dovecot-1.0-stable/src/imap/cmd-list.c Wed Jul 20 16:36:51 2005 @@ -293,6 +293,8 @@ list_flags = ctx->list_flags; if (*ns->prefix == '\0' || ns->inbox) list_flags |= MAILBOX_LIST_INBOX; + if (ns->showdotfiles) + list_flags |= MAILBOX_LIST_DOTFILES; ctx->list_ctx = mail_storage_mailbox_list_init(ns->storage, cur_ref, cur_mask, list_flags); diff -ruN dovecot-1.0-stable.orig/src/imap/namespace.c dovecot-1.0-stable/src/imap/namespace.c --- dovecot-1.0-stable.orig/src/imap/namespace.c Mon Jan 31 16:37:54 2005 +++ dovecot-1.0-stable/src/imap/namespace.c Wed Jul 20 16:36:51 2005 @@ -31,7 +31,7 @@ { struct namespace *ns; const char *sep, *type, *prefix; - int inbox, hidden, subscriptions; + int inbox, hidden, showdotfiles, subscriptions; ns = p_new(pool, struct namespace, 1); @@ -40,6 +40,7 @@ prefix = getenv(t_strdup_printf("NAMESPACE_%u_PREFIX", num)); inbox = getenv(t_strdup_printf("NAMESPACE_%u_INBOX", num)) != NULL; hidden = getenv(t_strdup_printf("NAMESPACE_%u_HIDDEN", num)) != NULL; + showdotfiles = getenv(t_strdup_printf("NAMESPACE_%u_SHOWDOTFILES", num)) != NULL; subscriptions = getenv(t_strdup_printf("NAMESPACE_%u_SUBSCRIPTIONS", num)) != NULL; @@ -58,6 +59,7 @@ ns->prefix = p_strdup(pool, prefix); ns->inbox = inbox; ns->hidden = hidden; + ns->showdotfiles = showdotfiles; ns->subscriptions = subscriptions; ns->storage = mail_storage_create_with_data(data, user); if (ns->storage == NULL) { diff -ruN dovecot-1.0-stable.orig/src/imap/namespace.h dovecot-1.0-stable/src/imap/namespace.h --- dovecot-1.0-stable.orig/src/imap/namespace.h Mon Jan 31 16:37:54 2005 +++ dovecot-1.0-stable/src/imap/namespace.h Wed Jul 20 16:36:51 2005 @@ -16,7 +16,7 @@ char *prefix; size_t prefix_len; - int inbox, hidden, subscriptions; + int inbox, hidden, showdotfiles, subscriptions; struct mail_storage *storage; }; diff -ruN dovecot-1.0-stable.orig/src/lib-storage/index/mbox/mbox-list.c dovecot-1.0-stable/src/lib-storage/index/mbox/mbox-list.c --- dovecot-1.0-stable.orig/src/lib-storage/index/mbox/mbox-list.c Mon Jan 31 16:37:55 2005 +++ dovecot-1.0-stable/src/lib-storage/index/mbox/mbox-list.c Wed Jul 20 16:36:51 2005 @@ -225,13 +225,19 @@ enum imap_match_result match, match2; int ret, noselect; - /* skip all hidden files */ - if (fname[0] == '.') + /* skip all hidden files unless show dotfiles is set*/ + if (fname[0] == '.' && !(ctx->flags & MAILBOX_LIST_DOTFILES)) return 0; /* skip all .lock files */ len = strlen(fname); if (len > 5 && strcmp(fname+len-5, ".lock") == 0) + return 0; + + /* skip ".imap" "." and ".." directories in case we show dotfiles */ + if (((len == 5) && strcmp(fname, ".imap") == 0) || + ((len == 1) && fname[0] == '.') || + ((len == 2) && strcmp(fname, "..") == 0)) return 0; /* check the mask */ diff -ruN dovecot-1.0-stable.orig/src/lib-storage/mail-storage.h dovecot-1.0-stable/src/lib-storage/mail-storage.h --- dovecot-1.0-stable.orig/src/lib-storage/mail-storage.h Mon Jan 31 16:37:55 2005 +++ dovecot-1.0-stable/src/lib-storage/mail-storage.h Wed Jul 20 16:36:51 2005 @@ -16,7 +16,8 @@ MAILBOX_LIST_SUBSCRIBED = 0x01, MAILBOX_LIST_FAST_FLAGS = 0x02, MAILBOX_LIST_CHILDREN = 0x04, - MAILBOX_LIST_INBOX = 0x04 + MAILBOX_LIST_INBOX = 0x04, + MAILBOX_LIST_DOTFILES = 0x08 }; enum mailbox_flags { diff -ruN dovecot-1.0-stable.orig/src/master/mail-process.c dovecot-1.0-stable/src/master/mail-process.c --- dovecot-1.0-stable.orig/src/master/mail-process.c Tue Feb 8 10:30:11 2005 +++ dovecot-1.0-stable/src/master/mail-process.c Wed Jul 20 16:36:51 2005 @@ -175,6 +175,8 @@ env_put(t_strdup_printf("NAMESPACE_%u_INBOX=1", i)); if (ns->hidden) env_put(t_strdup_printf("NAMESPACE_%u_HIDDEN=1", i)); + if (ns->showdotfiles) + env_put(t_strdup_printf("NAMESPACE_%u_SHOWDOTFILES=1", i)); t_pop(); } } diff -ruN dovecot-1.0-stable.orig/src/master/master-settings.c dovecot-1.0-stable/src/master/master-settings.c --- dovecot-1.0-stable.orig/src/master/master-settings.c Mon Jan 31 16:37:55 2005 +++ dovecot-1.0-stable/src/master/master-settings.c Wed Jul 20 16:36:51 2005 @@ -194,6 +194,7 @@ DEF(SET_STR, location), DEF(SET_BOOL, inbox), DEF(SET_BOOL, hidden), + DEF(SET_BOOL, showdotfiles), { 0, NULL, 0 } }; diff -ruN dovecot-1.0-stable.orig/src/master/master-settings.h dovecot-1.0-stable/src/master/master-settings.h --- dovecot-1.0-stable.orig/src/master/master-settings.h Mon Jan 31 16:37:55 2005 +++ dovecot-1.0-stable/src/master/master-settings.h Wed Jul 20 16:36:51 2005 @@ -163,6 +163,7 @@ int inbox; int hidden; + int showdotfiles; }; struct server_settings {

3 5

[Dovecot] subscriptions vs mailboxlist
by Marc Perkel 22 Jul '05

22 Jul '05

Is there a way to change the name of the .subscriptions file that dovecot looks for to .mailboxlist to be compatible with WU-IMAP? It would be handy for testing and conversion if you had that. BTW - I'm a new user and your software is very well designed. -- Marc Perkel - marc(a)perkel.com Spam Filter: http://www.junkemailfilter.com My Blog: http://marc.perkel.com

2 1