- dovecot - dovecot.org

[dovecot] 0.99.0 released
by Timo Sirainen 24 Nov '02

24 Nov '02

I figured now would be a good time for new release. All the major changes are pretty much done now. v0.99.0 2002-11-24 Timo Sirainen <tss(a)iki.fi> + Replaced hash file with binary tree file which makes Dovecot stay fast with large mailboxes after expunging multiple mails. + Several speed improvements with SEARCH + SEARCH CHARSET support using iconv(), although case-insensitive searching is currently supported only for ASCII characters. + OpenSSL support. + Support for regenerating Diffie Hellman and RSA parameters with specified intervals. NOTE: currently doesn't work with OpenSSL. + Support for each login connection being handled in it's own process. This is the default as it's more safe especially with SSL. + mbox locking is now safe, other processes can't modify the mbox file while we're reading it. + Notify clients with "EXISTS" almost immediately after new mail is received. + Rawlog: Support for saving user connections into files - useful for debugging. + Content-Language is finally parsed correctly + Lots of smaller speed optimizations - Partial BODY[] fetches weren't working properly - BODY[section] was buggy with message/rfc822 MIME parts - STARTTLS wasn't working - \* flag was missing from PERMANENTFLAGS. - Comments inside <> mail addresses crashed. - imap-login printed UTC timestamps to logfiles - passwd-file wasn't reread the the file changed - PAM authentication was implemented wrong, which caused it to break with some PAM plugins. - Lots of smaller fixes, mostly to do with reliability

1 0

[dovecot] Seen bug?
by Simon Lindgren 13 Nov '02

13 Nov '02

Hi, I'm using Kmail and dovecot for my IMAP pleasure, and have had some problems, which are evident from the following exchange with some kmail developers. My patch to the imap protocol in KDE fixed my problem, but should dovecot be changed instead? Thank you for any insight! I've included the discussion on kmail-devel so far: [ I wrote: ] -- I've recently switched to IMAP for email (using dovecot as the server, http://dovecot.procontrol.fi/) and have a small problem: Clicking a message brings it up, but doesn't seem to mark it read on the server. I need to select 'Message -> Mark Message -> Mark Message as Read' in order for it to "stay read" when I check for new mails on the server. Is anyone else seeing this behaviour? Is this a bug, or a feature? I've assigned a shortcut key for 'Mark Message as Read' so I get by, but with a large number of messages and folders, it quickly becomes annoying. -- [ Carsten Burghardt <burghardt(a)kde.org> wrote in reply: ] -- As your imap-server is not really common you should check with ethereal the communication between kmail and your server. -- [ I wrote in reply: ] -- Thanks for the replies... snooping with ethereal, I get the following: (quoting only the data sent from kmail) When viewing a mail in kmail: UID FETCH 203 (UID RFC822) When specifically flagging it as 'read' in kmail: UID STORE 203 -FLAGS.SILENT (\SEEN \ANSWERED \FLAGGED \DRAFT) UID STORE 203 +FLAGS.SILENT (\SEEN) It would seem that my problem would be solved if kmail also sent the +FLAGS stuff when viewing a message... but I don't know if this would be according to the IMAP specification - or if the server should flag the message as 'seen' when the client does the 'fetch' ? I do have 'Mark selected message as read after 0 sec' checked - and the message is flagged in kmail as read, but without specifically marking it as read in kmail, it's still 'unread' after the next 'check mail' run. I hope I'm being clear. If you still think this is a bug with my (non standard, I agree!) IMAP server I'll try another one. -- [ I wrote another reply: ] -- I made a change to imap4.cc, and it magically works for me - I've not tested this one very much, but it seems to do the trick so far. I would be very interested if someone knowledgeable would tell me just how wrong this piece of code is, as I'm working on learning these things. --- imap4.cc Tue Oct 15 21:52:43 2002 +++ imap4.cc Tue Nov 12 01:36:05 2002 @@ -289,6 +289,11 @@ } completeQueue.removeRef (cmd); + + if (aSection.find ("ENVELOPE", 0, false) == -1) { + doCommand (imapCommand:: + clientStore (aSequence, "+FLAGS.SILENT", "\\SEEN")); + } } } -- [ Carsten Burghardt <burghardt(a)kde.org> wrote: ] -- No, everything is alright here, you should check a different server. -- [ Bo Thorsen <bo(a)sonofthor.dk> wrote: ] -- It is definately a bug in the server. The IMAP RFC specifies that a server must only once serve a file flagged as new. So your server violates this principle.

2 1

[dovecot] Re: pam + radius
by Timo Sirainen 03 Nov '02

03 Nov '02

On Thu, 2002-10-31 at 21:32, Hielke Christian Braun wrote: > It helped a bit. Now the first login to dovecot works fine. The domainpart > of my username is not changed. But after the first login, dovecot > always sents the username from the first login to the radius server even when > i login from a different client with a complete different username. OK, fixed now in CVS. I implemented PAM support pretty wrong.

1 0

[dovecot] Re: pam + radius
by Timo Sirainen 29 Oct '02

29 Oct '02

On Tue, 2002-10-29 at 06:00, Hielke Christian Braun wrote: > > You probably don't have the users in /etc/passwd file too, right? > > I have the users in the passwd and shadow files as i need that for quotas > to work. Though in the shadow file i don't have the password and only a x. > The problem must be something else. Well .. I don't know then really. Since you did get it to work by changing PAM to use shadow auth, Dovecot is doing it at least partly right. Maybe the radius PAM module requires something that Dovecot didn't do.. Looking at Courier's PAM handling, it does pam_setcred() which dovecot doesn't. You could try if doing that helps: src/auth/userinfo-pam.c, around line 169, insert between pam_authenticate() and pam_acct_mgmt(): if ((status = pam_setcred(pamh, PAM_ESTABLISH_CRED)) != PAM_SUCCESS) { if (status == PAM_ABORT) i_fatal("pam_setcred_mgmt() requested abort"); return FALSE; } > Maybe it dovecot sets a realm, which is then mistakenly used by > the pam radius module, but not by the passwd/shadow module? PAM doesn't have any support for realms AFAIK.

1 0

[dovecot] Re: pam + radius
by Timo Sirainen 29 Oct '02

29 Oct '02

On Tue, 2002-10-29 at 05:01, Hielke Christian Braun wrote: > i am trying to use dovecot with pam and radius. My users have names > in the format joe(a)somedomain.com. When i have pam configured to use > the normal passwd/shadow files it works fine. With radius it does not. > I see at the radius server that the domain part of my usernames > is always replaced with the same domain @nikojet.com. I don't think > it is a problem with the pam radius, as the same library works fine > with the solid state pop3 server. Is this a fundamental problem and > dovecot/imap does not work with usernames which have a domain part or > is it a bug? You probably don't have the users in /etc/passwd file too, right? Dovecot currently wants the users to exist there too to get their UID, GID and home directory. I'll change this later so that you could give "gid=123 uid=456 homeroot=/var/mail" options to pam auth and it'd use them for all users. Or did PAM also support getting that information in some way? I'm not sure exactly..

1 0

[dovecot] vsnprintf()
by Thomas Wouters 23 Oct '02

23 Oct '02

I think I stumbled upon a bug in the i_snprintf() function. In the case of vnsprintf() being available, it depends on vnsprintf() returning -1 when the string was longer than the passed-in limit (or it won't terminate the string.). But this isn't the C99-standardized behaviour, and newer glibc's don't do that anymore either, so you can end up with a non-terminated string. This patch should fix it, I think. Index: strfuncs.c =================================================================== RCS file: /home/cvs/dovecot/src/lib/strfuncs.c,v retrieving revision 1.14 diff -c -u -r1.14 strfuncs.c --- strfuncs.c 20 Oct 2002 03:19:10 -0000 1.14 +++ strfuncs.c 23 Oct 2002 11:19:39 -0000 @@ -401,7 +401,7 @@ va_end(args); t_pop(); - if (ret < 0) { + if (ret < 0 || ret >= max_chars) { str[max_chars-1] = '\0'; ret = strlen(str); } -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

3 4

[dovecot] anonymous CVS access
by Timo Sirainen 23 Oct '02

23 Oct '02

OK, added anon-pserver: cvs -d :pserver:anonymous@dovecot.procontrol.fi:/home/cvs co dovecot

1 0

[dovecot] New mailing list: dovecot-cvs
by Timo Sirainen 23 Oct '02

23 Oct '02

Installed syncmail program what Thomas was wanting. It sends a mail of each cvs commit, including the actual changes in source. Otherwise the mailing list is read-only. Subscribing happens the same way as to dovecot list, ie. dovecot-cvs-request(a)procontrol.fi with subscribe in subject. Hmm. I think I should also separate the private/public cvs repositories running in that computer and provide anonymous pserver access..

1 0

[dovecot] Architectural questions
by Thomas Wouters 22 Oct '02

22 Oct '02

Greetings, I have some architectural questions regarding dovecot, and though I've half answered them by looking at the source, I'm also interested in hearing whether my (our) wishes and suggestions are already being considered (or can be considered, once built) for inclusion in dovecot itself. Let me first explain why I'm doing this. I work for XS4ALL, a fairly large ISP in the Netherlands. We provide a wide variety of services, including shell access, pop3, webmail, et cetera. We use Sendmail on several clusters of FreeBSD machines (loadbalanced using layer-4 ethernet switches) and several NetApp Filers (dedicated NFS servers with fail-safe disk-arrays and such) for backend. Several years ago (when we were a lot smaller) we noticed the typical use of the mailboxes included leaving much old email on the server, at least for a while, and that this is a bothersome thing when using mbox mailboxes. (The mboxes basically have to be copied over whenever the status of an email changes, leading to a lot of I/O.) We briefly played with modifying sendmail and the pop server to avoid the full copy in the common case (only status changes) by doing in-place edits of a pre-generated Status line, as well as avoid full scanning of the mbox file by creating special headers to mark the 'real' length of an email. It worked, for a while, but it wasn't going to scale very well. So we switched to maildir mailboxes for the mail spool. A modified mail.local (which we need for other reasons as well) delivers in /var/spool/mail/u/s/username, and mutt, uqwk, a modified pine and qmail's pop3 daemon read it from there. Until last week our clients could choose to have mbox inboxes, to use with 'elm' or 'mail', but we decided to discontinue that support. Our new shell servers, which are in test, don't have elm installed anymore anyway. We still have support for mbox mailboxes in a user's homedirectory though, by using procmail and such. So when we needed an IMAP server for use with our webmail (based on SquirrelMail), we were forced to go with the UW-IMAP server, with the maildir patch that's been scattered around the 'net. This worked, for a while; we also use the maildir patch with pine after all. However, the maildir patch is not very good. Not at all, even, and it only seems to work by pure chance. Pine works for the average user who does not get a lot of new mail while his pine is open or does not use procmail, and fortunately a lot of the people that do get a lot of email use mutt, which does work properly. The UW-IMAP server worked fine because SquirrelMail only uses (used) a small subset of the available functionality. But that's changing, as SquirrelMail gets actively developed, and we're also considering other IMAP-based services. But we can't switch to Courier or Cyrus because we need mbox support. And while looking for mbox patches for either of those two, I ran across dovecot. Yay! :) Dovecot is not everything we'd want, but it comes very close, and contrary to UW-IMAP both the design and the actual source code are clean, readable and logical, which means we can add the features we need and support them. What we need and want to add is fairly simple, but I've only been looking at dovecot since yesterday so I'd be happy to hear if any of it is possible, feasible, unwise or unacceptable. - First off, we need the maildir support to be 'correct' in that it does not rely on the naming of the files in the mailbox, other than the very loose specification DJB gives (doesn't contain a colon or slash and doesn't start with a dot.) The pine/UW-imap patch breaks here because it depends on the first part of the filename being time() or something else that, when sorted alphanumerically, puts new mail at the end. Our LDA does this, but procmail does not, and it shouldn't have to. - Second, we need the maildir support to be 'correct' in that it does not rely on the directory order being persistant. The NetApp Filers use btree-indexed directories, so the order of readdir() can change completely whenever a file is added or removed. The pine/uw-imap patch relies on the '.uidvalidity' file being modified whenever the maildir sort order changed, and this isn't happening. I *think*, from reading the sources, both of those are correct already. If they aren't, I'd strongly urge you to fix it, as #1 is a problem for anyone using procmail and #2 is a problem for anyone with 'indexed' directories (including such new filesystems as reiserfs, and I assume FreeBSD's new hashed directories.) - We need to avoid using fcntl(). The Netapps support it, but file-locking over NFS is very, very poorly designed and we've had too much problems of various kinds before, with fcntl. We also don't like the idea of having thousands of fcntl locks at the same time ;P Instead, we've switched to the locking method described in the Linux open(2) manpage under O_EXCL. (We call it 'dot-locking', I'm not sure where the name came from.) The actual implementation of that method is pretty simple, and I have a C version and a Python version hanging around here somewhere (the Python version is being used by GNU Mailman, last I looked.) If we're going to use dovecot, we will replace most, if not all, fcntl()s with dot-locking, the question is whether you want it contributed to dovecot :) - Every user's incoming mailbox is /var/spool/u/s/username. Other mailboxes are in /home/u/username/mail or /home/u/username/Mail (the second if the first does not exist.) We are not yet certain whether we want the inbox to be able to have subdir-mailboxes, as /var/spool and /home have different quotas and we urge people not to store their mail on /var/spool. (for one thing, it doesn't get backed-up.) We want these things to work without magical symlinks or empty files, because people _will_ delete them and cause unnecessary helpdesk calls :) Again, the question is mostly whether this is desirable in dovecot (or something enough like it to reduce local changes.) - We have over 300k mailboxes at the moment. We expect that number to keep growing. The indexer process (as described by design.txt) does not sound as a good idea in our case :) How necessary is it, really ? Especially since we do not expect more than 10% of those mailboxes to be actually used by IMAP, not even once. If disabling the indexer completely just means longer startup times for IMAP sessions, we can live with that. - The UW-IMAP maildir patch stores UID's in the indiviual filenames, using a 'U' flag. Will this interfere with dovecot ? We don't really need dovecot and UW-IMAP to share UIDs, but we would like to have an as painless transition as possible, without having to rename millions of files to remove the U flag and other flags :P It would also be nice to keep pine using the existing maildir patch, even though very few IMAP-users would use pine. - Would dovecot scale, architecturally speaking, to 500k+ active mailboxes ? The amount of hardware is not really an issue, we can add a lot of machines (off-the-shelve intel hadware) to each cluster, but if each dovecot process has to load in an index of all possible mailboxes... that would be a problem. Doing an inordinate number of file-accesses over NFS would also be a problem, but I haven't seen any indication of that in the source, yet. In case it wasn't clear yet, I'm very happy to have found dovecot. The lack of a decent mbox IMAP server has always dismayed me, let alone an mbox+maildir one :) I should also point out that even though XS4ALL is a commercial company, we would contribute our changes even if the licence didn't require it, and we want to contribute them back the way you want them, not necessarily the way it's easiest for us. We have a lot of experience with opensource software, as a simple google on my name should indicate ;P Regards, -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

3 24

[dovecot] Re: Architectural questions
by Thomas Wouters 21 Oct '02

21 Oct '02

On Sat, Oct 19, 2002 at 12:42:25PM -0400, Charlie Brady wrote: > > So when we needed an IMAP server for use with our webmail (based on > > SquirrelMail), we were forced to go with the UW-IMAP server, with the > > maildir patch that's been scattered around the 'net. This worked, for a > > while; we also use the maildir patch with pine after all. However, the > > maildir patch is not very good. Not at all, even, and it only seems to > > work by pure chance. > The best one I've found is the patch last modified AFAICT by Miquel van > Smoorenburg, which has maildir filenames like: > time.pid.host,U=xxx,W=yyy:2,flags We use two different patches, both of which have a bit of Mike in them. One is for the IMAP server, which is an old version ('uw-imap-2000' or something like that comes to mind, but it came from the pine 4.10 source) from before the mailbox->append prototype changed (but with Mike's bilennium-patch.) A big problem with later patches was that the append method changed in the pine source, but not in the maildir patch. We considered using a newer uw-imap but decided that the current one works good enough ;P For pine we currently use the patch that comes with Debian's 'pine-tracker' package (which is an installer for pine with patches, since you aren't allowed to distributed modified binaries.) This patch also has some Mike in it, but I'm not sure howmuch, as at least parts of it seem to be backed out later. This patch works okay except for the two problems I noted in my original mail: it depends on directory order not to change except when '.uidvalidity' gets touched, and it depends on alphanumerical sort order of files matching chronological (or at least uid-based, which should be the same) sort order. The latter breaks with (standard) procmail, the former occasionally with btree and (presumably) hashed directory indices. > The storage of RFC822.SIZE, aka the on-the-wire size, in the filename > makes a very big difference to performance. That's interesting. I'll keep that in mind for when we begin to see performance issues with UW-IMAP. (I'm hoping I never have to look at the pine source again, though.) > But even this patch (from, e.g. > http://www.star.le.ac.uk/~tjg/misc/uw_imap-2001a_maildir-02.patch) has a > number of bugs. I've fixed a few of them. You can find a source RPM at > ftp://ftp.e-smith.org/pub/e-smith/dev/5.6dev/SRPMS/. The fixes are: <snip fixes> I couldn't find the source RPM for pine or (uw-)imap in that directory, but it doesn't sound like your changes solve our fundamental problems. It's not that big a deal, I think we've decided internally (I know _I_ have :) that we can't offer real IMAP services based on UW-IMAP; we'd sooner go for Cyrus or Courier, even if it does mean disallowing mboxes with IMAP. But dovecot is an even better alternative, once it has all the features we need :) -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

1 0