[Dovecot] 1.0-test75 and roadmap for v1.0
Keywords are finally stored in maildir filenames and listed in "dovecot-keywords" file. It should be possible to just rename 0.99.x's .customflags file to dovecot-keywords (but it can't be renamed back after modification).
Also fixed another uid/sequence mixup bug with setting keywords in general (in mbox too).
Fixed SHA1 checksum generation with big-endian machines (used only in passwords).
I think I'll forget about the master/config rewrite for a while and leave it for Dovecot v2.0. That would mean the next release will be called 1.0-alpha1. After that there should be only bugfixes and some smaller features and optimizations left.
I looked through my TODO and found the following missing features / optimizations that pretty much have to be done before v1.0. Anything important missing?
- keywords:
- add some limits to how many there can be
- send FLAGS/PERMANENTFLAGS untagged replies when they change (required by IMAP RFC)
- remove unused keywords from keyword list? (only when adding new
ones)
- mail cache file
- cache _all_ headers that are marked to be cached when headers are being parsed, not just the ones client is requesting at that
time. - compression should drop fields with last_used < (latest_mail_index_date - month) - when parsing mbox or saving message, parse the mail through index-mail so things gets saved into cache immediately
- mbox
- when we're updating flags with lazy writing, we're still parsing
the mbox, just not writing to it! - always add empty line after mail. - make the parser require it too? syncing should make sure there always exists two LFs at end of file. raw-mbox-stream should make sure the last message ends with LF even if it doesn't exist in the file - COPY doesn't work to same mailbox (lock assert crash) - keep mbox lock for two extra seconds after sync to make sure we notice changes made by other mbox writer software
maildir
- hardlink copying doesn't update indexes (does it even work?)
deliver needs quota integration somehow so it returns out-of-quota failure as fatal instead of temporary
Probably after v1.0:
- Full NFS support
- Try to handle out-of-quota/disk space situations well
Timo Sirainen wrote:
I think I'll forget about the master/config rewrite for a while and leave it for Dovecot v2.0. That would mean the next release will be called 1.0-alpha1. After that there should be only bugfixes and some smaller features and optimizations left.
Sounds reasonable. Mind you, it feels scary migrating our 20,000 users to an "alpha" product! (This week I managed to switch ~120 University Admin staff, who are some of our heaviest users, to use Dovecot 1.0-stable with mbox, instead of UW-IMAP, without them noticing! So far, it seems that Dovecot is reading one third of the disk blocks and using half the CPU than UW-IMAP, for them.)
I looked through my TODO and found the following missing features / optimizations that pretty much have to be done before v1.0. Anything important missing?
<snip>
- mail cache file
- cache _all_ headers that are marked to be cached when headers are being parsed, not just the ones client is requesting at that time.
- compression should drop fields with last_used < (latest_mail_index_date - month)
- when parsing mbox or saving message, parse the mail through
index-mail so things gets saved into cache immediately
I'm still not convinced the cache file is shrinking properly for me. Do entries get expired after a set time at the moment (in 1.0-stable)? Anyway, it would be nice to be able to configure the lifetime (e.g. may be less than a month). I've noticed that if you've got a lot of messages
- and I'm being deliberately naughty with my INBOX ;) - doing a full
search will lead to a big cache file, even though the user is less likely to read the oldest messages.
Best Wishes, Chris
On 1.7.2005, at 02:27, Chris Wakelin wrote:
I'm still not convinced the cache file is shrinking properly for me. Do entries get expired after a set time at the moment (in 1.0-stable)?
No. I've been a bit lazy in updating 1.0-stable.. Committed this fix to it now.
Anyway, it would be nice to be able to configure the lifetime (e.g. may be less than a month). I've noticed that if you've got a lot of messages - and I'm being deliberately naughty with my INBOX ;) - doing a full search will lead to a big cache file, even though the user is less likely to read the oldest messages.
Hmm. I don't know .. For repeated header searches the cache will help a lot since there's no need to read all the messages again.
Currently Dovecot does two types of caching. One is "cache for a week" after which it will be dropped, another is "cache forever". These are mostly done to differentiate between clients having local caches and clients not having local caches. There are two rules that make a cache-for-a-week rule change to cache-forever: /* a) nonordered access within this session. if client doesn't request messages in growing order, we assume it doesn't have a permanent local cache. b) accessing message older than one week. assume it's a client with no local cache. if it was just a new client generating the local cache for the first time, we'll drop back to TEMP within few months. */
On Thursday 30 June 2005 23:29, Timo Sirainen wrote:
Keywords are finally stored in maildir filenames and listed in "dovecot-keywords" file. It should be possible to just rename 0.99.x's .customflags file to dovecot-keywords (but it can't be renamed back after modification).
Also fixed another uid/sequence mixup bug with setting keywords in general (in mbox too).
Fixed SHA1 checksum generation with big-endian machines (used only in passwords).
I think I'll forget about the master/config rewrite for a while and leave it for Dovecot v2.0. That would mean the next release will be called 1.0-alpha1. After that there should be only bugfixes and some smaller features and optimizations left.
A 1.0 release would be nice :-)
I looked through my TODO and found the following missing features / optimizations that pretty much have to be done before v1.0. Anything important missing?
- keywords:
- add some limits to how many there can be
- send FLAGS/PERMANENTFLAGS untagged replies when they change (required by IMAP RFC)
- remove unused keywords from keyword list? (only when adding
new ones)
- mail cache file
- cache _all_ headers that are marked to be cached when headers
are being parsed, not just the ones client is requesting at that time. - compression should drop fields with last_used < (latest_mail_index_date - month) - when parsing mbox or saving message, parse the mail through index-mail so things gets saved into cache immediately
- mbox
- when we're updating flags with lazy writing, we're still
parsing the mbox, just not writing to it! - always add empty line after mail. - make the parser require it too? syncing should make sure there always exists two LFs at end of file. raw-mbox-stream should make sure the last message ends with LF even if it doesn't exist in the file - COPY doesn't work to same mailbox (lock assert crash) - keep mbox lock for two extra seconds after sync to make sure we notice changes made by other mbox writer software
maildir
- hardlink copying doesn't update indexes (does it even work?)
deliver needs quota integration somehow so it returns
out-of-quota failure as fatal instead of temporary
Probably after v1.0:
- Full NFS support
- Try to handle out-of-quota/disk space situations well
Any chance of kqueue support for us BSD users? I think the diff I have could probably be the basis of support for it.
Thanks,
Dominic GoodforBusiness.co.uk I.T. Services for SMEs in the UK.
On 1.7.2005, at 02:55, Dominic Marks wrote:
Any chance of kqueue support for us BSD users? I think the diff I have could probably be the basis of support for it.
Could you fix the XXX part of it? There are no longer any priorities in I/O loop so it could be made faster I think?
Also what exactly do you mean with this:
* changing this to i_fatal is debatable, however
* if you do so you will potentially expose the case
* where a process receives the client from a socket
* then closes the listening socket, experiences an
* error and calls exit, dropping the client we just
* picked up
io_loop_handle_remove() should always be called before close(), so kevent() should never fail. What am I missing?
On Sunday 03 July 2005 22:49, Timo Sirainen wrote:
On 1.7.2005, at 02:55, Dominic Marks wrote:
Any chance of kqueue support for us BSD users? I think the diff I have could probably be the basis of support for it.
Could you fix the XXX part of it? There are no longer any priorities in I/O loop so it could be made faster I think?
Sure.
Also what exactly do you mean with this:
* changing this to i_fatal is debatable, however * if you do so you will potentially expose the case * where a process receives the client from a socket * then closes the listening socket, experiences an * error and calls exit, dropping the client we just * picked upio_loop_handle_remove() should always be called before close(), so kevent() should never fail. What am I missing?
I'll investigate, and see what I can come up with.
Cheers,
Dominic Marks
On Sunday 03 July 2005 23:01, Dominic Marks wrote:
On Sunday 03 July 2005 22:49, Timo Sirainen wrote:
On 1.7.2005, at 02:55, Dominic Marks wrote:
Any chance of kqueue support for us BSD users? I think the diff I have could probably be the basis of support for it.
Could you fix the XXX part of it? There are no longer any priorities in I/O loop so it could be made faster I think?
I had a crack at this, but I couldn't get things working right. I'll keep on fiddling with it.
Sure.
Also what exactly do you mean with this:
* changing this to i_fatal is debatable, however * if you do so you will potentially expose the case * where a process receives the client from a socket * then closes the listening socket, experiences an * error and calls exit, dropping the client we just * picked upio_loop_handle_remove() should always be called before close(), so kevent() should never fail. What am I missing?
Seems to be the other way around, since I can work around the warning by using fstat to check if the descriptor is in use. I currently have this:
if (kevent(data->kq, &data->event, 1, NULL, 0, NULL) < 0) { if (fstat(fd, NULL) == 0) { /* * Trying to remove filters on a closed descriptor * will cause kevent(2) to return an error. If we * sure that descriptors are ALWAYS closed before * this function we can delete the entire function * body. */ i_warning("couldn't remove filter on fd %d with kqueue: %m", fd); } }
I'll investigate, and see what I can come up with.
Cheers,
Thanks,
Dominic Marks
On Tue, 2005-07-12 at 16:07 +0100, Dominic Marks wrote:
io_loop_handle_remove() should always be called before close(), so kevent() should never fail. What am I missing?
Seems to be the other way around, since I can work around the warning by using fstat to check if the descriptor is in use. I currently have this:
if (kevent(data->kq, &data->event, 1, NULL, 0, NULL) < 0) { if (fstat(fd, NULL) == 0) { /* * Trying to remove filters on a closed descriptor * will cause kevent(2) to return an error. If we * sure that descriptors are ALWAYS closed before * this function we can delete the entire function * body. */ i_warning("couldn't remove filter on fd %d with kqueue: %m", fd); }
Uhm. I think Dovecot's code should rather be fixed so the function doesn't get called with closed file descriptors. I'm not sure if it's possible everywhere without larger changes though..
Anyway I don't think you should add the extra fstat() there, not giving a warning if EBADF error comes is just as good without extra syscall.
On Tuesday 12 July 2005 17:28, Timo Sirainen wrote:
On Tue, 2005-07-12 at 16:07 +0100, Dominic Marks wrote:
io_loop_handle_remove() should always be called before close(), so kevent() should never fail. What am I missing?
Seems to be the other way around, since I can work around the warning by using fstat to check if the descriptor is in use. I currently have this:
if (kevent(data->kq, &data->event, 1, NULL, 0, NULL) < 0) { if (fstat(fd, NULL) == 0) { /* * Trying to remove filters on a closed descriptor * will cause kevent(2) to return an error. If we * sure that descriptors are ALWAYS closed before * this function we can delete the entire function * body. */ i_warning("couldn't remove filter on fd %d with kqueue: %m", fd); }
Uhm. I think Dovecot's code should rather be fixed so the function doesn't get called with closed file descriptors. I'm not sure if it's possible everywhere without larger changes though..
Ok.
Anyway I don't think you should add the extra fstat() there, not giving a warning if EBADF error comes is just as good without extra syscall.
Makes sense, I've changed the code.
My work in progress:
http://www.helenmarks.co.uk/~dom/dovecot/ioloop-kqueue.c
I'll keep tinkering with the io prioritisation stuff, but I can't get it run properly as yet.
Thanks,
Dominic Marks
On Sun, Jul 03, 2005 at 11:01:30PM +0100, Dominic Marks wrote:
On Sunday 03 July 2005 22:49, Timo Sirainen wrote:
On 1.7.2005, at 02:55, Dominic Marks wrote:
Any chance of kqueue support for us BSD users? I think the diff I have could probably be the basis of support for it.
Could you fix the XXX part of it? There are no longer any priorities in I/O loop so it could be made faster I think?
Sure.
Also what exactly do you mean with this:
* changing this to i_fatal is debatable, however * if you do so you will potentially expose the case * where a process receives the client from a socket * then closes the listening socket, experiences an * error and calls exit, dropping the client we just * picked upio_loop_handle_remove() should always be called before close(), so kevent() should never fail. What am I missing?
I'll investigate, and see what I can come up with.
Cheers,
Dominic Marks
and what ever happened with the kqueue() diff for the I/O event handler?
On 1.7.2005, at 10:32, Alan Schmitt wrote:
Le 1 juil. 05 à 00:29, Timo Sirainen a écrit :
- maildir - hardlink copying doesn't update indexes (does it even work?)
This is a bit scary, as I had this enabled (under 74) and I did not notice anything bad. Could you elaborae a little?
OK, so it works then :) It just doesn't update indexes directly, meaning the next mailbox synchronization is slower than it could be.
On Fri, 1 Jul 2005, Timo Sirainen wrote:
What's the status of Dovecot-LDA? Should/can it become part of the Dovecot v1.0 package?
Bye,
Steffen Kaiser wrote:
What's the status of Dovecot-LDA? Should/can it become part of the Dovecot v1.0 package?
Version in dovecot-lda CVS reposity works pretty well, I have been using it on a test basis for couple months now. Code in dovecot-lda reposity includes Sieve support, LDA code in test75 package does not.
LDA reads Sieve code from .dovecot.sieve file which is stored in users home directory, when first run LDA compiles Sieve to binary form which is then stored in .dovecot.sievec.
In addition to standard mail filtering you can do "out of office" replies with Sieve vacation extension, no need for 3rd party software anymore. One major feature still missing from LDA is quota enforcing, but this is not issue if file system quota is used.
On Thu, 2005-07-07 at 12:18 -0400, Brad wrote:
On Fri, Jul 01, 2005 at 01:29:53AM +0300, Timo Sirainen wrote:
Probably after v1.0:
- Fix IPv6 support to not rely upon v4-mapped addresses
Did this mean that binding should listen in both v4 and v6 addresses, or something else? I already applied a patch containing this code, does it help?
/* if using IPv6, bind both on the IPv4 and IPv6 addresses */
#ifdef IPV6_V6ONLY if (so.sin.sin_family == AF_INET6) { opt = 0; setsockopt(fd, IPPROTO_IPV6, IPV6_V6ONLY, &opt, sizeof(opt)); } #endif
On Tue, Jul 12, 2005 at 07:15:47PM +0300, Timo Sirainen wrote:
On Thu, 2005-07-07 at 12:18 -0400, Brad wrote:
On Fri, Jul 01, 2005 at 01:29:53AM +0300, Timo Sirainen wrote:
Probably after v1.0:
- Fix IPv6 support to not rely upon v4-mapped addresses
Did this mean that binding should listen in both v4 and v6 addresses, or something else? I already applied a patch containing this code, does it help?
/* if using IPv6, bind both on the IPv4 and IPv6 addresses */ #ifdef IPV6_V6ONLY if (so.sin.sin_family == AF_INET6) { opt = 0; setsockopt(fd, IPPROTO_IPV6, IPV6_V6ONLY, &opt, sizeof(opt)); } #endif
Dovecot still only creates a single socket.
On Wed, Jul 13, 2005 at 09:27:02PM -0400, Brad wrote:
On Tue, Jul 12, 2005 at 07:15:47PM +0300, Timo Sirainen wrote:
On Thu, 2005-07-07 at 12:18 -0400, Brad wrote:
On Fri, Jul 01, 2005 at 01:29:53AM +0300, Timo Sirainen wrote:
Probably after v1.0:
- Fix IPv6 support to not rely upon v4-mapped addresses
Did this mean that binding should listen in both v4 and v6 addresses, or something else? I already applied a patch containing this code, does it help?
/* if using IPv6, bind both on the IPv4 and IPv6 addresses */ #ifdef IPV6_V6ONLY if (so.sin.sin_family == AF_INET6) { opt = 0; setsockopt(fd, IPPROTO_IPV6, IPV6_V6ONLY, &opt, sizeof(opt)); } #endif
Dovecot still only creates a single socket.
To clarify, Dovecot should create a socket listening for * (v4) and another for :: (v6). That's fairly simplified since you should also be able to specify "listen = [1-2 v4 IP's] [1-2 v6 IP's]" for example and it would also create the required v4/v6 sockets.
participants (8)
-
Alan Schmitt
-
Brad
-
Chris Wakelin
-
Dominic Marks
-
Marten Lehmann
-
Steffen Kaiser
-
Timo Sirainen
-
Tomi Hakala