[Dovecot] v1.1.alpha4 released / about dbox
http://dovecot.org/releases/1.1/alpha/dovecot-1.1.alpha4.tar.gz http://dovecot.org/releases/1.1/alpha/dovecot-1.1.alpha4.tar.gz.sig
Lots of fixes since alpha3. Also includes the dbox rewrite. Currently dbox works properly only in one-message-per-file mode. I'm not actually even sure how good idea it would be to use it with multiple-messages-per-file mode. But I guess it won't hurt to finish its implementation and see. :)
As you can see I haven't been replying much to the mailing list lately. My school started again, which meant that I had to start sleeping again at night instead of at day :) Doing that always takes a few weeks of feeling too tired to get much of anything done. Hopefully that goes away soon.
A bit more about dbox:
Originally I had thought that it'd be good to have multiple messages per file. The bad thing about that is that it works badly without fcntl locks, which don't work all that great with NFS. So I changed my plans a bit so that dbox also supports one-message-per-file behavior where it doesn't need locking much. (Multiple-messages-per-file is still supported by the format, but currently the code is not fully working.)
dbox files begin with a header, then they have the message data and at the end of file there's a metadata block. Metadata contains things like message flags, keywords, received date and so on.
When a flag/keyword is updated, the change is normally done only to index files. This is why dovecot.index and dovecot.index.log files are important with dbox format, they contain flag changes that haven't been written anywhere else.
But since there's still a chance that index files could break (although v1.1 tries harder than v1.0 to fix problems), it would be nice if the flags/keywords were written to metadata block once in a while. So if the index files are lost, flag changes wouldn't be completely lost. Metadata updates of course use disk I/O so they shouldn't be done too often.
I was thinking that the metadata could be updated:
- if IMAP connection has been idling for 4 hours (not changing flags)
- when closing mailbox and there are changes older than 4h
- immediately if there are changes older than 24h (whenever mailbox is being synced, e.g. SELECT/NOOP/STATUS)
Or something like that. Those rules can of course be changed, but I'm not sure if I should bother making them configurable from dovecot.conf. There are already too many settings.
Timo Sirainen wrote:
But since there's still a chance that index files could break (although v1.1 tries harder than v1.0 to fix problems), it would be nice if the flags/keywords were written to metadata block once in a while. So if the index files are lost, flag changes wouldn't be completely lost. Metadata updates of course use disk I/O so they shouldn't be done too often.
I was thinking that the metadata could be updated:
- if IMAP connection has been idling for 4 hours (not changing flags)
- when closing mailbox and there are changes older than 4h
- immediately if there are changes older than 24h (whenever mailbox is being synced, e.g. SELECT/NOOP/STATUS)
Or something like that. Those rules can of course be changed, but I'm not sure if I should bother making them configurable from dovecot.conf. There are already too many settings.
what about a maintanance srcipt/daemon which can be run from cron and every sysadm can decided when and how often he'd like to update metadatas?
-- Levente "Si vis pacem para bellum!"
On Sat, 2007-09-08 at 15:04 +0200, Farkas Levente wrote:
Timo Sirainen wrote:
But since there's still a chance that index files could break (although v1.1 tries harder than v1.0 to fix problems), it would be nice if the flags/keywords were written to metadata block once in a while. So if the index files are lost, flag changes wouldn't be completely lost. Metadata updates of course use disk I/O so they shouldn't be done too often.
I was thinking that the metadata could be updated:
- if IMAP connection has been idling for 4 hours (not changing flags)
- when closing mailbox and there are changes older than 4h
- immediately if there are changes older than 24h (whenever mailbox is being synced, e.g. SELECT/NOOP/STATUS)
Or something like that. Those rules can of course be changed, but I'm not sure if I should bother making them configurable from dovecot.conf. There are already too many settings.
what about a maintanance srcipt/daemon which can be run from cron and every sysadm can decided when and how often he'd like to update metadatas?
That would require keeping the "<mailbox>: <last updated>" information in some central database. Otherwise if you had lots of mailboxes it would waste a lot of disk I/O in such run. And I'm not really interested in creating such a database at least yet. :)
On Vas, Szeptember 9, 2007 06:01, Timo Sirainen wrote:
On Sat, 2007-09-08 at 15:04 +0200, Farkas Levente wrote:
But since there's still a chance that index files could break (although v1.1 tries harder than v1.0 to fix problems), it would be nice if the flags/keywords were written to metadata block once in a while. So if
Timo Sirainen wrote: the
index files are lost, flag changes wouldn't be completely lost. Metadata updates of course use disk I/O so they shouldn't be done too often.
I was thinking that the metadata could be updated:
- if IMAP connection has been idling for 4 hours (not changing flags)
- when closing mailbox and there are changes older than 4h
- immediately if there are changes older than 24h (whenever mailbox is being synced, e.g. SELECT/NOOP/STATUS)
Or something like that. Those rules can of course be changed, but I'm not sure if I should bother making them configurable from dovecot.conf. There are already too many settings.
what about a maintanance srcipt/daemon which can be run from cron and every sysadm can decided when and how often he'd like to update metadatas?
That would require keeping the "<mailbox>: <last updated>" information in some central database. Otherwise if you had lots of mailboxes it would waste a lot of disk I/O in such run. And I'm not really interested in creating such a database at least yet. :)
than may ber move this code to lda also helps. my main reason to suugest the above is the same why i like lda: try to distribute the system load in time. that's the main problem with indexing at reading time. every morning when most people start to read his mail dovecot start to index all mail which creates high load with lda we distribute this load from mail read time to mail arrival time which is much better place! the same should have to be do with metadata update. somehow distribute the load eg. move to some time which is not so important during the night or move to lda which happend during arrival time in stead of read time or any other place but not during read.
--
It appears configure.in is still set to 1.1.alpha3 instead of alpha4.
Great work on the dbox implementation.
- David
-- Love feeling your best ever, all day, every day? Email join@RadicalHealth.com for the easy way.
On Sep 8, 2007, at 3:08, Timo Sirainen wrote:
Lots of fixes since alpha3. I tried it on a SuSE 10.2, running alpha 3. Same configure, same config files. I'm getting a lot of: dovecot: Sep 09 20:32:31 Fatal: execv(/usr/libexec/dovecot/pop3- login) failed: T oo many open files dovecot: Sep 09 20:32:31 Fatal: execv(/usr/libexec/dovecot/imap- login) failed: T oo many open files in the log file and no one is able to access the mail. I reverted to alpha3 and everything is working fine now.
My dovecot -n (seen from the alpha3) looks like this: baobab:/var/log/dovecot # dovecot -n # 1.1.alpha3: /etc/dovecot.conf log_path: /var/log/dovecot/logfile info_log_path: /var/log/dovecot/logfile.info protocols: imap imaps pop3 pop3s ssl_listen: * ssl_ca_file: /etc/ssl/ca.crt disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_greeting: Dovecot at Baobab ready. login_max_processes_count: 512 first_valid_uid: 100 first_valid_gid: 0 mail_location: mbox:/var/spool/mailboxes/%u:INBOX=/var/spool/mail/%u maildir_copy_with_hardlinks: no mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): pop3_enable_last(default): no pop3_enable_last(imap): no pop3_enable_last(pop3): yes pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: mechanisms: plain login cache_size: 3072 username_format: %Lu verbose: yes debug: yes debug_passwords: yes passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix plugin: quota: maildir::dirsize quota_rule: *:storage=300M
luciano.
/"\ /Via A. Salaino, 7 - 20144 Milano (Italy) \ / ASCII RIBBON CAMPAIGN / PHONE : +39 2 485781 FAX: +39 2 48578250 X AGAINST HTML MAIL / E-MAIL: posthamster@sublink.sublink.ORG / \ AND POSTINGS / WWW: http://www.mannucci.ORG/
On Mon, 2007-09-10 at 01:04 +0200, luciano mannucci wrote:
dovecot: Sep 09 20:32:31 Fatal: execv(/usr/libexec/dovecot/pop3- login) failed: T oo many open files dovecot: Sep 09 20:32:31 Fatal: execv(/usr/libexec/dovecot/imap- login) failed: T oo many open files in the log file and no one is able to access the mail. I reverted to alpha3 and everything is working fine now.
alpha5 should have fixed this.
participants (4)
-
David Favor
-
Farkas Levente
-
luciano mannucci
-
Timo Sirainen