[Dovecot] 1.0alpha5 released
We have now some new big features:
Beginnings of dbox mailbox format. It should work, but if you happen to delete/break index files, the mails aren't found again. So really not a good idea to use it for more than testing.
GSSAPI (Kerberos 5) support by Jelmer Vernooij with some fixes by pod@herald.ox.ac.uk
Added dictionary API to Dovecot, similar to what Postfix has. It can be used for getting/setting key=value pairs, which can be useful for many different purposes. For example it could be used to store mailbox metadata for ANNOTATEMORE extension. Implemented SQL backend for it. In future there should be berkeley DB backend at least.
Also there could be some kind of proxy backend which would use UNIX socket to connect to dictionary server doing the actual work. This way each imap/lda process wouldn't have to connect to SQL server themself.
Quota plugin is now included. We also have now dictionary based quota support, which means the quota limits and values can be stored in SQL. However I haven't actually tested these changes yet, so there's a good chance the quota plugin doesn't actually work at all.
Created a new trash plugin which works only together with quota plugin. The point of it is to expunge mails from a list of configurable mailboxes whenever saving new mails would cause the account to go over quota. The oldest mails are expunged first, and only as many mails are expunged as is needed to save the new mail.
For configuration it first looks up for ~/.dovecot.trash.conf, and then fallbacks to /etc/dovecot-trash.conf. The file format is "<priority> <mailbox name>". It's possible to have multiple mailboxes with same priority. Lower number = higher priority. This plugin is also currently untested.
The plugins are currently installed as shared libraries to plugin directory. At some point I'll add a configure option to select which plugins to compile directly into binaries. Possibly even do it by default for some plugins (quota).
And smaller new features:
If LDAP lookup fails because connection gets closed, try retrying it again after reconnect. This is however untested, hopefully didn't break LDAP code completely..
%w variable in dovecot-auth contains password. Can be used to do eg.: password_query = select null as password from users where userid = '%u' and password = password('%w')
syslog_facility setting
And fixes:
- auth_cache was quite broken
- adding keywords caused sometimes assert-crash
- If login_greeting_capability = yes we could have sent the capability before dovecot-auth returned what auth mechanisms were available, forcing clients to use plaintext authentication
- Lots of mbox fixes again..
- userdb prefetch wasn't supposed to be uncommented in dovecot-example.conf in alpha4. Comment it out if you don't need it.
- Don't crash with "out of memory" in 64bit systems
- IDLE command might have crashed when finishing itself if there were lots of external changes
- pop3_reuse_xuidl setting wasn't actually working
Timo Sirainen wrote:
We have now some new big features:
Wow. And seems to work very fine.
But a problem reported a while ago remains: Using epoll with TLS is somehow broken:
Dec 11 01:57:54 ymmv dovecot: imap-login: Login: user=<jh>, method=CRAM-MD5, rip=84.56.237.53, lip=85.214.25.134, TLS Dec 11 01:57:55 ymmv last message repeated 3 times Dec 11 02:11:35 ymmv dovecot: imap-login: io_loop_handle_add: epoll_ctl(3, 13): No such file or directory Dec 11 02:11:35 ymmv dovecot: child 19064 (login) returned error 89 Dec 11 02:11:36 ymmv dovecot: imap-login: Login: user=<jh>, method=CRAM-MD5, rip=84.56.237.53, lip=85.214.25.134, TLS Dec 11 02:11:36 ymmv last message repeated 3 times Dec 11 02:11:37 ymmv dovecot: imap-login: io_loop_handle_add: epoll_ctl(3, 9): No such file or directory Dec 11 02:11:37 ymmv dovecot: child 19412 (login) returned error 89 Dec 11 02:11:38 ymmv dovecot: imap-login: Login: user=<jh>, method=CRAM-MD5, rip=84.56.237.53, lip=85.214.25.134, TLS Dec 11 02:11:38 ymmv last message repeated 2 times Dec 11 02:11:38 ymmv dovecot: imap-login: io_loop_handle_add: epoll_ctl(3, 9): No such file or directory Dec 11 02:11:38 ymmv dovecot: child 19417 (login) returned error 89 Dec 11 02:11:39 ymmv dovecot: imap-login: Login: user=<jh>, method=CRAM-MD5, rip=84.56.237.53, lip=85.214.25.134, TLS Dec 11 02:11:39 ymmv dovecot: imap-login: io_loop_handle_add: epoll_ctl(3, 9): No such file or directory Dec 11 02:11:39 ymmv dovecot: child 19421 (login) returned error 89 [...]
using poll instead of epoll or not using TLS/SSL (i.e. starttls and imaps) fixes that.
And btw, I get a lot of "Warning: Timeout leak: 0x8080944" with the current CVS of the lda.
Great news!
One question, what is dbox?
Thanks Oliver
Timo Sirainen wrote:
We have now some new big features:
- Beginnings of dbox mailbox format. It should work, but if you happen to delete/break index files, the mails aren't found again. So really not a good idea to use it for more than testing.
-- Oliver Schulze L. oliver@samera.com.py
On Sun, 2005-12-11 at 09:04 -0300, Oliver Schulze L. wrote:
Great news!
One question, what is dbox?
Dovecot's own mailbox format:
http://dovecot.org/pipermail/dovecot/2005-September/009161.html http://dovecot.org/pipermail/dovecot/2005-September/009294.html
The design has still changed a bit since those mails though.
Really interesting. I also think that there could be a better solution between mbox and maildir. Later, if you could port it to procmail, it should be very popular.
Thanks for the links. Oliver
Timo Sirainen wrote:
On Sun, 2005-12-11 at 09:04 -0300, Oliver Schulze L. wrote:
Great news!
One question, what is dbox?
Dovecot's own mailbox format:
http://dovecot.org/pipermail/dovecot/2005-September/009161.html http://dovecot.org/pipermail/dovecot/2005-September/009294.html
The design has still changed a bit since those mails though.
-- Oliver Schulze L. oliver@samera.com.py
Oliver Schulze L. wrote:
Really interesting. I also think that there could be a better solution between mbox and maildir. Later, if you could port it to procmail, it should be very popular.
Why procmail? Dovecot has its own LDA which has similar filtering capabilities than procmail and it keeps indexes up to date.
-- Tomi Hakala
Thanks for the links. Oliver
Timo Sirainen wrote:
On Sun, 2005-12-11 at 09:04 -0300, Oliver Schulze L. wrote:
Great news!
One question, what is dbox?
Dovecot's own mailbox format:
http://dovecot.org/pipermail/dovecot/2005-September/009161.html http://dovecot.org/pipermail/dovecot/2005-September/009294.html
The design has still changed a bit since those mails though.
Tomi Hakala wrote:
Oliver Schulze L. wrote:
Really interesting. I also think that there could be a better solution between mbox and maildir. Later, if you could port it to procmail, it should be very popular.
Why procmail? Dovecot has its own LDA which has similar filtering capabilities than procmail and it keeps indexes up to date. I mean, for compatibility and migration scenarios. I for example was using: sendmail, procmail(mbox) and wu-imap. Now, I'm using sendmail, procmail(mbox) and dovecot because the migracion was too easy and fast(small downtime).
I have some filters in procmail and if procmail can write to dbox, I think it will help a little.
Regards, Oliver
- Oliver Schulze L. oliver@samera.com.py
Tomi Hakala:
Oliver Schulze L. wrote:
Really interesting. I also think that there could be a better solution between mbox and maildir. Later, if you could port it to procmail, it should be very popular.
Why procmail? Dovecot has its own LDA which has similar filtering capabilities than procmail and it keeps indexes up to date.
Ok, I'll bite.
Procmail has been very widely deployed for a long, long time. Not only has it its proven track record gained system administators' confidence, but there's also the not-so-small matter of having existing procmail filters, doing their thing.
With a few thousand users, I find that only a relatively small number of them has personalized procmail filtering set up. Still, that "relatively small number" may amount to a hundred procmailrc files or so. The users that wrote up a filter for themselves typically are power users, and their procmail recipes can be involved, so say the least.
Now if there were a way in which LDA could be used in conjunction with a site-wide filter that chains procmail whenever a user has a .procmailrc, that might be a nice step. You'd get LDA delivery for those without their own procmail, and procmail delivery for those with, right?
Not using LDA, I'm not sure whether this is possible, though. If it is, I'd say the scenario deserves mention on Dovecot's wiki, as a migration scenario that might be relevant for many of us.
On Wed, Dec 14, 2005 at 09:45:03 +0100, Jeroen Scheerder wrote:
Procmail has been very widely deployed for a long, long time. Not only has it its proven track record gained system administators' confidence, but there's also the not-so-small matter of having existing procmail filters, doing their thing.
We also have hundreds of users using procmail and I was thinking about the following: what about modifying procmail to use dovecot's lib-storage? we would then have support for ~/.procmailrc and also updates of the dovecot index files. I'll have a look at that and see how difficult it would be. Unfortunately, the procmail source code seems to be pretty messy...
Cheers David
David Schweikert | phone: +41 44 632 7019 System manager ISG.EE | walk: ETH Zentrum, ETL F24.1 ETH Zurich, Switzerland | web: http://people.ee.ethz.ch/dws
David Schweikert wrote:
On Wed, Dec 14, 2005 at 09:45:03 +0100, Jeroen Scheerder wrote:
Procmail has been very widely deployed for a long, long time. Not only has it its proven track record gained system administators' confidence, but there's also the not-so-small matter of having existing procmail filters, doing their thing.
We also have hundreds of users using procmail and I was thinking about the following: what about modifying procmail to use dovecot's lib-storage? we would then have support for ~/.procmailrc and also updates of the dovecot index files. I'll have a look at that and see how difficult it would be. Unfortunately, the procmail source code seems to be pretty messy...
This is what I mean :)
Of course, I can drop procmail when dovecot's dbox is ready. I can live with dovecot LDA. As noted by Jeroen, too few users use procmailrc
Oliver
-- Oliver Schulze L. oliver@samera.com.py
Jeroen Scheerder wrote:
With a few thousand users, I find that only a relatively small number of them has personalized procmail filtering set up. Still, that "relatively small number" may amount to a hundred procmailrc files or so. The users that wrote up a filter for themselves typically are power users, and their procmail recipes can be involved, so say the least.
There is a flip side to that argument: procmail has a fairly high learning curve, because the "recipes" are actually a terse programming language. Even though procmail has been around for a long time, there are no decent [web] editors for recipes (yes, there are some, but not standalone), so very few ordinary users take advantage of them.
The whole point of SIEVE was that it was a [mostly] capable, easily scripted language that *ordinary* users could employ. Asking your power users to rewrite their procmail recipes in SIEVE, so that a much more significant percentage of your entire user base could have the ability to use server side filtering, it entirely reasonable, IMNSHO.
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
On Wed, 2005-12-14 at 09:45 +0100, Jeroen Scheerder wrote:
Now if there were a way in which LDA could be used in conjunction with a site-wide filter that chains procmail whenever a user has a .procmailrc, that might be a nice step. You'd get LDA delivery for those without their own procmail, and procmail delivery for those with, right?
If you're using the exim mailserver you can just check for a .procmailrc in the users homedir, call procmail if it's there and call dovecot-lda if it isn't.
Or write a simple perl script as delivery agent that does this.
Mike.
Miquel van Smoorenburg:
On Wed, 2005-12-14 at 09:45 +0100, Jeroen Scheerder wrote:
Now if there were a way in which LDA could be used in conjunction with a site-wide filter that chains procmail whenever a user has a .procmailrc, that might be a nice step. You'd get LDA delivery for those without their own procmail, and procmail delivery for those with, right?
If you're using the exim mailserver you can just check for a .procmailrc in the users homedir, call procmail if it's there and call dovecot-lda if it isn't.
Or write a simple perl script as delivery agent that does this.
How about running LDA, in conjunction with a system-wide SIEVE script that invokes procmail as a filter if the presence of $HOME/.procmailrc is detected?
Am Sonntag, den 11.12.2005, 00:36 +0200 schrieb Timo Sirainen:
And smaller new features:
- If LDAP lookup fails because connection gets closed, try retrying it again after reconnect. This is however untested, hopefully didn't break LDAP code completely..
about complete LDAP breakdown ... is the problem that I posted for alpha3 and 4 solved now (or did I maybe miss something important):
---------CUT----------
All of our data is ldap based and everything is working as before for our IMAP customers, but our POP3 users fail.
Obviously the pop3 login process does not read the data correctly that comes from ldap.
after enabling mail_debug, I saw the following error in our log:
--------CUT-------- dovecot: Nov 02 18:01:32 Info: pop3(some.one.at.work): mbox:
--------CUT--------
The INBOX should be /virtual/at.work/var/mail/some.one.at.work, yet
data=/virtual/at.work/home/some.one/mail/:INBOX=/var/mail/some.one.at.work the
POP3 process obviously uses the default_mail_env settings.
the relevant line in our dovecot-ldap configuration looks like this: -------CUT-------- user_attrs = uid,homeDirectory,MailMessageStore,,uidNumber,gidNumber -------CUT--------
As I said, the IMAP process uses the ldap data correctly. ---------CUT----------
thanks
Udo Rader
-- BestSolution.at EDV Systemhaus GmbH http://www.bestsolution.at
After an absence ('Where is Timo?'), he comes storming back!
Timo Sirainen wrote:
We have now some new big features:
Beginnings of dbox mailbox format. It should work, but if you happen to delete/break index files, the mails aren't found again. So really not a good idea to use it for more than testing.
GSSAPI (Kerberos 5) support by Jelmer Vernooij with some fixes by pod@herald.ox.ac.uk
Added dictionary API to Dovecot, similar to what Postfix has. It can be used for getting/setting key=value pairs, which can be useful for many different purposes. For example it could be used to store mailbox metadata for ANNOTATEMORE extension. Implemented SQL backend for it. In future there should be berkeley DB backend at least.
Also there could be some kind of proxy backend which would use UNIX socket to connect to dictionary server doing the actual work. This way each imap/lda process wouldn't have to connect to SQL server themself.
Quota plugin is now included. We also have now dictionary based quota support, which means the quota limits and values can be stored in SQL. However I haven't actually tested these changes yet, so there's a good chance the quota plugin doesn't actually work at all.
Created a new trash plugin which works only together with quota plugin. The point of it is to expunge mails from a list of configurable mailboxes whenever saving new mails would cause the account to go over quota. The oldest mails are expunged first, and only as many mails are expunged as is needed to save the new mail.
For configuration it first looks up for ~/.dovecot.trash.conf, and then fallbacks to /etc/dovecot-trash.conf. The file format is "<priority> <mailbox name>". It's possible to have multiple mailboxes with same priority. Lower number = higher priority. This plugin is also currently untested.
The plugins are currently installed as shared libraries to plugin directory. At some point I'll add a configure option to select which plugins to compile directly into binaries. Possibly even do it by default for some plugins (quota).
And smaller new features:
If LDAP lookup fails because connection gets closed, try retrying it again after reconnect. This is however untested, hopefully didn't break LDAP code completely..
%w variable in dovecot-auth contains password. Can be used to do eg.: password_query = select null as password from users where userid = '%u' and password = password('%w')
syslog_facility setting
And fixes:
- auth_cache was quite broken
- adding keywords caused sometimes assert-crash
- If login_greeting_capability = yes we could have sent the capability before dovecot-auth returned what auth mechanisms were available, forcing clients to use plaintext authentication
- Lots of mbox fixes again..
- userdb prefetch wasn't supposed to be uncommented in dovecot-example.conf in alpha4. Comment it out if you don't need it.
- Don't crash with "out of memory" in 64bit systems
- IDLE command might have crashed when finishing itself if there were lots of external changes
- pop3_reuse_xuidl setting wasn't actually working
--
Stewart Dean, Unix System Admin, Henderson Computer Resources
Center of Bard College, Annandale-on-Hudson, New York 12504
sdean@bard.edu voice: 845-758-7475, fax: 845-758-7035
BIG THANK YOU TIMO!
I was waiting for 64bit fix for loooong time!
BR, Alen
Timo Sirainen wrote:
We have now some new big features:
Beginnings of dbox mailbox format. It should work, but if you happen to delete/break index files, the mails aren't found again. So really not a good idea to use it for more than testing.
GSSAPI (Kerberos 5) support by Jelmer Vernooij with some fixes by pod@herald.ox.ac.uk
Added dictionary API to Dovecot, similar to what Postfix has. It can be used for getting/setting key=value pairs, which can be useful for many different purposes. For example it could be used to store mailbox metadata for ANNOTATEMORE extension. Implemented SQL backend for it. In future there should be berkeley DB backend at least.
Also there could be some kind of proxy backend which would use UNIX socket to connect to dictionary server doing the actual work. This way each imap/lda process wouldn't have to connect to SQL server themself.
Quota plugin is now included. We also have now dictionary based quota support, which means the quota limits and values can be stored in SQL. However I haven't actually tested these changes yet, so there's a good chance the quota plugin doesn't actually work at all.
Created a new trash plugin which works only together with quota plugin. The point of it is to expunge mails from a list of configurable mailboxes whenever saving new mails would cause the account to go over quota. The oldest mails are expunged first, and only as many mails are expunged as is needed to save the new mail.
For configuration it first looks up for ~/.dovecot.trash.conf, and then fallbacks to /etc/dovecot-trash.conf. The file format is "<priority> <mailbox name>". It's possible to have multiple mailboxes with same priority. Lower number = higher priority. This plugin is also currently untested.
The plugins are currently installed as shared libraries to plugin directory. At some point I'll add a configure option to select which plugins to compile directly into binaries. Possibly even do it by default for some plugins (quota).
And smaller new features:
If LDAP lookup fails because connection gets closed, try retrying it again after reconnect. This is however untested, hopefully didn't break LDAP code completely..
%w variable in dovecot-auth contains password. Can be used to do eg.: password_query = select null as password from users where userid = '%u' and password = password('%w')
syslog_facility setting
And fixes:
- auth_cache was quite broken
- adding keywords caused sometimes assert-crash
- If login_greeting_capability = yes we could have sent the capability before dovecot-auth returned what auth mechanisms were available, forcing clients to use plaintext authentication
- Lots of mbox fixes again..
- userdb prefetch wasn't supposed to be uncommented in dovecot-example.conf in alpha4. Comment it out if you don't need it.
- Don't crash with "out of memory" in 64bit systems
- IDLE command might have crashed when finishing itself if there were lots of external changes
- pop3_reuse_xuidl setting wasn't actually working
participants (11)
-
Alen Salamun
-
David Schweikert
-
Jakob Hirsch
-
Jeroen Scheerder
-
John Peacock
-
Miquel van Smoorenburg
-
Oliver Schulze L.
-
Stewart Dean
-
Timo Sirainen
-
Tomi Hakala
-
Udo Rader