Released Pigeonhole v0.4.7.rc1 for Dovecot v2.2.16.rc1
Hello Dovecot users,
Last time I had a few stupid problems in the releases, so I'll follow Timo's example and I release an RC first.
The highlights include the implementation of the index and metadata extensions. Quite a few bugs are fixed as well.
Changelog v0.4.7:
- editheader extension: Made protection against addition and deletion of
headers configurable separately. Also, the
Received' andAuto-Submitted' headers are no longer protected against addition by default. - Turned message envelope address parse errors into warnings.
- The interpreter now accepts non-standard domain names, e.g. containing '_'.
- Implemented the Sieve index extension (RFC 5260).
- Implemented support for the mboxmetadata and servermetadata extensions (RFC 5490).
- Implemented new sieve commands for the doveadm command line utility. These commands are currently limited to ManageSieve operations, but the other current sieve tools will be migrated to doveadm in the near future as well.
- Added more debug output to binary up-to-date checking.
- The Sieve interpreter now flushes duplicate database during start phase of result execution rather than commit phase. This makes sure locks on the duplicate database are released as soon as possible, preventing contention.
- Performed a few optimizations in the lexical scanner of the language.
- Fixed bug in `:matches' match-type that made a pattern without wildcards match as if there were a '*' at the beginning.
- file storage: Restructured storage initialization to address backwards compatibility issues.
- Fixed crash in validation of the string parameter of the comparator tag.
- extprograms extension: Made sure supplemental group privileges are also dropped. This was a problem reported by Debian lintian.
- Fixed bug in handling of binary errors for action side-effects and message overrides.
The release is available as follows:
http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.7.rc1.... http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.7.rc1....
Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for more information. Have fun testing this new release and don't hesitate to notify me when there are any problems.
Regards,
-- Stephan Bosch stephan@rename-it.nl
Hi Stephan,
I'm running the current stable version (Dovecot 2.2.15 and Pigeonhole 0.4.6), all works fine except an issue.
In my configuration I have this default sieve script (I'm using Maildir):
=== require ["vnd.dovecot.duplicate", "fileinto", "mailbox"];
if header :matches "X-Spam-Status" "Yes, *" { fileinto "Spam"; }
if duplicate :seconds 60 { fileinto "Trash"; }
when in the Maildir/cur/ there are more than (about) 100.000 messages dovecot-lda crash and the email is not delivered (remains in the queue) and in the log I found this error:
=========== Feb 16 03:23:41 mx03 dovecot: lda(user@domain.com): Panic: file mail-transaction-log.c: line 271 (mail_transaction_log_rotate): assertion failed: (file->locked) Feb 16 03:23:41 mx03 dovecot: lda(user@domain.com): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x7319a) [0x7f27cef8b19a] -> /usr/lib64/dovecot/libdovecot.so.0(i_syslog_fatal_handler+0x33) [0x7f27cef8b253] -> /usr/lib64/dovecot/libdovecot.so.0(+0x24b1a) [0x7f27cef3cb1a] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0xd3965) [0x7f27cf2ce965] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_write+0x1f5) [0x7f27cf2cd7a5] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x2c4) [0x7f27cf2c5214] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x510e8) [0x7f27cf24c0e8] -> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_transaction_save_commit_pre+0x40f) [0x7f27cf2477ef] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0xb39c8) [0x7f27cf2ae9c8] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_transaction_commit_full+0x9f) [0x7f27cf2bc62f] -> /usr/lib64/dovecot/libdovecot-storage.so.0(index_transaction_commit+0xa5) [0x7f27cf2ae585] -> /usr/lib64/dovecot/lib20_zlib_plugin.so(+0x302f) [0x7f27cd90d02f] -> /usr/lib64/dovecot/lib10_quota_plugin.so(+0xac52) [0x7f27cdf34c52] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x96b46) [0x7f27cf291b46] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit_get_changes+0x4f) [0x7f27cf279def] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit+0x16) [0x7f27cf279ea6] -> /usr/lib64/dovecot/libdovecot-sieve.so.0(+0x418c7) [0x7f27cca328c7] -> /usr/lib64/dovecot/libdovecot-sieve.so.0(+0x382a1) [0x7f27cca292a1] -> /usr/lib64/dovecot/libdovecot-sieve.so.0(sieve_result_execute+0x1e2) [0x7f27cca295f2] -> /usr/lib64/dovecot/libdovecot-sieve.so.0(sieve_execute+0x56) [0x7f27cca3a346] -> /usr/lib64/dovecot/lib90_sieve_plugin.so(+0x2e1e) [0x7f27ccc8de1e] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver+0x45) [0x7f27cf51fda5] -> /usr/libexec/dovecot/dovecot-lda(main+0x6ff) [0x402caf] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f27ceba2d5d] -> /usr/libexec/dovecot/dovecot-lda() [0x402339]
If I remove from default.sieve "if duplicate" rule all works fine.
Do you think this problem can be solved in the new version?
Thanks
Il 10/03/2015 03:27, Stephan Bosch ha scritto:
Hello Dovecot users,
Last time I had a few stupid problems in the releases, so I'll follow Timo's example and I release an RC first.
The highlights include the implementation of the index and metadata extensions. Quite a few bugs are fixed as well.
Changelog v0.4.7:
Alessio Cecchi schreef op 10-3-2015 om 9:26:
=========== Feb 16 03:23:41 mx03 dovecot: lda(user@domain.com): Panic: file mail-transaction-log.c: line 271 (mail_transaction_log_rotate): assertion failed: (file->locked) Feb 16 03:23:41 mx03 dovecot: lda(user@domain.com): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x7319a) [0x7f27cef8b19a] -> /usr/lib64/dovecot/libdovecot.so.0(i_syslog_fatal_handler+0x33) [0x7f27cef8b253] -> /usr/lib64/dovecot/libdovecot.so.0(+0x24b1a) [0x7f27cef3cb1a] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0xd3965) [0x7f27cf2ce965] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_write+0x1f5) [0x7f27cf2cd7a5] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x2c4) [0x7f27cf2c5214] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x510e8) [0x7f27cf24c0e8] -> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_transaction_save_commit_pre+0x40f) [0x7f27cf2477ef] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0xb39c8) [0x7f27cf2ae9c8] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_transaction_commit_full+0x9f) [0x7f27cf2bc62f] -> /usr/lib64/dovecot/libdovecot-storage.so.0(index_transaction_commit+0xa5) [0x7f27cf2ae585] -> /usr/lib64/dovecot/lib20_zlib_plugin.so(+0x302f) [0x7f27cd90d02f] -> /usr/lib64/dovecot/lib10_quota_plugin.so(+0xac52) [0x7f27cdf34c52] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x96b46) [0x7f27cf291b46] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit_get_changes+0x4f) [0x7f27cf279def] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit+0x16) [0x7f27cf279ea6] -> /usr/lib64/dovecot/libdovecot-sieve.so.0(+0x418c7) [0x7f27cca328c7] -> /usr/lib64/dovecot/libdovecot-sieve.so.0(+0x382a1) [0x7f27cca292a1] -> /usr/lib64/dovecot/libdovecot-sieve.so.0(sieve_result_execute+0x1e2) [0x7f27cca295f2] -> /usr/lib64/dovecot/libdovecot-sieve.so.0(sieve_execute+0x56) [0x7f27cca3a346] -> /usr/lib64/dovecot/lib90_sieve_plugin.so(+0x2e1e) [0x7f27ccc8de1e] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver+0x45) [0x7f27cf51fda5] -> /usr/libexec/dovecot/dovecot-lda(main+0x6ff) [0x402caf] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f27ceba2d5d] -> /usr/libexec/dovecot/dovecot-lda() [0x402339]
If I remove from default.sieve "if duplicate" rule all works fine.
Do you think this problem can be solved in the new version?
Oh, looks like you reported this before. Will discuss this with Timo.
Regards,
Stephan.
On 10 Mar 2015, at 10:26, Alessio Cecchi alessio@skye.it wrote:
Hi Stephan,
I'm running the current stable version (Dovecot 2.2.15 and Pigeonhole 0.4.6), all works fine except an issue.
Feb 16 03:23:41 mx03 dovecot: lda(user@domain.com): Panic: file mail-transaction-log.c: line 271 (mail_transaction_log_rotate): assertion failed: (file->locked)
This is fixed by v2.2.16.rc1. I'll make v2.2.16 release this week.
If I remove from default.sieve "if duplicate" rule all works fine.
I think it's pure chance that it happens to crash that way. I don't see how the duplicate checking could have affected this crash.
I think it's pure chance that it happens to crash that way. I don't see how the duplicate checking could have affected this crash. Oh! Do you mean Timo, that Computer Science is non-deterministic! Is
On 10/03/15 11:40, Timo Sirainen wrote: there chance, therefore luck, to it? Just kidding!
Thanks for all the the good work you and your associates are doing. For us in the education sector and with dwindling money supplies you are a life saver. Thanks.
Andreas
-- Andreas Kasenides Senior IT Officer Dept. of Computer Science, University of Cyprus Tel: 22892714, Fax: 22892701 (5B4ANK)
Last time I had a few stupid problems in the releases, so I'll follow Timo's example and I release an RC first.
The highlights include the implementation of the index and metadata extensions. Quite a few bugs are fixed as well.
When I compiled and installed this, Sieve scripts were being ignored. Not sure if it's my own stupid mistake, but when I put v0.4.6 back in place, it worked fine. No configuration changes, only make install on the different sources and restart dovecot.
On 3/10/2015 8:11 PM, E.B. wrote:
Last time I had a few stupid problems in the releases, so I'll follow Timo's example and I release an RC first.
The highlights include the implementation of the index and metadata extensions. Quite a few bugs are fixed as well. When I compiled and installed this, Sieve scripts were being ignored. Not sure if it's my own stupid mistake, but when I put v0.4.6 back in place, it worked fine. No configuration changes, only make install on the different sources and restart dovecot.
Could you show your dovecot -n output?
Also, if you enable mail_debug, what sieve-related debug lines are shown?
Regards,
Stephan.
Last time I had a few stupid problems in the releases, so I'll follow Timo's example and I release an RC first.
The highlights include the implementation of the index and metadata extensions. Quite a few bugs are fixed as well. When I compiled and installed this, Sieve scripts were being ignored. Not sure if it's my own stupid mistake, but when I put v0.4.6 back in place, it worked fine. No configuration changes, only make install on the different sources and restart dovecot.
Could you show your dovecot -n output?
Also, if you enable mail_debug, what sieve-related debug lines are shown?
OK, I re-tested and it's still ignoring personal scripts (but not global ones). No .svbin gets generated, no errors, just nothing. However, I do see that Sieve was accessing the user home directory because for some reason now it just created a ".pki" directory therein, which inside of it has an empty "nssdb" directory. That never happened before...? Not a big problem, but I'd prefer not to have that there.
Re: mail_debug, this relates to another post I made that didn't get any replies - can I not override settings such as that (and log_path) from a userdb lookup? Hmm, I WAS able to override mail_debug from userdb, but not log_path?
Sieve-related mail_debug, then?
This looks like the relevant log info:
dovecot: lmtp(testuser@example.com): Debug: Be3h7iRf/1TnUw2PM/SpMA: sieve: file storage: Storage path `/vmail/example.com/testuser/sieve' not found dovecot: lmtp(testuser@example.com): Debug: Be3h7iRf/1TnUw2PM/SpMA: sieve: No default script configured for user dovecot: lmtp(testuser@example.com): Debug: Be3h7iRf/1TnUw2PM/SpMA: sieve: User has no personal script
I'll check on 0.4.6 and report if I see anything interesting, but I will assume for the moment that since personal scripts work in 0.4.6 that this log info won't be there. It is correct that there is no "sieve" file or directory in the user's home dir. This wasn't a problem in 0.4.6. Is it a requirement?
Also, if you didn't see my post a few days ago, while I have your attention, I thank you for the extdata plugin and vote to keep it alive. Only caveat is I hope to see the dict mechanism in Dovecot become more flexible... at a minimum, would like to be able to pass at least one parameter (beside the implicit username) and indicate what field to test it against (I would like at least a simply-configured WHERE clause) if not full query customization. :) Perhaps not easy stuff, but extdata as is still helps me out.
doveconf -n
# 2.2.16.rc1: /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.7.rc1 (c74220e16e0f+) # OS: Linux 3.10.0-123.20.1.el7.x86_64 x86_64 CentOS Linux release 7.0.1406 (Core) xfs dict { sieve = mysql:/usr/local/etc/dovecot/pigeonhole-sql.dict.ext } mail_location = mdbox:/vmail/%d/%1Mn/%1.1Mn/%n mail_plugins = " zlib" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { sieve = file:~/sieve;active=~/.dovecot.sieve sieve_before = /usr/local/var/dovecot/sieve/ sieve_execute_bin_dir = /usr/local/var/dovecot/sieve-extscripts/ sieve_extdata_dict_uri = proxy::sieve sieve_global_extensions = +vnd.dovecot.extdata +vnd.dovecot.execute sieve_plugins = sieve_extdata sieve_extprograms zlib_save = gz zlib_save_level = 9 } service auth-worker { user = $default_internal_user } service dict { unix_listener dict { group = vmail mode = 0660 } } service imap-login { inet_listener imap { address = 127.0.0.1 } } service lmtp { inet_listener lmtp { address = 127.0.0.1 port = 24 } } ssl_cert =
On 3/11/2015 3:16 AM, E.B. wrote:
Last time I had a few stupid problems in the releases, so I'll follow Timo's example and I release an RC first.
The highlights include the implementation of the index and metadata extensions. Quite a few bugs are fixed as well. When I compiled and installed this, Sieve scripts were being ignored. Not sure if it's my own stupid mistake, but when I put v0.4.6 back in place, it worked fine. No configuration changes, only make install on the different sources and restart dovecot. Could you show your dovecot -n output?
Also, if you enable mail_debug, what sieve-related debug lines are shown? OK, I re-tested and it's still ignoring personal scripts (but not global ones). No .svbin gets generated, no errors, just nothing. However, I do see that Sieve was accessing the user home directory because for some reason now it just created a ".pki" directory therein, which inside of it has an empty "nssdb" directory. That never happened before...? Not a big problem, but I'd prefer not to have that there.
Sieve doesn't do that. I don't think Dovecot does that either, but I am not sure.
Re: mail_debug, this relates to another post I made that didn't get any replies - can I not override settings such as that (and log_path) from a userdb lookup? Hmm, I WAS able to override mail_debug from userdb, but not log_path?
Sieve-related mail_debug, then?
This looks like the relevant log info:
dovecot: lmtp(testuser@example.com): Debug: Be3h7iRf/1TnUw2PM/SpMA: sieve: file storage: Storage path `/vmail/example.com/testuser/sieve' not found dovecot: lmtp(testuser@example.com): Debug: Be3h7iRf/1TnUw2PM/SpMA: sieve: No default script configured for user dovecot: lmtp(testuser@example.com): Debug: Be3h7iRf/1TnUw2PM/SpMA: sieve: User has no personal script
I'll check on 0.4.6 and report if I see anything interesting, but I will assume for the moment that since personal scripts work in 0.4.6 that this log info won't be there. It is correct that there is no "sieve" file or directory in the user's home dir. This wasn't a problem in 0.4.6. Is it a requirement?
Well, since your config says the following:
sieve = file:~/sieve;active=~/.dovecot.sieve
It expects a sieve storage directory at ~/sieve (created when ManageSieve is used to upload a script). Also, a symbolic link pointing to the active script will be located at ~/.dovecot.sieve once a script is activated (i.e. through ManageSieve or doveadm sieve).
I wonder how this would have worked before with 0.4.6. Is the ~/.dovecot.sieve a normal script file perhaps (rather than a symlink)? This would mean that the following config would work (e.g. if you don't use ManageSieve):
sieve = file:~/.dovecot.sieve
Also, if you didn't see my post a few days ago, while I have your attention, I thank you for the extdata plugin and vote to keep it alive. Only caveat is I hope to see the dict mechanism in Dovecot become more flexible... at a minimum, would like to be able to pass at least one parameter (beside the implicit username) and indicate what field to test it against (I would like at least a simply-configured WHERE clause) if not full query customization. :) Perhaps not easy stuff, but extdata as is still helps me out.
I haven't looked at that plugin in a while. I could add it to the main pigeonhole package, since it doesn't depend on anything else. It is still a work in progress though, so don't expect the syntax of the extensions to remain the same in the future when I would continue development.
I don't know whether it is acceptable to change the dict functionality like that. Timo will know...
Regards,
Stephan.
When I compiled and installed this, Sieve scripts were being ignored. Not sure if it's my own stupid mistake, but when I put v0.4.6 back in place, it worked fine. No configuration changes, only make install on the different sources and restart dovecot. Could you show your dovecot -n output?
Also, if you enable mail_debug, what sieve-related debug lines are shown? OK, I re-tested and it's still ignoring personal scripts (but not global ones). No .svbin gets generated, no errors, just nothing. However, I do see that Sieve was accessing the user home directory because for some reason now it just created a ".pki" directory therein, which inside of it has an empty "nssdb" directory. That never happened before...? Not a big problem, but I'd prefer not to have that there.
Sieve doesn't do that. I don't think Dovecot does that either, but I am not sure.
Odd. Some lib Sieve uses? These directories do not appear in user home directories unless I install the newest Sieve (and not until a delivery via LMTP happens). No other changes. No other software is currently accessing user home locations at all.
Re: mail_debug, this relates to another post I made that didn't get any replies - can I not override settings such as that (and log_path) from a userdb lookup? Hmm, I WAS able to override mail_debug from userdb, but not log_path?
Sieve-related mail_debug, then?
This looks like the relevant log info:
dovecot: lmtp(testuser@example.com): Debug: Be3h7iRf/1TnUw2PM/SpMA: sieve: file storage: Storage path `/vmail/example.com/testuser/sieve' not found dovecot: lmtp(testuser@example.com): Debug: Be3h7iRf/1TnUw2PM/SpMA: sieve: No default script configured for user dovecot: lmtp(testuser@example.com): Debug: Be3h7iRf/1TnUw2PM/SpMA: sieve: User has no personal script
I'll check on 0.4.6 and report if I see anything interesting, but I will assume for the moment that since personal scripts work in 0.4.6 that this log info won't be there. It is correct that there is no "sieve" file or directory in the user's home dir. This wasn't a problem in 0.4.6. Is it a requirement?
Well, since your config says the following:
sieve = file:~/sieve;active=~/.dovecot.sieve
It expects a sieve storage directory at ~/sieve (created when ManageSieve is used to upload a script). Also, a symbolic link pointing to the active script will be located at ~/.dovecot.sieve once a script is activated (i.e. through ManageSieve or doveadm sieve).
I wonder how this would have worked before with 0.4.6. Is the ~/.dovecot.sieve a normal script file perhaps (rather than a symlink)? This would mean that the following config would work (e.g. if you don't use ManageSieve):
sieve = file:~/.dovecot.sieve
The configuration for that was not of my doing (doesn't that mean it shouldn't have shown up in doveconf -n?). Yes, the .sieve scripts in user home are regular files. Strange 0.4.6 didn't mind this situation, but seems easy to put the configuration right and move on.
However, I do see that Sieve was accessing the user home directory because for some reason now it just created a ".pki" directory therein, which inside of it has an empty "nssdb" directory. That never happened before...? Not a big problem, but I'd prefer not to have that there.
Sieve doesn't do that. I don't think Dovecot does that either, but I am not sure.
Odd. Some lib Sieve uses? These directories do not appear in user home directories unless I install the newest Sieve (and not until a delivery via LMTP happens). No other changes. No other software is currently accessing user home locations at all.
Using today's hg version, these .pki directories aren't created. I wonder if Timo could shine a light on this.
On 12 Mar 2015, at 05:48, E.B. emailbuilder88@yahoo.com wrote:
However, I do see that Sieve was accessing the user home directory because for some reason now it just created a ".pki" directory therein, which inside of it has an empty "nssdb" directory. That never happened before...? Not a big problem, but I'd prefer not to have that there.
Sieve doesn't do that. I don't think Dovecot does that either, but I am not sure.
Odd. Some lib Sieve uses? These directories do not appear in user home directories unless I install the newest Sieve (and not until a delivery via LMTP happens). No other changes. No other software is currently accessing user home locations at all.
Using today's hg version, these .pki directories aren't created. I wonder if Timo could shine a light on this.
Probably some PAM plugin or some other strange automation. Dovecot definitely doesn't create those itself.
On 3/12/2015 4:23 AM, E.B. wrote:
I'll check on 0.4.6 and report if I see anything interesting, but I will assume for the moment that since personal scripts work in 0.4.6 that this log info won't be there. It is correct that there is no "sieve" file or directory in the user's home dir. This wasn't a problem in 0.4.6. Is it a requirement?
Well, since your config says the following:
sieve = file:~/sieve;active=~/.dovecot.sieve
It expects a sieve storage directory at ~/sieve (created when ManageSieve is used to upload a script). Also, a symbolic link pointing to the active script will be located at ~/.dovecot.sieve once a script is activated (i.e. through ManageSieve or doveadm sieve).
I wonder how this would have worked before with 0.4.6. Is the ~/.dovecot.sieve a normal script file perhaps (rather than a symlink)? This would mean that the following config would work (e.g. if you don't use ManageSieve):
sieve = file:~/.dovecot.sieve The configuration for that was not of my doing (doesn't that mean it shouldn't have shown up in doveconf -n?). Yes, the .sieve scripts in user home are regular files. Strange 0.4.6 didn't mind this situation, but seems easy to put the configuration right and move on.
Well, even though your config is a bit strange, Pigeonhole should recognize this rather than fail.
I will fix this.
Regards,
Stephan.
participants (5)
-
Alessio Cecchi
-
Andreas Kasenides
-
E.B.
-
Stephan Bosch
-
Timo Sirainen