v2.3.3 rc1 - Error: sieve: !!BUG!!: Binary compiled from dovecot.sieve is still corrupt
I have a sieve problem with dovecot v2.3.3 rc1 with either of pigeonhole 0.5.2 or 0.5.3.rc1. The the change that triggers this is in dovecot not pigeonhole. Sieve filtering has been working with v2.3.2 and many earlier versions.
Log report: Sep 24 10:36:13 mailhost dovecot: [ID 702911 mail.error] lmtp(user@domain.tld)<15816><RnikKw2wqFvIPQAAs831Jw>: Error: sieve: !!BUG!!: Binary compiled from /path/to/domain/user/dovecot.sieve is still corrupt; bailing out and reverting to default delivery
This is a sieve script with a vacation handler. My other sieve filters are working. My actual sieve filter worked before and was written by Roundcube web mail. My failing test script is cut-n-paste from the sieve examples https://wiki2.dovecot.org/Pigeonhole/Sieve/Examples so should not be in doubt.
The text script compiles to dovecot.svbin and newly compiled files are identical to older versions, the compilation has not changed.
James.
Can you try with the sieve-test command-line tool, in particular with trace debugging enabled?
Op 24-9-2018 om 12:03 schreef James:
I have a sieve problem with dovecot v2.3.3 rc1 with either of pigeonhole 0.5.2 or 0.5.3.rc1. The the change that triggers this is in dovecot not pigeonhole. Sieve filtering has been working with v2.3.2 and many earlier versions.
Log report: Sep 24 10:36:13 mailhost dovecot: [ID 702911 mail.error] lmtp(user@domain.tld)<15816><RnikKw2wqFvIPQAAs831Jw>: Error: sieve: !!BUG!!: Binary compiled from /path/to/domain/user/dovecot.sieve is still corrupt; bailing out and reverting to default delivery
This is a sieve script with a vacation handler. My other sieve filters are working. My actual sieve filter worked before and was written by Roundcube web mail. My failing test script is cut-n-paste from the sieve examples https://wiki2.dovecot.org/Pigeonhole/Sieve/Examples so should not be in doubt.
The text script compiles to dovecot.svbin and newly compiled files are identical to older versions, the compilation has not changed.
James.
On 24/09/2018 11:51, Stephan Bosch wrote:
Can you try with the sieve-test command-line tool, in particular with trace debugging enabled?
Thank you for the debugging tip, is this useful?
$ sieve-test -D -c dovecot.conf sieve message
sieve-test(james): Debug: sieve: Pigeonhole version 0.5.3.rc1 (716b1b49)
initializing
sieve-test(james): Debug: sieve: include: sieve_global is not set; it is
currently not possible to include :global' scripts. debug: file storage: Using Sieve script path: /home/james/sieve. debug: file script: Opened script sieve' from /home/james/sieve'. debug: Script sieve' from /home/james/sieve successfully compiled.
sieve-test(james): Info: corrupt binary deleted.
sieve-test(james): Error: unlink((null)) failed: Bad address (in
sieve-test.c:426)
sieve-test(james): Info: final result: failed; resolved with successful
implicit keep
$ cat dovecot.conf mail_uid = james mail_gid = staff mail_location = maildir:/path/to/%d/%n/Maildir postmaster_address = postmaster@domain.tld
$ cat sieve require ["vacation"]; vacation # Reply at most once a day to a same sender :days 1 :subject "Out of office reply" # List of additional recipient addresses which are included in the auto replying. # If a mail's recipient is not the envelope recipient and it's not on this list, # no vacation reply is sent for it. :addresses ["j.doe@company.dom", "john.doe@company.dom"] "I'm out of office, please contact Joan Doe instead. Best regards John Doe";
$ cat message Subject: test From: me@domain.tld
testing 123
On 24/09/2018 12:34, James wrote:
$ cat sieve require ["vacation"]; vacation # Reply at most once a day to a same sender :days 1
...removing the ":days 1" stops the corruption:
$ sieve-test -D -c dovecot.conf sieve.2 message
sieve-test(james): Debug: Effective uid=1001, gid=10, home=/home/james
sieve-test(james): Debug: maildir++: root=/path/to//james/Maildir,
index=, indexpvt=, control=, inbox=/path/to//james/Maildir, alt=
sieve-test(james): Debug: sieve: Pigeonhole version 0.5.3.rc1 (716b1b49)
initializing
sieve-test(james): Debug: sieve: include: sieve_global is not set; it is
currently not possible to include :global' scripts. debug: file storage: Using Sieve script path: /home/james/sieve.2. debug: file script: Opened script sieve.2' from /home/james/sieve.2'. debug: Script sieve.2' from /home/james/sieve.2 successfully compiled.
Performed actions:
- send vacation message: => seconds : 604800 => subject : Out of office reply => handle : I'm out of office, please contact Joan Doe instead. Best regards John DoeOut of office reply<default-from><NO-MIME>
START MESSAGE I'm out of office, please contact Joan Doe instead. Best regards John Doe END MESSAGE
Implicit keep:
- store message in folder: INBOX
sieve-test(james): Info: final result: success
$ cat sieve.2 require ["vacation"]; vacation # Reply at most once a day to a same sender # :days 1 :subject "Out of office reply" # List of additional recipient addresses which are included in the auto replying. # If a mail's recipient is not the envelope recipient and it's not on this list, # no vacation reply is sent for it. :addresses ["j.doe@company.dom", "john.doe@company.dom"] "I'm out of office, please contact Joan Doe instead. Best regards John Doe";
On 24/09/2018 12:54, James wrote:
Remove 2.3.3 rc1, install 2.3.2 and 0.5.2, success with sieve-test including ":days 1"
You can enable -d -' and -t - -Tlevel=matching' as well.
Op 24-9-2018 om 13:34 schreef James:
On 24/09/2018 11:51, Stephan Bosch wrote:
Can you try with the sieve-test command-line tool, in particular with trace debugging enabled?
Thank you for the debugging tip, is this useful?
$ sieve-test -D -c dovecot.conf sieve message sieve-test(james): Debug: sieve: Pigeonhole version 0.5.3.rc1 (716b1b49) initializing sieve-test(james): Debug: sieve: include: sieve_global is not set; it is currently not possible to include
:global' scripts. debug: file storage: Using Sieve script path: /home/james/sieve. debug: file script: Opened scriptsieve' from/home/james/sieve'. debug: Scriptsieve' from /home/james/sieve successfully compiled. sieve-test(james): Info: corrupt binary deleted. sieve-test(james): Error: unlink((null)) failed: Bad address (in sieve-test.c:426) sieve-test(james): Info: final result: failed; resolved with successful implicit keep$ cat dovecot.conf mail_uid = james mail_gid = staff mail_location = maildir:/path/to/%d/%n/Maildir postmaster_address = postmaster@domain.tld
$ cat sieve require ["vacation"]; vacation # Reply at most once a day to a same sender :days 1 :subject "Out of office reply" # List of additional recipient addresses which are included in the auto replying. # If a mail's recipient is not the envelope recipient and it's not on this list, # no vacation reply is sent for it. :addresses ["j.doe@company.dom", "john.doe@company.dom"] "I'm out of office, please contact Joan Doe instead. Best regards John Doe";
$ cat message Subject: test From: me@domain.tld
testing 123
On 24/09/2018 13:35, Stephan Bosch wrote:
You can enable
-d -' and-t - -Tlevel=matching' as well.
$ sieve-test -D -d - -t - -Tlevel=matching -c dovecot.conf sieve message
sieve-test(james): Debug: Effective uid=1001, gid=10, home=/home/james
sieve-test(james): Debug: maildir++: root=/path/to//james/Maildir,
index=, indexpvt=, control=, inbox=/path/to//james/Maildir, alt=
sieve-test(james): Debug: sieve: Pigeonhole version 0.5.2 (7704de5e)
initializing
sieve-test(james): Debug: sieve: include: sieve_global is not set; it is
currently not possible to include :global' scripts. debug: file storage: Using Sieve script path: /home/james/sieve. debug: file script: Opened script sieve' from /home/james/sieve'. debug: Script sieve' from /home/james/sieve successfully compiled.
- Script metadata (block: 0):
class = file class.version = 0 location = /home/james/sieve
Required extensions (block: 1):
0: vacation (id: 9)
Main program (block: 2):
Address Line Code 00000000: DEBUG BLOCK: 3 00000001: EXTENSIONS [1]: 00000002: vacation 00000004: 2: VACATION 00000007: 4: seconds: NUM 5 00000009: Binary is corrupt.
## Started executing script 'sieve'
VACATION: #ERROR#: unknown optional operand
2: [[EXECUTION ABORTED]]
## Finished executing script 'sieve'sieve-test(james): Info: corrupt binary deleted. sieve-test(james): Error: unlink((null)) failed: Bad address (in sieve-test.c:426) sieve-test(james): Info: final result: failed; resolved with successful implicit keep
Ok, will fix...
Op 24-9-2018 om 15:06 schreef James:
On 24/09/2018 13:35, Stephan Bosch wrote:
You can enable
-d -' and-t - -Tlevel=matching' as well.$ sieve-test -D -d - -t - -Tlevel=matching -c dovecot.conf sieve message sieve-test(james): Debug: Effective uid=1001, gid=10, home=/home/james sieve-test(james): Debug: maildir++: root=/path/to//james/Maildir, index=, indexpvt=, control=, inbox=/path/to//james/Maildir, alt= sieve-test(james): Debug: sieve: Pigeonhole version 0.5.2 (7704de5e) initializing sieve-test(james): Debug: sieve: include: sieve_global is not set; it is currently not possible to include
:global' scripts. debug: file storage: Using Sieve script path: /home/james/sieve. debug: file script: Opened scriptsieve' from/home/james/sieve'. debug: Scriptsieve' from /home/james/sieve successfully compiled.
- Script metadata (block: 0):
class = file class.version = 0 location = /home/james/sieve
- Required extensions (block: 1):
0: vacation (id: 9)
- Main program (block: 2):
Address Line Code 00000000: DEBUG BLOCK: 3 00000001: EXTENSIONS [1]: 00000002: vacation 00000004: 2: VACATION 00000007: 4: seconds: NUM 5 00000009: Binary is corrupt.
## Started executing script 'sieve' VACATION: #ERROR#: unknown optional operand 2: [[EXECUTION ABORTED]] ## Finished executing script 'sieve'
sieve-test(james): Info: corrupt binary deleted. sieve-test(james): Error: unlink((null)) failed: Bad address (in sieve-test.c:426) sieve-test(james): Info: final result: failed; resolved with successful implicit keep
Hi,
Something mightily weird is going on at your end. It doesn't fail here
(see below). First of all, what is your configuration (output from
dovecot -n)?
Also, can you make a hex dump of the binary (using sieve-dump -h <your-script-binary>.svbin). See below for an example at my end.
Finally, can you try to explicitly delete the binary (preferably after preserving it elsewhere) so that it is guaranteed to be created fresh? That should not matter, but you never know.
## Success at my end:
$ sieve-test -D -d - -t - -Tlevel=matching ~/frop.sieve ~/message.eml
sieve-test(stephan)<27390><>: Debug: Loading modules from directory:
/usr/lib/dovecot/modules
sieve-test(stephan)<27390><>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib01_acl_plugin.so
sieve-test(stephan)<27390><>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_virtual_plugin.so
sieve-test(stephan): Debug: Effective uid=1000, gid=1000, home=/home/stephan
sieve-test(stephan): Debug: acl: No acl_shared_dict setting - shared
mailbox listing is disabled
sieve-test(stephan): Debug: Namespace inbox: type=private,
prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
location=maildir:~/Maildir
sieve-test(stephan): Debug: maildir++: root=/home/stephan/Maildir,
index=, indexpvt=, control=, inbox=/home/stephan/Maildir, alt=
sieve-test(stephan): Debug: acl: initializing backend with data: vfile
sieve-test(stephan): Debug: acl: acl username = stephan
sieve-test(stephan): Debug: acl: owner = 1
sieve-test(stephan): Debug: acl vfile: Global ACLs disabled
sieve-test(stephan): Debug: Namespace : type=private, prefix=Archive/,
sep=/, inbox=no, hidden=no, list=yes, subscriptions=yes
location=maildir:~/MailArchive
sieve-test(stephan): Debug: maildir++: root=/home/stephan/MailArchive,
index=, indexpvt=, control=, inbox=, alt=
sieve-test(stephan): Debug: acl: initializing backend with data: vfile
sieve-test(stephan): Debug: acl: acl username = stephan
sieve-test(stephan): Debug: acl: owner = 1
sieve-test(stephan): Debug: acl vfile: Global ACLs disabled
sieve-test(stephan): Debug: Namespace : type=public, prefix=Public/,
sep=/, inbox=no, hidden=no, list=yes, subscriptions=no
location=maildir:/var/mail/public
sieve-test(stephan): Debug: maildir++: root=/var/mail/public, index=,
indexpvt=, control=, inbox=, alt=
sieve-test(stephan): Debug: acl: initializing backend with data: vfile
sieve-test(stephan): Debug: acl: acl username = stephan
sieve-test(stephan): Debug: acl: owner = 0
sieve-test(stephan): Debug: acl vfile: Global ACLs disabled
sieve-test(stephan): Debug: Namespace : type=private, prefix=virtual/,
sep=/, inbox=no, hidden=no, list=yes, subscriptions=yes
location=virtual:~/Maildir/virtual
sieve-test(stephan): Debug: fs: root=/home/stephan/Maildir/virtual,
index=, indexpvt=, control=, inbox=, alt=
sieve-test(stephan): Debug: acl: initializing backend with data: vfile
sieve-test(stephan): Debug: acl: acl username = stephan
sieve-test(stephan): Debug: acl: owner = 1
sieve-test(stephan): Debug: acl vfile: Global ACLs disabled
sieve-test(stephan): Debug: Namespace : type=private, prefix=, sep=,
inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
sieve-test(stephan): Debug: none: root=, index=, indexpvt=, control=,
inbox=, alt=
sieve-test(stephan): Debug: sieve: Pigeonhole version 0.5.3.rc1
(b29d6a24) initializing
sieve-test(stephan): Debug: sieve: Sieve imapsieve plugin for Pigeonhole
version 0.5.3.rc1 (b29d6a24) loaded
sieve-test(stephan): Debug: sieve: Sieve Extprograms plugin for
Pigeonhole version 0.5.3.rc1 (b29d6a24) loaded
sieve-test(stephan): Debug: sieve: vnd.dovecot.pipe extension: no bin or
socket directory specified; extension is unconfigured (both
sieve_pipe_bin_dir and sieve_pipe_socket_dir are not set)
sieve-test(stephan): Debug: sieve: vnd.dovecot.filter extension: no bin
or socket directory specified; extension is unconfigured (both
sieve_filter_bin_dir and sieve_filter_socket_dir are not set)
debug: file storage: Using Sieve script path: /home/stephan/frop.sieve.
debug: file script: Opened script frop' from /home/stephan/frop.sieve'.
debug: binary save: not saving binary /home/stephan/frop.svbin, because
it is already stored.
- Script metadata (block: 0):
class = file class.version = 0 location = /home/stephan/frop.sieve
- Required extensions (block: 1):
0: vacation (id: 9)
- Main program (block: 2):
Address Line Code 00000000: DEBUG BLOCK: 3 00000001: EXTENSIONS [1]: 00000002: vacation 00000004: 3: VACATION 00000007: 5: seconds: NUM 86400 0000000c: 6: subject: STR[19] "Out of office reply" 00000023: 10: addresses: STRLIST [2] (end: 00000054) 00000029: STR[17] "j.doe@company.dom" 0000003d: STR[20] "john.doe@company.dom" 00000055: 11: reason: STR[75] "I'm out of office, please contact Joan Doe instead.??Best regards??John Doe" 000000a3: 3: handle: STR[117] "I'm out of office, please contact Joan Doe instead.??Best regards??John DoeOu... 0000011b: 3: [End of code]
sieve-test(stephan): Debug: acl: No acl_shared_dict setting - shared mailbox listing is disabled sieve-test(stephan): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= sieve-test(stephan): Debug: Mailbox /home/stephan/message.eml: Opened mail UID=1 because: header Message-ID (Cache file is unusable) sieve-test(stephan): Debug: Mailbox /home/stephan/message.eml: Opened mail UID=1 because: header Sender (Cache file is unusable) sieve-test(stephan): Debug: Mailbox /home/stephan/message.eml: Opened mail UID=1 because: header Envelope-To (Cache file is unusable) sieve-test(stephan): Debug: Mailbox /home/stephan/message.eml: Opened mail UID=1 because: header To (Cache file is unusable)
## Started executing script 'frop' 3: vacation action 3: auto-reply with message `I'm out of office, please contact Joan Doe instead.??Best regards??John Doe' ## Finished executing script 'frop'
Performed actions:
* send vacation message: => seconds : 86400 => subject : Out of office reply => handle : I'm out of office, please contact Joan Doe instead. Best regards John DoeOut of office reply<default-from><NO-MIME>
START MESSAGE I'm out of office, please contact Joan Doe instead. Best regards John Doe END MESSAGE
Implicit keep:
* store message in folder: INBOX
sieve-test(stephan): Info: final result: success
## Binary hex dump:
$ sieve-dump -h ~/frop.svbin sieve-dump(stephan)<27527><>: Debug: Loading modules from directory: /usr/lib/dovecot/modules sieve-dump(stephan)<27527><>: Debug: Module loaded: /usr/lib/dovecot/modules/lib01_acl_plugin.so sieve-dump(stephan)<27527><>: Debug: Module loaded: /usr/lib/dovecot/modules/lib20_virtual_plugin.so sieve-dump(stephan): Debug: Effective uid=1000, gid=1000, home=/home/stephan sieve-dump(stephan): Debug: acl: No acl_shared_dict setting - shared mailbox listing is disabled
- Binary blocks (count: 4):
0: size: 33 bytes 1: size: 13 bytes 2: size: 283 bytes 3: size: 26 bytes
- Block 0 (33 bytes, file offset 00000054):
00000000 04 66 69 6c 65 00 00 18 2f 68 6f 6d 65 2f 73 74 |.file.../home/st| 00000010 65 70 68 61 6e 2f 66 72 6f 70 2e 73 69 65 76 65 |ephan/frop.sieve| 00000020 00 |.| 00000021
- Block 1 (13 bytes, file offset 00000080):
00000000 01 08 76 61 63 61 74 69 6f 6e 00 00 00 |..vacation...| 0000000d
- Block 2 (283 bytes, file offset 00000098):
00000000 03 01 00 00 0d 00 01 01 85 a3 00 02 02 13 4f 75
|..............Ou|
00000010 74 20 6f 66 20 6f 66 66 69 63 65 20 72 65 70 6c |t of office
repl|
00000020 79 00 04 03 00 00 00 30 02 02 11 6a 2e 64 6f 65
|y......0...j.doe|
00000030 40 63 6f 6d 70 61 6e 79 2e 64 6f 6d 00 02 14 6a
|@company.dom...j|
00000040 6f 68 6e 2e 64 6f 65 40 63 6f 6d 70 61 6e 79 2e
|ohn.doe@company.|
00000050 64 6f 6d 00 00 02 4b 49 27 6d 20 6f 75 74 20 6f |dom...KI'm
out o|
00000060 66 20 6f 66 66 69 63 65 2c 20 70 6c 65 61 73 65 |f office,
please|
00000070 20 63 6f 6e 74 61 63 74 20 4a 6f 61 6e 20 44 6f | contact
Joan Do|
00000080 65 20 69 6e 73 74 65 61 64 2e 0d 0a 42 65 73 74 |e
instead...Best|
00000090 20 72 65 67 61 72 64 73 0d 0a 4a 6f 68 6e 20 44 |
regards..John D|
000000a0 6f 65 00 02 75 49 27 6d 20 6f 75 74 20 6f 66 20 |oe..uI'm out
of |
000000b0 6f 66 66 69 63 65 2c 20 70 6c 65 61 73 65 20 63 |office,
please c|
000000c0 6f 6e 74 61 63 74 20 4a 6f 61 6e 20 44 6f 65 20 |ontact Joan
Doe |
000000d0 69 6e 73 74 65 61 64 2e 0d 0a 42 65 73 74 20 72
|instead...Best r|
000000e0 65 67 61 72 64 73 0d 0a 4a 6f 68 6e 20 44 6f 65 |egards..John
Doe|
000000f0 4f 75 74 20 6f 66 20 6f 66 66 69 63 65 20 72 65 |Out of
office re|
00000100 70 6c 79 3c 64 65 66 61 75 6c 74 2d 66 72 6f 6d
|ply
- Block 3 (26 bytes, file offset 000001bc):
00000000 17 00 12 00 19 00 02 04 01 17 00 1c 00 54 00 65 |.............T.e| 00000010 00 02 8f ff ff ff 78 01 4e 00 |......x.N.| 0000001a
Regards,
Stephan.
Op 24/09/2018 om 15:06 schreef James:
On 24/09/2018 13:35, Stephan Bosch wrote:
You can enable
-d -' and-t - -Tlevel=matching' as well.$ sieve-test -D -d - -t - -Tlevel=matching -c dovecot.conf sieve message sieve-test(james): Debug: Effective uid=1001, gid=10, home=/home/james sieve-test(james): Debug: maildir++: root=/path/to//james/Maildir, index=, indexpvt=, control=, inbox=/path/to//james/Maildir, alt= sieve-test(james): Debug: sieve: Pigeonhole version 0.5.2 (7704de5e) initializing sieve-test(james): Debug: sieve: include: sieve_global is not set; it is currently not possible to include
:global' scripts. debug: file storage: Using Sieve script path: /home/james/sieve. debug: file script: Opened scriptsieve' from/home/james/sieve'. debug: Scriptsieve' from /home/james/sieve successfully compiled.
- Script metadata (block: 0):
class = file class.version = 0 location = /home/james/sieve
- Required extensions (block: 1):
0: vacation (id: 9)
- Main program (block: 2):
Address Line Code 00000000: DEBUG BLOCK: 3 00000001: EXTENSIONS [1]: 00000002: vacation 00000004: 2: VACATION 00000007: 4: seconds: NUM 5 00000009: Binary is corrupt.
## Started executing script 'sieve' VACATION: #ERROR#: unknown optional operand 2: [[EXECUTION ABORTED]] ## Finished executing script 'sieve'
sieve-test(james): Info: corrupt binary deleted. sieve-test(james): Error: unlink((null)) failed: Bad address (in sieve-test.c:426) sieve-test(james): Info: final result: failed; resolved with successful implicit keep
On 25/09/2018 22:39, Stephan Bosch wrote:
Something mightily weird is going on at your end. It doesn't fail here (see below). First of all, what is your configuration (output from
dovecot -n)?
You have dovecot.conf but...
$ dovecot -c dovecot.conf -n # 2.3.3.rc1 (14e4920d8): dovecot.conf # Pigeonhole version 0.5.2 (7704de5e) # OS: SunOS 5.11 i86pc # Hostname: mailhost doveconf: Warning: please set ssl_dh= /etc/opt/xxx/dovecot/dh.pem mail_debug = yes mail_gid = staff mail_location = maildir:/path/to/%d/%n/Maildir mail_uid = james managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext postmaster_address = postmaster@domain.tld ssl_dh = # hidden, use -P to show it
Also, can you make a hex dump of the binary (using
sieve-dump -h <your-script-binary>.svbin).
As said the svbin is identical to the one create by the previous version.
Comparing the dump: Block 0 differs because it has the source file name. Block 1 is identical Block 2 is identical. It is this block that is declared corrupt.
Finally, can you try to explicitly delete the binary (preferably after preserving it elsewhere) so that it is guaranteed to be created fresh?
I did each time.
## Success at my end:
Spot the difference...
Address Line Code 00000000: DEBUG BLOCK: 3 00000001: EXTENSIONS [1]: 00000002: vacation 00000004: 3: VACATION 00000007: 5: seconds: NUM 86400
Address Line Code 00000000: DEBUG BLOCK: 3 00000001: EXTENSIONS [1]: 00000002: vacation 00000004: 2: VACATION 00000007: 4: seconds: NUM 5 00000009: Binary is corrupt.
The line numbers differs and 86400 is read as 5. It is like it has forgotten the size of an integer or is confused about endianness. There is something strange, like an #if that guesses wrong. At least I have somewhere to start looking.
Thank you for checking at your end, I was worried the RC had introduced an error and your result suggests not. RCs are for testing and I am.
Op 27-9-2018 om 11:42 schreef James:
On 25/09/2018 22:39, Stephan Bosch wrote:
Something mightily weird is going on at your end. It doesn't fail here (see below). First of all, what is your configuration (output from
dovecot -n)?You have dovecot.conf but...
$ dovecot -c dovecot.conf -n # 2.3.3.rc1 (14e4920d8): dovecot.conf # Pigeonhole version 0.5.2 (7704de5e) # OS: SunOS 5.11 i86pc # Hostname: mailhost doveconf: Warning: please set ssl_dh= /etc/opt/xxx/dovecot/dh.pem mail_debug = yes mail_gid = staff mail_location = maildir:/path/to/%d/%n/Maildir mail_uid = james managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext postmaster_address = postmaster@domain.tld ssl_dh = # hidden, use -P to show it
Also, can you make a hex dump of the binary (using
sieve-dump -h <your-script-binary>.svbin).As said the svbin is identical to the one create by the previous version.
Comparing the dump: Block 0 differs because it has the source file name. Block 1 is identical Block 2 is identical. It is this block that is declared corrupt.
Finally, can you try to explicitly delete the binary (preferably after preserving it elsewhere) so that it is guaranteed to be created fresh?
I did each time.
## Success at my end:
Spot the difference...
Address Line Code 00000000: DEBUG BLOCK: 3 00000001: EXTENSIONS [1]: 00000002: vacation 00000004: 3: VACATION 00000007: 5: seconds: NUM 86400
Address Line Code 00000000: DEBUG BLOCK: 3 00000001: EXTENSIONS [1]: 00000002: vacation 00000004: 2: VACATION 00000007: 4: seconds: NUM 5 00000009: Binary is corrupt.
The line numbers differs and 86400 is read as 5. It is like it has forgotten the size of an integer or is confused about endianness. There is something strange, like an #if that guesses wrong. At least I have somewhere to start looking.
Thank you for checking at your end, I was worried the RC had introduced an error and your result suggests not. RCs are for testing and I am.
The number is stored as a chain of bytes of which the most significant bit indicates whether the next byte still belongs to the number. If this bit is somehow interpreted wrong, the first byte of this number would read as 5, thereby returning '5' as the result and ignoring subsequent bytes (causing corruption at the next item to read).
Since you're using SunOS, your compiler may be doing something funky. Which compiler is used anyway? Perhaps different versions for the Dovecot releases that do and don't work?
Does it help when you change the "> 0" at the following code position to "!= 0" ?
https://github.com/dovecot/pigeonhole/blob/master/src/lib-sieve/sieve-binary...
Regards,
Stephan.
On 27/09/2018 13:40, Stephan Bosch wrote:
Address Line Code 00000000: DEBUG BLOCK: 3 00000001: EXTENSIONS [1]: 00000002: vacation 00000004: 2: VACATION 00000007: 4: seconds: NUM 5 00000009: Binary is corrupt.
The line numbers differs and 86400 is read as 5. It is like it has forgotten the size of an integer or is confused about endianness. There is something strange, like an #if that guesses wrong. At least I have somewhere to start looking.
Thank you for checking at your end, I was worried the RC had introduced an error and your result suggests not. RCs are for testing and I am.
The number is stored as a chain of bytes of which the most significant bit indicates whether the next byte still belongs to the number. If this bit is somehow interpreted wrong, the first byte of this number would read as 5, thereby returning '5' as the result and ignoring subsequent bytes (causing corruption at the next item to read).
Since you're using SunOS, your compiler may be doing something funky. Which compiler is used anyway? Perhaps different versions for the Dovecot releases that do and don't work?
It was studio cc. gcc doesn't make it through configure and I didn't ask why. I have some other things to do but will look at this again later. Thank you for the byte code explanations. The coding at this point is hard to follow with the pointers-to-functions and #defines.
James.
On 27 Sep 2018, at 16.53, James list@xdrv.co.uk wrote:
On 27/09/2018 13:40, Stephan Bosch wrote:
Address Line Code 00000000: DEBUG BLOCK: 3 00000001: EXTENSIONS [1]: 00000002: vacation 00000004: 2: VACATION 00000007: 4: seconds: NUM 5 00000009: Binary is corrupt.
The line numbers differs and 86400 is read as 5. It is like it has forgotten the size of an integer or is confused about endianness. There is something strange, like an #if that guesses wrong. At least I have somewhere to start looking.
Thank you for checking at your end, I was worried the RC had introduced an error and your result suggests not. RCs are for testing and I am.
The number is stored as a chain of bytes of which the most significant bit indicates whether the next byte still belongs to the number. If this bit is somehow interpreted wrong, the first byte of this number would read as 5, thereby returning '5' as the result and ignoring subsequent bytes (causing corruption at the next item to read).
Since you're using SunOS, your compiler may be doing something funky. Which compiler is used anyway? Perhaps different versions for the Dovecot releases that do and don't work?
It was studio cc. gcc doesn't make it through configure and I didn't ask why. I have some other things to do but will look at this again later. Thank you for the byte code explanations. The coding at this point is hard to follow with the pointers-to-functions and #defines.
Can you share a little bit more info on how did the compile (or configure even) fail with gcc on Solaris 11?
as I have no problems in compiling dovecot and pigeonhole on my Solaris 11.3 system with gcc. The version that ships with my Solaris is 4.5.2.
I also have Sun Studio 12.5 installed but I have not even tried to compile dovecot wit that yet.
Sami
On 27/09/2018 16:14, Sami Ketola wrote:
It was studio cc. gcc doesn't make it through configure and I didn't ask why.
Can you share a little bit more info on how did the compile (or configure even) fail with gcc on Solaris 11?
$ ./configure $ARGS ... checking Linux compatible mremap()... no checking whether shared mmaps get updated by write()s... no checking whether fd passing works... no configure: error: fd passing is required for Dovecot to work
Which in the log corresponds to:
configure:22685: ./conftest ./configure[2026]: eval: line 1: 22335: Memory fault(coredump)
Appears to the option "-mfunction-return=thunk" that cause the problem, remove and no core dump. Older gccs do not have -mfunction-return.
as I have no problems in compiling dovecot and pigeonhole on my Solaris 11.3 system with gcc. The version that ships with my Solaris is 4.5.2.
Strictly speaking Solaris 11 does not ship with gcc, one can install it [from the OS vendor] with pkg and there is a choice of versions.
# pkg list -a | grep gcc-c
I have gcc versions installed: 4.9.5, 5.5.0, 6.4.0, 7.3.0 and 8.2.0.
I also have Sun Studio 12.5 installed but I have not even tried to compile dovecot wit that yet.
Current Release - Oracle Developer Studio 12.6.
James.
On 28.09.2018 12:38, James wrote:
On 27/09/2018 16:14, Sami Ketola wrote:
It was studio cc. gcc doesn't make it through configure and I didn't ask why.
Can you share a little bit more info on how did the compile (or configure even) fail with gcc on Solaris 11?
$ ./configure $ARGS ... checking Linux compatible mremap()... no checking whether shared mmaps get updated by write()s... no checking whether fd passing works... no configure: error: fd passing is required for Dovecot to work
Which in the log corresponds to:
configure:22685: ./conftest ./configure[2026]: eval: line 1: 22335: Memory fault(coredump)
Appears to the option "-mfunction-return=thunk" that cause the problem, remove and no core dump. Older gccs do not have -mfunction-return.
as I have no problems in compiling dovecot and pigeonhole on my Solaris 11.3 system with gcc. The version that ships with my Solaris is 4.5.2.
Strictly speaking Solaris 11 does not ship with gcc, one can install it [from the OS vendor] with pkg and there is a choice of versions.
# pkg list -a | grep gcc-c
I have gcc versions installed: 4.9.5, 5.5.0, 6.4.0, 7.3.0 and 8.2.0.
I also have Sun Studio 12.5 installed but I have not even tried to compile dovecot wit that yet.
Current Release - Oracle Developer Studio 12.6.
James.
Appears we need more thoroughly ensure -mfunction-return=thunk actually works.
Aki
On 28.09.2018 13:44, Aki Tuomi wrote:
On 28.09.2018 12:38, James wrote:
On 27/09/2018 16:14, Sami Ketola wrote:
It was studio cc. gcc doesn't make it through configure and I didn't ask why. Can you share a little bit more info on how did the compile (or configure even) fail with gcc on Solaris 11?
$ ./configure $ARGS ... checking Linux compatible mremap()... no checking whether shared mmaps get updated by write()s... no checking whether fd passing works... no configure: error: fd passing is required for Dovecot to work
Which in the log corresponds to:
configure:22685: ./conftest ./configure[2026]: eval: line 1: 22335: Memory fault(coredump)
Appears to the option "-mfunction-return=thunk" that cause the problem, remove and no core dump. Older gccs do not have -mfunction-return.
as I have no problems in compiling dovecot and pigeonhole on my Solaris 11.3 system with gcc. The version that ships with my Solaris is 4.5.2. Strictly speaking Solaris 11 does not ship with gcc, one can install it [from the OS vendor] with pkg and there is a choice of versions.
# pkg list -a | grep gcc-c
I have gcc versions installed: 4.9.5, 5.5.0, 6.4.0, 7.3.0 and 8.2.0.
I also have Sun Studio 12.5 installed but I have not even tried to compile dovecot wit that yet. Current Release - Oracle Developer Studio 12.6.
James. Appears we need more thoroughly ensure -mfunction-return=thunk actually works.
Aki
You can use --disable-harderning to disable these things.
Aki
On 28/09/2018 12:38, Aki Tuomi wrote:
It was studio cc. gcc doesn't make it through configure and I didn't ask why. Can you share a little bit more info on how did the compile (or configure even) fail with gcc on Solaris 11?
$ ./configure $ARGS ... checking Linux compatible mremap()... no checking whether shared mmaps get updated by write()s... no checking whether fd passing works... no configure: error: fd passing is required for Dovecot to work
Which in the log corresponds to:
configure:22685: ./conftest ./configure[2026]: eval: line 1: 22335: Memory fault(coredump)
Appears to the option "-mfunction-return=thunk" that cause the problem, remove and no core dump.
If you thunk that was all think again. My gcc builds fail to link ending with lots of undefined symbols. It is probably libtool obstructing portability but I have an easy solution of using another compiler and ignoring libtool.
You can use --disable-harderning to disable these things.
--disable-hardening
On Thu, Sep 27, 2018 at 10:42:16 +0100, James wrote:
On 25/09/2018 22:39, Stephan Bosch wrote:
Something mightily weird is going on at your end. It doesn't fail here (see below). First of all, what is your configuration (output from
dovecot -n)?You have dovecot.conf but...
$ dovecot -c dovecot.conf -n # 2.3.3.rc1 (14e4920d8): dovecot.conf # Pigeonhole version 0.5.2 (7704de5e) # OS: SunOS 5.11 i86pc
Out of curiosity, is this a Solaris system or an illumos system?
Jeff.
-- I'm somewhere between geek and normal. - Linus Torvalds
On 27/09/2018 14:55, Josef 'Jeff' Sipek wrote:
On Thu, Sep 27, 2018 at 10:42:16 +0100, James wrote:
$ dovecot -c dovecot.conf -n # 2.3.3.rc1 (14e4920d8): dovecot.conf # Pigeonhole version 0.5.2 (7704de5e) # OS: SunOS 5.11 i86pc
Out of curiosity, is this a Solaris system or an illumos system?
Not Illumos. Test builds on 10 and 11.3 - RC exercise, I'll add Sparc when my machine is powered on. Tested run on Solaris 10 and 11.3 - RC exercise. Reports from 11.3, results so far the same either way. Packages built on Solaris 10 as LCD unless there is good reason to have a package per rev.
On 25/09/2018 22:39, Stephan Bosch wrote:
Hello,
Something mightily weird is going on at your end. It doesn't fail here
This is correct. I believe there to be a compiler problem. Removing optimisation from sieve-binary-code.c gives success.
Thank you to Stephan Bosch for testing this for me and pointing me in the right direction.
My debugging was confused because dovecot flags are passed to pigeonhole, eg $DOVECOT_CFLAGS. This is why the build of dovecot affects pigeonhole. Once I removed these I could control the pigeonhole build and I was able to isolate the problem.
James.
Op 1-10-2018 om 10:43 schreef James:
On 25/09/2018 22:39, Stephan Bosch wrote:
Hello,
Something mightily weird is going on at your end. It doesn't fail here
This is correct. I believe there to be a compiler problem. Removing optimisation from sieve-binary-code.c gives success.
Thank you to Stephan Bosch for testing this for me and pointing me in the right direction.
My debugging was confused because dovecot flags are passed to pigeonhole, eg $DOVECOT_CFLAGS. This is why the build of dovecot affects pigeonhole. Once I removed these I could control the pigeonhole build and I was able to isolate the problem.
Ok, good.
I'd still like to know whether doing this helps somehow:
Does it help when you change the "> 0" at the following code position to "!= 0" ?
https://github.com/dovecot/pigeonhole/blob/master/src/lib-sieve/sieve-binary...
Regards,
Stephan.
On 01/10/2018 13:37, Stephan Bosch wrote:
I'd still like to know whether doing this helps somehow:
Does it help when you change the "> 0" at the following code position to "!= 0" ?
https://github.com/dovecot/pigeonhole/blob/master/src/lib-sieve/sieve-binary...
--- ../original/src/lib-sieve/sieve-binary-code.c 2018-02-05 19:45:53.000000000 +0000 +++ src/lib-sieve/sieve-binary-code.c 2018-10-01 14:35:18.553169973 +0000 @@ -297,7 +297,7 @@ return FALSE; /* Read first integer bytes [1xxxxxxx] */ - while ( (ADDR_DATA_AT(address) & 0x80) > 0 ) { + while ( (ADDR_DATA_AT(address) & 0x80) != 0 ) { if ( ADDR_BYTES_LEFT(address) > 0 && bits > 0) { integer |= ADDR_DATA_AT(address) & 0x7F; ADDR_JUMP(address, 1); Quick reply: It does. I can't see why. James.
participants (5)
-
Aki Tuomi
-
James
-
Josef 'Jeff' Sipek
-
Sami Ketola
-
Stephan Bosch