PDFs getting mangled

[Stephan Bosch]

Op 1/20/2018 om 11:01 PM schreef Adam Weinberger:
>> On 20 Jan, 2018, at 10:05, Adam Weinberger <adamw at adamw.org> wrote:
>>
>>
>>> On 19 Jan, 2018, at 4:39, Aki Tuomi <aki.tuomi at dovecot.fi> wrote:
>>>
>>>
>>>
>>> On 19.01.2018 04:35, Adam Weinberger wrote:
>>>> Since upgrading to 2.3.0 / 0.5.0.1, incoming PDFs are getting mangled.
>>>> It seems to be happening when I use vnd.dovecot.filter. When I comment
>>>> out the block, things come through fine.
>>>>
>>>> My filter block looks like this:
>>>>        require "vnd.dovecot.filter";
>>>>        filter "bogofilter_filter";
>>>>
>>>>         if header :contains "X-Bogosity" [
>>>>                 "Spam, tests=bogofilter, spamicity=1.00",
>>>>                 "Spam, tests=bogofilter, spamicity=0.99"
>>>>         ] {
>>>>             fileinto "spam/totally";
>>>>             stop;
>>>>         }
>>>>         elsif header :contains "X-Bogosity" "Spam," {
>>>>             fileinto "spam/probably";
>>>>             stop;
>>>>         }
>>>>         elsif header :contains "X-Bogosity" "Unsure," {
>>>>             fileinto "spam/maybe";
>>>>             stop;
>>>>         }
>>>>
>>>> filter/bogofilter_filter looks like this:
>>>>         #!/bin/sh
>>>>         cat /dev/stdin | /usr/local/bin/bogofilter -p -e -c
>>>> /path/to/bogofilter.cf
>>>>
>>>> Images seem to come through fine. I didn't have this problem before
>>>> the
>>>> upgrade. It could easily be a problem with bogofilter, but I figured
>>>> that I'd check here first to see if anybody else has an
>>>> attachment-mangling issue with vnd.dovecot.filter.
>>>>
>>>> # Adam
>>>>
>>>>
>>>> -- 
>>>> Adam Weinberger
>>>> adamw at adamw.org
>>>> http://www.adamw.org
>>> It might work better with
>>>
>>> exec /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf
>>>
>>> Aki
>>
>> After looking into this some more, what's happening is that messages
>> are getting truncated. Regardless of whether it's an attachment or
>> just a long message body, the message gets truncated.
>>
>> To show this, I made a fileinto :copy rule to save a message copy
>> before it gets sent off to bogofilter.
>>
>> You can see from the below pastings that the message is definitely
>> getting truncated. When I replace my filter script with "cat
>> /dev/stdin", it works fine. When I send the message to bogofilter on
>> the command-line, it works fine. Something about the combination of
>> vnd.dovecot.filter + bogofilter is causing the messages to truncate,
>> and I don't know whether it's from truncating on the send or receive
>> of the filter.
>>
>> Can you give me any advice?
>>
>> 1)
>> $ jot 200000 1 > numbers.txt
>> $ du -a . | grep numbers
>> 1260    ./numbers-sent.txt
>> 1248    ./numbers-received.txt
>>
>> 2)
>> root at imap:~# ll test-*
>> 125 -rw-------  1 root  wheel   123K Jan 20 09:35 test-afterbogo.msg
>> 149 -rw-------  1 root  wheel   136K Jan 20 09:35 test-beforebogo.msg
>
> The more I look into it, the more it looks to me like pigeonhole is
> somehow losing the last 4-6K of messages over 100K.
>
> When my filter script is:
> cat /dev/stdin | tee /tmp/input | bogofilter[...] | tee /tmp/output
> Then /tmp/output is the full message, but what pigeonhole gets back is
> truncated at by about 4K.
>
> Same with
> [...] bogofilter[...] > /tmp/output; cat /tmp/output
> The message is truncated there too.
>
> I've tried everything I can think of to make sure it's not a buffering
> problem:
> [...] bogofilter[...] | cat
> [...] bogofilter[...] | dd bs=1k
> [...] bogofilter[...] ; a=$?; sleep 2; exit $a
>
> But everything I do comes up with the same problem.
>
> But a simple "cat /dev/stdin" and the message comes through fine.
>
> Your help is really appreciated. I'm in the process of buying a house,
> so the mangled attachments is not coming at a great time :-)

We are looking into it.

Regards,

Stephan.