PDFs getting mangled

[Adam Weinberger]

> On 20 Jan, 2018, at 10:05, Adam Weinberger <adamw at adamw.org> wrote:
>
>
>> On 19 Jan, 2018, at 4:39, Aki Tuomi <aki.tuomi at dovecot.fi> wrote:
>>
>>
>>
>> On 19.01.2018 04:35, Adam Weinberger wrote:
>>> Since upgrading to 2.3.0 / 0.5.0.1, incoming PDFs are getting mangled.
>>> It seems to be happening when I use vnd.dovecot.filter. When I comment
>>> out the block, things come through fine.
>>>
>>> My filter block looks like this:
>>>        require "vnd.dovecot.filter";
>>>        filter "bogofilter_filter";
>>>
>>>         if header :contains "X-Bogosity" [
>>>                 "Spam, tests=bogofilter, spamicity=1.00",
>>>                 "Spam, tests=bogofilter, spamicity=0.99"
>>>         ] {
>>>             fileinto "spam/totally";
>>>             stop;
>>>         }
>>>         elsif header :contains "X-Bogosity" "Spam," {
>>>             fileinto "spam/probably";
>>>             stop;
>>>         }
>>>         elsif header :contains "X-Bogosity" "Unsure," {
>>>             fileinto "spam/maybe";
>>>             stop;
>>>         }
>>>
>>> filter/bogofilter_filter looks like this:
>>>         #!/bin/sh
>>>         cat /dev/stdin | /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf
>>>
>>> Images seem to come through fine. I didn't have this problem before the
>>> upgrade. It could easily be a problem with bogofilter, but I figured
>>> that I'd check here first to see if anybody else has an
>>> attachment-mangling issue with vnd.dovecot.filter.
>>>
>>> # Adam
>>>
>>>
>>> --
>>> Adam Weinberger
>>> adamw at adamw.org
>>> http://www.adamw.org
>> It might work better with
>>
>> exec /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf
>>
>> Aki
>
> After looking into this some more, what's happening is that messages are  
> getting truncated. Regardless of whether it's an attachment or just a  
> long message body, the message gets truncated.
>
> To show this, I made a fileinto :copy rule to save a message copy before  
> it gets sent off to bogofilter.
>
> You can see from the below pastings that the message is definitely  
> getting truncated. When I replace my filter script with "cat /dev/stdin",  
> it works fine. When I send the message to bogofilter on the command-line,  
> it works fine. Something about the combination of vnd.dovecot.filter +  
> bogofilter is causing the messages to truncate, and I don't know whether  
> it's from truncating on the send or receive of the filter.
>
> Can you give me any advice?
>
> 1)
> $ jot 200000 1 > numbers.txt
> $ du -a . | grep numbers
> 1260	./numbers-sent.txt
> 1248	./numbers-received.txt
>
> 2)
> root at imap:~# ll test-*
> 125 -rw-------  1 root  wheel   123K Jan 20 09:35 test-afterbogo.msg
> 149 -rw-------  1 root  wheel   136K Jan 20 09:35 test-beforebogo.msg

The more I look into it, the more it looks to me like pigeonhole is somehow  
losing the last 4-6K of messages over 100K.

When my filter script is:
cat /dev/stdin | tee /tmp/input | bogofilter[...] | tee /tmp/output
Then /tmp/output is the full message, but what pigeonhole gets back is  
truncated at by about 4K.

Same with
[...] bogofilter[...] > /tmp/output; cat /tmp/output
The message is truncated there too.

I've tried everything I can think of to make sure it's not a buffering  
problem:
[...] bogofilter[...] | cat
[...] bogofilter[...] | dd bs=1k
[...] bogofilter[...] ; a=$?; sleep 2; exit $a

But everything I do comes up with the same problem.

But a simple "cat /dev/stdin" and the message comes through fine.

Your help is really appreciated. I'm in the process of buying a house, so  
the mangled attachments is not coming at a great time :-)

# Adam


--
Adam Weinberger
adamw at adamw.org
http://www.adamw.org