PDFs getting mangled

[Stephan Bosch]

Op 1/22/2018 om 8:58 PM schreef Stephan Bosch:
> Op 1/21/2018 om 4:34 PM schreef Stephan Bosch:
>> Op 1/20/2018 om 11:01 PM schreef Adam Weinberger:
>>>> On 20 Jan, 2018, at 10:05, Adam Weinberger <adamw at adamw.org> wrote:
>>>>
>>>>
>>>>> On 19 Jan, 2018, at 4:39, Aki Tuomi <aki.tuomi at dovecot.fi> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 19.01.2018 04:35, Adam Weinberger wrote:
>>>>>> Since upgrading to 2.3.0 / 0.5.0.1, incoming PDFs are getting mangled.
>>>>>> It seems to be happening when I use vnd.dovecot.filter. When I comment
>>>>>> out the block, things come through fine.
>>>>>>
>>>>>> My filter block looks like this:
>>>>>>        require "vnd.dovecot.filter";
>>>>>>        filter "bogofilter_filter";
>>>>>>
>>>>>>         if header :contains "X-Bogosity" [
>>>>>>                 "Spam, tests=bogofilter, spamicity=1.00",
>>>>>>                 "Spam, tests=bogofilter, spamicity=0.99"
>>>>>>         ] {
>>>>>>             fileinto "spam/totally";
>>>>>>             stop;
>>>>>>         }
>>>>>>         elsif header :contains "X-Bogosity" "Spam," {
>>>>>>             fileinto "spam/probably";
>>>>>>             stop;
>>>>>>         }
>>>>>>         elsif header :contains "X-Bogosity" "Unsure," {
>>>>>>             fileinto "spam/maybe";
>>>>>>             stop;
>>>>>>         }
>>>>>>
>>>>>> filter/bogofilter_filter looks like this:
>>>>>>         #!/bin/sh
>>>>>>         cat /dev/stdin | /usr/local/bin/bogofilter -p -e -c
>>>>>> /path/to/bogofilter.cf
>>>>>>
>>>>>> Images seem to come through fine. I didn't have this problem before
>>>>>> the
>>>>>> upgrade. It could easily be a problem with bogofilter, but I figured
>>>>>> that I'd check here first to see if anybody else has an
>>>>>> attachment-mangling issue with vnd.dovecot.filter.
>>>>>>
>>>>>> # Adam
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Adam Weinberger
>>>>>> adamw at adamw.org
>>>>>> http://www.adamw.org
>>>>> It might work better with
>>>>>
>>>>> exec /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf
>>>>>
>>>>> Aki
>>>> After looking into this some more, what's happening is that messages
>>>> are getting truncated. Regardless of whether it's an attachment or
>>>> just a long message body, the message gets truncated.
>>>>
>>>> To show this, I made a fileinto :copy rule to save a message copy
>>>> before it gets sent off to bogofilter.
>>>>
>>>> You can see from the below pastings that the message is definitely
>>>> getting truncated. When I replace my filter script with "cat
>>>> /dev/stdin", it works fine. When I send the message to bogofilter on
>>>> the command-line, it works fine. Something about the combination of
>>>> vnd.dovecot.filter + bogofilter is causing the messages to truncate,
>>>> and I don't know whether it's from truncating on the send or receive
>>>> of the filter.
>>>>
>>>> Can you give me any advice?
>>>>
>>>> 1)
>>>> $ jot 200000 1 > numbers.txt
>>>> $ du -a . | grep numbers
>>>> 1260    ./numbers-sent.txt
>>>> 1248    ./numbers-received.txt
>>>>
>>>> 2)
>>>> root at imap:~# ll test-*
>>>> 125 -rw-------  1 root  wheel   123K Jan 20 09:35 test-afterbogo.msg
>>>> 149 -rw-------  1 root  wheel   136K Jan 20 09:35 test-beforebogo.msg
>>> The more I look into it, the more it looks to me like pigeonhole is
>>> somehow losing the last 4-6K of messages over 100K.
>>>
>>> When my filter script is:
>>> cat /dev/stdin | tee /tmp/input | bogofilter[...] | tee /tmp/output
>>> Then /tmp/output is the full message, but what pigeonhole gets back is
>>> truncated at by about 4K.
>>>
>>> Same with
>>> [...] bogofilter[...] > /tmp/output; cat /tmp/output
>>> The message is truncated there too.
>>>
>>> I've tried everything I can think of to make sure it's not a buffering
>>> problem:
>>> [...] bogofilter[...] | cat
>>> [...] bogofilter[...] | dd bs=1k
>>> [...] bogofilter[...] ; a=$?; sleep 2; exit $a
>>>
>>> But everything I do comes up with the same problem.
>>>
>>> But a simple "cat /dev/stdin" and the message comes through fine.
>>>
>>> Your help is really appreciated. I'm in the process of buying a house,
>>> so the mangled attachments is not coming at a great time :-)
>> We are looking into it.
> Problem found; working on a fix.

Fix pending:

https://github.com/stephanbosch/dovecot-core/commits/fix-program-client-big-io

This is a feature branch of 9 commits containing the fix and also
cleanups and reliability improvements. This is of course based on master.

Regards,

Stephan.