PDFs getting mangled

[Adam Weinberger]

> On 19 Jan, 2018, at 4:39, Aki Tuomi <aki.tuomi at dovecot.fi> wrote:
>
>
>
> On 19.01.2018 04:35, Adam Weinberger wrote:
>> Since upgrading to 2.3.0 / 0.5.0.1, incoming PDFs are getting mangled.
>> It seems to be happening when I use vnd.dovecot.filter. When I comment
>> out the block, things come through fine.
>>
>> My filter block looks like this:
>>        require "vnd.dovecot.filter";
>>        filter "bogofilter_filter";
>>
>>         if header :contains "X-Bogosity" [
>>                 "Spam, tests=bogofilter, spamicity=1.00",
>>                 "Spam, tests=bogofilter, spamicity=0.99"
>>         ] {
>>             fileinto "spam/totally";
>>             stop;
>>         }
>>         elsif header :contains "X-Bogosity" "Spam," {
>>             fileinto "spam/probably";
>>             stop;
>>         }
>>         elsif header :contains "X-Bogosity" "Unsure," {
>>             fileinto "spam/maybe";
>>             stop;
>>         }
>>
>> filter/bogofilter_filter looks like this:
>>         #!/bin/sh
>>         cat /dev/stdin | /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf
>>
>> Images seem to come through fine. I didn't have this problem before the
>> upgrade. It could easily be a problem with bogofilter, but I figured
>> that I'd check here first to see if anybody else has an
>> attachment-mangling issue with vnd.dovecot.filter.
>>
>> # Adam
>>
>>
>> --
>> Adam Weinberger
>> adamw at adamw.org
>> http://www.adamw.org
> It might work better with
>
> exec /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf
>
> Aki

After looking into this some more, what's happening is that messages are  
getting truncated. Regardless of whether it's an attachment or just a long  
message body, the message gets truncated.

To show this, I made a fileinto :copy rule to save a message copy before it  
gets sent off to bogofilter.

You can see from the below pastings that the message is definitely getting  
truncated. When I replace my filter script with "cat /dev/stdin", it works  
fine. When I send the message to bogofilter on the command-line, it works  
fine. Something about the combination of vnd.dovecot.filter + bogofilter is  
causing the messages to truncate, and I don't know whether it's from  
truncating on the send or receive of the filter.

Can you give me any advice?

1)
$ jot 200000 1 > numbers.txt
$ du -a . | grep numbers
1260	./numbers-sent.txt
1248	./numbers-received.txt

2)
root at imap:~# ll test-*
125 -rw-------  1 root  wheel   123K Jan 20 09:35 test-afterbogo.msg
149 -rw-------  1 root  wheel   136K Jan 20 09:35 test-beforebogo.msg

root at imap:~# tail -20 test-beforebogo.msg
IAowMDAwMTAxMTUzIDAwMDAwIG4gCjAwMDAxMDEyMDYgMDAwMDAgbiAKMDAwMDEwMTIzMiAwMDAw
MCBuIAowMDAwMTAxMjc0IDAwMDAwIG4gCjAwMDAxMDEyOTMgMDAwMDAgbiAKdHJhaWxlcgo8PCAv
U2l6ZSAxOSAvUm9vdCAxMiAwIFIgL0luZm8gMSAwIFIgL0lEIFsgPGM2YTE5OTc3MWRmODZjODk0
YzJhNDIwZDUzZjU3YzAzPgo8YzZhMTk5NzcxZGY4NmM4OTRjMmE0MjBkNTNmNTdjMDM+IF0gPj4K
c3RhcnR4cmVmCjEwMTQ1NwolJUVPRgo=
--Apple-Mail=_6BE3438D-2544-4A5B-8D67-44E4A7E33BA5
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
         charset=us-ascii;
         delsp=yes;
         format=flowed


Adam Weinberger
adamw at adamw.org
http://www.adamw.org


--Apple-Mail=_6BE3438D-2544-4A5B-8D67-44E4A7E33BA5--

root at imap:~# tail -20 test-afterbogo.msg
CkGE66EHXl45NsGeItQOxE5P4ONNu/mZPUXdOF/pOOBI9ACqvQxFvW162vb9cG7hXCioruIo+2ES
Wtn+UT2t6Yv2YgpTlOhTXD5riRQ4vlEWepl6tvU6ZOj+Hhpm5INo8B2Nz6XHiU2Ohz333pyvZj6M
9cQeY0Oi8pzB5qzHlH16DZiMeth/a66YRspD4uvuvziGSiLxxx2ghYaFOL6VhVuI9YOhuyZrfZ3C
drtt27RNfwboY6iERO0zZKE0dzqDx3WcaxkzABLu6vgjBo6OeG5O4nwidEzSUciJU41Yh1q9rjWQ
z9GBrRJxpSOhzZQubWwQR3xJfGAOjgDDcHQJ56jNpGdVjoG3pvgu5lEKxmKwTsHGrxkuh5WT4g/O
dXsiK8IvxjVJmx0oKF3OAtdvGUxACg/3onJx6uiKFRCI/GL4AtahLiEUHXPVcXDSSkJ0XY9lxXxY
Ucu+H5xcbSVe98Eh12HxXpHFQjdBvJf79XM0kYhi/KHyGMCqrYZNBsRg6lOucyClarTvwKC6GfwB
2RTwhE+nCnmIgx3A4JBrYA92Qy0wSCrO6VyWt0D3eiZYEseLIflww0Q6IoPuK89g7Y7QdIH3MRQ7
QghQ7WWefQ9A4NE5Hoj0To5G6QBOsfmT2ROmEACdIlYdiH1qblfCh8inz6Ox16XdP941NC4f8PPz
isEgdId5LlPyJDi9fRgIdURTYzqwnk4YhRKtUnDNDbDojLk3qbPs9XTfVCXb6lKfkDKgnECsautd
Rj8ikTwlPC/eA4vDE0AAtB4Erm3AaTC8wh7/AJMn8dnYKLty7ZmZIU1smt7OHvxPRfpWtE9HsnwH
JFPp9UguEIhweW3AAAAHAYeguJjkDyI8mJqCpIZtWpDk3rwEFVUE5HqjonCeumwGF5N+NJr1cKTx
DGnpOo8NPGEIri13UJySpHjpqZxXe+/D+/T/APAr/pPT/ndn8DahqidSDjkL5V6YyNaU4eT2H4PV
MKRUpASwDr1UCmBtXBKra8rH8EfvHUTgZyp9gPxnB4RTxr9k9wrwejJKRG3MPKcHVDDOaVQxC30D
TNR6wJy5UEHY6A8lfbNyJEUBo4VN74JrYgzjSOgQatEZ474e14GgvHJFNhFm2Usa3iI7yG/M95gc
XEKI8I9TIg087R/s+uPjTR0AaHy359TDzr0861/J5fx28TEIO4Jsh7AB8Y1wvymNfwLnF6Bgj8nf
1taKM9U8NO0nWHRgFTWaNrRyRV37BhnjLrmxFHXtjxm6iaKuxfI+KIO89G1M7rSa89ZghRuqA/Kz
IZNNURgeCvjOp5SHqq1snTUjcFJ4XZNA4SwE50lTNBajww6boqGlslH0UJz3wGd9p9nIv0NIhED0
CQm9W7mGJ9zgfcB1G6


--
Adam Weinberger
adamw at adamw.org
http://www.adamw.org