rawlog data in a lua script

[michael.zork at feierfighter.de]

Hi Aki,

On 08.08.22 13:54, Aki Tuomi wrote:
> Hi, Michael, did you consider my suggestion to use raw events instead of rawlogs for this?

I was writing an answer to you next :-)

As far as I can see, the "Event Export" only exports events of the 
requests, but not the full raw responses, correct?
https://doc.dovecot.org/configuration_manual/event_export/

I need the complete rawlog that currently is written to the rawlog 
directory, which means the raw requests (IMAP, POP3 commands), and the 
raw response lines (for example a FETCH response -> a 20 MB mail 
content). Everything that could be seen on the wire via tcpdump after 
authentication (which is the rawlog of Dovecot as far as I can see).

I need the rawlog feature, but not written to multiple files (which I 
have to collect in realtime with some black magic), but for example in a 
lua-script, which would make it a lot easier to analyse and/or send it 
to an HTTP endpoint.

Maybe there are other possibilities, for example sending the rawlog of a 
user to a single file (or pipe/socket), where I can easily receive the 
raw logs for that user and send it to an HTTP endpoint. That's a lot 
easier than to "watch" a directory for new files, detect changes to 
existing files, collect them and send them via HTTP.

I somehow need to send the raw log of specific users in realtime (maybe 
with a few seconds delay) to an HTTP endpoint (where each request or 
response is a single HTTP request, maybe we could also batch some 
requests and responses to reduce the HTTP requests to the endpoint).

The current implementation of the rawlog feature is nice for manually 
debugging a single user, but when debugging/monitoring multiple users 
automatically, collect the logs and send them to a central place, it's 
hard to use ("watching" directories for changes via inotify, and run 
"tail" on the files for hours and days is not fun and can easily break).

Michael


>> On 08/08/2022 14:52 EESTmichael.zork at feierfighter.de  wrote:
>>
>>
>> Hi,
>>
>> as far as I know I cannot configure Dovecot to pipe the rawlog into rsyslog. Or can I, how?
>>
>> The rawlog feature in Dovecot writes multiple files (two for each connection, one for raw requests and one for raw responses) into a predefined directory for the user. This generates dozens or hundreds of files per user per day, each file with a timestamp in it, so the filename is not predictable.
>>
>> Even if it works, I'm not sure if syslog (rsyslog or syslog-ng) should be (ab)used to collect the rawlog file contents, which might be hundreds of MB per minute if someone FETCHes all his emails while setting up a new account in Thunderbird or so. That sounds like a suboptional idea. Syslog cannot handle binary text I guess, and it might have limits like "line length limits" or similar. It sounds like the wrong tool for the job.
>>
>> Michael
>>
>>
>> Am 28-Jul-2022 15:28:16 +0200 schriebdovecot at ptld.com:
>>>> I'm searching for a possibility to have the rawlog feature in lua, which would be much easier for processing.
>>>>
>>>> It would be much easier to hook to the "raw request and response events" inside Dovecot and have the rawlog-data in a lua script,
>>>> where I can prepare it and send it to another maschine for monitoring/collection/analysis/statistics or similar, for example via HTTP.
>>> rsyslog has this feature (omprog) allowing you to setup any script/program for it to pipe logs to in real time.
>>>
>>> https://www.rsyslog.com/doc/master/configuration/modules/omprog.html
>>> https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md
>>>
>>> Works similar in concept to postfix policy servers if you are familiar with them.