Hi all,
I run logwatch on a few servers and the script on there which parses the dovecot logs provides waaaaaay too much detail, and contains a bunch of stuff I'm not interested in. There's not much point in having a log summary if its 100Kb long! The one for exim is also terrible, but I've replaced that with eximstats, and a bit of bash scripting and have something I can work with. I might at some point look at replacing that too.
Now to my question. I haven't found anything in the dovecot family which compares to eximstats. I was wondering if there possibly is a report tool, or a way of extracting stats from doveadm just for key metrics such as 'number of POP logins in the last 24 hours', number of failed passwords, etc. Things that might be interesting on a logwatch report. Or maybe someone has written a script already that can be used in logwatch in perl, python, bash, whatever. Any pointers, suggestions, or ideas for useful summary metrics to pull out of the logfiles are appreciated.
I suspect I might end up writing one myself anyway, but I'm just keen to see what already exists, get the benefit of anyone else's experiences in this matter before I start.
P.
On 24/05/2021 10:20 Plutocrat plutocrat@gmail.com wrote:
Hi all,
I run logwatch on a few servers and the script on there which parses the dovecot logs provides waaaaaay too much detail, and contains a bunch of stuff I'm not interested in. There's not much point in having a log summary if its 100Kb long! The one for exim is also terrible, but I've replaced that with eximstats, and a bit of bash scripting and have something I can work with. I might at some point look at replacing that too.
Now to my question. I haven't found anything in the dovecot family which compares to eximstats. I was wondering if there possibly is a report tool, or a way of extracting stats from doveadm just for key metrics such as 'number of POP logins in the last 24 hours', number of failed passwords, etc. Things that might be interesting on a logwatch report. Or maybe someone has written a script already that can be used in logwatch in perl, python, bash, whatever. Any pointers, suggestions, or ideas for useful summary metrics to pull out of the logfiles are appreciated.
I suspect I might end up writing one myself anyway, but I'm just keen to see what already exists, get the benefit of anyone else's experiences in this matter before I start.
P.
with recent 2.3.14
metric auth_failed { filter=event=auth_request_finished and (not success=yes) group_by = service }
service stats { inet_listener http { port = 9900 } }
should produce
auth_failed_pop3 etc. which you can collect with prometheus, or you can use doveadm stats dump
Aki
OK, so I started playing with this. I'm running dovecot 2.3.7 on this server (Ubuntu 20.04 LTS), so when I created a conf.d/95-plugins.conf file with the content below, it wasn't happy about the metric stanza. However I dug around in the documentation and found another few metrics to get going with that my older version was happy with. I left it overnight.
When I came back in the morning, doveadm stats dump gave me ... a big line of zeros! Oh.
Looking around further, I found that I needed to enable the mail_plugins variable, both in the main config, and also under the imap and pop protocols. So that looked like this:
# In 10-master.conf mail_plugins = stats
# In 20-imap.conf protocol imap { #mail_plugins = quota imap_quota mail_plugins = $mail_plugins quota imap_quota } So now a config dump gives protocol imap { mail_plugins = stats quota imap_quota ... and then the same edit in 20-pop.conf
Right. Restarted the server, all looking good, but now 'doveadm stats dump' gives me Fatal: Plugin 'stats' not found from directory /usr/lib/dovecot/modules
And sure enough, the stats plugin isn't there. And systemctl status dovecot shows there is no stats daemon running.
Now the strange bit. I can't apt install dovecot-stats, because there doesn't seem to be a dovecot-stats package available for Ubuntu!
Not sure how to proceed here. Will start looking for another repo, but at this point, I've already spend a few hours hacking around. I could have knocked up some log processing bash script in half that time! Isn't that always the way ...
P.
On 24/05/2021 15.52, Aki Tuomi wrote:
On 24/05/2021 10:20 Plutocrat plutocrat@gmail.com wrote:
Hi all,
I run logwatch on a few servers and the script on there which parses the dovecot logs provides waaaaaay too much detail, and contains a bunch of stuff I'm not interested in. There's not much point in having a log summary if its 100Kb long! The one for exim is also terrible, but I've replaced that with eximstats, and a bit of bash scripting and have something I can work with. I might at some point look at replacing that too.
Now to my question. I haven't found anything in the dovecot family which compares to eximstats. I was wondering if there possibly is a report tool, or a way of extracting stats from doveadm just for key metrics such as 'number of POP logins in the last 24 hours', number of failed passwords, etc. Things that might be interesting on a logwatch report. Or maybe someone has written a script already that can be used in logwatch in perl, python, bash, whatever. Any pointers, suggestions, or ideas for useful summary metrics to pull out of the logfiles are appreciated.
I suspect I might end up writing one myself anyway, but I'm just keen to see what already exists, get the benefit of anyone else's experiences in this matter before I start.
P.
with recent 2.3.14
metric auth_failed { filter=event=auth_request_finished and (not success=yes) group_by = service }
service stats { inet_listener http { port = 9900 } }
should produce
auth_failed_pop3 etc. which you can collect with prometheus, or you can use
doveadm stats dumpAki
Hi!
Seems ubuntu is missing this for whatever odd reason. You can try https://repo.dovecot.org/.
The metrics filter syntax was changed in 2.3.12.
Aki
On 26/05/2021 06:31 Plutocrat plutocrat@gmail.com wrote:
OK, so I started playing with this. I'm running dovecot 2.3.7 on this server (Ubuntu 20.04 LTS), so when I created a conf.d/95-plugins.conf file with the content below, it wasn't happy about the metric stanza. However I dug around in the documentation and found another few metrics to get going with that my older version was happy with. I left it overnight.
When I came back in the morning, doveadm stats dump gave me ... a big line of zeros! Oh.
Looking around further, I found that I needed to enable the mail_plugins variable, both in the main config, and also under the imap and pop protocols. So that looked like this:
# In 10-master.conf mail_plugins = stats
# In 20-imap.conf protocol imap { #mail_plugins = quota imap_quota mail_plugins = $mail_plugins quota imap_quota } So now a config dump gives protocol imap { mail_plugins = stats quota imap_quota ... and then the same edit in 20-pop.conf
Right. Restarted the server, all looking good, but now 'doveadm stats dump' gives me Fatal: Plugin 'stats' not found from directory /usr/lib/dovecot/modules
And sure enough, the stats plugin isn't there. And systemctl status dovecot shows there is no stats daemon running.
Now the strange bit. I can't apt install dovecot-stats, because there doesn't seem to be a dovecot-stats package available for Ubuntu!
Not sure how to proceed here. Will start looking for another repo, but at this point, I've already spend a few hours hacking around. I could have knocked up some log processing bash script in half that time! Isn't that always the way ...
P.
On 24/05/2021 15.52, Aki Tuomi wrote:
On 24/05/2021 10:20 Plutocrat plutocrat@gmail.com wrote:
Hi all,
I run logwatch on a few servers and the script on there which parses the dovecot logs provides waaaaaay too much detail, and contains a bunch of stuff I'm not interested in. There's not much point in having a log summary if its 100Kb long! The one for exim is also terrible, but I've replaced that with eximstats, and a bit of bash scripting and have something I can work with. I might at some point look at replacing that too.
Now to my question. I haven't found anything in the dovecot family which compares to eximstats. I was wondering if there possibly is a report tool, or a way of extracting stats from doveadm just for key metrics such as 'number of POP logins in the last 24 hours', number of failed passwords, etc. Things that might be interesting on a logwatch report. Or maybe someone has written a script already that can be used in logwatch in perl, python, bash, whatever. Any pointers, suggestions, or ideas for useful summary metrics to pull out of the logfiles are appreciated.
I suspect I might end up writing one myself anyway, but I'm just keen to see what already exists, get the benefit of anyone else's experiences in this matter before I start.
P.
with recent 2.3.14
metric auth_failed { filter=event=auth_request_finished and (not success=yes) group_by = service }
service stats { inet_listener http { port = 9900 } }
should produce
auth_failed_pop3 etc. which you can collect with prometheus, or you can use
doveadm stats dumpAki
After testing this again, I noticed that it is there:
$ apt-cache show dovecot-core Package: dovecot-core Architecture: amd64 Version: 1:2.3.7.2-1ubuntu3.3 Priority: optional Section: mail Source: dovecot Origin: Ubuntu
$ dpkg -L dovecot-core | grep stats /usr/lib/dovecot/modules/lib90_old_stats_plugin.so /usr/lib/dovecot/modules/old-stats /usr/lib/dovecot/modules/old-stats/libold_stats_mail.so /usr/lib/dovecot/modules/old-stats/libstats_auth.so /usr/lib/dovecot/old-stats /usr/lib/dovecot/stats /usr/share/man/man1/doveadm-stats.1.gz
$ doveadm stats dump Fatal: net_connect_unix(/var/run/dovecot/stats-reader) failed: No such file or directory
Maybe you have mismatch between running dovecot and doveadm somehow?
Aki
On 26/05/2021 07:54 Aki Tuomi aki.tuomi@open-xchange.com wrote:
Hi!
Seems ubuntu is missing this for whatever odd reason. You can try https://repo.dovecot.org/.
The metrics filter syntax was changed in 2.3.12.
Aki
On 26/05/2021 06:31 Plutocrat plutocrat@gmail.com wrote:
OK, so I started playing with this. I'm running dovecot 2.3.7 on this server (Ubuntu 20.04 LTS), so when I created a conf.d/95-plugins.conf file with the content below, it wasn't happy about the metric stanza. However I dug around in the documentation and found another few metrics to get going with that my older version was happy with. I left it overnight.
When I came back in the morning, doveadm stats dump gave me ... a big line of zeros! Oh.
Looking around further, I found that I needed to enable the mail_plugins variable, both in the main config, and also under the imap and pop protocols. So that looked like this:
# In 10-master.conf mail_plugins = stats
# In 20-imap.conf protocol imap { #mail_plugins = quota imap_quota mail_plugins = $mail_plugins quota imap_quota } So now a config dump gives protocol imap { mail_plugins = stats quota imap_quota ... and then the same edit in 20-pop.conf
Right. Restarted the server, all looking good, but now 'doveadm stats dump' gives me Fatal: Plugin 'stats' not found from directory /usr/lib/dovecot/modules
And sure enough, the stats plugin isn't there. And systemctl status dovecot shows there is no stats daemon running.
Now the strange bit. I can't apt install dovecot-stats, because there doesn't seem to be a dovecot-stats package available for Ubuntu!
Not sure how to proceed here. Will start looking for another repo, but at this point, I've already spend a few hours hacking around. I could have knocked up some log processing bash script in half that time! Isn't that always the way ...
P.
On 24/05/2021 15.52, Aki Tuomi wrote:
On 24/05/2021 10:20 Plutocrat plutocrat@gmail.com wrote:
Hi all,
I run logwatch on a few servers and the script on there which parses the dovecot logs provides waaaaaay too much detail, and contains a bunch of stuff I'm not interested in. There's not much point in having a log summary if its 100Kb long! The one for exim is also terrible, but I've replaced that with eximstats, and a bit of bash scripting and have something I can work with. I might at some point look at replacing that too.
Now to my question. I haven't found anything in the dovecot family which compares to eximstats. I was wondering if there possibly is a report tool, or a way of extracting stats from doveadm just for key metrics such as 'number of POP logins in the last 24 hours', number of failed passwords, etc. Things that might be interesting on a logwatch report. Or maybe someone has written a script already that can be used in logwatch in perl, python, bash, whatever. Any pointers, suggestions, or ideas for useful summary metrics to pull out of the logfiles are appreciated.
I suspect I might end up writing one myself anyway, but I'm just keen to see what already exists, get the benefit of anyone else's experiences in this matter before I start.
P.
with recent 2.3.14
metric auth_failed { filter=event=auth_request_finished and (not success=yes) group_by = service }
service stats { inet_listener http { port = 9900 } }
should produce
auth_failed_pop3 etc. which you can collect with prometheus, or you can use
doveadm stats dumpAki
Well I didn't really find what I was looking for around the web, so I just sat down and hacked out an attempt in bash which gives me a starting point. I guess I'll add things when I need them and fix things when it breaks.
Its here if anyone can use it. Any suggestions for improvement welcome. Use at your own risk etc.
https://gist.github.com/plutocrat/8a2033923e14670dd13611fc0b51fc0f
To add to logwatch you
- put it in the scripts dir as eg /etc/logwatch/scripts/services/dovecot_summary
- add a config file in /etc/logwatch/conf/services/dovecot_summary.conf
- Add a line in /etc/logwatch/conf/logwatch.conf like "Service = dovecot_summary"
P.
I run logwatch on a few servers and the script on there which parses the dovecot logs provides waaaaaay too much detail, and contains a bunch of stuff I'm not interested in. There's not much point in having a log summary if its 100Kb long! The one for exim is also terrible, but I've replaced that with eximstats, and a bit of bash scripting and have something I can work with. I might at some point look at replacing that too>> Now to my question. I haven't found anything in the dovecot family which compares to eximstats. I was wondering if there possibly is a report tool, or a way of extracting stats from doveadm just for key metrics such as 'number of POP logins in the last 24 hours', number of failed passwords, etc. Things that might be interesting on a logwatch report. Or maybe someone has written a script already that can be used in logwatch in perl, python, bash, whatever. Any pointers, suggestions, or ideas for useful summary metrics to pull out of the logfiles are appreciated.
I suspect I might end up writing one myself anyway, but I'm just keen to see what already exists, get the benefit of anyone else's experiences in this matter before I start.
On 02 Jun 2021, at 02:10, Plutocrat plutocrat@gmail.com wrote:
Its here if anyone can use it. Any suggestions for improvement welcome. Use at your own risk etc.
https://gist.github.com/plutocrat/8a2033923e14670dd13611fc0b51fc0f
This looks good. My only comment is the script doesn't account for compressed logs.
LOG1="/var/log/dovecot.log.0.bz2" LOG2="/var/log/dovecot.log" bzcat $LOG1 | sed "0,/^$STARTTIME/d" > $TEMPFILE cat $LOG2 >> $TEMPFILE
Well, another comment, I think you meant these tow lines to read like this:
STARTDATE=$(head -n 1 $TEMPFILE | awk '{print $1 " " $2 " " $3 }') ENDDATE=$(tail -n 1 $TEMPFILE | awk '{print $1 " " $2 " " $3 }')
Though I don't think the awk is necessary, the first 15 characters of the first and last lines contain the info you want to display.
Of course, it would begetter to handle this all transparently, but … meh.
-- "Are you pondering what I'm pondering?" "I think so, Brain! But do I have what it take to be the 'Lord of the Dance'?"
On 2.6.2021 11.41, @lbutlr wrote:
On 02 Jun 2021, at 02:10, Plutocrat plutocrat@gmail.com wrote:
Its here if anyone can use it. Any suggestions for improvement welcome. Use at your own risk etc.
https://gist.github.com/plutocrat/8a2033923e14670dd13611fc0b51fc0f This looks good. My only comment is the script doesn't account for compressed logs.
LOG1="/var/log/dovecot.log.0.bz2" LOG2="/var/log/dovecot.log" bzcat $LOG1 | sed "0,/^$STARTTIME/d" > $TEMPFILE cat $LOG2 >> $TEMPFILE
Well, another comment, I think you meant these tow lines to read like this:
STARTDATE=$(head -n 1 $TEMPFILE | awk '{print $1 " " $2 " " $3 }') ENDDATE=$(tail -n 1 $TEMPFILE | awk '{print $1 " " $2 " " $3 }')
Though I don't think the awk is necessary, the first 15 characters of the first and last lines contain the info you want to display.
Of course, it would begetter to handle this all transparently, but … meh.
In case one happens to use dovecot v2.3.12+ you can do some of this with the new metrics (some of it works before that too, but 2.3.12+ all of below should work):
## puts events json formatted into dovecot log for easier parsing event_exporter log { transport = log format = json format_args = time-unix }
metric auth_failures { filter=event=auth_request_finished AND (NOT success=yes) exporter = log }
metric auth_successes { filter=event=auth_request_finished AND success=yes exporter = log }
## generates logins_service (as total of succeeded) ## generates logins_service_pop3, logins_service_imap etc. (for the first level) ## also generates logins_service_pop3_110, login_service_pop3_990 etc. (as the last level) metric logins_service { filter=event=auth_request_finished AND success=yes group_by=service port }
Sure, it was meant to be a quick and dirty solution for my particular needs, and I'm sure there are many things that could be done better, neater, tidier. But I just thought I'd stick it up there in case it helped anyone.
For handling both compressed and uncompressed files, I believe you can use 'zcat -f', which might be easier. In my particular case, I have weekly log rotation, so I only needed the two named ones to guarantee 24 hours of logs.
Fair commend about the TEMPFILE. Must have pasted an older version to the one I eventually used on the server. Will correct it.
Aki, I did look at the dovecot stats module, but after spending a few hours without success, I decided to take the path of least resistance and just hack up a script which probably took me an hour. Anyway thanks for your help and patience on that. I'd probably persevere with that approach if I was in corporate mode, and especially if I needed historical / comparative data.
P.
On 02/06/2021 16.41, @lbutlr wrote:
On 02 Jun 2021, at 02:10, Plutocrat plutocrat@gmail.com wrote:
Its here if anyone can use it. Any suggestions for improvement welcome. Use at your own risk etc.
https://gist.github.com/plutocrat/8a2033923e14670dd13611fc0b51fc0f
This looks good. My only comment is the script doesn't account for compressed logs.
LOG1="/var/log/dovecot.log.0.bz2" LOG2="/var/log/dovecot.log" bzcat $LOG1 | sed "0,/^$STARTTIME/d" > $TEMPFILE cat $LOG2 >> $TEMPFILE
Well, another comment, I think you meant these tow lines to read like this:
STARTDATE=$(head -n 1 $TEMPFILE | awk '{print $1 " " $2 " " $3 }') ENDDATE=$(tail -n 1 $TEMPFILE | awk '{print $1 " " $2 " " $3 }')
Though I don't think the awk is necessary, the first 15 characters of the first and last lines contain the info you want to display.
Of course, it would begetter to handle this all transparently, but … meh.
participants (3)
-
＠lbutlr
-
Aki Tuomi
-
Plutocrat