Dovecot 2.3.18 and OpenMetrics-Prometheus problem

[Christian Rößner]

Hi,

I have defined several metrics in Dovecot and activated the metrics service. After the I tried to include this in Prometheues, but I get strange errors. I gave up now after several hours in the hope that this might be a bug. So I ask for help here:

Distribution: Rocky 8
Official Dovecot Repo
Version: 2.3.18

Config for metrics, service and events:
-----------------------------------------------------------------
metric auth_success {
  exporter = log
  filter = event=auth_request_finished AND success=yes
}

metric auth_failures {
  exporter = log
  filter = event=auth_request_finished AND NOT success=yes
}

metric auth_client_userdb {
  filter = event=auth_client_userdb_lookup_finished
}

metric auth_master {
  exporter = log
  filter = event=auth_master_client_login_finished
}

metric client_connected {
  filter = event=client_connection_connected
}

metric client_disconnected {
  filter = event=client_connection_disconnected
  group_by = reason
}

metric imap_command {
  filter = event=imap_command_finished
  group_by = cmd_name tagged_reply_state
}

metric lmtp_command {
  filter = event=smtp_server_command_finished AND protocol=lmtp
  group_by = cmd_name status_code duration:exponential:1:5:10
}

metric imap_hibernate {
  filter = event=imap_client_hibernated
  group_by = error
}

metric imap_unhibernate {
  filter = event=imap_client_hibernated
  group_by = reason error
}

metric mail_delivery {
  filter = event=mail_delivery_finished
  group_by = duration:exponential:1:5:10
}

metric sql_query {
  filter = event=sql_query_finished
  group_by = error_code
}

metric sieve_actions {
  filter = event=sieve_action_finished
  group_by = action_name error
}

metric managesieve {
  filter = event=managesieve_command_finished
  group_by = script_name
}

metric dict_lookups {
  filter = event=dict_lookup_finished
  group_by = driver error
}

service stats {
  inet_listener http {
    port = 9900
  }
}

event_exporter log {
  format = json
  format_args = time-rfc3339
  transport = log
}
-----------------------------------------------------------------

Shortened curl output:
-----------------------------------------------------------------
# HELP process_start_time_seconds Timestamp of service start
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1645743604
# HELP dovecot_build Dovecot build information
# TYPE dovecot_build info
dovecot_build_info{version="2.3.18",revision="9dd8408c18"} 1
# HELP dovecot_auth_success Total number of all events of this kind
# TYPE dovecot_auth_success counter
dovecot_auth_success_total 1669
# HELP dovecot_auth_success_duration_seconds Total duration of all events of this kind
# TYPE dovecot_auth_success_duration_seconds counter
dovecot_auth_success_duration_seconds_total 1.317570
# HELP dovecot_auth_failures Total number of all events of this kind
# TYPE dovecot_auth_failures counter
dovecot_auth_failures_total 1
# HELP dovecot_auth_failures_duration_seconds Total duration of all events of this kind
# TYPE dovecot_auth_failures_duration_seconds counter
...
# EOF
-----------------------------------------------------------------

Prometheus yml config snippet:
-----------------------------------------------------------------
global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: dovecot_exporter
    static_configs:
      - targets:
        - 172.30.10.1:9900
-----------------------------------------------------------------

Prometheus (Docker stable latest) with debug level show these lines:
-----------------------------------------------------------------
prometheus_1     | ts=2022-02-25T10:22:00.840Z caller=scrape.go:1307 level=debug component="scrape manager" scrape_pool=dovecot_exporter target=http://172.30.10.1:9900/metrics msg="Append failed" err="expected label name or left brace, got \"INVALID\""
-----------------------------------------------------------------

Therefor the service is considered as being down.

I really have no ide, what I can do here.

Thanks for any help in advance

Christian Rößner
-- 
Rößner-Network-Solutions
Zertifizierter ITSiBe / CISO
Karl-Bröger-Str. 10, 36304 Alsfeld
Fax: +49 6631 78823409, Mobil: +49 171 9905345
USt-IdNr.: DE225643613, https://roessner.website
PGP fingerprint: 658D 1342 B762 F484 2DDF 1E88 38A5 4346 D727 94E5