- dovecot - dovecot.org

Fields not showing in openmetrics statistics
by Rybkin Vladimir 17 Aug '22

17 Aug '22

Hello I'm trying to configure stats with openmetrics Dovecot version: 2.3.19 I make the settings as follows: ``` # cat /home/beget/dovecot/conf/conf.d/90-openmetrics.conf | grep -v -E "^#" service stats { inet_listener http { port = 9900 } } event_exporter log { format = json format_args = time-rfc3339 transport = log } metric imap_commands { exporter = log filter = event=* } metric imap_command { filter = event=imap_command_finished fields = bytes_in bytes_out group_by = cmd_name tagged_reply_state } ``` The problem is that no matter what configuration I do, the fields parameter is not displayed in openmetrics ``` # curl http://192.168.20.252:9900/metrics | grep -v -E "^#\ " | grep imap_command % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4891 100 4891 0 0 285k 0 --:--:-- --:--:-- --:--:-- 341k dovecot_imap_commands_total 388 dovecot_imap_commands_duration_seconds_total 24.225760 dovecot_imap_command_total{cmd_name="NAMESPACE"} 1 dovecot_imap_command_total{cmd_name="NAMESPACE",tagged_reply_state="OK"} 1 dovecot_imap_command_total{cmd_name="ID"} 4 dovecot_imap_command_total{cmd_name="ID",tagged_reply_state="OK"} 4 dovecot_imap_command_total{cmd_name="ENABLE"} 4 dovecot_imap_command_total{cmd_name="ENABLE",tagged_reply_state="OK"} 4 dovecot_imap_command_total{cmd_name="SELECT"} 3 dovecot_imap_command_total{cmd_name="SELECT",tagged_reply_state="OK"} 3 dovecot_imap_command_total{cmd_name="GETQUOTAROOT"} 3 dovecot_imap_command_total{cmd_name="GETQUOTAROOT",tagged_reply_state="OK"} 3 dovecot_imap_command_total{cmd_name="UID FETCH"} 19 dovecot_imap_command_total{cmd_name="UID FETCH",tagged_reply_state="OK"} 19 dovecot_imap_command_total{cmd_name="LIST"} 2 dovecot_imap_command_total{cmd_name="LIST",tagged_reply_state="OK"} 2 dovecot_imap_command_total{cmd_name="IDLE"} 1 dovecot_imap_command_total{cmd_name="IDLE",tagged_reply_state="OK"} 1 dovecot_imap_command_total{cmd_name="STATUS"} 2 dovecot_imap_command_total{cmd_name="STATUS",tagged_reply_state="OK"} 2 dovecot_imap_command_count 39 dovecot_imap_command_duration_seconds_total{cmd_name="NAMESPACE"} 0.000218 dovecot_imap_command_duration_seconds_total{cmd_name="NAMESPACE",tagged_reply_state="OK"} 0.000218 dovecot_imap_command_duration_seconds_total{cmd_name="ID"} 0.000900 dovecot_imap_command_duration_seconds_total{cmd_name="ID",tagged_reply_state="OK"} 0.000900 dovecot_imap_command_duration_seconds_total{cmd_name="ENABLE"} 0.000332 dovecot_imap_command_duration_seconds_total{cmd_name="ENABLE",tagged_reply_state="OK"} 0.000332 dovecot_imap_command_duration_seconds_total{cmd_name="SELECT"} 0.028734 dovecot_imap_command_duration_seconds_total{cmd_name="SELECT",tagged_reply_state="OK"} 0.028734 dovecot_imap_command_duration_seconds_total{cmd_name="GETQUOTAROOT"} 0.127621 dovecot_imap_command_duration_seconds_total{cmd_name="GETQUOTAROOT",tagged_reply_state="OK"} 0.127621 dovecot_imap_command_duration_seconds_total{cmd_name="UID FETCH"} 0.097117 dovecot_imap_command_duration_seconds_total{cmd_name="UID FETCH",tagged_reply_state="OK"} 0.097117 dovecot_imap_command_duration_seconds_total{cmd_name="LIST"} 0.003738 dovecot_imap_command_duration_seconds_total{cmd_name="LIST",tagged_reply_state="OK"} 0.003738 dovecot_imap_command_duration_seconds_total{cmd_name="IDLE"} 7.397259 dovecot_imap_command_duration_seconds_total{cmd_name="IDLE",tagged_reply_state="OK"} 7.397259 dovecot_imap_command_duration_seconds_total{cmd_name="STATUS"} 0.041169 dovecot_imap_command_duration_seconds_total{cmd_name="STATUS",tagged_reply_state="OK"} 0.041169 dovecot_imap_command_duration_seconds_sum 7.697088 ``` It is displayed through doveadm ``` # altdocker exec $(altdocker ps -aq -f name=dove) doveadm stats dump metric_name field count sum min max avg median stddev %95 imap_commands duration 449 1043812019 20 116763630 2324748.37 2382 12137052.43 608328 imap_command duration 49 145807071 72 46020784 2975654.51 3381 11041798.07 46017747 imap_command bytes_in 49 1419 0 179 28.96 37 28.93 55 imap_command bytes_out 49 1867985 0 424616 38122.14 167 74553.96 158613 imap_command_NAMESPACE duration 1 218 218 218 218.00 218 0.00 218 imap_command_NAMESPACE bytes_in 1 0 0 0 0.00 0 0.00 0 imap_command_NAMESPACE bytes_out 1 86 86 86 86.00 86 0.00 86 imap_command_NAMESPACE_OK duration 1 218 218 218 218.00 218 0.00 218 imap_command_NAMESPACE_OK bytes_in 1 0 0 0 0.00 0 0.00 0 imap_command_NAMESPACE_OK bytes_out 1 86 86 86 86.00 86 0.00 86 imap_command_ID duration 4 900 145 288 225.00 233 59.07 288 imap_command_ID bytes_in 4 168 42 42 42.00 42 0.00 42 imap_command_ID bytes_out 4 275 66 75 68.75 67 3.63 75 imap_command_ID_OK duration 4 900 145 288 225.00 233 59.07 288 imap_command_ID_OK bytes_in 4 168 42 42 42.00 42 0.00 42 imap_command_ID_OK bytes_out 4 275 66 75 68.75 67 3.63 75 imap_command_ENABLE duration 4 332 72 110 83.00 75 15.65 110 imap_command_ENABLE bytes_in 4 44 11 11 11.00 11 0.00 11 imap_command_ENABLE bytes_out 4 147 36 37 36.75 37 0.44 37 imap_command_ENABLE_OK duration 4 332 72 110 83.00 75 15.65 110 imap_command_ENABLE_OK bytes_in 4 44 11 11 11.00 11 0.00 11 imap_command_ENABLE_OK bytes_out 4 147 36 37 36.75 37 0.44 37 imap_command_SELECT duration 3 28734 4662 16200 9578.00 7872 4862.39 16200 imap_command_SELECT bytes_in 3 33 7 14 11.00 12 2.94 14 imap_command_SELECT bytes_out 3 1034 309 416 344.67 309 50.44 416 imap_command_SELECT_OK duration 3 28734 4662 16200 9578.00 7872 4862.39 16200 imap_command_SELECT_OK bytes_in 3 33 7 14 11.00 12 2.94 14 imap_command_SELECT_OK bytes_out 3 1034 309 416 344.67 309 50.44 416 imap_command_GETQUOTAROOT duration 6 171676 12353 74871 28612.67 17219 22308.42 74871 imap_command_GETQUOTAROOT bytes_in 6 66 7 14 11.00 12 2.94 14 imap_command_GETQUOTAROOT bytes_out 6 983 159 167 163.83 165 3.18 167 imap_command_GETQUOTAROOT_OK duration 6 171676 12353 74871 28612.67 17219 22308.42 74871 imap_command_GETQUOTAROOT_OK bytes_in 6 66 7 14 11.00 12 2.94 14 imap_command_GETQUOTAROOT_OK bytes_out 6 983 159 167 163.83 165 3.18 167 imap_command_UID_FETCH duration 20 97998 881 16915 4899.90 3304 4306.81 13912 imap_command_UID_FETCH bytes_in 20 922 13 179 46.10 37 34.76 103 imap_command_UID_FETCH bytes_out 20 1864818 40 424616 93240.90 100838 92111.34 174020 imap_command_UID_FETCH_OK duration 20 97998 881 16915 4899.90 3304 4306.81 13912 imap_command_UID_FETCH_OK bytes_in 20 922 13 179 46.10 37 34.76 103 imap_command_UID_FETCH_OK bytes_out 20 1864818 40 424616 93240.90 100838 92111.34 174020 imap_command_LIST duration 2 3738 1715 2023 1869.00 1869 154.00 2023 imap_command_LIST bytes_in 2 56 10 46 28.00 28 18.00 46 imap_command_LIST bytes_out 2 315 85 230 157.50 157 72.50 230 imap_command_LIST_OK duration 2 3738 1715 2023 1869.00 1869 154.00 2023 imap_command_LIST_OK bytes_in 2 56 10 46 28.00 28 18.00 46 imap_command_LIST_OK bytes_out 2 315 85 230 157.50 157 72.50 230 imap_command_IDLE duration 4 145454703 7397259 46020784 36363675.75 46018330 16723768.54 46020784 imap_command_IDLE bytes_in 4 32 8 8 8.00 8 0.00 8 imap_command_IDLE bytes_out 4 91 10 27 22.75 27 7.36 27 imap_command_IDLE_OK duration 4 145454703 7397259 46020784 36363675.75 46018330 16723768.54 46020784 imap_command_IDLE_OK bytes_in 4 32 8 8 8.00 8 0.00 8 imap_command_IDLE_OK bytes_out 4 91 10 27 22.75 27 7.36 27 imap_command_STATUS duration 2 41169 16692 24477 20584.50 20584 3892.50 24477 imap_command_STATUS bytes_in 2 92 45 47 46.00 46 1.00 47 imap_command_STATUS bytes_out 2 236 117 119 118.00 118 1.00 119 imap_command_STATUS_OK duration 2 41169 16692 24477 20584.50 20584 3892.50 24477 imap_command_STATUS_OK bytes_in 2 92 45 47 46.00 46 1.00 47 imap_command_STATUS_OK bytes_out 2 236 117 119 118.00 118 1.00 119 imap_command_NOOP duration 3 7603 1654 3177 2534.33 2772 644.07 3177 imap_command_NOOP bytes_in 3 6 2 2 2.00 2 0.00 2 imap_command_NOOP bytes_out 3 0 0 0 0.00 0 0.00 0 imap_command_NOOP_OK duration 3 7603 1654 3177 2534.33 2772 644.07 3177 imap_command_NOOP_OK bytes_in 3 6 2 2 2.00 2 0.00 2 imap_command_NOOP_OK bytes_out 3 0 0 0 0.00 0 0.00 0 ``` I can’t understand what I’m doing wrong so that field data is displayed in openmetrics.

1 0

Re: dovecot/config processes open, and consuming all memory
by Joseph Tam 17 Aug '22

17 Aug '22

From: Aki Tuomi <aki.tuomi(a)open-xchange.com> > The *default* configuration for service config is usually just fine. > Is there some reason you decided to modify it in first place? Are you asking me, or the OP? I guess the blunt answer is ignorance on my part. However, I pointed out that the docs about this setting is somewhat misleading -- it's used to limit any potential memory leaks for long-lived processes by ensuring it terminates periodically. I read this and thought to myself "this is a good thing to do", without realizing that it would have the opposite effect as lingering clients could delay service termination indefinitely. The side effect is many new processes are spawned to handle new clients, and eventually the maximum process limit is reached, and chaos ensues. So for services like imap-login and others that can have lingering clients, the only sensible values for service_limit is {0,1}. If you set service_limit>1, the asymptotic behaviour is like service_limit=1, and process_limit would have to be adjusted accordingly. The docs can explain this rather wordy and subtle explanation of service_limit, or service_limit can be constrained to values {0,1} so that others don't blunder along the same path I did. Joseph Tam <jtam.home(a)gmail.com>

2 1

Re: dovecot/config processes open, and consuming all memory
by Joseph Tam 16 Aug '22

16 Aug '22

> I'm having strange behavior in dovecot 2.3.16. > It's opening dozens of dovecot/config process and consuming all server > memory. Normally each process consumes between 700Mb and 1Gb of ram. > > Would anyone have an idea about this? > > service config { > vsz_limit = 2048M > idle_kill = 60s > service_count = 1024 > } Not sure it's related, but if you have service_count not 0 or 1, there is a strange interaction with other limits that could cause processes to hang around. My description of problem https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html Your situation is slgihtly different (service not imap_login, and idle_kill timeout should reduce lingering processes that caused my problem), but try setting service_limit to either 0 or 1 and see if your problem goes away, or gets worse. You can also see how many file descriptors are being held by the config process, and see the behaviour over time (e.g. monitor /proc/{pid}/fd/*); maybe that will give you a clue as to what the config process is doing. Joseph Tam <jtam.home(a)gmail.com>

3 5

Slow IMAP SORT
by Ewald Dieterich 16 Aug '22

16 Aug '22

The following IMAP SORT with 2 sort criteria is very slow on an mdbox mailbox with lots of mails (more than 10,000) and an empty cache: . SORT (ARRIVAL DATE) UTF-8 ALL Each mail is opened when the SORT is executed, which is slow, given the number of mails in the mailbox. Debug: Mailbox INBOX: UID 1003: Opened mail because: header stream <repeated for each mail> Repeating the SORT opens all mails again and again: Debug: Mailbox INBOX: UID 1003: Opened mail because: header stream <repeated for each mail> If I just sort by ARRIVAL, the SORT is fast and I don't see any "Opened mail because" messages in the debug log. . SORT (ARRIVAL) UTF-8 ALL If I just sort by DATE, the first SORT is slow. . SORT (DATE) UTF-8 ALL The debug log looks like this: Debug: Mailbox INBOX: UID 1003: Opened mail because: header Date (Mail has other cached fields, reset_id=1633683216) <repeated for each mail> This fills the cache and if I sort by DATE again, the SORT is fast and I don't get any more "Opened mail because" messages in the debug log. If I now repeat "SORT (ARRIVAL DATE)", the SORT is fast, the mails are not opened. If I clear the cache, "SORT (ARRIVAL DATE)" goes back to being slow, not filling the cache. This doesn't seem right. "SORT (ARRIVAL DATE)" should also fill the cache to execute repeated SORTS faster, right?

3 2

dovecot/config processes open, and consuming all memory
by filipe＠digirati.com.br 15 Aug '22

15 Aug '22

I'm having strange behavior in dovecot 2.3.16. It's opening dozens of dovecot/config process and consuming all server memory. Normally each process consumes between 700Mb and 1Gb of ram. Would anyone have an idea about this? service config { vsz_limit = 2048M idle_kill = 60s service_count = 1024 } pstree systemd─┬─ModemManager───2*[{ModemManager}] ├─agetty ├─cron ├─dbus-daemon ├─dovecot─┬─anvil │ ├─6*[auth] │ ├─46*[config] │ ├─1212*[imap] │ ├─155*[imap-login] │ ├─12*[lmtp] │ ├─38*[log] │ ├─10*[managesieve] │ ├─19*[pop3] │ ├─3*[pop3-login] │ └─18*[stats] root 45831 0.0 1.1 774688 752732 ? S 09:31 0:31 dovecot/config root 388792 0.0 1.1 775060 753276 ? S 14:00 0:15 dovecot/config root 510685 0.0 1.1 775384 753604 ? S 15:06 0:20 dovecot/config root 675638 0.0 1.1 775348 753620 ? S 16:56 0:15 dovecot/config root 795375 0.0 1.1 775460 753516 ? S 18:03 0:07 dovecot/config root 798754 0.2 1.1 775592 753712 ? S 18:05 0:30 dovecot/config root 1082696 0.2 1.1 774892 753216 ? S 21:10 0:07 dovecot/config root 1098433 0.4 1.1 774924 753244 ? S 21:33 0:07 dovecot/config root 1109255 0.9 1.1 774924 753344 ? S 21:50 0:07 dovecot/config root 1112976 2.0 1.1 774956 753528 ? S 21:57 0:07 dovecot/config root 1114137 3.0 1.1 775028 753308 ? S 21:59 0:07 dovecot/config root 1115382 5.4 1.1 774924 753496 ? S 22:01 0:06 dovecot/config root 1883627 0.0 1.1 759120 728832 ? S Aug11 0:07 dovecot/config root 1889705 0.0 1.8 1251460 1221872 ? S Aug11 0:11 dovecot/config root 1895022 0.0 1.8 1253280 1224284 ? S Aug11 0:11 dovecot/config root 1900690 0.0 1.8 1255684 1227528 ? S Aug11 0:12 dovecot/config root 1905648 0.0 1.8 1257880 1229912 ? S Aug11 0:12 dovecot/config root 1910857 0.0 1.8 1259156 1231552 ? S Aug11 0:12 dovecot/config root 1914332 0.0 1.1 764328 736552 ? S Aug11 0:20 dovecot/config root 2343896 0.0 1.8 1259472 1231516 ? S Aug11 0:12 dovecot/config root 2346351 0.0 1.8 1259472 1231836 ? S Aug11 0:13 dovecot/config root 2348559 0.0 1.1 764704 736440 ? S Aug11 0:14 dovecot/config root 2445701 0.0 1.1 764276 736540 ? S Aug11 0:19 dovecot/config root 2572525 0.0 1.1 764640 736880 ? S Aug11 0:18 dovecot/config root 2734251 0.0 1.1 764776 737696 ? S Aug11 0:08 dovecot/config root 2740980 0.0 1.1 764768 737244 ? S Aug11 0:17 dovecot/config root 2899925 0.0 1.1 764624 737760 ? S Aug11 0:43 dovecot/config root 3517063 0.0 1.1 764984 738004 ? S Aug11 0:09 dovecot/config root 3541465 0.0 1.1 765224 738756 ? S Aug11 0:07 dovecot/config root 3545589 0.0 1.1 766452 740408 ? S Aug11 0:06 dovecot/config root 3549259 0.0 1.1 766796 741048 ? S Aug11 0:07 dovecot/config root 3553902 0.0 1.1 767812 742284 ? S Aug11 0:07 dovecot/config root 3558080 0.0 1.1 768440 743524 ? S Aug11 0:08 dovecot/config root 3562091 0.0 1.1 769224 744424 ? S Aug11 0:07 dovecot/config root 3568721 0.0 1.1 769388 744888 ? S Aug11 0:07 dovecot/config root 3573024 0.0 1.1 770048 745904 ? S Aug11 0:08 dovecot/config root 3578416 0.0 1.1 770836 746736 ? S Aug11 0:06 dovecot/config root 3581765 0.0 1.1 771948 748492 ? S Aug11 0:06 dovecot/config root 3585837 0.0 1.1 772828 749860 ? S Aug11 0:07 dovecot/config root 3590276 0.0 1.1 773560 750520 ? S Aug11 0:07 dovecot/config root 3594640 0.0 1.1 774612 752552 ? S Aug11 0:08 dovecot/config root 3597417 0.0 1.1 774968 753076 ? S Aug11 0:35 dovecot/config root 4116314 0.0 1.9 1280496 1258580 ? S 05:31 0:13 dovecot/config root 4118115 0.0 1.1 774688 752556 ? S 05:33 0:16 dovecot/config

3 4

convert mdbox to maildir
by lutz.niederer＠gmx.net 14 Aug '22

14 Aug '22

Hi! We need to move all users from one (pretty old) installation of dovecot to a new one. The old one uses mdbox for users' mailboxes and maildir for shared/public mailboxes. The new one must be maildir only. I believe that I can just copy the shared/public maildir structure to the new installation without problems. Am I right? Do I have to expect any trouble? (Set owner, permissions etc...) The real problem is that we must not use the running, old dovecot installation. So we are not able to connect to the old server, pull all folders and mails and create a new maildir structure. Currently, we can't do anything against it. What we get are the users' mdbox files. Is there any way to convert mdbox files and structures to maildir directly from filesystem? Or do we have to build a copy of the old machine (dovecot only, or -maybe better- a vm) and then use doveadm backup? Or is it ok to just set up the completely new installation, set mail_location to where the new Maildirs will be, like maildir:~/Maildir and then run something like doveadm backup mdbox:/tmp/$user/mdbox -u $user? Will this transfer all mails and folders or do we have to keep an eye on some specific things? I thank you very much and cheers! -lutzn

9 14

Submission auth gssapi: line too long
by Carlos Mogas da Silva 14 Aug '22

14 Aug '22

Hey guys. Dovecot is complaining about a "Line too long" when the clients send the AUTH command with GSSAPI authentication. Known bug? # dovecot --version 2.3.19.1 (9b53102964) This is right after the EHLO command (which is after STARTTLS): [1]: No more commands pending [1]: Sending replies [1]: No more commands pending [1]: Client sent invalid command: Command line is too long [1]: command [unknown]: Invalid command [1]: command [unknown]: 500 reply: Submitted [1]: command [unknown]: Replied [1]: command [unknown]: Ready to reply [1]: Trigger output [1]: Sending replies [1]: command [unknown]: Next to reply [1]: command [unknown]: Completed [1]: command [unknown]: 500 reply: Sent: 500 5.5.2 Line too long [1]: command [unknown]: Finished [1]: command [unknown]: Destroy [1]: command [unknown]: 500 reply: Destroy [1]: Trigger output

1 0

Aw: Re: convert mdbox to maildir
by lutz.niederer＠gmx.net 14 Aug '22

14 Aug '22

1 0

mdbox: purge only if it's worthwhile
by Sven Strickroth 12 Aug '22

12 Aug '22

Hi, with the mdbox storage one needs to regularly run purge to really clean up deleted mails from the storage. Running purge can be an IO intensive task as the files are append-only and even if in a 10 MB mdbox file only a small mail is deleted all other mails need to be copied to a new file. Therefore, I wonder whether there is the option to apply a threshold before a mdbox file is cleaned up, e.g. only purge an old mail if this affects > 10% of the mdbox filesize or if at least X bytes will be freed in an mdbox file. Is this possible ATM? Otherwise this can be seen as a feature request. Best, Sven PS: From the docs it's unclear to me how the mdbox rotating algorithm works. Is only to the latest mdbox file appended or it is possible that a small mail might be appended to an older mdbox file?

1 0

pre-define or alter list of IMAP keywords
by spi 11 Aug '22

11 Aug '22

I am using Dovecot 2.3.4.1 with sdbox and mostly Thunderbird on Linux and FairEmail on Android as IMAP clients. I use a defined set of IMAP keywords to allow different users to mark and filter their mails based on keywords. That works pretty well but I stumble across an issue from time to time: In TB I can pre-define a list of keywords which the user can choose one from. FairEmail shows all those keywords Dovecot presents - there is nothing like the list in TB. Issue 1: If there are no mails flagged in a folder, Dovecot doesn't present any keywords and therefore FairEmail shows none - the user has to remember and type in the correct keyword. Issue 2: After some time FairEmail shows a huge list of keywords for the INBOX, even keywords not in use for quite some time. To get rid of these keywords I deleted the index files on Dovecot (which with sdbox are not meant to be deleted?). It seems, Dovecot caches the keywords in the index files. Question: 1) Is there a way on Dovecot to define a list of keywords Dovecot presents to the IMAP clients? 2) If not, is there a nicer way to clean the keyword lists from old entries not assigned to any mail in that folder? -- Cheers spi

3 11