- dovecot - dovecot.org

Re: dovecot/config processes open, and consuming all memory
by Joseph Tam 17 Aug '22

17 Aug '22

From: Aki Tuomi <aki.tuomi(a)open-xchange.com> > The *default* configuration for service config is usually just fine. > Is there some reason you decided to modify it in first place? Are you asking me, or the OP? I guess the blunt answer is ignorance on my part. However, I pointed out that the docs about this setting is somewhat misleading -- it's used to limit any potential memory leaks for long-lived processes by ensuring it terminates periodically. I read this and thought to myself "this is a good thing to do", without realizing that it would have the opposite effect as lingering clients could delay service termination indefinitely. The side effect is many new processes are spawned to handle new clients, and eventually the maximum process limit is reached, and chaos ensues. So for services like imap-login and others that can have lingering clients, the only sensible values for service_limit is {0,1}. If you set service_limit>1, the asymptotic behaviour is like service_limit=1, and process_limit would have to be adjusted accordingly. The docs can explain this rather wordy and subtle explanation of service_limit, or service_limit can be constrained to values {0,1} so that others don't blunder along the same path I did. Joseph Tam <jtam.home(a)gmail.com>

2 1

Re: dovecot/config processes open, and consuming all memory
by Joseph Tam 16 Aug '22

16 Aug '22

> I'm having strange behavior in dovecot 2.3.16. > It's opening dozens of dovecot/config process and consuming all server > memory. Normally each process consumes between 700Mb and 1Gb of ram. > > Would anyone have an idea about this? > > service config { > vsz_limit = 2048M > idle_kill = 60s > service_count = 1024 > } Not sure it's related, but if you have service_count not 0 or 1, there is a strange interaction with other limits that could cause processes to hang around. My description of problem https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html Your situation is slgihtly different (service not imap_login, and idle_kill timeout should reduce lingering processes that caused my problem), but try setting service_limit to either 0 or 1 and see if your problem goes away, or gets worse. You can also see how many file descriptors are being held by the config process, and see the behaviour over time (e.g. monitor /proc/{pid}/fd/*); maybe that will give you a clue as to what the config process is doing. Joseph Tam <jtam.home(a)gmail.com>

3 5

Slow IMAP SORT
by Ewald Dieterich 16 Aug '22

16 Aug '22

The following IMAP SORT with 2 sort criteria is very slow on an mdbox mailbox with lots of mails (more than 10,000) and an empty cache: . SORT (ARRIVAL DATE) UTF-8 ALL Each mail is opened when the SORT is executed, which is slow, given the number of mails in the mailbox. Debug: Mailbox INBOX: UID 1003: Opened mail because: header stream <repeated for each mail> Repeating the SORT opens all mails again and again: Debug: Mailbox INBOX: UID 1003: Opened mail because: header stream <repeated for each mail> If I just sort by ARRIVAL, the SORT is fast and I don't see any "Opened mail because" messages in the debug log. . SORT (ARRIVAL) UTF-8 ALL If I just sort by DATE, the first SORT is slow. . SORT (DATE) UTF-8 ALL The debug log looks like this: Debug: Mailbox INBOX: UID 1003: Opened mail because: header Date (Mail has other cached fields, reset_id=1633683216) <repeated for each mail> This fills the cache and if I sort by DATE again, the SORT is fast and I don't get any more "Opened mail because" messages in the debug log. If I now repeat "SORT (ARRIVAL DATE)", the SORT is fast, the mails are not opened. If I clear the cache, "SORT (ARRIVAL DATE)" goes back to being slow, not filling the cache. This doesn't seem right. "SORT (ARRIVAL DATE)" should also fill the cache to execute repeated SORTS faster, right?

3 2

dovecot/config processes open, and consuming all memory
by filipe＠digirati.com.br 15 Aug '22

15 Aug '22

I'm having strange behavior in dovecot 2.3.16. It's opening dozens of dovecot/config process and consuming all server memory. Normally each process consumes between 700Mb and 1Gb of ram. Would anyone have an idea about this? service config { vsz_limit = 2048M idle_kill = 60s service_count = 1024 } pstree systemd─┬─ModemManager───2*[{ModemManager}] ├─agetty ├─cron ├─dbus-daemon ├─dovecot─┬─anvil │ ├─6*[auth] │ ├─46*[config] │ ├─1212*[imap] │ ├─155*[imap-login] │ ├─12*[lmtp] │ ├─38*[log] │ ├─10*[managesieve] │ ├─19*[pop3] │ ├─3*[pop3-login] │ └─18*[stats] root 45831 0.0 1.1 774688 752732 ? S 09:31 0:31 dovecot/config root 388792 0.0 1.1 775060 753276 ? S 14:00 0:15 dovecot/config root 510685 0.0 1.1 775384 753604 ? S 15:06 0:20 dovecot/config root 675638 0.0 1.1 775348 753620 ? S 16:56 0:15 dovecot/config root 795375 0.0 1.1 775460 753516 ? S 18:03 0:07 dovecot/config root 798754 0.2 1.1 775592 753712 ? S 18:05 0:30 dovecot/config root 1082696 0.2 1.1 774892 753216 ? S 21:10 0:07 dovecot/config root 1098433 0.4 1.1 774924 753244 ? S 21:33 0:07 dovecot/config root 1109255 0.9 1.1 774924 753344 ? S 21:50 0:07 dovecot/config root 1112976 2.0 1.1 774956 753528 ? S 21:57 0:07 dovecot/config root 1114137 3.0 1.1 775028 753308 ? S 21:59 0:07 dovecot/config root 1115382 5.4 1.1 774924 753496 ? S 22:01 0:06 dovecot/config root 1883627 0.0 1.1 759120 728832 ? S Aug11 0:07 dovecot/config root 1889705 0.0 1.8 1251460 1221872 ? S Aug11 0:11 dovecot/config root 1895022 0.0 1.8 1253280 1224284 ? S Aug11 0:11 dovecot/config root 1900690 0.0 1.8 1255684 1227528 ? S Aug11 0:12 dovecot/config root 1905648 0.0 1.8 1257880 1229912 ? S Aug11 0:12 dovecot/config root 1910857 0.0 1.8 1259156 1231552 ? S Aug11 0:12 dovecot/config root 1914332 0.0 1.1 764328 736552 ? S Aug11 0:20 dovecot/config root 2343896 0.0 1.8 1259472 1231516 ? S Aug11 0:12 dovecot/config root 2346351 0.0 1.8 1259472 1231836 ? S Aug11 0:13 dovecot/config root 2348559 0.0 1.1 764704 736440 ? S Aug11 0:14 dovecot/config root 2445701 0.0 1.1 764276 736540 ? S Aug11 0:19 dovecot/config root 2572525 0.0 1.1 764640 736880 ? S Aug11 0:18 dovecot/config root 2734251 0.0 1.1 764776 737696 ? S Aug11 0:08 dovecot/config root 2740980 0.0 1.1 764768 737244 ? S Aug11 0:17 dovecot/config root 2899925 0.0 1.1 764624 737760 ? S Aug11 0:43 dovecot/config root 3517063 0.0 1.1 764984 738004 ? S Aug11 0:09 dovecot/config root 3541465 0.0 1.1 765224 738756 ? S Aug11 0:07 dovecot/config root 3545589 0.0 1.1 766452 740408 ? S Aug11 0:06 dovecot/config root 3549259 0.0 1.1 766796 741048 ? S Aug11 0:07 dovecot/config root 3553902 0.0 1.1 767812 742284 ? S Aug11 0:07 dovecot/config root 3558080 0.0 1.1 768440 743524 ? S Aug11 0:08 dovecot/config root 3562091 0.0 1.1 769224 744424 ? S Aug11 0:07 dovecot/config root 3568721 0.0 1.1 769388 744888 ? S Aug11 0:07 dovecot/config root 3573024 0.0 1.1 770048 745904 ? S Aug11 0:08 dovecot/config root 3578416 0.0 1.1 770836 746736 ? S Aug11 0:06 dovecot/config root 3581765 0.0 1.1 771948 748492 ? S Aug11 0:06 dovecot/config root 3585837 0.0 1.1 772828 749860 ? S Aug11 0:07 dovecot/config root 3590276 0.0 1.1 773560 750520 ? S Aug11 0:07 dovecot/config root 3594640 0.0 1.1 774612 752552 ? S Aug11 0:08 dovecot/config root 3597417 0.0 1.1 774968 753076 ? S Aug11 0:35 dovecot/config root 4116314 0.0 1.9 1280496 1258580 ? S 05:31 0:13 dovecot/config root 4118115 0.0 1.1 774688 752556 ? S 05:33 0:16 dovecot/config

3 4

convert mdbox to maildir
by lutz.niederer＠gmx.net 14 Aug '22

14 Aug '22

Hi! We need to move all users from one (pretty old) installation of dovecot to a new one. The old one uses mdbox for users' mailboxes and maildir for shared/public mailboxes. The new one must be maildir only. I believe that I can just copy the shared/public maildir structure to the new installation without problems. Am I right? Do I have to expect any trouble? (Set owner, permissions etc...) The real problem is that we must not use the running, old dovecot installation. So we are not able to connect to the old server, pull all folders and mails and create a new maildir structure. Currently, we can't do anything against it. What we get are the users' mdbox files. Is there any way to convert mdbox files and structures to maildir directly from filesystem? Or do we have to build a copy of the old machine (dovecot only, or -maybe better- a vm) and then use doveadm backup? Or is it ok to just set up the completely new installation, set mail_location to where the new Maildirs will be, like maildir:~/Maildir and then run something like doveadm backup mdbox:/tmp/$user/mdbox -u $user? Will this transfer all mails and folders or do we have to keep an eye on some specific things? I thank you very much and cheers! -lutzn

9 14

Submission auth gssapi: line too long
by Carlos Mogas da Silva 14 Aug '22

14 Aug '22

Hey guys. Dovecot is complaining about a "Line too long" when the clients send the AUTH command with GSSAPI authentication. Known bug? # dovecot --version 2.3.19.1 (9b53102964) This is right after the EHLO command (which is after STARTTLS): [1]: No more commands pending [1]: Sending replies [1]: No more commands pending [1]: Client sent invalid command: Command line is too long [1]: command [unknown]: Invalid command [1]: command [unknown]: 500 reply: Submitted [1]: command [unknown]: Replied [1]: command [unknown]: Ready to reply [1]: Trigger output [1]: Sending replies [1]: command [unknown]: Next to reply [1]: command [unknown]: Completed [1]: command [unknown]: 500 reply: Sent: 500 5.5.2 Line too long [1]: command [unknown]: Finished [1]: command [unknown]: Destroy [1]: command [unknown]: 500 reply: Destroy [1]: Trigger output

1 0

Aw: Re: convert mdbox to maildir
by lutz.niederer＠gmx.net 14 Aug '22

14 Aug '22

1 0

mdbox: purge only if it's worthwhile
by Sven Strickroth 12 Aug '22

12 Aug '22

Hi, with the mdbox storage one needs to regularly run purge to really clean up deleted mails from the storage. Running purge can be an IO intensive task as the files are append-only and even if in a 10 MB mdbox file only a small mail is deleted all other mails need to be copied to a new file. Therefore, I wonder whether there is the option to apply a threshold before a mdbox file is cleaned up, e.g. only purge an old mail if this affects > 10% of the mdbox filesize or if at least X bytes will be freed in an mdbox file. Is this possible ATM? Otherwise this can be seen as a feature request. Best, Sven PS: From the docs it's unclear to me how the mdbox rotating algorithm works. Is only to the latest mdbox file appended or it is possible that a small mail might be appended to an older mdbox file?

1 0

pre-define or alter list of IMAP keywords
by spi 11 Aug '22

11 Aug '22

I am using Dovecot 2.3.4.1 with sdbox and mostly Thunderbird on Linux and FairEmail on Android as IMAP clients. I use a defined set of IMAP keywords to allow different users to mark and filter their mails based on keywords. That works pretty well but I stumble across an issue from time to time: In TB I can pre-define a list of keywords which the user can choose one from. FairEmail shows all those keywords Dovecot presents - there is nothing like the list in TB. Issue 1: If there are no mails flagged in a folder, Dovecot doesn't present any keywords and therefore FairEmail shows none - the user has to remember and type in the correct keyword. Issue 2: After some time FairEmail shows a huge list of keywords for the INBOX, even keywords not in use for quite some time. To get rid of these keywords I deleted the index files on Dovecot (which with sdbox are not meant to be deleted?). It seems, Dovecot caches the keywords in the index files. Question: 1) Is there a way on Dovecot to define a list of keywords Dovecot presents to the IMAP clients? 2) If not, is there a nicer way to clean the keyword lists from old entries not assigned to any mail in that folder? -- Cheers spi

3 11

Sieve rule for whitelisting domains after spam processing
by Nicolae Iuoras 10 Aug '22

10 Aug '22

Hello, My mail server has a global Sieve rule defined in sieve_before to file messages classified as spam by spamassassin into the Spam folder: require ["regex", "fileinto", "imap4flags"]; if allof (header :regex "X-Spam-Status" "^Yes") { fileinto "Spam"; } This works great, except I'd like the ability to whitelist certain senders or domains whose emails end up in Spam, usually due to a misconfigured SPF DNS TXT record or something like that. I use Roundcube as my Webmail client, so the easiest way to whitelist would be to create a rule for each domain directly in Roundcube using its built-in Sieve integration to move legitimate messages out of the Spam folder. I already have several rules set up to flag and sort non-spam messages, and they're working fine. I therefore tried adding a simple rule to move Spam into the Inbox: # rule:[Not Spam] if allof (header :contains "from" "user(a)good-domain.com") { fileinto "INBOX"; } However, the above rule does not work as I expected. When I receive an email from good-domain.com that's classified as Spam by spamassassin, the global rule sends it to the Spam folder and it is not re-filed into the Inbox. The second rule doesn't seem to do anything. I'm probably doing something simple wrong, but I can't figure out what (I'm a Sieve newb). Any suggestions please? Thanks, Nick

2 1