- dovecot - dovecot.org

Massive LMTP Problems with dovecot
by Ralf Hildebrandt 17 Oct '16

17 Oct '16

Currently I'm having massive problems with LMTP delivery into dovcot. dovecot/lmtp processes are piling up, eas using considerable amounts of CPU: # ps auxwww|fgrep dove root 20537 0.0 0.0 18124 1196 ? Ss 15:18 0:00 /usr/sbin/dovecot -c /etc/dovecot/dovecot.conf dovecot 20541 0.0 0.0 9620 1084 ? S 15:18 0:00 dovecot/anvil root 20542 0.0 0.0 9752 1264 ? S 15:18 0:00 dovecot/log root 20544 0.0 0.0 21168 2276 ? S 15:18 0:00 dovecot/config copymail 20580 72.8 0.0 39556 7036 ? R 15:18 2:00 dovecot/lmtp dovecot 20582 0.0 0.0 18568 1756 ? S 15:18 0:00 dovecot/auth copymail 20597 77.2 0.0 35688 5136 ? R 15:18 2:06 dovecot/lmtp copymail 20598 39.3 0.0 38060 5596 ? R 15:18 1:04 dovecot/lmtp copymail 20613 62.3 0.0 38036 5600 ? R 15:18 1:41 dovecot/lmtp copymail 20619 56.4 0.0 37732 7448 ? R 15:18 1:31 dovecot/lmtp copymail 20620 75.9 0.0 35872 5336 ? R 15:18 2:03 dovecot/lmtp copymail 20627 37.8 0.0 36480 5892 ? R 15:18 1:01 dovecot/lmtp copymail 20838 60.5 0.0 35640 5036 ? R 15:19 0:59 dovecot/lmtp copymail 20840 66.3 0.0 35920 5296 ? R 15:19 1:04 dovecot/lmtp copymail 20841 66.0 0.0 37456 6852 ? R 15:19 1:04 dovecot/lmtp copymail 20842 64.5 0.0 36424 5808 ? R 15:19 1:02 dovecot/lmtp copymail 20843 67.6 0.0 39612 7064 ? R 15:19 1:05 dovecot/lmtp doveadm stop won't stop these, I have to use kill -9 on them. I already tried disabling fts (entirely), still things won't speed up. I can't strace: # strace -p 20841 Process 20841 attached (and that's it) # dpkg -l|grep dovecot ii dovecot-core 2:2.2.25-1~auto+57 amd64 secure POP3/IMAP server - core files ii dovecot-imapd 2:2.2.25-1~auto+57 amd64 secure POP3/IMAP server - IMAP daemon ii dovecot-lmtpd 2:2.2.25-1~auto+57 amd64 secure POP3/IMAP server - LMTP server ii dovecot-lucene 2:2.2.25-1~auto+57 amd64 secure POP3/IMAP server - Lucene support ii dovecot-sieve 2:2.2.25-1~auto+57 amd64 secure POP3/IMAP server - Sieve filters support I also tried deleting the mdboxes, that also didn't change anything. Ideas? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

2 1

Re: First steps in Dovecot; IMAP not working
by Marnaud 17 Oct '16

17 Oct '16

Sorry, my previous message got mangled. I'm re-writing it, quoting manually. I apologize for the traffic. >(Sorry I read this list in digest form so frequently I'm half a step behind.) No problem. >No, it's quite explicit. User "webuser" has uid/gid = 1001(webuser)/1000(ftpusers). Your mail spool has permission uid/gid = root(0)/mail(8), neither of which allows webuser to write to this mail spool to creates its own mail folder. You're right (I don't have enough Unix habits, it seems...). I couldn't change this user (it must be in the ftpusers group for other purposes), so I tried adding another user for testing mail. "mailtest", the new user, is in group mail(8). In addition, I've added write permission for "others" to /var/mail. Now, I'm trying to send a message to "mailtest" from another, working, e-mail account and nothing happens. This time, "doveadm log errors" is empty. In short, I don't get any error but no mail either. >Aki Tuomi <aki.tuomi(a)dovecot.fi> replies with several solutions: >>In your configuration, dovecot uses whatever user/group returned by PAM. Since the webuser has never logged in, it has no directory under /var/mail. If you want, you can >>a) override mail_uid and mail_gid in userdb/passdb >>b) pre-create /var/mail/webuser and chown it to webuser:ftpusers >>c) you can let ftpusers write to /var/mail. Ok, I thought I had to do all of them (and didn't understand step a)). So I've done step c) by allowing everyone write access. >Or if you dynamically/frequently onboard mail accounts, and users cannot arbitrarily write into this directory, you can "chmod 1777 /var/mail/" and let dovecot auto-create it (might also want to set "lda_mailbox_autocreate = yes". I've done it right now; same problem. Since "doveadm log errors" returns an empty result, where should I look for the problem? Thank you.

1 0

Re: First steps in Dovecot; IMAP not working
by Joseph Tam 17 Oct '16

17 Oct '16

(Sorry I read this list in digest form so frequently I'm half a step behind.) > For sake of ?security?, I chose to not allow plaintext communication > (being new to this, I think being strict is a good choice). I?ve tried > with the openssl option and it successfully logged in. Yes, you've included some more log entries, which makes the problem clearer, as it usually does. > Oct 13 05:56:28 imap(webuser): Error: open(/var/mail/webuser) failed: > Permission denied (euid=1001(webuser) egid=1000(ftpusers) missing +w perm: > /var/mail, we're not in group 8(mail), dir owned by 0:8 mode=0775) > ... > I checked, using ls -l /var, and I get this: > drwxrwsr-x 2 root mail 4096 Apr 27 11:27 mail > so the group looks to be correctly set to 'mail', despite what the log says, > right? No, it's quite explicit. User "webuser" has uid/gid = 1001(webuser)/1000(ftpusers). Your mail spool has permission uid/gid = root(0)/mail(8), neither of which allows webuser to write to this mail spool to creates its own mail folder. Aki Tuomi <aki.tuomi(a)dovecot.fi> replies with several solutions: > In your configuration, dovecot uses whatever user/group returned by > PAM. Since the webuser has never logged in, it has no directory under > /var/mail. If you want, you can > > a) override mail_uid and mail_gid in userdb/passdb > b) pre-create /var/mail/webuser and chown it to webuser:ftpusers > c) you can let ftpusers write to /var/mail. Or if you dynamically/frequently onboard mail accounts, and users cannot arbitrarily write into this directory, you can "chmod 1777 /var/mail/" and let dovecot auto-create it (might also want to set "lda_mailbox_autocreate = yes". Joseph Tam <jtam.home(a)gmail.com>

2 1

Re: First steps in Dovecot; IMAP not working
by Aki Tuomi 17 Oct '16

17 Oct '16

On 13.10.2016 12:42, Marnaud wrote: > > "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: > >> doveconf -n shows what's there. if you have ssl=no somewhere else in the >> config after you set it to required, it gets overwritten. >> >> Aki > > Thanks, Aki. > It means I have to open each conf file (e.g. using nano) and search > for ssl=no; I'm right or there's a specific file to check? I see you replied to me only, please keep your replies on-list. Try grep -r ssl.*no /etc/dovecot Aki

4 13

Latest HG Changes (fac92b5) affect Sieve-Plugin/LMTP
by Thomas Leuxner 15 Oct '16

15 Oct '16

# 2.2.devel (c73322f): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.devel (fac92b5) # OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.6 ==> /var/log/dovecot/dovecot.log <== Oct 15 09:50:15 nihlus dovecot: lmtp(11447): Connect from local Oct 15 09:50:15 nihlus dovecot: lmtp(tlx(a)leuxner.net): Panic: file lda-sieve-plugin.c: line 447 (lda_sieve_execute_scripts): assertion failed: (script != NULL) Oct 15 09:50:15 nihlus dovecot: lmtp(tlx(a)leuxner.net): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x938ae) [0x7fd161fc18ae] -> /usr/lib/dovecot/libdovecot.so.0(+0x9399c) [0x7fd161fc199c] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fd161f5b6de] -> /usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x3af8) [0x7fd15fdf8af8] -> /usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x49) [0x7fd16258bb39] -> dovecot/lmtp [DATA tlx(a)leuxner.net](+0x724e) [0x7fd1629bc24e] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x4c) [0x7fd161fd5e4c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10a) [0x7fd161fd730a] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7fd161fd5ed5] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7fd161fd6078] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fd161f61be3] -> dovecot/lmtp [DATA tlx(a)leuxner.net](main+0x1a2) [0x7fd1629ba382] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fd161ba4b45] -> dovecot/lmtp [DATA tlx(a)leuxner.net](+0x5430) [0x7fd1629ba430] Oct 15 09:50:15 nihlus dovecot: lmtp(tlx(a)leuxner.net): Fatal: master: service(lmtp): child 11447 killed with signal 6 (core not dumped)

3 9

Re: First steps in Dovecot; IMAP not working
by Joseph Tam 14 Oct '16

14 Oct '16

Moi <anic297(a)mac.com> wrote: > I've made some more tests and I still can't receive mails; sending them > still works. I don't receive any error message, just the mails that are > supposed to be received won't come. > In the mail logs, I find only this relevant line: > dovecot: imap-login: Aborted login (no auth attempts in 2 secs): user=<> Did you post doveconf -n (I didn't catch the head of this thread)? That would be step 0. A good first step is to test whether you have basic authentication working (to separate out if you have a server or client issue). I assume you allow plaintext communication, but if not, substitute telnet with "openssl sclient -connect your-server:993": C: # telnet your-server 143 S: * OK [CAPABILITY ... C: x1 login testuser theirpassword If you get an OK response to this, it may be a client issue (check settings on client). If you get an error or failure, look inward: check logs and config. Joseph Tam <jtam.home(a)gmail.com>

2 1

Maildir Expunged GUID mismatch for UID
by Steven Xu 14 Oct '16

14 Oct '16

Dovecot version:2.2.25 Since we used to keep our email files on widows server, I made the following changes in maildir-storage.h #define MAILDIR_EXTRA_SEP ',' #define MAILDIR_INFO_SEP_S ":" to "+". Everything seems working except EXPUNG, The dovecot log is flooded by messages like following: imap(xxxxx): Error: Mailbox INBOX: Expunged GUID mismatch for UID 7039 Then I read the source code, and found the following lines in maildir-sync-index.c T_BEGIN { guid = maildir_uidlist_lookup_ext(ctx->mbox->uidlist, uid, MAILDIR_UIDLIST_REC_EXT_GUID); if (guid == NULL) guid = t_strcut(filename, ':'); mail_generate_guid_128_hash(guid, guid_128); } T_END; I have to change the code to guid = t_strcut(filename, '+'); So, should MAILDIR_EXTRA_SEP be used here instead of ':'? Thanks, Steven

2 1

update dovecot-acl in all subfolders in a public folder
by Lukas Henrich 14 Oct '16

14 Oct '16

Hello everyone, right now I have problem and can't find a proper solution. But first: dovecot-version: 2.2.13 Now to my problem: A client of mine uses a public folder called "groups". In this Folder are several subfolders like "Archive", "projects", "sales" und so on. Unfortunately this client (and his employees) created thousands of subfolders within these folders. The folder "groups" is stored in /data/vmail/domain.com/. So, if i type "tree -a -L 1 /data/vmail/domain.com/groups" I get the following outpout: /data/vmail/domain.com/groups/ ??? .Archive ??? .Archive.subfolder1 ??? .Archive.subfolder2 ??? .Archive.subfolder2.subfolder3 ??? .Archive.subfolder2.subfolder4 (....) ??? .projects ??? .projects.subfolder1 ??? .projects.subfolder2 ??? .projects.subfolder2.subfolder3 ??? .projects.subfolder2.subfolder4 (...) In this groups-folder are at the moment more than 3400 folders! Now to the permissions: When the folders "Archive", "projects" and so on had been created every folder got a dovecot-acl for the permission for each user, e.g.: user=user1 kxeilprwts user=user2 kxeilprwts This worked finde, as the dovecot-acl got copied from the parent folder when the employees created new subfolders. Now to my problem I'm facing right now: This client got 2 new employees. So how can I edit all these dovecot-acl files in all subfolders where these 2 new employess schould get access to? Thank you in advance! Yours sincerely, Lukas Henrich Furthermore here is the output of dovecot -n: # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 4.4.6-1-pve x86_64 Debian 8.4 auth_username_format = %Ln disable_plaintext_auth = no lda_mailbox_autocreate = yes mail_home = /data/vmail/domain.com/%Ln mail_location = maildir:~ mail_plugins = " acl" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { hidden = no ignore_on_failure = no inbox = no list = children location = maildir:/data/vmail/domain.com/%%n:INDEXPVT=/data/vmail/domain.com/%n/share… prefix = shared/%%n/ separator = / subscriptions = yes type = shared } namespace { hidden = no ignore_on_failure = no inbox = no list = yes location = maildir:/data/vmail/domain.com/groups:INDEXPVT=/data/vmail/domain.com/%n/gr… prefix = groups/ separator = / subscriptions = yes type = public } namespace inbox { inbox = yes location = mailbox Archiv { special_use = \Archive } mailbox Archive { auto = subscribe special_use = \Archive } mailbox Archives { special_use = \Archive } mailbox "Deleted Messages" { special_use = \Trash } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Entwürfe { special_use = \Drafts } mailbox "Gelöschte Elemente" { special_use = \Trash } mailbox "Gelöschte Objekte" { special_use = \Trash } mailbox Gesendet { special_use = \Sent } mailbox "Gesendete Elemente" { special_use = \Sent } mailbox "Gesendete Objekte" { special_use = \Sent } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Papierkorb { special_use = \Trash } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX/ separator = / subscriptions = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } passdb { args = scheme=CRYPT username_format=%Ln /etc/dovecot/users driver = passwd-file } plugin { acl = vfile acl_shared_dict = file:/var/lib/dovecot/db/shared-mailboxes.db sieve = ~/dovecot.sieve sieve_dir = ~/sieve } postmaster_address = admin(a)domain.com protocols = " imap lmtp sieve sieve" service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } unix_listener auth-userdb { group = vmail user = vmail } } service imap-login { inet_listener imap { port = 143 } } service lmtp { inet_listener lmtp { address = 127.0.0.1 port = 24 } unix_listener /var/spool/postfix/private/lmtp-dovecot { group = postfix user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } } ssl_cert = </etc/ssl/certs/kmux-dovecot.intern.domain.com.crt ssl_key = </etc/ssl/private/kmux-dovecot.intern.domain.com.key userdb { args = /etc/dovecot/dovecot-ldap.conf.ext default_fields = uid=vmail gid=vmail home=/data/vmail/domain.com/%n driver = ldap } userdb { args = username_format=%Ln /etc/dovecot/users driver = passwd-file } protocol lmtp { mail_plugins = " acl sieve" } protocol lda { mail_plugins = " acl sieve" } protocol imap { mail_plugins = " acl imap_acl" }

1 0

fix SIS attachment errors
by Webert de Souza Lima 13 Oct '16

13 Oct '16

Hi, I've sent some e-mails about this before but since there was no answers I'll write it differently, with different information. I'm using SIS (Single Instance Attachment Storage). For any reason that is not relevant now, many attachments are missing and the messages can't be fetched: Error: read(attachments-connector(zlib(/dovecot/mdbox/bar.example/foo/storage/m.1))) failed: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: open(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36) failed: No such file or directory in this specific case, the /dovecot/attach/bar.example/23/ae/ director doesn't exist. In other cases, just one file is missing so I would assume the hardlink could be recreated and it would work. If I create the missing file (with touch or whatever), I get the following errors: Error: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is smaller than expected (0 < 483065) Error: read(attachments-connector(zlib(/dovecot/mdbox/bar.example/foo/storage/m.1))) failed: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is smaller than expected (0 < 483065) Error: fetch(body) failed for box=INBOX uid=15: BUG: Unknown internal error If I try to fill the file with the amount of bytes it complains about with the following command: $ dd if=/dev/zero of=/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36 bs=1 count=483065 then I get the following error: Error: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is larger than expected (483928 > 483065, eof=0) Error: read(attachments-connector(zlib(/srv/dovecot/mdbox/bar.example/foo/storage/m.1))) failed: read(//dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is larger than expected (483928 > 483065, eof=0) Error: fetch(body) failed for box=INBOX uid=15: BUG: Unknown internal error Based on this I have a few questions: 1. Is there a way, or a tool to scan all mailboxes to get all the messages that have compromised attachments? 2. is there a way to "fix" the missing files (even if creating fake files or removing the attachments information from the messages) 3. What I need is to migrate these boxes using doveadm backup/sync, which fails when these errors occur. Is is possible to ignore them or is there another tool that would do it? Thank you. Webert Lima Belo Horizonte, Brasil

1 1

First steps in Dovecot; IMAP not working
by Marnaud 13 Oct '16

13 Oct '16

Hello, I'm new in Dovecot and am having troubles making it working. I'm trying using Outlook and Apple's Mail as the mail clients. Outlook says it can't establish a secured connection to the server (for the IMAP protocol). I'm guessing sending e-mails works but I can't check. This is my current configuration (using dovecot -n): # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-042stab116.1 x86_64 Debian 8.6 mail_location = mbox:~/mail:INBOX=/var/mail/%u namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } passdb { driver = pam } protocols = " imap" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } service imap-login { inet_listener imaps { port = 993 ssl = yes } } ssl = no ssl_cert = </etc/ssl/certs/mail.crt ssl_key = </etc/ssl/private/mail.key userdb { driver = passwd } userdb { driver = passwd } I find abnormal I'm seeing "ssl = no" in this configuration despite the fact that I have "ssl = required" in the /etc/dovecot/conf.d/10-ssl.conf file, but I'm new to this... I have looked around the web; finally, I'm asking here, hoping it's the correct place to ask. Arnaud

2 1