February 2016 - dovecot

Pigeonhole and duplicate checking
by Dan Ragle 16 Feb '16

16 Feb '16

I spent some time setting up Dovecot LMTP with Pigeonhole this weekend. I got most of the things I wanted to working but have a few issues that I'm hoping that someone might be able to help with. First, I'm running the stock version of the tools from the core repos for CentOS/6, which means Dovecot 2.0.9 and Pigeonhole 0.2.2. Unfortunately, I don't expect to be able to upgrade those anytime soon, so I'm hoping I can find workarounds for my existing versions. My mailserver is sendmail, I linked to Dovecot LMTP by adding: FEATURE(`local_lmtp', `[IPC]', `FILE /var/run/dovecot/lmtp')dnl to my sendmail.mc. In my prior procmail setup, I auto-forwarded all my SPAM E-mail to a separate user on the system with their own account (called spamuser). In the Pigeonhole setup it looks like I have to redirect those messages (i.e., redirect "spamuser(a)mydomain.com";). That works, but I'm having issues with "duplicate" messages getting delivered directly to my inbox instead of getting redirected. IOW, my regular user's .dovecot.sieve has: if header :contains "X-Spam-Flag" "YES" { redirect "spamuser(a)mydomain.com"; stop; } and the spamuser's .dovecot.sieve has: require ["imap4flags"]; setflag "\\seen"; (this system is entirely for personal use, no concerns with a user's Spam being seen via the separate spamuser account). Now, the problem I'm having is that sometimes a message arrives with a duplicate message-ID. Here's an abbreviated example from my maillog: Feb 15 00:30:46 myhost sendmail[26844]: u1F5UeBP026844: from=<ciuehazt(a)autoonecareers.com>, size=613, class=0, nrcpts=1, msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON(a)hverovvxzb.arkansas-email.com>, proto=SMTP, daemon=MTA, relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may be forged) Feb 15 00:31:05 myhost dovecot: lmtp(26856): Connect from local Feb 15 00:31:07 myhost sendmail[26867]: u1F5UvFi026867: from=<ciuehazt(a)autoonecareers.com>, size=613, class=0, nrcpts=1, msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON(a)hverovvxzb.arkansas-email.com>, proto=SMTP, daemon=MTA, relay=117.27.191.61.broad.static.hf.ah.cndata.com [61.191.27.117] (may be forged) Feb 15 00:31:10 myhost dovecot: lmtp(26856, dmr): 7Hj/LoBiwVboaAAACXJZQA: sieve: msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON(a)hverovvxzb.arkansas-email.com>: forwarded to <spamuser(a)mydomain.com> Feb 15 00:31:10 myhost dovecot: lmtp(26856): Disconnect from local: Client quit Feb 15 00:31:10 myhost dovecot: lmtp(26893): Connect from local Feb 15 00:31:10 myhost dovecot: lmtp(26893, spamuser): 9TPnN55iwVYNaQAACXJZQA: sieve: msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON(a)hverovvxzb.arkansas-email.com>: stored mail into mailbox 'INBOX' Feb 15 00:31:11 myhost dovecot: lmtp(26893): Disconnect from local: Client quit Feb 15 00:31:18 myhost dovecot: lmtp(26893): Connect from local Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr): +TPnN55iwVYNaQAACXJZQA: sieve: msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON(a)hverovvxzb.arkansas-email.com>: discarded duplicate forward to <spamuser(a)biblestuph.com> Feb 15 00:31:18 myhost dovecot: lmtp(26893, dmr): +TPnN55iwVYNaQAACXJZQA: sieve: msgid=<68895654496651-HLTYGGRLPDJDVTZRHFIKGAON(a)hverovvxzb.arkansas-email.com>: stored mail into mailbox 'INBOX' So the first message comes in and is redirected properly to my spamuser and stored in the inbox. The second one comes in, pigeonhole sees it's a duplicate, refuses to redirect it, and stores it in my inbox instead. I tried forcing the .dovecot.lda-dupes file to be a symlink to /dev/null just to see if it would work, but unfortunately dovecot just recreates it as a normal file the next time it delivers to that user. Ideally, I'd like to just discard the duplicates. It looks like there is duplicate testing functionality available, but not until later versions of Dovecot/Pigeonhole. Is there anyway I can either just discard the duplicates, or get them to be redirected to the spamuser? Also, some other questions I came up with along the way: Is there any way I can force a message to fileinto a different user's Mailbox? I'm guessing no since it appears that the lmtp drops root privileges before the global sieve script is interpreted, but thought I'd ask anyway. I did try: fileinto "/var/mail/spamuser"; But Dovecot complained, something about the mailbox pattern being invalid. If it were possible, I would think it would want something like fileinto "spamuser:INBOX"; instead, but I don't know if that is even possible. In a global sieve script, is there anyway to know/test which system user is the targeted user for delivery? So in a global sieve_before script I could test the target user, and if it's spamuser just file it immediately and stop with no further testing? Thanks for any assistance! Dan

2 2

Health check for dsync?
by Rich Wales 16 Feb '16

16 Feb '16

I am syncing two Dovecot sites using dsync. Are there any tools available to confirm that dsync is (or is not) running properly between the two sites? (E.g., something I could run periodically in Nagios?) Rich Wales richw(a)richw.org

1 0

Redundant and Geobalancing setup
by Cedric Malitte 15 Feb '16

15 Feb '16

Hi, I use dovecot for a long time now, but only as a single isolated server each time. I joined a company a few years back. We had trouble with compagnies hosting our mail, supposedly full redundant and so on. The company is small, but we have many dealers around the world, and it's growing. Mail became the fist choice for clients to contact the dealers. No mail, and we loose sales. For now we have a single server ( with a backup ) on east coast. And sometimes peoples from EU complain about speed.... ah users :) What I'd like to implement is a redundant system with 2 servers, one in NA, one in EU. And I'd also like to be able to add another server if needed on the west coast. Idea is, that if a server goes down, the users will be able to still receive and send mails, and never loose mails. For geobalacing and failover, I read that I can do it with DNS ( I'm with easydns ). I'm at the first stage where I collect informations that I try to understand and foresee a solution. First idea is to set up servers with a mysql master, slaves and a glusterfs in replica mode on the servers. I tried glusterfs on FreeBSD and OMG, it's slow as hell ! ( well maybe it's a trouble on the VMs nics ) On centos it's way better. But I read there might be trouble/index corruption for the mail storage on "shared" space using maildir. I also had a look at dsync, but I wonder if it can be used on more than 2 servers. I found many pages on dovecot clusters using shared storage NFS mounted, but I feel it's not really what I need as the servers will be in different datacenters. So any guide, clue hint would be really appreciated for me to do my homework ! Regards. Cedric

3 5

Re: Question on setting up dovecot
by Jeffery Small 15 Feb '16

15 Feb '16

The problem is solved! A great thanks to everyone for all the valuable information and suggestions you offered. I learned quite a bit, especially about talking to the dovecot server using the telnet/openssl suggestions and the available POP3 commands. Many of you were correct about your diagnosis, but I had some things happening that were creating confusion. Here is a brief summary of what I discovered. There were a few initial dovecot configuration issues which I eventually ironed out and, as I was reporting at the end of last week, the diagnostic logs seemed to indicate that the connection was successful. Yes, it turned out that Outlook was indeed set to leave a copy of the emails on the server for two weeks -- a setting hidden away on a sub-menu. Now, why this did not seem to be the case was because every time Outlook connected to the Solaris system to retrieve mail, all the messages were immediately deleted (despite this setting), so I was expecting the same behavior on the Linux machine and wasn't looking for the setting. There is something different about how the pop3 handler is working on the two systems. The second problem, which some of you guessed, was that the emails left on the server were not being downloaded because when I wasn't looking, Outlook had already downloaded and tagged them. So, even when I thought there was new mail (new to me anyway) in the inbox, Outlook had beaten me to them so that when I looked, I could see no new mail flowing over to the Windows machine. Disabling the retention setting removed the ambiguity and now I see any new mail in the inbox moving to the other system. Thanks again to everyone! -- Jeff

1 0

ANNOTATE plugin? Squirrel uses it for EXPIRATION information
by Heiko Schlittermann 14 Feb '16

14 Feb '16

Hi, it seems that Squirrel mail uses Mailbox annotations for storing Expire times on the Server. It's an Cyrus server currently. (I've no clue how cyrexpire is able to read it's information from the annotiations) Does dovecot support some similiar mechanism? Or - does anybody know anything about how it's supposed to work on Cyrus and how it can be emulated/simulated with dovecot? Thanks in advance, Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

2 2

Segmentation fault on doveadm search -A with a huge user base
by Heiko Schlittermann 13 Feb '16

13 Feb '16

Hi, I'm using dovecot 2.2.9 with a director/backend setup. The user base is about 4711 users currently. If I start at one of the directors doveadm search -A all savedbefore 5000d it terminates with doveadm(1rrissma): Error: doveadm server disconnected before handshake: EOF doveadm(1rrissma): Error: 2001:638:913:f33::5:ff:24245: Command search failed for 1phaaman: EOF doveadm(1rrissma): Error: doveadm server failure doveadm: Error: Failed to iterate through some users The backend logs about that ... Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '5' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '5' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '0' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '0' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '0' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- 'd' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Fatal: master: service(doveadm): child 11235 killed with signal 11 (core dumped) Is this a known behaviour and fixed already, or do I need to do more investigation? (PS: Running the same command on one of the backends works w/o failure) Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

2 2

Lots of zero-byte hard link files in cur (and new/tmp), cannot see messages in folder
by Charles Marcus 12 Feb '16

12 Feb '16

Hello, I have an el-cheapo shared hosting account on Dreamhost, and have had it for a very long time. For the most part everything usually works fairly well, considering I do keep a lot of folders, and mail, on some of my accounts. They are running dovecot, but still don't have a response as to the version, or doveconf -n output yet. My problem is, one of my most used folders, which was working fine up until a week or so ago, stopped loading the messages, and after some frustrating troubleshooting via email with people who don't listen very well, I finally got a tarball of this folder, and they are using maildir. There are about 24,000 messages in there (non-zero-byte files). This number sounds about right. All other folders (including INBOX, Sent, etc) are still working fine. The problem, though, is there are over 815,000 zero-byte-files in the cur directory, all showing as hardlinks (looks like maybe a whole bunch of duplicates for each of the real message files in the cur directory). There are also 43 non-zero-byte message files in the new directory, and 1,515 of these zero-byte hardlinks (to message files in the new directory). There are also no non-zero-byte message files in the tmp directory, but there are 52 of the hardlinks, linked to something in the new directory. I've never seen any of these kinds of zero-byte files before on the one server I managed for a long time (not shared, just used for a single domain). Anyone ever seen this before? Would running: doveadm index -u myuser * or doveadm force-resync -u myuser * be appropriate commands to try to repair the damage (whatever it is)? Any other commands I could suggest running? Thanks. I know I haven't given much to go on. Charles /* */

2 2

Multiple quota rules from LDAP userdb?
by Heiko Schlittermann 12 Feb '16

12 Feb '16

Hello, I'm just asking myself, how I get multiple quota rules into the LDAP userdb. Dovecot is 2.2.9 userdb_attrs = … fooQuotaRule=quota_rule=%$ which allows me to have an user db entry as fooQuotaRule: *:storage=1000:messages=50 fooQuotaRule: Trash:storage=500 But, if I'd like to have another quota rule for a submailbox of that user? doveadm user returns the first field only field value uid 500 gid 500 home /volumes/dovecot/home/4b/heiko mail maildir:~:INBOX=/volumes/dovecot/inbox/4b/heiko:INDEX=/volumes/dovecot/cache/4b/heiko quota_rule *:storage=1000:messages=50 Inventing additional LDAP attributs fooQuotaRule2, ... doesn't scale well. Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

2 2

Autoexpunge on INBOX? And disabling autoexpunge for a mailbox via userdb_import?
by Tom Johnson 12 Feb '16

12 Feb '16

I have autoexpunge working for various special_use mailboxes, but I'd like it to optionally be used for users for their inbox as well. I'd also like to be able to disable autoexpunge for some users for certain directories. Is it possible to specify an autoexpunge period for the main inbox? And can autoexpunge be turned off via userdb_import? Here's some detail behind my questions: I have the following in 15-mailboxes.conf, and all is working great for the autoexpunge feature for the Junk and Trash mailboxes: namespace inbox { separator = . mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk autoexpunge = 15d } mailbox Trash { auto = subscribe special_use = \Trash autoexpunge = 15d } mailbox Sent { auto = subscribe special_use = \Sent } } And I can change the autoexpunge field for those mailboxes via userdb_import in a SQL query - that's working great. But I would like for some users to expire the main inbox and the Sent mailbox, and for others for there to be no expiration. Imagine a scenario where there are users who get a maximum of 15 days of storage - everything older than that should expire out. Other users are standard email users - their inboxes, personal folders, Sent mail should never expire. Adding autoexpunge to the main mailbox gives an error: namespace inbox { separator = . autoexpunge = 999d ... ... } doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/15-mailboxes.conf line 48: Unknown setting: autoexpunge So is there a way to add an autoexpunge setting to the main inbox? And can I set autoexpunge to 0 via userdb_import to disable autoexpunge for a particular mailbox? That doesn't give any error, but does it properly disable autoexpunge for that mailbox? Thanks- Tom

2 1

Re: Question on setting up dovecot
by Jeffery Small 12 Feb '16

12 Feb '16

Many thanks to Joseph Tam and Steffen Kaiser for pointing me to the logging information that I should have been using in the first place. Now that I have separate log files and debugging information, I'm still at a loss to see why the mail is not being pulled over from the Linux system to the Windows 7 machine. To recap, here is the output from doveconf -n: ---------------------------------------------------------------------- # 2.2.18: /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.8 (0c4ae064f307+) # OS: Linux 4.2.0-27-generic x86_64 Ubuntu 15.10 auth_debug = yes auth_verbose = yes disable_plaintext_auth = no info_log_path = /var/log/dovecot.info log_path = /var/log/dovecot mail_debug = yes mail_location = mbox:~/lib/Mail:INBOX=/var/mail/%u passdb { driver = pam } protocols = imap pop3 ssl_cert = </etc/dovecot/dovecot.pem ssl_key = </etc/dovecot/private/dovecot.pem userdb { driver = passwd } userdb { driver = passwd } verbose_ssl = yes ---------------------------------------------------------------------- Here is the configuration logging/debug info: ---------------------------------------------------------------------- log_path = /var/log/dovecot info_log_path = /var/log/dovecot.info auth_verbose = yes auth_debug = yes mail_debug = yes verbose_ssl = yes ---------------------------------------------------------------------- And here is the complete set of Debug/Info information logged when Outlook on the Win7 system attempts a Send/Receive operation. Long lines broken for readability: ---------------------------------------------------------------------- Feb 11 08:47:49 auth: Debug: Loading modules from directory: \ /usr/lib/dovecot/modules/auth Feb 11 08:47:49 auth: Debug: Read auth token secret from \ /var/run/dovecot/auth-token-secret.dat Feb 11 08:47:49 auth: Debug: auth client connected (pid=7861) Feb 11 08:47:49 auth: Debug: client in: AUTH 1 PLAIN \ service=pop3 session=dF2ST4ErtQDMa1sC \ lip=<server_IP> rip=<client_IP> \ lport=110 rport=54453 resp=<hidden> Feb 11 08:47:49 auth-worker(7864): Debug: Loading modules from directory: \ /usr/lib/dovecot/modules/auth Feb 11 08:47:49 auth-worker(7864): Debug: pam(sherril,<client_IP>): \ lookup service=dovecot Feb 11 08:47:49 auth-worker(7864): Debug: pam(sherril,<client_IP>): \ #1/1 style=1 msg=Password: Feb 11 08:47:49 auth: Debug: client passdb out: OK 1 user=sherril Feb 11 08:47:49 auth: Debug: master in: REQUEST 1975517185 7861 1 \ b48910c85f39498e2fe83277d7e94028 session_pid=7866 Feb 11 08:47:49 auth-worker(7864): Debug: pam(sherril,<client_IP>): lookup Feb 11 08:47:49 auth: Debug: master userdb out: USER 1975517185 \ sherril system_groups_user=sherril uid=1001 gid=1001 \ home=/u/sherril Feb 11 08:47:49 pop3-login: Info: Login: user=<sherril>, method=PLAIN, \ rip=<client_IP>, lip=<server_IP>, mpid=7866, \ session=<dF2ST4ErtQDMa1sC> Feb 11 08:47:49 pop3(sherril): Debug: Effective uid=1001, gid=1001, \ home=/u/sherril Feb 11 08:47:49 pop3(sherril): Debug: fs: root=/u/sherril/lib/Mail, \ index=, indexpvt=, control=, inbox=/var/mail/sherril, alt= Feb 11 08:47:49 pop3(sherril): Info: Disconnected: Logged out top=0/0, \ retr=0/0, del=0/3, size=36928 ---------------------------------------------------------------------- There are no errors. We have a pop3 connection. The user's name, uid, gid, password, home directory are all correct. The inbox is listed as /var/mail/sherril which is correct, and this file has the correct ownership and permissions: # ls -l /var/mail/sherril -rw-rw---- 1 sherril mail 194900 Feb 11 09:00 /var/mail/sherril Yet, no mail is pulled over. Are there any clues here? Thanks, -- Jeff

5 6