May 2013 - dovecot - dovecot.org

[Dovecot] Error: dict client sent broken reply
by Chris Richards 15 Aug '13

15 Aug '13

I've been mucking about, experimenting with the expire plugin and using a dictionary. I've got the iteration query working when I do a normal expunge using: doveadm expunge -A mailbox "INBOX.Trash" savedbefore 1w and expunging works as expected. However, I've got over 12,000 accounts on this server, so I was hoping using the expire plugin to could help out. I've configured the plugin, and things kinda work, except that somewhere between 3700 and 3800 users, I abort with this: doveadm(someuseraccount@somedomain): Error: dict client (/var/run/dovecot/dict) sent broken reply doveadm(someuseraccount@somedomain): Error: Dictionary iteration failed doveadm: Error: Failed to iterate through some users It consistently fails at the same user. If I delete that user from the expire database, then it appears to fail on the next user. I also see this in the logs: dovecot: dict: Error: dict client: COMMIT: Can't commit while iterating I've absolutely no idea where to go from here to troubleshoot this. Any guidance would be appreciated. Thanks, Chris doveconf -n: # 2.1.12: /etc/dovecot/dovecot.conf # OS: Linux 3.7.5-hardened-r1 x86_64 Gentoo Base System release 2.1 ext4 auth_master_user_separator = * auth_mechanisms = plain login auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@& auth_verbose_passwords = plain default_process_limit = 200 dict { quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no login_greeting = Awaiting command... mail_location = maildir:/home/vmail/%d/%n/Maildir mail_plugins = " quota" mail_privileged_group = 100 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { expire = Trash 7 expire2 = Trash/* 7 expire3 = Spam 7 quota = dict:User quota:%u:proxy::quota quota_rule = *:storage=200M quota_warning = storage=99%% quota-warning 99 %n %d quota_warning2 = storage=95%% quota-warning 95 %n %d quota_warning3 = storage=80%% quota-warning 80 %n %d quota_warning4 = -storage=95%% quota-warning 'less than 95' %n %d sieve = ~/.dovecot.sieve sieve_default = /home/vmail/dovecot/sieve/default.sieve sieve_dir = ~/sieve sieve_global_dir = /home/vmail/dovecot/sieve } protocols = imap pop3 sieve lmtp service auth-worker { user = $default_internal_user } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = dovecot mode = 0666 user = dovecot } user = $default_internal_user } service dict { unix_listener dict { mode = 0600 user = vmail } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service quota-warning { executable = script /etc/dovecot/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } ssl_cert = </etc/ssl/dovecot/server.pem ssl_key = </etc/ssl/dovecot/server.key userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_plugins = sieve quota postmaster_address = postmaster@domain } protocol lda { mail_plugins = " quota sieve quota" } protocol imap { mail_max_userip_connections = 20 mail_plugins = " quota quota imap_quota" } protocol pop3 { mail_plugins = " quota quota" } >From dovecot-dict-sql.conf.ext: connect = host=localhost dbname=maildb user=dbuser password=dbpass # CREATE TABLE quota ( # username varchar(100) not null, # bytes bigint not null default 0, # messages integer not null default 0, # primary key (username) # ); map { pattern = priv/quota/storage table = quota_usage username_field = address value_field = quota_bytes } map { pattern = priv/quota/messages table = quota_usage username_field = address value_field = quota_messages } # CREATE TABLE expires ( # username varchar(100) not null, # mailbox varchar(255) not null, # expire_stamp integer not null, # primary key (username, mailbox) # ); map { pattern = shared/expire/$user/$mailbox table = expires value_field = expire_stamp fields { address = $user folder = $mailbox } } dovecot-sql.conf.ext driver = mysql connect = host=/var/run/mysqld/mysqld.sock dbname=maildb user=dbuser password=dbpass default_pass_scheme = PLAIN user_query = SELECT homedir AS home, maildir AS mail, uid AS uid, gid AS gid, quota_rule AS quota_rule FROM email WHERE address = CONVERT('%u' USING latin1) AND is_alias=0; password_query = SELECT address AS user, NULL as password, homedir AS userdb_home, maildir as userdb_mail, uid AS userdb_uid, gid AS userdb_gid, quota_rule AS userdb_quota_rule, 'Y' AS nopassword FROM email WHERE address = CASE WHEN ('%d' = '') THEN CONCAT (CONVERT('%n' USING latin1), '@bordernet.com.au') ELSE CONVERT('%u' USING latin1) END AND is_alias=0 AND CheckPasswordFunc(CONVERT('%n' USING latin1), '%d', CONVERT('%w' USING latin1), '%r'); iterate_query = SELECT address AS user FROM email WHERE is_alias=0 AND length(password) > 1

3 7

[Dovecot] execvp /usr/local/libexec/dovecot/dovecot-lda: Permission denied
by Edward Carraro 09 Aug '13

09 Aug '13

I'm trying to follow the instructions for configuring Dovecot LMTP with postfix for virtual domains but whenever mail arrives, in /var/log/maillog, I see Sep 23 16:01:20 mybox postfix/pipe[572]: 999C83206C6: to=< user(a)mytest.domain.com>, relay=dovecot, delay=0.08, delays=0.04/0.01/0/0.03, dsn=4.3.0, status=deferred (temporary failure. Command output: pipe: fatal: pipe_command: execvp /usr/local/libexec/dovecot/dovecot-lda: Permission denied ) ls -lh for /usr/local/libexec/dovecot/dovecot-lda -rwxr-xr-x 1 root staff 62K Sep 20 15:15 dovecot-lda It will deliver mail when I do the sudo invocation of dovecot-lda, but I thought LMTP was suppose to be an alternative to that postfix/main.cf: virtual_transport = dovecot virtual_mailbox_domains = mytest.domain.com dovecot_destination_recipient_limit = 1 /etc/postfix/master.cf: dovecot unix - n n - - pipe flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -d ${recipient} /usr/local/etc/dovecot/conf.d/10-master.conf: service auth { unix_listener auth-userdb { mode = 0600 user = vmail #group = vmail } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } inet_listener lmtp { port = 24 } user = vmail } Did I miss something in my config?

7 9

[Dovecot] Wrong remote IP (rip) in mail.log using IMAP login
by tyli 02 Aug '13

02 Aug '13

Dear list users While trying to secure our dovecot server with fail2ban I came across the following problem: We use dovecot (1.2.9, ubuntu package) behind a NAT, and failed login attempts are logged with our firewall as the remote ip. Example: Apr 15 08:36:26 mail dovecot: imap-login: Disconnected (auth failed, 6 attempts): user=<xy>, method=PLAIN, rip=192.168.0.1, lip=192.168.0.3 Therefore I would ban 192.168.0.1 which means that I ban EVERY user. Funny thing is that POP3 login attempts are logged correctly: Apr 13 11:05:50 mail dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<sgvyniwx>, method=PLAIN, rip=217.81.27.55, lip=192.168.0.3 Any ideas how to change this? Thanks in advance tyli

5 5

[Dovecot] how to setup different quota for multiple namespaces
by Andreas Oster 29 Jul '13

29 Jul '13

Hello all, I am using dovecot for some time now but always had very basic configurations. I recently have configured a new, additional namespace in order to move archived mails (Thunderbird Archives) to another storage system. namespace { type = private separator = . prefix = Archives. location = maildir:/var/vmail/archive/%Ln/Maildir subscriptions = yes list = children inbox = no } This works well but unfortunately I cannot find a solution to either ignore this new namespace from quota computation or add a separate quota value for it. I have integrated user authentication with LDAP and fetch some user attributes from LDAP entries: user_attrs = =home=/var/vmail/%Ln/Maildir,=mail=maildir:/var/vmail/%Ln/Maildir/,otherPager=quota_rule=*:storage=%$ I configured the quota plugin in the following way: plugin { quota = maildir:User quota:ns= quota2 = maildir:Archive quota:ns=Archives. quota_rule = *:storage=1G quota_rule2 = ns=Archives.:ignore } unfortunately this does not give the desired result. Folders under the Archives namespace use the same quota as INBOX. Does anyone have an idea how to fix this ? Thank you for your kind help Andreas

4 19

[Dovecot] Proxying improvements in v2.1.2
by Timo Sirainen 16 Jul '13

16 Jul '13

I just committed a couple of features that will make life easier for some types of proxying setups: 1. IMAP proxying has already for a while supported sending local/remote IP/port to backend server, which can use it for logging and other purposes. I've now implemented this for POP3 as well, although only the remote IP/port is forwarded, not local IP/port. I implemented this also for LMTP in v2.2 tree, but haven't bothered to backport that change. Both POP3 and LMTP uses XCLIENT command that is compatible to Postfix's (XCLIENT ADDR=1.2.3.4 PORT=110). 2. proxy_maybe=yes + host=host.example.com actually works now. As long as host.example.com DNS lookup returns one IP that belongs to the current server the proxying is skipped. 3. auth_proxy_self = 1.2.3.4 setting means that if proxy_maybe=yes and host=1.2.3.4 then Dovecot assumes that this is a local login and won't proxy it, even if 1.2.3.4 isn't the actual local IP. This can be helpful if the host field contains load balancer's IP address instead of the server's. You can add more than one IP (space separated) and of course everything related to this works just as well with hostnames as with IPs (even when hostname expands to multiple IPs).

2 2

[Dovecot] Zero-sized Maildir files and index corruption on over-quota
by Edgar Fuß 28 Jun '13

28 Jun '13

We are using Postfix as an MTA delivering via Dovecot's LDA (with sieve). We also use Dovecot as a POP/IMAP server. Mail storage is Maildir on NFS, indexes are stored locally. Quotas are FS quotas enforced by the NFS server. The Dovecot version is 1.2.11. Recently, for one user being over quota, some attempts to deliver mail bounced as expected while some others resulted in zero-sized Maildir files. I can verify zero-sized files in cur, while I cannot tell whether they were already zero-sized in new. However, since nobody except Dovecot is accessing the mail storage, it's either Dovecot's LDA or Dovecot's IMAP producing the zero-sized files. Logs show successful delivery (deliver: stored mail into mailbox 'INBOX') but ,,Corrupted index cache file /var/db/dovecot/indexes/XXX/.INBOX/dovecot.index.cache: Corrupted virtual size for uid=YYY: 0 != NNNN'' alerts on IMAP access. It looks like all mails which correctly got bounced are >64KB while those resulting in zero-sized files are <64KB (there is one >32KB). My first guess would be that maybe for large files, write() fails, while for small ones, close() fails or there is some buffering involved for large files.

3 10

[Dovecot] traffic statistics
by Martin Waschbüsch 28 Jun '13

28 Jun '13

Hi all, Does anyone know a good tool to gather traffic statistics for imap? And what level of logging would I need to set for dovecot to make it happen. Ideally this would be able to filter per domain, too. ;-) Thanks a lot, Martin -- Martin Waschbüsch IT-Dienstleistungen Lautensackstr. 16 80687 München Telefon: +49 89 57005708 Fax: +49 89 57868023 Mobil: +49 170 2189794 martin(a)waschbuesch.de http://martin.waschbuesch.de

4 5

[Dovecot] dsync assert failure in 2.2.2
by Tomas Olsson 25 Jun '13

25 Jun '13

Hi, with 2.2.2 and today's hg, dsync crashes with dsync(root): Panic: file ../../../../../src/lib-storage/index/mbox/mbox-lock.c: line 797 (mbox_lock): assertion failed: (lock_type == F_RDLCK || mbox->mbox_lock_type != F_RDLCK) when I run USER=root 2.2-hg/bin/dsync -c etc/dovecot.conf -f -o mail_location=mbox:/tmp/imap/fwadmin.tmp:INBOX=/tmp/imap/fwadmin.tmp/INBOX mirror mdbox:/tmp/imap/fwadmin It seems to happen for all mailboxes that are not empty. 2.2.1 completes the conversion without any complaints. config: # 2.2.2 (a3b5b762639a): etc/dovecot.conf # OS: Linux 3.2.0-32-generic i686 Ubuntu 12.04.2 LTS default_internal_user = nobody default_login_user = nobody Backtrace (hg version): #0 i_panic (format=0x252530 "file %s: line %d (%s): assertion failed: (%s)") at ../../../src/lib/failures.c:255 #1 0x001c643f in mbox_lock (mbox=0x80d9b40, lock_type=<optimized out>, lock_id_r=0x811352c) at ../../../../../src/lib-storage/index/mbox/mbox-lock.c:797 #2 0x001c695c in mbox_mail_seek (mail=0x81188e0) at ../../../../../src/lib-storage/index/mbox/mbox-mail.c:82 #3 0x001c6ee1 in mbox_mail_init_stream (mail=0x81188e0) at ../../../../../src/lib-storage/index/mbox/mbox-mail.c:335 #4 mbox_mail_get_stream (_mail=0x81188e0, get_body=false, hdr_size=0x0, body_size=0x0, stream_r=0xbffff77c) at ../../../../../src/lib-storage/index/mbox/mbox-mail.c:375 #5 0x001e65b0 in mail_get_hdr_stream (mail=0x81188e0, hdr_size=0x0, stream_r=0xbffff77c) at ../../../src/lib-storage/mail.c:226 #6 0x00215532 in index_mail_get_header_stream (_mail=0x81188e0, headers=0x8106fd0, stream_r=0xbffff7e0) at ../../../../src/lib-storage/index/index-mail-headers.c:837 #7 0x001e649d in mail_get_header_stream (mail=0x81188e0, headers=0x8106fd0, stream_r=0xbffff7e0) at ../../../src/lib-storage/mail.c:190 #8 0x08084bf2 in dsync_mail_get_hdr_hash (mail=0x81188e0, hdr_hash_r=0xbffff8ec) at ../../../../src/doveadm/dsync/dsync-mail.c:32 #9 0x080736a8 in importer_try_next_mail (wanted_uid=1, importer=0x81136e0) at ../../../../src/doveadm/dsync/dsync-mailbox-import.c:515 #10 importer_next_mail (importer=0x81136e0, wanted_uid=1) at ../../../../src/doveadm/dsync/dsync-mailbox-import.c:536 #11 0x0807729a in dsync_mailbox_import_changes_finish (importer=0x81136e0) at ../../../../src/doveadm/dsync/dsync-mailbox-import.c:1754 #12 0x08072e3b in dsync_brain_recv_mail_change (brain=0x80beaf8) at ../../../../src/doveadm/dsync/dsync-brain-mails.c:102 #13 dsync_brain_sync_mails (brain=0x80beaf8) at ../../../../src/doveadm/dsync/dsync-brain-mails.c:319 #14 0x0806f11b in dsync_brain_run_real (changed_r=0xbffffae6, brain=0x80beaf8) at ../../../../src/doveadm/dsync/dsync-brain.c:440 #15 dsync_brain_run (brain=0x80beaf8, changed_r=0xbffffae6) at ../../../../src/doveadm/dsync/dsync-brain.c:469 #16 0x0806d1ef in cmd_dsync_run_local (ibc2=0x80be8d8, brain=0x80beaf8, user=0x80ba8f0, ctx=0x80b2ca8) at ../../../../src/doveadm/dsync/doveadm-dsync.c:356 #17 cmd_dsync_run (_ctx=0x80b2ca8, user=0x80ba8f0) at ../../../../src/doveadm/dsync/doveadm-dsync.c:543 #18 0x08055424 in doveadm_mail_next_user (error_r=0xbffffbcc, ctx=0x80b2ca8, input=<optimized out>) at ../../../src/doveadm/doveadm-mail.c:308 #19 doveadm_mail_next_user (ctx=0x80b2ca8, input=<optimized out>, error_r=0xbffffbcc) at ../../../src/doveadm/doveadm-mail.c:267 #20 0x080560fc in doveadm_mail_cmd (argv=0x80af1ec, argc=<optimized out>, cmd=0x80b226c) at ../../../src/doveadm/doveadm-mail.c:517 #21 doveadm_mail_try_run (cmd_name=0x80af286 "sync", argc=3, argv=0x80af1e4) at ../../../src/doveadm/doveadm-mail.c:609 #22 0x08055046 in main (argc=3, argv=0x80af1e4) at ../../../src/doveadm/doveadm.c:398 I'm attaching the mbox file. /t

1 1

[Dovecot] Autocreation the home folder
by Joan 14 Jun '13

14 Jun '13

Sorry if this has been already asked, but I haven't been able to find a solution, I am trying to migrate a dovecot 1.2 config to 2.1, the 'dovecot -n > dovecot.conf' migrated all the setup without issues, and the mail is working as expected. The only issue I've had is with the change with the mail_location, mail_home, that in the 2.1 versions can't be on the same path. After reading in the wiki the various config options ( http://wiki2.dovecot.org/VirtualUsers/Home) I opted to use the 3rd: Home directory under mail, for example: Maildir: home=/var/vmail/domain/user/home/ mail=/var/vmail/domain/user/ mbox: There's really no good and safe way to do it. So I changed from: mail_location = maildir:/home/vmail/%d/%n mail_home = /var/vmail/%d/%n To: mail_location = maildir:/home/vmail/%d/%n mail_home = /var/vmail/%d/%n/home The problem is that those home directories are not created (the mailboxes already existed) automatically, and I am getting these errors: ========================================================== May 10 17:08:59 server dovecot: lda(username(a)example.com): Debug: Home dir not found: /var/vmail/example.com/username/home May 10 17:08:59 server dovecot: lda(username(a)example.com): Debug: Quota root: name=User quota backend=maildir args= May 10 17:08:59 server dovecot: lda(username(a)example.com): Debug: Quota rule: root=User quota mailbox=* bytes=20480000000 messages=0 May 10 17:08:59 server dovecot: lda(username(a)example.com): Debug: Quota rule: root=User quota mailbox=Trash ignored May 10 17:08:59 server dovecot: lda(username(a)example.com): Debug: Quota warning: bytes=15360000000 (75%) messages=0 reverse=no command=/etc/dovecot/quota-warning.sh 75 May 10 17:08:59 server dovecot: lda(username(a)example.com): Debug: Quota warning: bytes=17408000000 (85%) messages=0 reverse=no command=/etc/dovecot/quota-warning.sh 85 May 10 17:08:59 server dovecot: lda(username(a)example.com): Debug: maildir++: root=/home/vmail/example.com/username, index=, control=, inbox=/home/vmail/example.com/username, alt= May 10 17:08:59 server dovecot: lda(username(a)example.com): Debug: Quota root: name=User quota backend=maildir args= May 10 17:08:59 server dovecot: lda(username(a)example.com): Debug: Quota rule: root=User quota mailbox=* bytes=1073741824 messages=0 May 10 17:08:59 server dovecot: lda(username(a)example.com): Debug: Quota warning: bytes=805306368 (75%) messages=0 reverse=no command=/etc/dovecot/quota-warning.sh 75 ======================================================== Am I supposed to create the folders via an external script? Or is there any option to make this automatic. Regards, Joan

4 10

[Dovecot] CATENATE/literal8 issue
by Michael M Slusarz 13 Jun '13

13 Jun '13

Using 2.2.2, I see this: C: 6 APPEND "INBOX" (\seen) "16-May-2013 22:05:14 -0600" CATENATE (URL "/INBOX;UIDVALIDITY=1255685337/;UID=48812/;SECTION=HEADER" TEXT ~{40} S: 6 NO [UNKNOWN-CTE] Binary input allowed only when the first part is binary. Why is there this limitation? It seems to me that CATENATE is confusing the content-type encoding of the data/part itself with the encoding of the IMAP literal. A literal 8 is nothing more than a series of OCTET's that *may* contain nulls, but not necessarily. i.e., in the above example the 40 octets of data are US-ASCII text, which is perfectly acceptable to send as a literal8. (Client rationale: If BINARY exists on the server, we don't bother to scan IMAP literal's for null data -- we just send them as literal8's. It's an optimization that I would hate to get rid of.) michael

2 8