March 2015 - dovecot - dovecot.org

v2.2.16 release candidate released
by Timo Sirainen 12 Mar '15

12 Mar '15

http://dovecot.org/releases/2.2/rc/dovecot-2.2.16.rc1.tar.gz http://dovecot.org/releases/2.2/rc/dovecot-2.2.16.rc1.tar.gz.sig Looks like it's been a long time since v2.2.15. There have been a ton of changes since it was released though, so here's a release candidate first to find out if somebody can find any bugs before the final v2.2.16. Unfortunately I haven't had time/energy to read Dovecot mailing list for a while now. I'm hoping this will change, but I don't really expect it to happen anytime soon. On the positive side for Dovecot, it's now becoming used in more and more multi-million user installations, which brings all kinds of nice new improvements. The largest changes since v2.2.15: * dbox: Resyncing (e.g. doveadm force-resync) no longer deletes dovecot.index.cache file. The cache file was rarely the problem so this just caused unnecessary slowness. * Mailbox name limits changed during mailbox creation: Each part of a hierarchical name (e.g. "x" or "y" in "x/y") can now be up to 255 chars long (instead of 200). This also reduces the max number of hierarchical levels to 16 (instead of 20) to keep the maximum name length 4096 (a common PATH_MAX limit). The 255 char limit is hopefully large enough for migrations from all existing systems. It's also the limit on many filesystems. + director: Added director_consistent_hashing setting to enable consistent hashing (instead of the mostly-random MD5 hashing). This causes fewer user moves between backends when backend counts are changed, which may improve performance (mainly due to caching). + director: Added support for "tags", which allows one director ring to serve multiple backend clusters with different sets of users. + LMTP server: Added lmtp_user_concurrency_limit setting to limit how many LMTP deliveries can be done concurrently for a single user. + LMTP server: Added support for STARTTLS command. + If logging data is generated faster than it can be written, log a warning about it and show information about it in log process's process title in ps output. Also don't allow a single service to flood too long at the cost of delaying other services' logging. + stats: Added support for getting global statistics. + stats: Use the same session IDs as the rest of Dovecot. + stats: Plugins can now create their own statistics fields + doveadm server: Non-mail related commands can now also be used via doveadm server (TCP socket). + doveadm proxying: passdb lookup can now override doveadm_port and change the username. + doveadm: Search query supports now "oldestonly" parameter to stop immediately on the first non-match. This can be used to optimize: doveadm expunge mailbox Trash savedbefore 30d oldestonly + doveadm: Added "save" command to directly save mails to specified mailbox (bypassing Sieve). + doveadm fetch: Added body.snippet field, which returns the first 100 chars of a message without whitespace or HTML tags. The result is stored into dovecot.index.cache, so it can be fetched efficiently. + dsync: Added -t <timestamp> parameter to sync only mails newer than the given received-timestamp. + dsync: Added -F [-]<flag> parameter to sync only mails with[out] the given flag/keyword. + dsync: Added -a <mailbox> parameter to specify the virtual mailbox containing user's all mails. If this mailbox is already found to contain the wanted mail (by its GUID), the message is copied from there instead of being re-saved. (This isn't efficient enough yet for incremental replication.) + dsync: -m parameter can now specify \Special-use names for mailboxes. + imapc: Added imapc_features=gmail-migration to help migrations from GMail. See http://wiki2.dovecot.org/Migration/Gmail + imapc: Added imapc_features=search to support IMAP SEARCH command. (Currently requires ESEARCH support from remote server.) + expire plugin: Added expire_cache=yes setting to cache most of the database lookups in dovecot index files. + quota: If overquota-flag in userdb doesn't match the current quota usage, execute a configured script. + redis dict: Added support for expiring keys (:expire_secs=n) and specifying the database number (:db=n) - auth: Don't crash if master user login is attempted without any configured master=yes passdbs - Parsing UTF-8 text for mails could have caused broken results sometimes if buffering was split in the middle of a UTF-8 character. This affected at least searching messages. - String sanitization for some logged output wasn't done properly: UTF-8 text could have been truncated wrongly or the truncation may not have happened at all. - fts-lucene: Lookups from virtual mailbox consisting of over 32 physical mailboxes could have caused crashes.

4 7

Released Pigeonhole v0.4.7.rc1 for Dovecot v2.2.16.rc1
by Stephan Bosch 12 Mar '15

12 Mar '15

Hello Dovecot users, Last time I had a few stupid problems in the releases, so I'll follow Timo's example and I release an RC first. The highlights include the implementation of the index and metadata extensions. Quite a few bugs are fixed as well. Changelog v0.4.7: * editheader extension: Made protection against addition and deletion of headers configurable separately. Also, the `Received' and `Auto-Submitted' headers are no longer protected against addition by default. * Turned message envelope address parse errors into warnings. * The interpreter now accepts non-standard domain names, e.g. containing '_'. + Implemented the Sieve index extension (RFC 5260). + Implemented support for the mboxmetadata and servermetadata extensions (RFC 5490). + Implemented new sieve commands for the doveadm command line utility. These commands are currently limited to ManageSieve operations, but the other current sieve tools will be migrated to doveadm in the near future as well. + Added more debug output to binary up-to-date checking. - The Sieve interpreter now flushes duplicate database during start phase of result execution rather than commit phase. This makes sure locks on the duplicate database are released as soon as possible, preventing contention. - Performed a few optimizations in the lexical scanner of the language. - Fixed bug in `:matches' match-type that made a pattern without wildcards match as if there were a '*' at the beginning. - file storage: Restructured storage initialization to address backwards compatibility issues. - Fixed crash in validation of the string parameter of the comparator tag. - extprograms extension: Made sure supplemental group privileges are also dropped. This was a problem reported by Debian lintian. - Fixed bug in handling of binary errors for action side-effects and message overrides. The release is available as follows: http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.7.rc1… http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.7.rc1… Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for more information. Have fun testing this new release and don't hesitate to notify me when there are any problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl

5 12

Expunge (?)
by Muzaffer Tolga Ozses 12 Mar '15

12 Mar '15

Hi, I don't know if this is the correct term. Let me elaborate. From time to time, I want to expunge (?) the mail folder so that users' Trash'd mails don't occupy space. So, here are my questions: 1. Is this the correct term? 2. Is it a Good Idea™ to do it? Thanks, -- mto

2 2

Sieve security: Any way to protect credentials used in extprograms?
by E.B. 11 Mar '15

11 Mar '15

I need to connect to a database in a script called using Sieve extprograms plugin. When delivering mail, Sieve is running as the mail recipient user, which means any files, either the sieve script or the extprograms it invokes, are run under that user's permissions. What would be a way to hide the database credentials in a more restricted file? I can think of... * Store the credentials in the database itself, return them from a DICT lookup that is fed to the sieve script. The credentials would have to be duplicated for every user record in the database. * Somehow put the value in the Sieve environment, but I'm not sure how to get a value into the environment (just return it in a userdb lookup?) and then how to get at that value once I'm executing the sieve script (can I retrieve it with the "environment" extension?) Are there better ways? [Of course, if I could do more elaborate things like call a stored procedure from Dovecot DICT then I could avoid this situation]

2 1

Sieve reject with ORIG-TO vs TO
by E.B. 11 Mar '15

11 Mar '15

Hi, The bounce message generated by the reject extension has what looks like a hard coded message prefix that comes before the configurable reason text: "Your message to <username> was automatically rejected:" In some cases, the <username> is NOT the original-to address, which can cause confusion to the sender or expose private aliasing data that some people might want to hide. Is there a way to make the reject extension use the original-to address in that hard-coded message prefix? Is there a way to completely customize the FULL message? Is there a way to customize the headers for such messages?

2 1

FTS search with CLucene giving odd results/exposing config issue
by David Gessel 11 Mar '15

11 Mar '15

I recently got FTS search with Clucene working on FreeBSD 10.1 (I had some trouble with 9.3, postponed solving it, then restarted with 10.1 by manually installing libtextcat, a listed dependency on http://wiki2.dovecot.org/Plugins/FTS/Lucene, but not part of the FreeBSD makefile: maybe that helped, maybe it was irrelevant). However, I now get some anomalous results. I'm putting them together as they may be related. Or, maybe not. 1) Searching from Thunderbird returns 35x as many results for the same term as searching from the command line. I'm not sure if the indexes are created or enumerable. 2) There seems to be a glitch in the maildir naming structure that results in a double slash (//) which doesn't seem to impact IMAP serving or Sieve, but maybe is causing the problem above with Clucene. Any hints? If I run a search from Thunderbird and "run search on server" for a somewhat infrequently used term like "titanium," I get results in more or less every archive folder one by one over about 20 minutes of searching and get about 516 results. It seems to run a little faster without the "run search on server" checkbox selected, but Thunderbird on Windows is abandonware and the only useful datapoint is 516 results. Maybe it is time for Claws. Anyway, if I run: # doveadm search -u user(a)domain.com mailbox '*' body titanium doveadm(user(a)domain.com): Error: stat(/mail/domain.com/user//.dovecot.sieve/tmp) failed: Not a directory doveadm(user(a)domain.com): Error: Syncing mailbox dovecot.sieve failed: Internal error occurred. Refer to server log for more information. [2015-03-11 09:11:16] bcadc922ba26b953a25701002812f29a 812478 bcadc922ba26b953a25701002812f29a 9338037 bcadc922ba26b953a25701002812f29a 9342918 bcadc922ba26b953a25701002812f29a 9343205 bcadc922ba26b953a25701002812f29a 9346096 bcadc922ba26b953a25701002812f29a 9350977 bcadc922ba26b953a25701002812f29a 9351379 bcadc922ba26b953a25701002812f29a 9352318 bcadc922ba26b953a25701002812f29a 9352441 bcadc922ba26b953a25701002812f29a 9352501 3d5d982e3a89b853c12c01002812f29a 1901940 3d5d982e3a89b853c12c01002812f29a 1908073 3d5d982e3a89b853c12c01002812f29a 1941453 3d5d982e3a89b853c12c01002812f29a 2036962 3d5d982e3a89b853c12c01002812f29a 2038360 I get 15 results. # doveadm fetch -u user(a)domain.com "mailbox" mailbox-guid 3d5d982e3a89b853c12c01002812f29a uid 1908073 mailbox: INBOX All are from the inbox. This takes roughly 30 seconds (much faster, that's good, though still not super fast) and in the logs I see promising messages like: Mar 11 09:27:52 host dovecot: indexer-worker(user(a)domain.com): Indexed 3708 messages in Commerce but also slightly less promising messages like: Mar 11 09:27:52 host dovecot: indexer-worker(user(a)domain.com): Warning: Maildir /mail/domain.com/user//.Bulkmail: Expunged message reappeared, giving a new UID (old uid=9335155, file=1426089637.M350915P91305.host,S=23845,W=24282:2,) If I execute # doveadm index -A '*' I get doveadm(user(a)domain.com): Error: stat(/mail/domain.com/user//.dovecot.sieve/tmp) failed: Not a directory doveadm(user(a)domain.com): Error: Syncing mailbox dovecot.sieve failed: Internal error occurred. Refer to server log for more information. [2015-03-11 09:51:26] (there's nothing in the server log) But note in both previous results a double slash (//) in the directory structure. I don't think that's right. So: Does # doveadm search search subfolders? The internets say either '*' or INBOX should search the hierarchy and the search command does set off indexing operations on every subfolder as indicated by the logs, but the results are only from the top level folder.... perhaps that is related to... Why do some errors report a double slash (//) in directory names? I don't seem to be having any other symptoms (if, indeed, this is one) of incompatibilities. My sieve filters work just fine and are accessible from the managesieve plugin in Thunderbird. My config file is below: # 2.2.15: /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 (3e924b1b6c5c+) # OS: FreeBSD 10.1-RELEASE amd64 auth_default_realm = domain.com auth_mechanisms = plain login auth_realms = domain.com domain2.org disable_plaintext_auth = no first_valid_gid = 5000 first_valid_uid = 5000 hostname = domain.com imap_idle_notify_interval = 29 mins last_valid_gid = 5000 last_valid_uid = 5000 lda_mailbox_autocreate = yes listen = * mail_gid = 5000 mail_location = maildir:~:CONTROL=/var/no-quota/%u mail_plugins = quota mail_log notify fts fts_lucene mail_temp_dir = /var/tmp mail_uid = 5000 maildir_broken_filename_sizes = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate spamtest spamtestplus editheader mbox_lazy_writes = no namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { expire = Trash fts = lucene fts_autoindex = yes fts_lucene = whitespace_chars=@. mime_parts mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = maildir:User quota quota_exceeded_message = Storage quota for this account has been exceeded, please try again later. quota_rule = *:storage=50G quota_rule2 = Trash:storage=+30%% quota_rule3 = Sent:storage=+30%% quota_warning = storage=90%% quota-warning 90 %u quota_warning2 = storage=75%% quota-warning 75 %u sieve = ~/.dovecot.sieve sieve_before = /usr/local/etc/dovecot/sieve/ sieve_dir = ~/sieve sieve_extensions = +spamtest +spamtestplus +relational +comparator-i;ascii-numeric +editheader sieve_global_dir = /usr/local/etc/dovecot/sieve } postmaster_address = user(a)domain.com protocols = imap lmtp sieve sieve sendmail_path = /usr/local/sbin/sendmail service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 vsz_limit = 128 M } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } ssl = required ssl_cert = </usr/local/etc/ca/host-cert.pem ssl_key = </usr/local/etc/ca/host-unencrypted-key.pem ssl_prefer_server_ciphers = yes userdb { driver = prefetch } userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lda { mail_plugins = quota mail_log notify fts fts_lucene sieve } protocol imap { mail_max_userip_connections = 20 mail_plugins = quota mail_log notify fts fts_lucene imap_quota imap_zlib } protocol lmtp { mail_plugins = quota mail_log notify fts fts_lucene sieve } (END)

1 0

libdriver_msql.so
by kaniggl 11 Mar '15

11 Mar '15

Hello, i installed dovecot 2.11.1 on ubuntu Linux dadd3041 3.16.0-30-generic #40-Ubuntu SMP Mon Jan 12 22:07:11 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux To make it clear, architecture is PowerPC 64bit Then i installed dovecot via apt-get. But the file /usr/lib/dovecot/modules/auth/libdriver_mysql.so is missing. So i compiled dovecot myself on this system and got a file /usr/lib/dovecot/modules/auth/libdriver_mysql.so Then i restartetd the server and try to do a login via a webmailer (roundcube), but i get no connection to the database server. This is found in /var/log/mail.log Mar 11 15:19:04 dadd3041 dovecot: master: Dovecot v2.2.9 starting up (core dumps disabled) Mar 11 15:19:04 dadd3041 postfix/master[3367]: reload -- version 2.11.1, configuration /etc/postfix Mar 11 15:19:04 dadd3041 postfix/master[3367]: terminating on signal 15 Mar 11 15:19:04 dadd3041 postfix/master[3597]: daemon started -- version 2.11.1, configuration /etc/postfix Mar 11 15:19:25 dadd3041 dovecot: auth: Error: dlopen(/usr/lib/dovecot/modules/auth/libdriver_mysql.so) failed: /usr/lib/dovecot/modules/auth/libdriver_mysql.so: wrong ELF class: ELFCLASS32 Mar 11 15:19:25 dadd3041 dovecot: auth: Fatal: Unknown database driver 'mysql' Mar 11 15:19:25 dadd3041 dovecot: master: Error: service(auth): command startup failed, throttling for 2 secs Mar 11 15:19:25 dadd3041 dovecot: imap-login: Disconnected: Auth process broken (disconnected before auth was ready, waited 0 secs): user=<>, rip=::1, lip=::1, TLS handshaking, session=<rJpS9AMR9wAAAAAAAAAAAAAAAAAAAAAB> I know this means it is not a 64bit object but how could i achieve to get right one? best regards, kaniggl

3 4

dec2str ...
by Hardy Flor 11 Mar '15

11 Mar '15

a very stupid question: What reason is there for an output with printf until dec2str to convert the numeric value to a string and not to use the format identifier "%d" or "u%"?

2 1

full text index "per user"?
by Ralf Hildebrandt 11 Mar '15

11 Mar '15

Is there any way of disabling the creation of a full text index on a per user basis? -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

1 1

Deduplicate Public Folder
by Bobber 11 Mar '15

11 Mar '15

How do you perform the doveadm deduplicate command on public folders? If I specify a user, nothing is found. Using -A also doesn't seem to work. -- *Bob Wooldridge* Blog: http://kc0dxf.net/blog/

2 1