- dovecot - dovecot.org

[Dovecot] zlib config questions
by Terry Barnum 05 Dec '13

05 Dec '13

After nearly running out of space I swapped in larger disks and then saw the recent threads about zlib compression. Unfortunately I'm still confused after reading <http://wiki2.dovecot.org/Plugins/Zlib>. In order to compress new email being stored do I only need to change 10-mail.conf to this: mail_plugins = $mail_plugins zlib and 90-plugin.conf to this: plugin { zlib_save_level = 6 zlib_save = gz ... Or do I need to instead (or also?) add it to the list of plugins in 20-imap.conf? A simple 'sudo doveadm reload' to enable? Once enabled, is there an indication in the logs that compression is taking place? Do new mail files have a suffix like .Z to indicate they've been compressed? Any gotchas to be aware of? I'm running macports dovecot 2.2.5. Thanks for any help. -Terry Terry Barnum digital OutPost http://www.dop.com

1 0

[Dovecot] Syslog debug messages
by Bruno Galindro da Costa 05 Dec '13

05 Dec '13

Is there a way to configure dovecot to register in syslog the debug messages enabled by mail_debug=yes as a debug not as information? I need this because I want to send this kind of information only to a remote server. I've configured rsyslog to do this, but dovecot is registering those debug messages as information not as debug (as I expected). This is my rsyslog config: mail.info;\ mail.warning;\ mail.err;\ mail.crit -/var/log/mail.log mail.* :omrelp:192.168.2.162:2514 root@imap:~# dovecot -n # 1.2.9: /etc/dovecot/dovecot.conf Warning: fd limit 1024 is lower than what Dovecot can use under full load (more than 4748). Either grow the limit or change login_max_processes_count and max_mail_processes settings # OS: Linux 2.6.32-44-server x86_64 Ubuntu 10.04.4 LTS xfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps managesieve pop3s listen(default): *:143,[::]:143 listen(imap): *:143,[::]:143 listen(pop3): * listen(managesieve): * ssl_listen(default): *:993,[::]:993 ssl_listen(imap): *:993,[::]:993 ssl_listen(pop3): *:2221,[::]:2221 ssl_listen(managesieve): ssl_ca_file(default): /etc/dovecot/ca.crt ssl_ca_file(imap): /etc/dovecot/ca.crt ssl_ca_file(pop3): /etc/dovecot/ca.crt ssl_ca_file(managesieve): ssl_cert_file(default): /etc/dovecot/ufsc.br.crt ssl_cert_file(imap): /etc/dovecot/ufsc.br.crt ssl_cert_file(pop3): /etc/dovecot/ufsc.br.crt ssl_cert_file(managesieve): /etc/ssl/certs/dovecot.pem ssl_key_file(default): /etc/dovecot/ufsc.br.key ssl_key_file(imap): /etc/dovecot/ufsc.br.key ssl_key_file(pop3): /etc/dovecot/ufsc.br.key ssl_key_file(managesieve): /etc/ssl/private/dovecot.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_executable(managesieve): /usr/lib/dovecot/managesieve-login login_process_per_connection: no login_process_size: 256 login_max_processes_count: 900 login_max_connections: 1024 max_mail_processes: 2048 mail_max_userip_connections(default): 2048 mail_max_userip_connections(imap): 2048 mail_max_userip_connections(pop3): 10 mail_max_userip_connections(managesieve): 10 mail_privileged_group: mail mail_uid: vmail mail_gid: mail mail_location: maildir:/var/mail/mailboxes/%d/%n:INDEX=/var/mail/indexes/%d/%n mail_debug: yes fsync_disable(default): yes fsync_disable(imap): yes fsync_disable(pop3): yes fsync_disable(managesieve): no mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_executable(managesieve): /usr/lib/dovecot/managesieve mail_plugins(default): quota imap_quota autocreate trash zlib mail_plugins(imap): quota imap_quota autocreate trash zlib mail_plugins(pop3): quota mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve managesieve_implementation_string(default): dovecot managesieve_implementation_string(imap): dovecot managesieve_implementation_string(pop3): dovecot managesieve_implementation_string(managesieve): Cyrus timsieved v2.2.13 lda: fsync_disable: no postmaster_address: xxx(a)xxx.xxx.xx mail_plugins: quota sieve auth_socket_path: /var/run/dovecot/auth-master auth default: cache_size: 1024 cache_ttl: 600 cache_negative_ttl: 60 username_chars: abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@~ username_format: %Lu passdb: driver: checkpassword args: /etc/dovecot/passdb.v3.1.sh userdb: driver: prefetch userdb: driver: ldap args: /etc/dovecot/dovecot-ldap-userdb.conf socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 master: path: /var/run/dovecot/auth-master mode: 384 user: vmail group: mail plugin: quota_exceeded_message: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. quota: maildir:DefaultQuota quota_warning: storage=95%% /usr/local/bin/quota-warning.sh 95 %u quota_warning2: storage=90%% /usr/local/bin/quota-warning.sh 90 %u trash: /etc/dovecot/dovecot-trash.conf sieve: /var/mail/sieve_scripts/%d/%n/.dovecot.sieve sieve_dir: /var/mail/sieve_scripts/%d/%n/sieve mail_log_events: delete expunge copy mailbox_delete mail_log_fields: uid box msgid from subject size vsize autocreate: Spam autosubscribe: Spam -- Att. Bruno Galindro da Costa

2 1

[Dovecot] FTS solr issue with second private namespace
by Andreas Oster 05 Dec '13

05 Dec '13

Hi all, I have just stumbled over an issue with the FTS plugin of dovecot. For our primary namespace, INBOX and its subfolders, searching works without any problems but in a second namespace, which I have created for old/archived mails, searching of the mail text/body does not work. I can see the search request in the tomcat log but no result is created and the mail client (Thunderbird) keeps displaying the serach message without returning any results. Tomcat6 catalina.out: Dez 02, 2013 11:58:00 AM org.apache.solr.core.SolrCore execute INFO: [] webapp=/solr path=/select params={fl=uid,score&sort=uid+asc&q=body:"schiller"&fq=%2Bbox:1b4de60d0a6287522d0c00004424cda4+%2Buser:"testuser"&rows=2} hits=0 status=0 QTime=1 I am currently using Dovecot 2.2.9 and do not have any special FTS config: mail_plugins = quota mailbox_alias fts fts_solr plugin { fts = solr fts_solr = break-imap-search url=http://localhost:8080/solr/ fts_autoindex = yes } Thank you for your kind help best regards Andreas

1 2

[Dovecot] Full text search improvements
by Timo Sirainen 05 Dec '13

05 Dec '13

FTS indexing is something I hear quite often nowadays. I’ve added some hacks to make it work better for some installations, but it’s about time to think about the whole design and how it could be improved for everyone in future. Here are some of my initial thoughts. Currently Dovecot supports 3 full text search engines: Solr, CLucene and Dovecot Squat. CLucene plugin has various features built in, which should have been built in a generic way to work with all the engines (although Solr has most of those already built-in). Squat was abandoned a few years ago in favor of Solr/CLucene, but perhaps it could be brought back to life, since it looks like its index sizes could be smaller than Lucene's. Here's a list of things that should be added to generic Dovecot FTS code to improve all the backends: 1. Support for multiple languages. Use textcat while indexing to guess the language of the indexed data. (Perhaps run it separately for each paragraph to handle multi-language mails? Or at least many emails begin/end with different language than the text in the middle, e.g. "Foo Bar wrote:" is often in various languages.) Index the data using the detected language's stemming and other features. Keep track of which languages have been used in the index, and when searching stem the search words to all the used languages. Since each added language requires additional searches and there's the possibility of wrong detection, the list of allowed languages could be configurable. See also http://ntextcat.codeplex.com/ or at least change textcat to use UTF8. 2. Word stemming. This can be done for many languages with Snowball library. Solr has also implemented several other languages, perhaps its code can be somehow automatically translated to C(++) for use with Dovecot? 3. Don't index language-specific stopwords. We can get the word lists from e.g. Solr. 4. Try to detect compound words and index each part separately for languages that use them. http://wiki.apache.org/solr/LanguageAnalysis#Decompounding suggests two possible ways to do it. 5. Normalize words (e.g. drop diacritics). libicu can be used for this. 6. Drop (Unicode) characters that don't belong to the language? Or especially don't index most of the weird Unicode characters. This would avoid filling the index with unnecessary garbage. 7. Don't index non-text data? For example if there is large block of base64 data or something else that definitely doesn't look like text, it's pretty useless to index it. Then again, we do want to index all kinds of IDs that someone might want to search. This could be a bit difficult to implement well. 8. Index attachments separately, so it would be possible to search only attachments. (Should "SEARCH BODY word1 BODY word2" return matches if word1 and word2 are in different attachments?) 9. Attachments can be translated to indexable UTF-8 text already with fts_decoder setting by doing it via a conversion script. This could also support Apache Tika server directly. 10. It should be configurable which fields are indexed. Body and header would always be separately indexed. Optionally there could be also at least: attachments, From, To, Cc, Bcc and Subject. The From/To/Cc/Bcc could also be indexed together in one "addresses" field. The more fields there are, the larger the index, but better/faster search results. 11. Each indexed mail should have metadata: Mailbox GUID, mail UID and the language the mail was indexed with. For attachments there should also be the MIME part number. When matching results, drop results if returned language doesn't match the query language. Squat ----- Currently Squat index consists of a trie containing all the words and pointer to a file listing all the message UIDs that contain them. Each node in the trie has a pointer to the UIDs, so e.g. with "abc" the "a" node will contain UIDs of all mails that contain the "a" letter (e.g. 1,3-5,10). "ab" node will contain mails that have the "ab" substring. Since the "ab" is a subset of "a", the "ab" won't contain UIDs directly but instead it contains indexes to the "a" list to get a better compression (e.g. UID 3-5,10 -> 2-4 indexes in the "1,3-5,10" list). The "abc" node then similarly refers to the "ab" node's indexes. It's configurable how long words Squat will index. Also substring matching is configurable. By default both are 4 letters, so words longer than 4 letters will be split to 4 letter pieces which are indexed (e.g. "dovecot" -> "dove", "ovec", "veco", "ecot"). When searching these pieces are looked up and the results are merged. It's pretty pointless to do a search for 1-2 letter substrings. Most likely the user wants to find 1-2 letter word instead. Perhaps this is true also for 3 letters? The Squat index could be changed to only add results for the first 1-2 (or 1-3?) letters only for full words, not to word prefixes. This of course would mean that the "ab" referring to "a" UID list would no longer work for the first nodes. Substring searching likely wouldn't work very nicely for stemmed words. So Squat should probably index the full stemmed word and then also index the unstemmed word in the small 4 letter pieces. It should be possible to also disable substring searching entirely. Squat already attempts to reduce disk space by encoding the common characters with less bits than other characters. This is hardcoded for English language though. Each index compression could analyze the most common letters and dynamically use those. Perhaps different languages could even use separate Squat index files to get the most advantage of this? Although if there are a lot of languages it's a bit annoying to have many different index files. Squat currently indexes each Unicode character as one letter, so there can be quite a huge number of different 4 letter words. This substring indexing should probably be disabled for words that contain letters not used by the current language. The UID lists are not compressed very efficiently. See http://www2009.eprints.org/41/1/p401.pdf for alternatives. There is currently one Squat index per mailbox. There should be one per user. This requires adding a separate ID -> { mailbox GUID, UID, language, mime part } mapping file. Squat indexes should be updated in two parts. For old mails there are the large trie+uidlist base files in optimized format. For new mails there are small trie+uidlist files in a format where they are just appended to. When the trie or uidlist becomes large enough, it's first internally sorted/optimized, and then merged with the base files by recreating new optimized base files. This fixes the scalability problem with current large Squat indexes where with large unsorted indexes the sorting could have taken forever. Since substring searches in Squat produce only "maybe matches" UID lists, the mails still need to be opened and searched through. This step could probably also do the same language detection + stemming as is done during indexing to improve the search results. (And this feature could be enabled even if no FTS index is enabled.)

6 9

[Dovecot] Postfix & Dovecot Sieve Advice
by craig.dovecot＠noboost.org 05 Dec '13

05 Dec '13

Hi, I've configured up Dovecot Pigeonhole with Postfix for delivery (all working perfectly). Seeing as I'm about to use this in production, I'm just after any advice or tips people might have to safeguard email delivery from Postfix to Dovecot via LMTP? Spec: postfix-2.6.6-2.2.el6_1.x86_64 dovecot-2.0.9-5.el6.x86_64 dovecot-pigeonhole-2.0.9-5.el6.x86_64 Postfix: virtual_transport = lmtp:unix:private/dovecot-lmtp Dovecot (key settings): vim /etc/dovecot/conf.d/10-auth.conf auth_username_format = %Ln vim /etc/dovecot/conf.d/10-master.conf service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } vim /etc/dovecot/conf.d/20-lmtp.conf protocol lmtp { mail_plugins = $mail_plugins sieve } vim /etc/dovecot/conf.d/90-sieve.conf plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_vacation_min_period = 1h sieve_vacation_default_period = 10d sieve_vacation_max_period = 30d } Regards, Craig

1 0

[Dovecot] BUG: Authentication client sent unknown handshake command
by Emmanuel Dreyfus 04 Dec '13

04 Dec '13

Hi After upgrading the kernel, everything is fine, except dovecot authentication. I get this trange thing (data after REQUEST? changed just in case it contains anything sensitive): Nov 29 16:56:01 volanges dovecot: auth: Error: BUG: Authentication client sent unknown handshake command: REQUEST?6970356762?616?6?235264ef69dbd1665538af54d12fdaea?session_pid=453?req... Nov 29 16:56:01 volanges dovecot: imap: Error: Authentication server didn't send valid SPID as expected: MECH PLAIN plaintext Nov 29 16:56:01 volanges dovecot: imap: Error: Disconnected from auth server, aborting (client-pid=161 client-id=1) Nov 29 16:56:01 volanges dovecot: imap-login: Internal login failure (pid=161 id=1) (internal failure, 1 successful auths): user=<jdoe>, method=PLAIN, rip=192.0.2.251, lip=192.0.2.10, mpid=453, TLS, TLSv1 with cipher AES128-SHA (128/128 bits) Reverting to the previous kernel fixed the problem, but I have not been able to spot what the problem was. Any idea? -- Emmanuel Dreyfus manu(a)netbsd.org

3 5

[Dovecot] Ubuntu packages and broken link in docs
by Alan McGinlay - SICS 04 Dec '13

04 Dec '13

Hi, Two things: 1. Does anyone know of a good PPA with updated packages of Dovecot for Ubuntu? "For the latest Dovecot for latest Ubuntu LTS see Xiaoka APT Repository. (Outdated. Could someone provide a ppa for latest stable Dovecot, please?)" 2. On the http://wiki2.dovecot.org/PrebuiltBinaries#Ubuntu page there is a link to a supposedly outdated PPA, the ppa page now redirects to an ad for the guys services so should probably be removed. /Alan

2 1

[Dovecot] Different settings for SSL/non-SSL protocols
by Алексей Прокопчук 04 Dec '13

04 Dec '13

Good time of the day! It is possible to setup dovecot with different requirements for SSL and non-SSL protocols? What would I like to do: pop3/imap non-SSL = allowed plain text authentication. pop3/imap with SSL = allowed plain text authentication with required valid SSL certificates. I need to allow access from any IP address for first group of users, which have valid SSL certificates. And allow access from only local IP addresses for second group of users which doesn't have SSL certificates at all. I tried to setup dovecot for such task. SSL and non-SSL pop3/imap works together fine, but without require valid SSL certificates. If I set auth_ssl_require_client_cert = yes, non-SSL pop3/imap refuses authentication without valid SSL certificate. I tried to use different sections for protocol pop3 {} and protocol pop3s {} (and imap/imaps) but this seems doesn't work. Is there any way to solve this? Thanks for attention, with best regards, Alexey Prokopchuk (AP8686-RIPE)

1 2

[Dovecot] Complete migration from Cyrus on remote server
by Alan McGinlay - SICS 04 Dec '13

04 Dec '13

Hi All! I am working on a complete migration from an old solaris machine running Cyrus IMAP v2.3.14 and wish to migrate all users to a new ubuntu vm running Dovecot 2.0.19. What I have so far is a fully functional dovecot installation with LDAP / Dovecot SASL auth using Maildir++. My plan is to use Perdition IMAP proxy on a third host and migrate users a few at a time (see attached picture of my whiteboard). The proxy is in place and already used by webmail clients to connect to the old server, shortly I will change the dns to point smtp.domain.com and imap.domain.com etc to the proxy instead of the old server but I haven't finished testing yet. Once that is done it is a simple matter of changing the proxy configuration per user to make them hit the new server instead. The problem I am having with this is that I am using Imapsync (http://imapsync.lamiral.info) to perform the initial synchronization (i have scripted it to maintain sync every 10 minutes with the addtion of "--maxage 1" option) using the following command line options: /usr/local/bin/imapsync --noreleasecheck --useheader ALL --delete2 --expunge2 --tmpdir /var/vmail/cache --subscribe --subscribed --tls1 --tls2 --usecache --nofoldersizes --nocheckmessageexists --host1 oldserver.domain.com --authuser1 adminuser --user1 testuser(a)domain.com --authmech1 PLAIN --password1 <HIDDEN> --host2 newserver.domain.com --authuser2 adminuser --user2 testuser(a)domain.com --authmech2 PLAIN --password2 <HIDDEN> All email is copied over and stored in dovecot as you would expect. Then I switch the user to the new server on the imapproxy and restart the client (thunderbird or whatever). This is where all email headers are downloaded again as if they were new! There are *Many* users here who would complain intensely if this happened and I was really wanting the migration to be as invisible as possible. Is there a way to do what I have described without the clients redownloading headers / full messages? I have been working on this for some time now but have hit a roadblock and don't know where to turn, any help would be really appreciated! Notes: * The proxy is not used during imapsync. * Some users have 20 years worth of mail running into 10's of GB * I have looked at dsync but the docs state that the mailbox should not be accessed during sync, this is not possible for me (it doesn't say why this is the case however)

4 7

[Dovecot] file descriptor errors
by Christian Rößner 04 Dec '13

04 Dec '13

Hi, I use dovecot 2.2.9 on Gentoo Linux and I get lots of these erors in my log file: Dec 4 08:53:37 mx dovecot: auth: Error: close failed in file object destructor: Dec 4 08:53:37 mx dovecot: auth: Error: IOError: [Errno 9] Bad file descriptor What could cause these error messages? My mailboxes are stored on an ext4 filesystem: /dev/vdb1 on /srv/mail type ext4 (rw,noatime) doveconf -n: # OS: Linux 3.11.2-hardened x86_64 Gentoo Base System release 2.2 ext4 auth_master_user_separator = * auth_mechanisms = plain login auth_verbose = yes disable_plaintext_auth = no imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes mail_access_groups = mail mail_attachment_dir = /srv/mail/copymail/attachments mail_gid = mail mail_location = maildir:/srv/mail/virtual/%d/%n mail_plugins = quota acl zlib mail_log notify mail_privileged_group = mail mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk-E-Mail { auto = subscribe special_use = \Junk } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX. } passdb { args = /usr/local/sbin/ispcp-decrypt.py driver = checkpassword } plugin { acl = vfile acl_shared_dict = file:/srv/mail/virtual/shared-mailboxes.db quota = dict:User quota::file:/srv/mail/virtual/%d/%n/dovecot-quota quota_rule = *:storage=300M:messages=20000 quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 Mailbox is full quota_status_success = DUNNO quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster(a)deltaweb.de protocols = imap pop3 lmtp sieve service auth-worker { extra_groups = ssl-cert unix_listener auth-worker { mode = 0600 user = vmail } user = vmail } service auth { extra_groups = ssl-cert unix_listener /var/spool/postfix.relay/private/auth { group = postfix mode = 0660 user = postfix } unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { mode = 0660 user = vmail } user = vmail } service config { unix_listener config { mode = 0600 user = vmail } } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { inet_listener imaps { port = 993 ssl = yes } } service lmtp { inet_listener lmtp { address = ::1 port = 24 } } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } } service pop3-login { inet_listener pop3s { port = 995 ssl = yes } } service quota-status { client_limit = 1 executable = quota-status -p postfix inet_listener { port = 12340 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } ssl_cert = </etc/ssl/certs/mx_deltaweb_de.crt ssl_key = </etc/ssl/private/mx_deltaweb_de.key userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_plugins = quota acl zlib mail_log notify sieve } protocol lda { mail_plugins = quota acl zlib mail_log notify sieve } protocol imap { mail_max_userip_connections = 50 mail_plugins = quota acl zlib mail_log notify imap_quota imap_acl imap_zlib } Thanks in advance -Christian Rößner -- Bachelor of Science Informatik Erlenwiese 14, 36304 Alsfeld T: +49 6631 78823400, F: +49 6631 78823409, M: +49 176 93118939 USt-IdNr.: DE225643613, http://www.roessner-network-solutions.com

1 0