November 2011 - dovecot

[Dovecot] Fault tolerant architecture
by Marcelo Salhab Brogliato 30 Nov '11

30 Nov '11

Hi, I'm new to this list and want your help. I'm the mail admin for some domains in Rio de Janeiro - Brazil. Today we have only one machine running dovecot (imap+pop3) with local mail. We are migrating to two virtual machines in kvm running in separate hosts. Then we have two main problems: - How to share mail files to both dovecots? We've been thinking about NFS using local indexes. Is this a good approach? - How do we have a fault tolerant mail servers? Our first solutions is using two IP addresses in our DNS records. About NFS using local indexes, I'm configuring a test server. But how to configure local indexes when my mail_location comes from sql (userdb_home actually). I'm using dovecot 1.2.9. I guess you already had some of these problems or maybe in another architecture these neither exists. Thanks for any help, Marcelo

4 3

[Dovecot] Best setup for performance and fault tolerance
by Donny Brooks 30 Nov '11

30 Nov '11

Hello all. I am in need of some guidance. First a little background. Currently our mail server is on physical hardware (Dell server with 2x 2.8GHz Xeon w/ 4GB ram, raid5 array, single gigabit nic) running on Fedora 11 and postfix-2.5.6-3.fc11.x86_64 with dovecot-1.2.11-3.fc11.x86_64. Mailstore is via Maildir format that was converted from mbox about a year ago. This same machine is also our PDC with samba, Master LDAP, slave MySQL replication, primary DNS, and home server for about 20 users. We have approximately 200 end users that have mailboxes on the server ranging from 1KB to 20GB in size. Total mail store is currently at 300GB. About 75 of the users are currently POP access and their mail will be moved to the server soon and setup as IMAP. This is calculated to add roughly another 150GB of mail for a total of 450GB mail store. Being a state agency we have to keep the mail indefinitely for public record reasons. We use a mixture of Thunderbird as an IMAP client and SOGo for web access. Now to the problem: Recently we have been having super slow access to the mail server. Turns out the load was insanely high partially due to the samba home server portion, which is being moved off as we speak, and the other part is due to people searching their mail. Just yesterday one of our users nearly brought the entire agency to its knees by performing a search on her 8GB of mail via IMAP. Since the server is old in both hardware and software I have been tasked with moving it to newer hardware and a newer OS. We currently have 3 virtual servers running Xen and a SAN. The new setup will be placed in the virtual environment. I will probably run Fedora 16 as the OS but am open to Centos, Fedora, or Ubuntu. Now to the question: What is the best way to setup Dovecot so that it is tuned for performance and high available? We have been running with this single point of failure for years so as long as we are moving the mail server we might as well build in some redundancy. To solve the searching problem I thought of maybe setting up some type of indexing. I do kind of want to break the various services out on to separate virtual machines for a little more fault tolerance, but that is not totally necessary. What do you think of things like iRedmail? I see it's usefulness but the not being able to separate services kind of defeats the purpose, plus I want to setup a high available MySQL cluster and possibly OpenLDAP or 389 cluster so iRedMail may not be the best solution. Sorry for the long email but I am trying to get all the information out there at once so it will help get more directed responses in the shortest amount of time. I look forward to any and all input on this matter -- Donny B MDAH

2 1

[Dovecot] Indexes to MLC-SSD
by Patrick Westenberg 29 Nov '11

29 Nov '11

Hi all, is anyone on this list who dares/dared to store his index files on a MLC-SSD? Regards, Patrick

9 26

[Dovecot] 2.0.16 - auth with vpopmail won´t work anymore
by Werner Schöller 29 Nov '11

29 Nov '11

Hi, after upgrading from 2.0.15 to 2.0.16 authentication with vpopmail won´t work anymore. 2.0.x and 1.2.x works (this output is from 2.0.15): Nov 28 18:29:21 mail dovecot: auth: Debug: auth client connected (pid=3855) Nov 28 18:29:21 mail dovecot: auth: Debug: client in: AUTH 1 PLAIN service=imap secured lip=10.114.57.11 rip=10.114.57.11 lport=143 rport=37693 resp=d3NjaG9lbGxlckBodGwtcGVyZy5hYy5hdAB3c2Nob2VsbGVyQGh0bC1wZXJnLmFjLmF0AGRpYWJsbw== Nov 28 18:29:21 mail dovecot: auth-worker: Debug: vpopmail(wschoeller(a)htl-perg.ac.at,10.114.57.11): lookup user=wschoeller domain=htl-perg.ac.at Nov 28 18:29:21 mail dovecot: auth: Debug: client out: OK 1 user=wschoeller(a)htl-perg.ac.at Nov 28 18:29:21 mail dovecot: auth: Debug: master in: REQUEST 4001103873 3855 1 7a21a4d39cc6b187b3b14dc4cfbf077a Nov 28 18:29:21 mail dovecot: auth-worker: Debug: vpopmail(wschoeller(a)htl-perg.ac.at,10.114.57.11): lookup user=wschoeller domain=htl-perg.ac.at Nov 28 18:29:21 mail dovecot: auth: Debug: master out: USER 4001103873 wschoeller(a)htl-perg.ac.at uid=89 gid=89 home=/var/vpopmail/domains/htl-perg.ac.at/wschoeller Nov 28 18:29:21 mail dovecot: imap-login: Login: user=<wschoeller(a)htl-perg.ac.at>, method=PLAIN, rip=10.114.57.11, lip=10.114.57.11, mpid=3856, secured Nov 28 18:29:21 mail dovecot: imap(wschoeller(a)htl-perg.ac.at): Disconnected: Logged out bytes=29/405 Nov 28 18:29:21 mail dovecot: auth: Debug: auth client connected (pid=3858) 2.0.16 doesn´t work: Nov 28 18:43:15 mail dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/auth Nov 28 18:43:15 mail dovecot: auth: Debug: auth client connected (pid=27392) Nov 28 18:43:15 mail dovecot: auth: Debug: client in: AUTH 1 PLAIN service=imap secured lip=10.114.57.11 rip=10.114.57.11 lport=143 rport=42768 resp=d3NjaG9lbGxlckBodGwtcGVyZy5hYy5hdAB3c2Nob2VsbGVyQGh0bC1wZXJnLmFjLmF0AGRpYWJsbw== Nov 28 18:43:15 mail dovecot: auth-worker: Debug: Loading modules from directory: /usr/lib/dovecot/auth Nov 28 18:43:15 mail dovecot: auth-worker: Debug: vpopmail(wschoeller(a)htl-perg.ac.at,10.114.57.11): lookup user=wschoeller domain=htl-perg.ac.at Nov 28 18:43:15 mail dovecot: auth-worker: vpopmail(wschoeller(a)htl-perg.ac.at,10.114.57.11): Password mismatch Nov 28 18:43:15 mail dovecot: auth-worker: Error: md5_verify(wschoeller(a)htl-perg.ac.at): Not a valid MD5-CRYPT or PLAIN-MD5 password Nov 28 18:43:15 mail dovecot: auth-worker: Error: smd5_verify(wschoeller(a)htl-perg.ac.at): SMD5 password too short Nov 28 18:43:15 mail dovecot: auth-worker: Error: ssha_verify(wschoeller(a)htl-perg.ac.at): SSHA password too short Nov 28 18:43:15 mail dovecot: auth-worker: Error: ssha256_verify(wschoeller(a)htl-perg.ac.at): SSHA256 password too short Nov 28 18:43:15 mail dovecot: auth-worker: Error: ssha512_verify(wschoeller(a)htl-perg.ac.at): SSHA512 password too short Nov 28 18:43:15 mail dovecot: auth-worker: Warning: Invalid OTP data in passdb Nov 28 18:43:15 mail dovecot: auth-worker: Debug: vpopmail(wschoeller(a)htl-perg.ac.at,10.114.57.11): CLEARTEXT(mypassword) != '', try OTP scheme instead Nov 28 18:43:17 mail dovecot: auth: Debug: client out: FAIL 1 user=wschoeller(a)htl-perg.ac.at Nov 28 18:43:17 mail dovecot: imap-login: Aborted login (auth failed, 1 attempts): user=<wschoeller(a)htl-perg.ac.at>, method=PLAIN, rip=10.114.57.11, lip=10.114.57.11, secured Dovecot -n: # 2.0.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.36-gentoo-r8 i686 Gentoo Base System release 1.12.14 auth_debug = yes auth_debug_passwords = yes disable_plaintext_auth = no first_valid_uid = 89 listen = * mail_location = maildir:~/.maildir namespace { hidden = no inbox = yes list = yes location = prefix = INBOX. separator = . type = private } passdb { args = webmail=127.0.0.1 driver = vpopmail } service auth { user = root } ssl_cert = </etc/ssl/dovecot/server.pem ssl_key = </etc/ssl/dovecot/server.key userdb { driver = vpopmail } Just a wild guess - but is it possible that the reason could be found here: http://hg.dovecot.org/dovecot-2.0/rev/b99da8b1ecef Thanks for any suggestions. Best regards, Werner

1 0

[Dovecot] squat fts not working on 2.1rc1
by Metro Domain Admin 28 Nov '11

28 Nov '11

Hello, I tried to upgrade from 2.0.15 to 2.1.rc1 over the weekend but ran into some problems with squat. My two largest mailboxes were corrupted, producing this error: Nov 27 17:38:12 kerio1 dovecot[42860]: imap(bigmailbox): Error: Corrupted squat uidlist file /var/mail/metro-email.com/bigmailbox/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.search.uids: wrong indexid There is a good chance that I had shot my own foot, by forgetting to update postfix's dovecot-lda reference to the new instance. I had also enabled an inet based LMTP listener, although it is not being used yet. Even using non-corrupted indexes, it seemed that FTS was not being used. A basic search such as this one took many minutes to complete: doveadm search -u jsample mailbox INBOX body eps I found messages like this in mail.log after performing the search: Nov 27 18:21:39 kerio1 dovecot[42860]: indexer-worker(jsample): Indexed 0 messages in Drafts Nov 27 18:21:39 kerio1 dovecot[42860]: indexer-worker(jsample): Indexed 0 messages in INBOX When I rolled back to 2.0.15, the same search returned almost immediately. The larger mailboxes went quickly again, but only after they had a chance to rebuild. A few other notes about our configuration: We are on PPC, with the stats service disabled (because of kqueue issue discussed earlier). I made some minor changes in my config since the previous installed version (diff below). mail_location override is in place for all users via password file, e.g.: userdb_mail=mdbox:/var/mail/metro-email.com/jsample/mdbox Thanks, Tony kerio1:~ metroadmin$ /opt/dovecot-2.1.rc1/bin/doveconf -n -c /opt/dovecot-2.1.rc1/etc/dovecot/dovecot.conf # 2.1.rc1: /opt/dovecot-2.1.rc1/etc/dovecot/dovecot.conf # OS: Darwin 9.8.0 Power Macintosh auth_cache_negative_ttl = 0 auth_cache_size = 1 M auth_mechanisms = plain login lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes lmtp_proxy = yes mail_location = maildir:%h mail_plugins = fts fts_squat mail_log notify mail_privileged_group = mail maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave editheader mdbox_rotate_size = 10 M namespace { inbox = yes location = prefix = separator = / } passdb { args = scheme=CRYPT username_format=%u /opt/dovecot-2.1.rc1/etc/dovecot/users driver = passwd-file } plugin { fts = squat mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size from subject sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } postmaster_address = postmaster(a)metro-email.com protocols = imap lmtp service auth { unix_listener /var/spool/postfix/private/auth { group = _postfix mode = 0660 user = _postfix } unix_listener auth-userdb { group = mail mode = 0660 user = metroadmin } } service imap { executable = imap postlogin } service lmtp { inet_listener lmtp { address = 192.168.20.10 127.0.0.1 port = 24 } } service postlogin { executable = script-login rawlog -t } service stats { fifo_listener stats-mail { mode = 00 } } ssl_cert = </opt/local/apache2/conf/ssl_startcom/server.crt ssl_key = </opt/local/apache2/conf/ssl_startcom/server.key userdb { args = username_format=%u /opt/dovecot-2.1.rc1/etc/dovecot/users driver = passwd-file } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 16 } Here is a diff between the current/working and attempted configs: kerio1:~ metroadmin$ diff k1.curr k1.new2 1c1 < # 2.0.15: /opt/dovecot-2.0.15/etc/dovecot/dovecot.conf --- > # 2.1.rc1: /opt/dovecot-2.1.rc1/etc/dovecot/dovecot.conf 9a10 > lmtp_proxy = yes 15c16 < managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave --- > managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave editheader 24c25 < args = scheme=CRYPT username_format=%u /opt/dovecot-2.0.15/etc/dovecot/users --- > args = scheme=CRYPT username_format=%u /opt/dovecot-2.1.rc1/etc/dovecot/users 50a52,57 > service lmtp { > inet_listener lmtp { > address = 192.168.20.10 127.0.0.1 > port = 24 > } > } 53a61,65 > service stats { > fifo_listener stats-mail { > mode = 00 > } > } 57c69 < args = username_format=%u /opt/dovecot-2.0.15/etc/dovecot/users --- > args = username_format=%u /opt/dovecot-2.1.rc1/etc/dovecot/users kerio1:~ metroadmin$

1 0

[Dovecot] Which FTS to use!
by Henrique Santos Fernandes 27 Nov '11

27 Nov '11

Hello, i am currently using squat dovecot, but some users that have lots os emails ( 1GB ) are complaining about speed. Iam using dovecot 2.0.13 on debian 6 my squat conf is plugin { fts = squat fts_squat = partial=4 full=5 } I am thinking in maybe start to use Solr, so any one can help me with an decision? I am probably gonna update dovecot to 2.1 next year. and them maybe change the FTS plugin. This year i will get 10k disk only to store the dovecot index, rigth now they are saved on the same storage that the mails are and also change the "full" parameter to something like 10 or 15 I know i will use more disk space for it, but it will make a lot of differences right? it will match the message without the need of opening it for any search smaller than 10 character i guess. But this will be enough? or Solr or Lucene still better? Anyway, i am not current updating the indexes automatic, but i am planing on it, just not sure what is better, the command doveadm index or doveadm search... with one is better to do the task? Thanks! []'sf.rique

2 2

[Dovecot] Dovecot 1.2 ACL problem.
by Rajeev Sharma 26 Nov '11

26 Nov '11

Sir, I have migrate from courier-imap to dovecot without any problem but on courier in have shared folder which access on the basis on rights. I have manage these folders in dovecot but i have one problem all the shared folder is accessible by all user whether they have rights or not i have place dovecot-acl in shared folder and assign user my dovecot-acl is user=rajeev(a)teamsg.in tr. But all other user which is not listing on dovecot-acl also able to access these folder. Here is the output of dovecot -n. # 1.2.17: /usr/local/etc/dovecot.conf # OS: Linux 2.6.18-194.3.1.el5 i686 CentOS release 5.5 (Final) base_dir: /var/run/dovecot/ log_path: /var/log/dovecot.log ssl_listen: 192.168.0.251:993 ssl_cert_file: /var/qmail/control/servercert.pem ssl_key_file: /var/qmail/control/servercert.pem login_dir: /var/run/dovecot//login login_executable: /usr/local/libexec/dovecot/imap-login first_valid_uid: 508 last_valid_uid: 508 mail_location: maildir:~/Maildir namespace: type: private separator: . prefix: INBOX. inbox: yes list: yes subscriptions: yes namespace: type: shared separator: . prefix: INBOX.Shared. location: maildir:/home/vpopmail/domains/sgcricket.com/Maildir/Out:INDEX=~/Maildir/Sh… list: children auth default: user: vpopmail passdb: driver: vpopmail userdb: driver: vpopmail plugin: acl: vfile Please help whats wrong i am doing. Rajeev Sharma.

1 0

[Dovecot] Seen flag getting lost
by Edgar Fuß 25 Nov '11

25 Nov '11

We have two dovecot 1.2 instances sharing Maildirs on NFS. Indexes are local to the individual servers. Occasionally (no idea how to trigger this), the Seen flag gets lost on some messages. I've verified that actually the ``S'' is missing from the filename. I suspect something like server A caching the flags, server B setting Seen, and then server A flushing its cache for another change so overwriting what B changed. Any ideas short of switching to 2.0?

6 8

[Dovecot] v2.1.rc1 released
by Timo Sirainen 25 Nov '11

25 Nov '11

http://dovecot.org/releases/2.1/rc/dovecot-2.1.rc1.tar.gz http://dovecot.org/releases/2.1/rc/dovecot-2.1.rc1.tar.gz.sig I'll probably release v2.1.0 pretty soon, unless people report some new bugs. I know v2.1 is already being used to serve mails to tens (or hundreds?) of thousands of users, so it should be pretty stable. Most of the changes since v2.1.beta1 have been for rather small issues. Many of the fixed bugs exist also in v2.0, but since they are so minor I didn't bother backporting the fixes. As new features since v2.1.beta1 there are the things I did also for v2.0.16, and: * ssl_client_cert/key: Proxying can now send SSL certificate to backend server * ssl_crypto_device for enabling hardware encryption * SCRAM-SHA-1 authentication mechanism support by Florian Zeitz * passdb/userdb checkpassword: Export all auth %variables to AUTH_* environment. * maildir_broken_filename_sizes setting for people upgrading from other maildir servers that wrote tons of broken S=<size> values And as a reminder, here are the biggest new things since v2.0: * Plugins now use UTF-8 mailbox names rather than mUTF-7: acl, autocreate, expire, trash, virtual * auth_username_format default changed to %Lu. If you really want case sensitive usernames, set it back to empty. * Solr full text search backend changed to use mailbox GUIDs instead of mailbox names, requiring reindexing everything. solr_old backend can be used with old indexes to avoid reindexing, but it doesn't support some newer features. + imapc (= IMAP client) storage allows using a remote IMAP server to be used as storage. This allows using Dovecot as a smart (caching) proxy or using dsync to do migration from remote IMAP server. + Mailbox indexing via queuing indexer service (required for Lucene) + Lucene full text search (FTS) backend rewritten with support for different languages + FTS finally supports "OR" search operation + FTS supports indexing attachments via external programs + IMAP FUZZY extension, supported by Lucene and Solr FTS backends + Mailbox list indexes + Statistics tracking via stats service. Exported via doveadm stats. + Autocreate plugin creates/subscribes mailboxes physically only when the mailbox is opened for the first time. Mailbox listing shows the autocreated mailboxes even if they don't physically exist. + Password and user databases now support default_fields and override_fields settings to specify template defaults/overrides. - listescape plugin works perfectly now

6 8

[Dovecot] How to create home directories for virtual users?
by Olli Räisänen 24 Nov '11

24 Nov '11

Hello, I,m using Dovecot 1.2.9, now finishing a migration from Courier IMAP. Documentation (http://wiki.dovecot.org/VirtualUsers/Home) says that home directory should not be the same as mail directory. When I create new user accounts with PostfixAdmin the mail directory is not a problem but is there some smart way to create the home directory as well (locating it for instance in '/srv/vmail/%d/%n/home')? Regards, Olli

5 8