November 2011 - dovecot

Re: [Dovecot] Please advise on very fast search
by Alexander Chekalin 17 Nov '11

17 Nov '11

Thanks, Robert, will take a look at. What I'm afraid for is how database storage should be planned (storage, CPU, RAM, scaling when will be over-filled). When dealing with files (I'm using maildir), it is much easy to understand and to fix just about everything. Adding database involves tune it up too, and I'll have more points of 'tune it a bit' In fact work with Dovecot is pretty nice, but I think I can tune it to work faster. I now run it on FreeBSD (on UFS2), maybe I should change OS + FS, but need to test (really hope ZFS disks on SAS drives will help; still find no benchmarks on such a setup). Will also try to use full text search, but afraid of index size (and I need no search on body, just on headers). Anyway thank your for pointing me in right directions! Yours, Alexander

3 16

[Dovecot] Getting started with sieve and conversion from procmail
by Alex 17 Nov '11

17 Nov '11

Hi, I have a fedora15 system with dovecot-v2.0.15, dovecot-pigeonhole and and amavisd-new and would like to implement it on my system using postfix. I've read the basic wiki documentation, but I really can't figure out the proper starting point in my case. I've enabled "protocol lda" in 15-lda.conf and added "sieve" to the list of mail_plugins. I'm trying to adapt postfix to use deliver as its LDA for multiple virtual domains. I've added the following to master.cf: dovecot unix - n n - - pipe flags=DRhu user=dovecot:dovecot argv=/usr/libexec/dovecot/deliver -f ${sender} -d ${recipient} -a "${RECIPIENT}" Is that the proper format? Is it okay to use the dovecot user here, or should I create a new user? What permissions does this user need to possess? I've added the following to main.cf: dovecot_destination_recipient_limit = 1 virtual_transport = dovecot I have a number of procmail scripts and I've converted them to sieve, but they didn't convert very well. Can someone show me how to convert the following: :0 { :0 fhW | /home/alex/bin/process-mail.pl } The script skipped right over that. I'm also trying to convert the following: :0 * (RCVD_IN_DNSWL_HI|RCVD_IN_DNSWL_LOW|RCVD_IN_HOSTKARMA_W) * !USER_IN_WHITELIST * !BAYES_00 * !URIBL_ * !KHOP_RCVD_UNTRUST * !RCVD_IN_DNSWL_MED * !SPF_SOFTFAIL * SPF_HELO_PASS * SPF_PASS { LOG="DNSWL: " :0: whitelist/dnswl } The script only produced the following: if true { if true { if true { if true { if true { if true { if true { if true { if true { if true { fileinto "{"; fileinto " LOG="DNSWL: ""; fileinto " :0:"; fileinto " whitelist/dnswl"; fileinto "}"; } Any ideas greatly appreciated. Thanks, Alex

4 22

[Dovecot] v2.0.16 released
by Timo Sirainen 17 Nov '11

17 Nov '11

http://dovecot.org/releases/2.0/dovecot-2.0.16.tar.gz http://dovecot.org/releases/2.0/dovecot-2.0.16.tar.gz.sig I only now noticed that the VSZ limits weren't being enforced with earlier v2.0.x releases (or they were set 1024 times too high). So if a Dovecot process was leaking memory, it wasn't being killed by kernel. Now that this enforcing is done, some installations will probably start seeing errors about reaching these limits in normal operation. The default_vsz_limit is 256 MB. You may want to increase it in larger installations to 1 GB just in case. Besides the changes listed below, a lot of smaller fixes were done. v2.0.16 2011-11-17 Timo Sirainen <tss(a)iki.fi> * VSZ limits weren't being enforced for any processes. On server with large mailboxes you may now see errors about it if the limits aren't high enough. To fix them, either increase individual service { vsz_limit } values or simply increase the default_vsz_limit setting. * Proxying: If using ssl=yes or starttls=yes with a hostname (not IP) as proxy destination, require that the certificate matches the given hostname. * LMTP: Changed default client_limit to 1. This should improve LMTP throughput with default settings. * dsync: Quota is no longer enforced (i.e. dsync can't fail because user is over quota). + Added "auto" mail storage driver, which can be used to autodetect mailbox location and format. This behavior is already the default for empty mail_location setting, so this change is mainly useful for shared namespace's location setting. + checkpassword: Export all auth %variables to AUTH_* environment.

1 0

[Dovecot] Bus factor of dovecot / GIT
by Thomas Koch 16 Nov '11

16 Nov '11

Hi, somebody from the kolab groupware project recently explained me that the Bus- Factor[1] of around 1 would be one of their primary reasons not to use Dovecot and stick with Cyrus. What do you think about that? Is the bus factor much higher then 1? [1] http://en.wikipedia.org/wiki/Bus_factor Somehow related: Since the free software world[2] slowly converges towards GIT as the "one and only" VCS, have you thought about a switch? It's much more likely that somebody checks out your code, looks into it and provides patches if the VCS is already installed. [2] Android, Debian, Drupal, Eclipse, Fedora, Gnome, KDE, Linux Kernel, Perl, PHP (ongoing), PostgreSQL, Qt, Ruby on Rails, X.org (freedesktop.org) Best regards, Thomas Koch, http://www.koch.ro

6 6

[Dovecot] submission_host problem
by Bambero 16 Nov '11

16 Nov '11

Hello, I configured dovecot to use submission smtp host becouse of chroot. submission_host = 127.0.0.1 Unfortunatelly: Nov 12 05:11:15 myhost exim[23366]: 2011-11-12 05:11:15 SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "EHLO myhost" H=localhost [127.0.0.1] next input="MAIL FROM:<root@myhost>\\r\\n" Nov 12 05:11:15 myhost exim[23366]: 2011-11-12 05:11:15 SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "EHLO myhost" H=localhost [127.0.0.1] next input="MAIL FROM:<root@myhost>\\r\\n" Nov 12 05:11:15 myhost dovecot: lda(alias(a)apisoft.pl): Error: smtp(127.0.0.1): RCPT TO failed: 554 SMTP synchronization error Nov 12 05:11:15 myhost dovecot: lda(alias(a)apisoft.pl): Error: sieve: msgid=<E1RP4uv-00043S-L7@myhost>: failed to redirect message to <bambero(a)gmail.com> (refer to server log for more information) Does anyone know how to fix it ? Regards, Bambero

4 10

[Dovecot] Glued-together private namespaces
by Mark Moseley 16 Nov '11

16 Nov '11

I've been goofing with this all day with 2.0.15 and I'm starting to realize that either a) I'm not that smart, b) it's been so long since I messed with namespaces that I'm going about it completely wrong, or c) it's just not possible. I haven't posted 'doveconf -n' and other details, because mainly I'm just looking for 'yes, this is possible' or 'no, you're smoking crack' before posting further details. At this point, it's all maildir and moving to mdbox, while highly desirable in the future, is not possible in the near- to medium-term. I'm trying to glue a namespace underneath INBOX: namespace INBOX { type = private separator = . prefix = INBOX. # Yes, this used to be on Courier inbox = yes list = yes hidden = no subscriptions = yes location = maildir:~/Maildir } namespace archive { type = private separator = . prefix = INBOX.Archives. inbox = no list = children subscriptions = yes location = maildir:~/Maildir-Archive } I've tried putting namespace archive's 'prefix' as just "Archives", but Tbird doesn't seem to see this namespace, regardless of how much I futz with the imap settings in tbird. With the above setup, it actually seems to work correctly (provided ~/Maildir-Archive exists), though I'm sure a big gotcha is waiting in the wings. I can move messages around, create subfolders, subscribe to folders in ~/Maildir-Archive). The only thing I can't seem to get working is quotas. With my password_query like: password_query = ... CONCAT( '*:bytes=', "1M" ) AS 'userdb_quota_rule', \ CONCAT( '*:messages=10' ) AS 'userdb_quota_rule2', \ CONCAT( 'INBOX.Archives:bytes=+4900M' ) AS 'userdb_quota_rule3', \ CONCAT( 'INBOX.Archives:messages=+3900' ) AS 'userdb_quota_rule4' ... only the default quota seems to be in place for any subfolder of INBOX.Archives and for INBOX.Archives itself, i.e. *:bytes still applies to INBOX.Archives. The debug log show that: Debug: Quota root: name=User quota backend=maildir args= Debug: Quota rule: root=User quota mailbox=* bytes=1048576 messages=0 Debug: Quota rule: root=User quota mailbox=* bytes=1048576 messages=10 Debug: Quota rule: root=User quota mailbox=INBOX.Archives bytes=+5138022400 messages=0 Debug: Quota rule: root=User quota mailbox=INBOX.Archives bytes=+5138022400 messages=+3900 These are wildly stupid quotas but they're just there to test. With INBOX already at capacity (byte-wise; only set to a meg), copying large messages inside INBOX.Archives fails (only copying a 800k message but the quota should be 5gig now). Again, before I post configs, I'm just curious if what I'm trying to do isn't remotely possible, or that I'm approaching this entirely wrongly. Thanks!

2 6

Re: [Dovecot] Seg fault in dovecot/auth 2.0.15
by Timo Sirainen 16 Nov '11

16 Nov '11

Your message was discarded by mailman, probably because it was html-only. Anyway: On Wed, 2011-11-16 at 13:15 -0500, Mark Goldey wrote: > (gdb) p *auth_request->userdb->userdb->iface > $2 = {name = 0x807023e "vpopmail", preinit = 0x8065d60 > <vpopmail_preinit>, init = 0, deinit = 0, > lookup = 0x8066010 <vpopmail_lookup>, iterate_init = 0, iterate_next > = 0, iterate_deinit = 0} OK, so "auth_request->userdb->userdb->iface->lookup" should call vpopmail_lookup(), it's not NULL as this says: > > (gdb) bt full > > #0 0x00000000 in ?? () So something's going strangely wrong. Maybe gcc optimizations cause this and the real problem happens later. Try compiling auth process with optimizations: - Remove -O2 from src/auth/Makefile - cd src/auth ; make clean ; make ; make install Is the backtrace different?

1 0

[Dovecot] Missing public folder
by komodo 16 Nov '11

16 Nov '11

Hi i am using dovecot 2.0.13 and i can't see public folder in my mail client. But what is strange, it was working after instalation, and i don't know when it dissapeared. Maybe after some upgrade ? When i enable debug, then i can see in my log that folder exists and no errors. Please where should i find the problem ? Thanks Martin Here is log Nov 7 11:00:27 OVZ dovecot: imap(komodo): Debug: Namespace : type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir Nov 7 11:00:27 OVZ dovecot: imap(komodo): Debug: maildir++: root=/home/komodo/Maildir, index=, control=, inbox=/home/komodo/Maildir Nov 7 11:00:27 OVZ dovecot: imap(komodo): Debug: Namespace : type=public, prefix=spam_learner., sep=., inbox=no, hidden=no, list=yes, subscriptions=no location=maildir:/home/shared/Maildir Nov 7 11:00:27 OVZ dovecot: imap(komodo): Debug: maildir++: root=/home/shared/Maildir, index=, control=, inbox= Here is my conf # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-042stab037.1 x86_64 CentOS release 5.5 (Final) default_client_limit = 1300 disable_plaintext_auth = no hostname = somehost.com mail_location = maildir:~/Maildir managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mbox_write_locks = fcntl namespace { inbox = yes location = prefix = INBOX. separator = . type = private } namespace { location = maildir:/home/shared/Maildir prefix = spam_learner. separator = . subscriptions = no type = public } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { sieve = ~/.dovecot.sieve sieve_before = /etc/dovecot/sieve/before.d/ sieve_dir = ~/sieve } protocols = imap pop3 sieve sendmail_path = /usr/sbin/sendmail.exim service imap-login { process_limit = 1000 process_min_avail = 30 } ssl_cert = </etc/pki/dovecot/certs/host.pem ssl_key = </etc/pki/dovecot/private/host.pem userdb { driver = prefetch } protocol lda { mail_plugins = " sieve" } protocol imap { mail_max_userip_connections = 100 }

2 1

Re: [Dovecot] Please advise on very fast search
by Stan Hoeppner 16 Nov '11

16 Nov '11

On 11/16/2011 12:15 AM, Alexander Chekalin wrote: > Hello, Stan, > >> This is why I recommended mbox in the first place. If your only writes >> to these mailbox files are appends of new messages, mbox is the best >> format by far. It's faster at appending than any other format, and it's >> faster for searching than any other. > > I now seriously consider to use mdbox due to its nice self-regulation. > After all it I believe mdbox should do file compression on its own, no > cron scripts required. mbox and mdbox each has strengths and weaknesses. mbox will compress with a higher ratio than mdbox. You already have a nightly script that moves all mail from the day into a new file. Piping that through gzip or bzip2 is a no brainer. It'll add one line to your existing script, if that. Dovecot will decompress the file transparently when you access it via IMAP. And again since it's a single file searching it is much faster. With mbox you will have a single file for each day of emails. This seems ideal for archive purposes, one file per day. mdbox does fully transparent de/compression which is nice. The downside is that Dovecot does dbox compression on a per email basis, not a per file basis. So your compression ratio will be much less than with mbox, especially with bzip2 which works best on files over 900KB in size. Most emails are less than 8KB. Using mdbox will yield multiple files per day of emails instead of just one. Either format is much better than maildir for archiving. >> It's an archive. You're not going to use maildir so you don't need >> random IOPS performance. Thus RAID5/6 are a much better fit for an >> archive as you get better read performance, with more than adequate >> write performance, and you use less disks. And as this is an archive, >> you don't need real time automatic/transparent compression. Thus I >> recommend something like: >> >> 1. Debian 6 w/linux-image-2.6.39-bpo.2-amd64 or a custom rolled >> 2.6.39 or later kernel >> 2. hardware RAID5 w/large (2TB) SATA disks, 512B native sectors >> e.g. MegaRAID SAS 9261-8i, 4 Seagate Constellation ES ST2000NM0011 >> Specify a strip size of 256KB for the array >> Perma set /sys/block/sdX/read_ahead_kb to 512 so you're reading >> ahead 1024 sectors at a time instead of the default of 256. This >> will speed up your searches quite a bit. >> 3. XFS filesystem on the RAID device, created with mkfs.xfs defaults >> 4. mbox w/zlib plugin. Compress daily files each night with a script >> 5. You don't need LVM with a good RAID card (or with mdraid). This >> controller can expand the RAID5 up to 8 drives (up to 32 drives max >> using SAS expanders) > > We are considering to get HP DL180G6 server for 8 or 14 drives bays The P410 tops out at 8 drives, so get the 8 drive model. Start with 4 x 2TB drives in RAID5. Add 4 more drives when you need the capacity, and when drive prices are back down to normal (see below). http://h18004.www1.hp.com/products/quickspecs/13248_na/13248_na.html > (base model price is somewhat equal, but additional drives adds up cost) Especially right now in 2011. Flooding in Thailand, where 25% of the world's drives are produced, has doubled the cost of all hard drives worldwide. Now is a horrible time to buy spinning drives. I've read it may be 12 months before prices start coming back down... > with HP Smart Array P410 RAID controller (some servers are equipped with > this controller by default) with 256 Mb battery-backed cache, but I'll > check your suggestions! The P410 should be fine for a dedicated archive server. > What memory size should I plan in the server? You're talking about AMD64 > OS image, and 64-bit software are like to consume more memory that > 32-bit, so looks like your talking about pretty huge RAM, and I don't > believe it's necessary, or maybe I'm wrong? The memory footprint of 64bit binaries is nothing to worry about. The additional amount consumed is more than offset by the performance gained with direct access to RAM above 4GB compared to the performance of PAE. Keep in mind that 90% of your memory will be eaten by Linux buffer cache. Your binaries will account for less than 5% of your RAM consumption. If I understand correctly how you will use this archive server, then 8GB should be plenty. 8GB is standard on the 8 drive DL180 G6. http://h18004.www1.hp.com/products/quickspecs/13248_na/13248_na.html > Problem is I have no experience with XFS and not sure I can tune it in > the best way, so I'll go with mkfs.xfs defaults, I think. With only 4 drives and using a P410 w/cache and RAID5, doing manual XFS tuning isn't necessary for good performance, especially for an archive application which is data heavy, not metadata heavy. Setting sunit/swidth to match the RAID5 layout may increase performance slightly due to stripe aligned writes, but not enough that I'd worry about it. Just use the mkfs.xfs defaults. If you get the BBWC for the P410, enable the controller write cache, and mount XFS with 'nobarrier'. This will increase write performance quite a bit as fsyncs will complete instantly. > Hope we'll see Dovecot 2.1.x stable soon, as I'd like to use fts plugins > and 2.1 handle that much better, but I don't like the idea of use > unstable in production. Me neither. Speaking of archive/search, did you take look at Enkive yet? http://www.enkive.org/ > Thank you for taking your time on my case, You're welcome Alexander. -- Stan P.S. You may wish to implement dnswl.org ;)

1 0

[Dovecot] Chroot
by Bambero 16 Nov '11

16 Nov '11

Hello, I have successfully setup dovecot with chroot but there are two things yet 1. submission_host problem (I posted message 2 days ago) 2. Problem with acl's and sharing: all maildirs are located in: /var/mail/jdoe(a)example.com/Maildir so: mail_chroot=/var/mail I setup acls with: 10-mail.conf ... namespace { type = private separator = / prefix =. inbox = yes hidden = no list = yes subscriptions = yes } namespace { type = shared separator = / prefix = shared/%%u/ location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u subscriptions = yes list = yes } ... 90-acl.conf ... plugin { acl = vfile:cache_secs=5 } plugin { acl_shared_dict = file:/%i.db } ... Setting acl is no problem there are dovecot-acl dovecot-acl-list in Maildir. and dictionary /var/mail/xxx.db contains list od shares I can also see shared folder in my IMAP client but when trying to subscribe i got error: Error: userdb lookup: connect(/var/dovecot/auth-userdb) failed: No such file or directory Error: Namespace 'shared/': Could not lookup home for user jdoe(a)example.com /var/dovecot/auth-userdb is not accessible because chroot. But why userdb is necessary in this case ? I hope there is a solution without disabling chroot. Below my basic configuration. Regards, Bambero # 2.0.13: /etc/dovecot/dovecot.conf # OS: OpenBSD 5.0 i386 auth_mechanisms = plain login digest-md5 cram-md5 apop default_internal_user = _dovecot default_login_user = _dovenull first_valid_gid = 1000 first_valid_uid = 1000 last_valid_gid = 10000 last_valid_uid = 10000 mail_chroot = /var/mail mail_location = maildir:~/Maildir managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date imapflags notify mbox_write_locks = fcntl mmap_disable = yes passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { autocreate = Trash autocreate2 = Spam autocreate3 = Sent autocreate4 = Drafts autosubscribe = Trash autosubscribe2 = Spam autosubscribe3 = Sent autosubscribe4 = Drafts quota = maildir:User quota sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +notify +imapflags sieve_global_path = /etc/sieve/default.sieve } protocols = imap pop3 sieve sieve service auth { unix_listener auth-client { mode = 0660 user = _exim } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } ssl_ca = /etc/ssl/ca-bundle.crt ssl_cert = </etc/ssl/mycert.crt ssl_key = </etc/ssl/private/mycert.key submission_host = 127.0.0.1 userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } valid_chroot_dirs = /var/mail protocol lda { mail_plugins = sieve quota autocreate } protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags mail_plugins = quota imap_quota autocreate } protocol pop3 { mail_plugins = quota imap_quota autocreate pop3_client_workarounds = outlook-no-nuls oe-ns-eoh }

2 5