March 2010 - dovecot - dovecot.org

[Dovecot] 2.0.beta4, Panic: stream doesn't support seeking backwards
by Juergen Daubert 04 Jun '10

04 Jun '10

Hi, got the following with beta4 while uploading messages from a local mbox to the dovecot server in my LAN, tested with mutt and sylpheed, the same action works with 1.2.11: Mar 29 13:42:46 [dovecot] imap(juergen): Panic: stream doesn't support seeking backwards Mar 29 13:42:46 [dovecot] imap(juergen): Raw backtrace: /usr/lib/dovecot/libdovecot.so.0 [0xb76b7ec4] -> /usr/lib/dovecot/libdovecot.so.0 [0xb76b7f2c] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0xb76b8193] -> /usr/lib/dovecot/libdovecot.so.0 [0xb76c02cd] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_seek+0xbb) [0xb76be1db] -> /usr/lib/dovecot/libdovecot.so.0 [0xb76c08a9] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x6e) [0xb76be27e] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x40) [0xb76be570] -> /usr/lib/dovecot/libdovecot.so.0(message_parse_header_next+0x6d) [0xb76a7b6d] -> /usr/lib/dovecot/libdovecot.so.0 [0xb76a4302] -> /usr/lib/dovecot/libdovecot.so.0 [0xb76a498f] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x6e) [0xb76be27e] -> /usr/lib/dovecot/libdovecot.so.0 [0xb76a3f61] -> /usr/lib/dovecot/libdovecot.so.0 [0xb76a4072] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_seek+0xbb) [0xb76be1db] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x52) [0xb76be262] -> /usr/lib/dovecot/libdovecot.so.0 [0xb76c246d] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x6e) [0xb76be27e] -> /usr/lib/dovecot/libdovecot-storage.so.0(mbox_save_continue+0x4c) [0xb776ca6c] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_save_continue+0x14) [0xb7708964] -> dovecot/imap [0x804e32f] -> dovecot/imap [0x804ed31] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd1) [0xb76c38f1] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x29) [0xb76c29f9] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x25) [0xb76b1c45] -> dovecot/imap(main+0x1b3) [0x805da13] -> /lib/libc.so.6(__libc_start_main+0xdb) [0xb7559a3b] -> dovecot/imap [0x804e161] Mar 29 13:42:46 [dovecot] master: service(imap): child 21107 killed with signal 6 (core dumps disabled) # dovecot -n # 2.0.beta4: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32.10 i686 listen = * login_access_sockets = tcpwrap passdb { driver = shadow } service tcpwrap { unix_listener login/tcpwrap { mode = 0666 } } ssl_cert = </etc/ssl/certs/dovecot.crt ssl_key = </etc/ssl/keys/dovecot.key userdb { driver = passwd } Greetings Juergen -- Juergen Daubert | mailto:jue@jue.li Korb, Germany | http://jue.li/crux

3 16

[Dovecot] userdb prefetch not compatible with ssl_require_client_cert
by zhong ming wu 04 Jun '10

04 Jun '10

Dear List I looked into docs but couldn't see any where that says of this incompatibility. Anywhere this is confirmed in the doc? Thanks mr.wu

2 3

[Dovecot] v2.0.beta4: conversion of v1.2 configuration
by Pascal Volk 04 Jun '10

04 Jun '10

Hi Timo, I've just tried to convert my v1.2 configuration to a v2.0 configuration. Below a few obsolete warnings, among others "mail_executable has been replaced by service { executable }", was an interesting error: "doveconf: Error: duplicate listener: 198:" The broken parts of the converted configuration: # 2.0.beta4 (84ffb38839b9): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-trunk-amd64 x86_64 Debian squeeze/sid listen = 192.168.111.222 [2a01 service imap-login { inet_listener imap { port = 198 } } service managesieve-login { inet_listener managesieve { port = 198 } } service managesieve { executable = /usr/lib/dovecot/rawlog /usr/lib/dovecot/managesieve } service pop3-login { inet_listener pop3 { port = 198 } } # missing section: #protocol lda { # … #} # missing setting #postmaster_address The corresponding parts from v1.2 configuration: protocols = imap imaps managesieve pop3 pop3s listen = 192.168.111.222 [2a01:198:37a:1::1] protocol managesieve { mail_executable = /usr/lib/dovecot/rawlog /usr/lib/dovecot/managesieve } protocol lda { postmaster_address = f007ba11.1008605(a)localdomain.org mail_plugins = sieve } Why is the lda section missing? Because it is not listed in v1.2's protocols setting? (only an idea/wild guess) Regards, Pascal -- The trapper recommends today: deadbeef.1008606(a)localdomain.org

3 3

[Dovecot] RTFM: Manual pages for Dovecot v2.0
by Pascal Volk 01 Jun '10

01 Jun '10

Hello Dovecot admins. I've started writing manual pages for Dovecot v2.0. I'm sure they are not perfect yet. I'd like to ask you all to read the manual pages. It would be nice to get some ideas/suggestions/improvements. So that that the Dovecot v2.0.0 release could be distributed with /fine manuals/. The manual pages are available at: http://wiki2.dovecot.org/ManPages This is also the intended place for 'discussions' about the man pages. Regards, Pascal -- The trapper recommends today: cafebabe.1007713(a)localdomain.org

2 3

[Dovecot] Overly long email of miscellaneous Dovecot migration questions
by Mark Moseley 27 May '10

27 May '10

Hello to the list! I've been asked to spec out the feasibility and, if feasible, plan a migration from Courier to Dovecot for both POP3 and IMAP for about 4 million mailboxes. I've been trying to absorb all the dovecot-related info I could over the past couple of weeks from the docs and from the list and I'm sure that I've muddled at least some of it, so apologies in advance if I make grossly incorrect assumptions. BTW, sorry for the giant email but I've got a bunch of questions. This is the sort of overly wordy email that makes my eyes glaze over in mailing lists. But I've got things configured to where (I think) everything will 'just work' if we migrate (barring running POP3 mailbox conversions), so I'm not looking for configuration help, just gotchas and assumption-checking mostly. Me breaking 4 million mailboxes isn't really a good move for my job security ;) * Since Dovecot 2.0 seems like it's just around the corner, that's all I've been testing, and indeed all I've even looked at. * This is a pretty fantastic piece of software. I love the neat little details that make admins' lives easier, e.g. how easy it is to use multiple SSL certs in the same server. That made me smile. I hadn't looked at dovecot in a number of years, so I was really impressed at the full feature set. It's been a while since I've played with a new server application that I've enjoyed this much. * Background: All of our mail is stored on NFS and will be for the foreseeable future, all in maildir format. Courier POP3/IMAP runs on load balanced Linux servers, so clients will be hitting multiple servers, though stickiness based on client IP address is currently done. Webmail as well is done on top of a local Courier IMAP server accessing the same NFS servers but on an entirely different pool of servers. All authentication is done against SQL. Courier has been very, very good to us over the past 9 years, but our NFS servers (12 Netapps) are unfortunately being beaten to death. We've got lots of users with thousands of emails in their inbox (i.e. in ./Maildir/cur/ alone) and I've seen plenty of mailboxes with 10's of thousands in their inbox and I've even seen some really scary ones approaching 100k emails just in cur/. Thanks to an endless cycle of industry raising mailbox limits, we actually have to support this (though not so much the 100k message ones, at least). * Our #1 main motivation for looking Dovecot is relief for our currently overtaxed NFS servers, mostly in the form of the index files. Benchmarking dovecot looks great, even with the index files in the maildir. I know the cool thing to do would be to move the index files to a separate NFS server running entirely on SSDs for the fastest accesses possible. Building a separate index NFS cluster obviously adds complexity and performance with the index files in the maildir seems pretty great already. But I also know that real life often makes assumptions made during benchmarks look very foolish. My inclination is to try things out with the indexes in the maildir directory (and then start looking at moving to a dedicated NFS server for the indexes in a subsequent phase) but I don't want to regret that choice later. Anybody with large-scale NFS maildir stores have any advice either way? * Exim: We currently deliver all of our mail via Exim on separate servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail servers delivering to maildirs only do Exim. From what I've seen in the docs and various threads, from what I can gather, the best thing to do in that case would be to use Exim's built-in maildir handling, instead of using 'deliver'. That would be my preference anyway, but I wanted to make sure I didn't misinterpret things. * Any problems running Courier POP3 and Dovecot IMAP for a while, possibly Courier IMAP and Dovecot IMAP concurrently?: Since migrating POP mailboxes is going to be a mighty task, our gameplan would probably be to start migrating *just* IMAP services at first and likely on just a few servers at a time. Being load balanced, there are several scenarios where a user could theoretically access a mailbox on different servers and therefore end up hitting dovecot at one point in time and then later hitting courier or vice versa. Or they could use IMAP from one location and POP3 (during the span of time before we migrated our POP3 services) from another. I've not yet seen any side effects from switching mail servers like that (though I'm aware of what would happen if both dovecot and courier *POP3* were in service at the same time -- redownloaded mail, bleh). Has anyone seen otherwise? I'm not worried as much by the 0.1% of IMAP users who might be using advanced features but rather the other 99.9% who are using it pretty vanilla-ishly. * Union mailboxes: I'm pretty sure in a fairly recent thread that Timo said that something like a 'union' mailbox (at least with maildir) wasn't possible. I tried messing with multiple 'private' namespaces (i.e. a namespace called "ARCHIVE" with a "location" different than the INBOX location, ideally placed on slower but denser NFS servers) but even with 'hidden=no' and 'list=yes', only the main INBOX folders would show up, so I'm guessing that's not going to work. That would be a killer feature, to be able to serve an alternate namespace that would show up in a mail client's subscribable list that could be on slower storage than the main inbox (though I'm not sure a mail client can even handle multiple namespaces). * Any problems with keeping only quota limits in db and not current quota numbers? Our limits come out of a SQL table but the current counts just live in the maildir file. Trying to update quota counts in SQL for 4 million mailboxes is a non-starter for us at this point. >From what I've done, dovecot seems pretty happy to maintain the quota using the 'maildir' quota plugin but let it be overridden by the userdb lookup. So I think I'm fine on that count, but I'd love to hear from anyone knowing of extreme gotchas doing it that way. * Any problem with leaving the namespace in "Courier compatibility" mode? I.e. in namespace 'private', leaving "prefix = INBOX.". With 4 million mailboxes, FAQs all over the place, and support reps trained in a particular way of doing things in IMAP, it'd be hellish to try to change the prefix (I know I could leave the courier namespace around with 'hidden=yes' but retraining support staff is perhaps better left to phase #2). Do I lose anything besides tidiness by not changing it to "prefix =" as if I was deploying dovecot from scratch? Does it hurt performance in any significant way? Benchmarking doesn't look any different, so I'm guessing not. * One thing that threw me and might be good for a FAQ (unless it was just me misconfiguring things) was when I started playing with putting the index files in an alternate location. I was utterly perplexed why it'd create the directories for the indexes but they'd be empty. Based on their location and names when they're in the maildir, I was just looking for the same dovecot.index* files right in the alternate directory. It wasn't till I started strace'ing that I noticed that the index files were indeed getting created but in a subdirectory called .INBOX (and with me just doing 'ls'). * The courier-dovecot-migrate.pl does a fine job at converting our POP3 mailboxes. We'll definitely be doing a mass run of it across all of our mailboxes. But with this many mailboxes, there's no way to get around the time lag between the script getting run and new mail showing up. Which means that there'll be some number of new messages that will re-download in POP3 when the switchover to Dovecot POP3 is done. To keep from getting too many complaints from customers, I'm thinking that I might have to write a wrapper script to do something like compare the mtime of ./Maildir/new/ against dovecot-uidlist and overwrite if ./Maildir/new/ is newer or perhaps see if the mtime of courierpop3dsizelist is newer than dovecot-uidlist and only if that's the case execute the conversion script with --overwrite. Considering we get between 500-2000 POP3 logins/sec, running any script after a POP3 login terrifies me--even a 6 line bash script. Anybody have any opinions to share on that? With a pooled architecture, it's near impossible to only have some mailboxes hit Courier and some hit Dovecot and keep slowly incrementing. At best my available units of increments to migrate are a few thousand at a time and more realistically 10-20k at a time. If we're going to have to live with users complaining about a one-time redownload of just post-conversion mail, I'll need to get started convincing the higher-ups that that's life. A big thanks to anyone who even actually reads this entire tedious email and a tremendous thanks to anyone who actually replies to any of it!

6 15

[Dovecot] Busy looping of 2.0Beta4 on NetBSD-current
by Scott Ellis 26 May '10

26 May '10

I built and install Dovecot 2.0Beta4 on my NetBSD/amd64-current system, and am seeing the user-owned imap processes busy-looping. In Dovecot 1.2.x (and 1.x and 0.x..) I force ionotify and ioloop to kqueue, so I suspected that was the problem. Turns out it's not: Using poll or none also causes the same problem. This did not happen in 1.2.8 (didn't try anything later than that). After a quick visit to gdb, I added some i_debug() statements in ioloop-kqueue.c (when I had that as the notify and loop method) which showed that the loop was in io_loop_handler_run() . I haven't done any similar debugging for the other ioloop methods, but a quick look at the 2.0 changelog for those files doesn't raise any immediate alarms. Any ideas what to look for here? Timo, any ideas? 1.2.x worked perfectly so it's something "new" in 2.0. ScottE

3 5

[Dovecot] quota and lazy_expunge plugins both used: quotas go wrong with lazy_expunge'd mails
by Baptiste Malguy 26 May '10

26 May '10

Hello, I may have missed a point, but I could have found an issue between plugins quota and lazy_expunge. I have noticed that the "lazy_expunge"d mails are counted as part of the quota, while the documentation says it should not be. I have first tried to find out if this was a configuration mistake somewhere. Notably, you can notice that the expunged mails are in directory ~/expunged while the usual mail is in ~/Maildir : expunged mails are not in a subdirectory of ~/Maildir (this was just to make sure there were no side effect with this). I have deleted some expunged mails to be sure : at its next cycle, the file ~/Maildir/maildirsize contained expected value. So yes, I do know expunged mails in the expunged directory are counted (size, number of mails). So I've taken a look at the source code, adding some i_info(...) at different places of the quota and lazy_expunge plugins. >From the source code reading session, I understand that : 1. In src/lazy-expunge/lazy-expunge-plugin.c 1.1. lazy_expunge_mail_storage_init() sets the expunged mails namespace flag with NAMESPACE_FLAG_NOQUOTA 1.2. lazy_expunge_mail_storage_init() is called by lazy_expunge_mail_storage_created() 1.3. lazy_expunge_mail_storage_created() is part of the callback list hook_mail_storage_create 2. In src/plugins/quota/quota-storage.c: 2.1. The only place in the whole dovecot source code where NAMESPACE_FLAG_NOQUOTA is checked is in function quota_mailbox_list_created() 3. In src/plugins/quota/quota-plugin.c: 3.1 quota_plugin_init() add quota_mailbox_list_created() to the callback list hook_mailbox_list_created. >From my observations : - Callback functions in quota_mailbox_list_created are called _before_ callback functions in hook_mail_storage_create. - This observation led me to move the piece of code that sets NAMESPACE_FLAG_NOQUOTA from lazy_expunge_mail_storage_init() to lazy_expunge_mail_storage_created(). - But there, lazy_expunge_mail_storage_created() is called after quota_mail_storage_created() because names of the library files are lib10_quota_plugin.so and lib02_lazy_expunge_plugin.so. - There I have also learnt that callback functions of the same list are called in the reverse order compared to the library filenames. So see if my observations were right, I have renamed lib10_quota_plugin.so to lib01_quota_plugin.so, then it worked. But I doubt this is the right way to fix it. This also reverses the order other callbacks between the two plugins are called, which migh not be expected at all. I'm not event sure moving some code from lazy_expunge_mail_storage_init() to lazy_expunge_mail_storage_created() was right : I've improvized the reading the source code and do not understand the whole logic, so I suppose this could break something else. Timo, what do you propose ? Thanks for your great software. -- Baptiste MALGUY PGP fingerprint: 49B0 4F6E 4AA8 B149 B2DF 9267 0F65 6C1C C473 6EC2

2 2

[Dovecot] ACLs and public folders
by hyperbatus＠gmx.de 07 May '10

07 May '10

Dear list, I am using dovecot 1.0.15 on Debian Lenny. I have a public folder, I use ACL / vfile (without public ACL), and I use maildir / vmail. Now I have two questions: 1) http://wiki.dovecot.org/ACL states: "Mailboxes in public namespaces don't have owners, so by default no-one can access them." The same document, when explaining the meaning of the k flag in the ACLs, states: "Mailboxes can be created (or renamed) under this mailbox (there is no recursion, so creating a mailbox under this mailbox's child uses only the child's ACLs)" I have placed an ACL (dovecot-ACL) at the root of the public namespace (all rights to anyone). The public namespace was appearing correctly in my IMAP client. Then I have copied a large folder with hundreds of nested subfolders (nesting level > 3) from another account to the root of the public namespace. According to the documentation mentioned above, this should not have been possible (if I got the docs right): The first nesting level of folder should have been created, but not the messages in them and for sure not the deeper nested folders. There is no dovecot-acl within a single of the first level folders (I have verified this), so no one should have access to them. Nevertheless, in addition to the succesful copy, I can see every folder and every message in my IMAP client, I can delete and so on. Could some please explain if I got the docs wrong? 2) If there really is no ACL recursion, how are we supposed to copy large folder structures (perhaps thousands of folders, nested to 5 levels deep) to a public namespace? Do we need to create all folders by hand, then place the dovecot-acl in each folder, and then copy the messages to each folder by hand? Do we need to switch to global ACLs so that we can establish a master user for doing the act of filling the public space? Or is there a dovecot module which, immediately after creating a folder, looks into the parent folder of the new folder and copies the dovecot-acl from the parent folder to the new folder? Currently, I am very happy that dovecot didn't what it should do according to my understanding of the documentation (since I really needed to copy this folder structure to the public namespace), but on the other hand, I am a bit puzzled now not knowing if the documentation is wrong, my understanding of it is wrong or the source code is wrong :-) Thanks you very much for any help, Peter Here is the output of dovecot -n: # 1.0.15: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ log_path: /var/log/dovecot.log info_log_path: /var/log/dovecot-info.log log_timestamp: %Y-%m-%d %H:%M:%S protocols: imaps listen: 192.168.20.23 ssl_cert_file: /etc/dovecot/imap-ssl.home.omeganet.de.crt ssl_key_file: /etc/dovecot/imap-ssl.home.omeganet.de.key ssl_parameters_regenerate: 24 ssl_cipher_list: ALL:!LOW login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login login_processes_count: 1 login_max_processes_count: 32 max_mail_processes: 32 mail_privileged_group: mail mail_location: maildir:~/Maildir mail_cache_fields: mail_never_cache_fields: mail_plugins: acl namespace: type: private inbox: yes namespace: type: public prefix: Archive. location: maildir:/home/vmail/archive:INDEX=~/Maildir/archive auth default: cache_size: 1 cache_ttl: 600 worker_max_count: 4 passdb: driver: passwd-file args: /etc/dovecot/passdb userdb: driver: static args: uid=vmail gid=vmail home=/home/vmail/%u socket: type: listen master: path: /var/run/dovecot/auth-master mode: 384 user: root plugin: acl: vfile -- GMX DSL: Internet, Telefon und Entertainment für nur 19,99 EUR/mtl.! http://portal.gmx.net/de/go/dsl02

2 2

[Dovecot] dovecot user
by Timo Sirainen 04 May '10

04 May '10

Any thoughts on this: The primary use for "dovecot" user has been for login processes. But people keep misunderstanding this and try to use dovecot for accessing mails. For years I've been wondering about renaming this user to something else like dovelogin, but it never really seemed practical. So now with v2.0 there are a bunch of new processes, and for example anvil and dict are now run as dovecot user by default. But it's not really good that login processes can just go and kill those processes. And even worse, if drop_priv_before_exec=yes they could ptrace these processes. So I think we need two Dovecot users for v2.0: 1. Completely untrusted user for login processes. 2. Slightly more trusted internal Dovecot user. So "dovecot" could be reused for 2. And it would no longer be a mortal sin to use dovecot user for owning mail files. For 1. there would be a new user. I'd use "dovelogin", but apparently tools still don't much like usernames that are longer than 8 characters. Like ps could show numeric uid instead of 9 character long username. So .. any suggestions? "dovlogin" could be one possibility I guess. It would be nice if the name somehow reminded of login processes, but maybe something else could be used too, like: dovenil, dovenull, dovezero, dovenone, dovevoid, doveint, dovedown, dovein, dove0

17 33

[Dovecot] Shared mailboxes unix permissions
by Thomas Hummel 16 Apr '10

16 Apr '10

Hello Timo, I'm running dovecot-1.2.11/Maildir and plan to migrate to single UID mailboxes some day, but for now, I've got "system" users and I'm testing permissions handling in order to set up shared mailboxes. The private namespace mailboxes location is location = maildir:/courriel/boites/%u:CONTROL=/courriel/meta/%u:INDEX=/var/dovecot-test/indexes/%1u/%u I've . created a 'doveshared' unix group . added 'mail_access_groups = doveshared' . chmod/chgrp the maildir and control dirs like this : drwxrws--- 7 doveimap doveshared 4096 Mar 31 18:47 /courriel/boites/doveimap drwxrws--- 5 doveimap doveshared 4096 Mar 31 18:47 /courriel/meta/doveimap [which were empty] . added 'acl_shared_dict = file:/var/dovecot-test/dict/shared-mailboxes' with drwxrwxs-- 2 root doveshared 512 Mar 31 18:44 /var/dovecot-test/dict [which were empty] Everything gets created with the permission I was expecting, except : -rw------- 1 doveimap doveshared 8 Mar 31 18:47 /courriel/meta/doveimap/dovecot-uidvalidity -rw------- 1 doveimap doveshared 0 Mar 31 18:44 /courriel/meta/doveimap/dovecot-uidvalidity.4bb37be4 and -rw------- 1 doveimap doveshared 0 Mar 31 18:44 /var/dovecot-test/dict/shared-mailboxes I can't see the explanation in http://wiki.dovecot.org/SharedMailboxes/Permissions and the 'dovecot-shared' file doesn't help (besides, my understanding is that in 1.2x, it's for backward compatibility reason). Any idea ? Thanks -- Thomas Hummel | Institut Pasteur <hummel(a)pasteur.fr> | Pôle informatique - systèmes et réseau

2 3