- dovecot - dovecot.org

[Dovecot] Bug tracking system
by Timo Sirainen 26 Mar '07

26 Mar '07

There are three reasons why I haven't really bothered setting up a bug tracking system for Dovecot: 1) They're all kind of annoying to use. 2) They all require a lot of stuff like PHP, SQL server, etc. which I don't really want to put into dovecot.org. 3) Probably most importantly: If people start reporing bugs in there, I'm pretty much the only person who will ever read them and reply to them. Currently when people report bugs in here, it's quite useful to have other people read them too. They may already know the bug and say they've reported it, they may have an idea what's causing it, they may know a workaround, etc. All in all, moving things to BTS would most likely cause my life to be harder and it might be more difficult to get some bugs fixed. A couple of days ago I started thinking about this and then I figured out a way to solve all of this nicely with a new and great Dovecot based bug tracking system. :) Just so I won't forget it, here's the plan: Dovecot BTS ----------- The preferred way to report bugs is to send them to dovecot-bugs(a)dovecot.org. The only thing it does is prefix the subject line with [BUG #nnn] and forward it to dovecot(a)dovecot.org. Now everyone can reply to it just as it was a normal mailing list mail. As long the subject contains the "[BUG #nnn]" prefix, it's part of the bug. Existing mailing list threads can also be turned into bugs by replying to the thread's root message with To: dovecot-bugs(a)dovecot.org. This again causes the new reply to contain [BUG #nnn] prefix. Then comes the web part. There's no SQL database or anything, everything is stored in a mailbox in the IMAP server. The mailbox contains the whole dovecot(a)dovecot.org archive. Bug metadata such as priority, status, etc. is stored using IMAP ANNOTATE extension, which allows setting key=value pairs to messages. Unfortunately Dovecot doesn't yet support that extension. Bugs' root messages contain the annotations for the whole bug thread. So to get a list of all bugs, the BTS issues an IMAP SEARCH ANNOTATE command to list all messages with some specific "bug=yes" type of annotation. For those messages then it can fetch the subject and whatever else metadata it wants. Newly created bugs don't have annotations though. To fix that, the BTS also uses ANNOTATEMORE (or is it METADATA nowadays?) extension to keep "last seen message in BTS" state in the mailbox's annotations. It then issues a SEARCH SUBJECT "[bug #" command for the new messages and does THREAD command for them to see if they're the bug root, or if they were replies to another message which should be treated as the bug root. When viewing a bug, the bug metadata comes from the annotations. Annotation history could also be kept, possibly as messages in another history mailbox. The messages for a bug can be viewed by issuing a THREAD (REFERENCES) command and showing all the mails that exist under the bug root. So, any volunteers to implement the web part, if I implement the ANNOTATE extension? :) It would actually even be possible to implement multiple frontends. All I need to do is give you access (read-only, or read-write for some) to the bug tracking mailbox, and you can use whatever frontend you wish, even a non-web one!

17 39

[Dovecot] Dovecot LDA issues
by Steven F Siirila 26 Mar '07

26 Mar '07

We've seen two issues come up with Dovecot LDA, both of which have caused us problems: 1) If the user's home directory does not exist, or is not owned by them, deliver fails and causes the mail message to bounce back to the originator. In our environment this happens when our user is moved to another server (where we move their files, but for up to 24 hours afterwards, continue to re-mail their /var/mail INBOX). This no longer works because the user has no home directory, causing deliver to fail with "Permission denied" when attempting to create their home directory for the purposes of creating INBOX index files. 2) If the user is over quota (in their home directory), deliver fails with a temporary error, causing requeues until the user is back under quota (which in our environment could be a long time, days). Since we've never had quotas on the user's INBOX (in /var/mail), this is a problem for us. I'd propose for both of these cases that deliver issue WARNING messages to its logs, and simply not create index files for the INBOX. If the INBOX index files already exist, and the user is over quota in /home, neither deliver nor any other Dovecot process should attempt to update them, and instead issue WARNING messages. This allows the e-mail to be delivered instead of requeued, or worse yet, rejected. Index files are supposed to make things perform better, not worse, in ALL cases. :) -- Steven F. Siirila Office: Lind Hall, Room 130B Internet Services E-mail: sfs(a)umn.edu Office of Information Technology Voice: (612) 626-0244 University of Minnesota Fax: (612) 626-7593

2 3

[Dovecot] deliver: -m, sieve and -n
by Jim Horner 26 Mar '07

26 Mar '07

Hi, I took a stab at editing the wiki ... the changes disappeared? I guess I was overruled. The wiki (http://wiki.dovecot.org/LDA) states -m <mailbox>: Destination mailbox (default is INBOX). If the mailbox doesn't exist, it's created (unless -n is used). If message couldn't be saved to the mailbox for any reason, it's delivered to INBOX instead. I tried to add a little note that the sieve plugin takes precedence. For example if at the end of the a .dovecot.sieve there is a line like: fileinto "CompanyFolders.general"; then the -m <mailbox> is ignored. I am not sure about the use of 'stop'. -n: If the destination mailbox doesn't exist, don't create it. This affects both -m parameter and fileinfo action in Sieve scripts. The fallback is to deliver mail to INBOX. (typo: fileinto not fileinfo) My testing shows that if the mailbox does not exist then the mail is rejected like so: === Your message was automatically rejected by Dovecot Mail Delivery Agent. The following reason was given: Mailbox doesn't exist: mail-lists.debian === I like the wiki's solution better. Jim

2 3

[Dovecot] dovecot authentication & exim in rc27
by Richard P. Welty 26 Mar '07

26 Mar '07

[introduction: new to the list, used dovecot w/exim as MTA in simple ways for 2 years, blah blah blah] i'm trying to get exim to use dovecot auth with a copy of rc27 and am missing something that's probably quite simple. specifically, how does one configure the authenticators in dovecot.conf for rc27? the auth section of the default config file installed in rc27 doesn't entirely look like what i'm seeing in other documentation; instead of auth default { } i'm seeing auth = default followed by lots of standard flags prefixed by auth_, but i don't see how to set up the socket. i have the following; dovecot starts and doesn't complain but it doesn't create the socket either: auth_socket listen { client { path = /var/run/dovecot/auth-client mode = 0660 user = exim group = exim } } any suggestions? thanks, richard

2 1

[Dovecot] anonymous mailing list archive configuration?
by Johannes Berg 25 Mar '07

25 Mar '07

Hi, I'd like to run a mailing list archive via IMAP like for this list but my mails are in maildir and not mbox. It's easy to add a new user that has access to the maildir that mails get stored in, and I can simply link that up to that user's inbox. It's also easy to prevent putting mail into that folder. However, one thing I'm not clear on is how to prevent that user from creating new folders. I don't think I can do that with permissions to the ~/Maildir since that user needs to be able to create files there for indizes etc. Is that possible? Should/can I write a plugin that prevents things? johannes

2 3

[Dovecot] Design docs in wiki
by Timo Sirainen 25 Mar '07

25 Mar '07

If you've ever wondered how Dovecot works internally, I've now written something about it to the wiki. It's probably a bit confusing and there are a lot of things missing, but it's better than nothing. :) http://wiki.dovecot.org/Design

2 1

[Dovecot] 1.0.rc28 / v1.0 plans
by Timo Sirainen 24 Mar '07

24 Mar '07

I'll probably release 1.0.rc28 in a few days. Would be nice to get some testing for it first. http://dovecot.org/nightly/dovecot-latest.tar.gz contains pretty much what it's supposed to be. I've now replied to all the mails I was planning on replying, excluding the few patches that are v1.1 material. If you think I forgot to reply to some mail, please resend it. I think the documentation in Wiki is finally beginning to look pretty good. All the important things are done. The only things in my "would be nice TODO" list are: - Add a new page about index files and their configuration. I'll probably do this before rc28. - PasswordSchemes and Authentication/Mechanisms are pretty heavy reading. Could be made easier to understand. Possibly also create some mechanism-specific pages describing how to configure them in Dovecot and in clients. - Design documents would also be nice, but not very useful for non-programmers and pretty big job to write them I'm planning on adding the Wiki pages to doc/wiki/ in the tarball and changing: - http://wiki.dovecot.org/* links to doc/wiki/*.txt in dovecot-example.conf - doc/variables.txt could be removed and links to it updated to doc/wiki/Variables.txt - doc/auth-protocol.txt is also already in wiki. Maybe create doc/design/ and put the AuthProtocol.txt there and other design wiki pages there as well. As for v1.0, it looks like there aren't many bug reports anymore about rc27. If there are as few for rc28, I'll release v1.0 in about a month. So unless something special happens, I'll fix only the simplest bugs after rc28 and leave the rest to v1.1/v1.0.1. Hmm. Looks like 13.4. is Friday. Perhaps that's a good day for a release. :) Looks like I ended up adding a few more new features to rc28: * deliver + userdb static: Verify the user's existence from passdb, unless allow_all_users=yes * dovecot --exec-mail: Log to configured log files instead of stderr + When copying/syncing a lot of mails, send "* OK Hang in there" replies to client every 15 seconds so it doesn't just timeout the connection. + Added idxview and logview utilities to examine Dovecot's index files + passdb passwd and shadow support blocking=yes setting now also + mbox: If mbox file changes unexpectedly while we're writing to it, log an error. + deliver: Ignore -m "" parameter to make calling it easier. + deliver: Added new -n parameter to disable autocreating mailboxes. It affects both -m parameter and Sieve plugin's fileinto action - auth cache: If unknown user was found from cache, we didn't properly return "unknown user" status, which could have caused problems in deliver. - Index view syncing fixes - rc27 didn't compile with some non-GCC compilers - vpopmail support didn't compile in rc27 - NFS check with chrooting broke home direcotry for the first login - deliver: If user lookup returned "unknown user", it logged "BUG: Unexpected input" - convert plugin didn't convert INBOX

5 11

[Dovecot] idxview: Fatal: file hdr read() 41984 != 1170103198
by Troy Engel 24 Mar '07

24 Mar '07

Playing with the new idxview I got this error running against my INBOX dovecot.index.cache file. What exactly does it mean? Full output: # /usr/libexec/dovecot/idxview dovecot.index.cache -- INDEX: dovecot.index.cache version = 1.8 base header size = 0 header size = 1170103198 record size = 1170103283 compat flags = 7 index id = 0 flags = 26512 uid validity = 0 next uid = 2460778624 messages count = 0 recent messages count = 632 seen messages count = 1 deleted messages count = 1140057508 first recent uid lowwater = 4294966816 first unseen uid lowwater = 3 first deleted uid lowwater = 1836 log file seq = 0 log file int offset = 2 log file ext offset = 1140057513 sync size = 266287972361 sync stamp = 21 day stamp = 0 day first uid[0] = 1378709065 day first uid[1] = 2037149797 day first uid[2] = 980374573 day first uid[3] = 826620960 day first uid[4] = 2019113286 day first uid[5] = 808463731 day first uid[6] = 1279801136 day first uid[7] = 1078085165 Fatal: file hdr read() 41984 != 1170103198 ('idxview dovecot.index' and 'logview dovecot.index.log' come back without any Fatal type errors, just gobs of information) -te -- Troy Engel | Systems Engineer Fluid, Inc | http://www.fluid.com

2 2

[Dovecot] Dovecot LDA Logging
by Jonathan Stewart 23 Mar '07

23 Mar '07

I have Dovecot LDA logging fine right now but I really don't like having my log files world writable because I have users with shell access. Is there a way to have Dovecot LDA log to syslog or would it be possible to write a mini-daemon that could log to syslog on behalf of Deliver or something else altogether? Jonathan

2 2

[Dovecot] Stalled imaptest10 process
by Steffen Kaiser 23 Mar '07

23 Mar '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, in order to stress my Dovecot test installation a bit, I compiled imaptest10.c. Now and then I see: Logi List Stat Sele Fetc Fet2 Copy Stor Dele Expu Appe Logo 100% 50% 50% 100% 100% 100% 33% 50% 100% 100% 100% 100% 30% 5% 5% 0 0 0 0 0 0 0 0 0 0 0 0 1/ 1 0 0 0 0 0 0 0 0 0 0 0 0 1/ 1 0 0 0 0 0 0 0 0 0 0 0 0 1/ 1 0 0 0 0 0 0 0 0 0 0 0 0 1/ 1 - 46. stalled for 1784 secs in APPEND - 46. stalled for 1785 secs in APPEND - 46. stalled for 1786 secs in APPEND - 46. stalled for 1787 secs in APPEND - 46. stalled for 1788 secs in APPEND - 46. stalled for 1789 secs in APPEND - 46. stalled for 1790 secs in APPEND (Sometimes SELECT or COPY) The maximum stalled time in APPEND is 1790s; in SELECT and COPY the maximum time is less than 20s until now. The only IMAP process loops around a poll(): poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}, {fd=2, events=POLLERR|POLLHUP|POLLNVAL}, {fd=0, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}], 3, 0) = 0 gettimeofday({1174389105, 882286}, {4294967236, 0}) = 0 gettimeofday({1174389105, 882339}, NULL) = 0 poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}, {fd=2, events=POLLERR|POLLHUP|POLLNVAL}, {fd=0, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}], 3, 999) = 0 gettimeofday({1174389106, 882011}, {4294967236, 0}) = 0 gettimeofday({1174389106, 882123}, NULL) = 0 poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}, {fd=2, events=POLLERR|POLLHUP|POLLNVAL}, {fd=0, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}], 3, 0) = 0 So does imaptest itself: poll([{fd=3, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}, {fd=52, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}], 2, 0) = 0 gettimeofday({1174389079, 93585}, {4294967236, 0}) = 0 write(1, " 0 0 0 0 0 0 "..., 68) = 68 write(2, " - 46. stalled for 1698 secs in "..., 39) = 39 gettimeofday({1174389079, 93814}, NULL) = 0 poll([{fd=3, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}, {fd=52, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}], 2, 999) = 0 gettimeofday({1174389080, 93445}, {4294967236, 0}) = 0 gettimeofday({1174389080, 93497}, NULL) = 0 poll([{fd=3, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}, {fd=52, events=POLLIN|POLLPRI|POLLERR|POLLHUP|POLLNVAL}], 2, 0) = 0 gettimeofday({1174389080, 93607}, {4294967236, 0}) = 0 There are no crashes, nothing. The server is idle then. I first assumed that it is related to an out-of-quota condition, but this time there is no fs quota set for the user. When I played with the quota, I ran imaptest with standard settings and without copybox=. I used this command: /usr/local/src/imaptest user=dvtest pass=pass clients=115 mbox=test.mbox box=box0 copybox=box1 msgs=200 ./configure summary: Install prefix ...................... : /usr/local/dovecot.rc27 File offsets ........................ : 64bit I/O loop method ..................... : poll File change notification method ..... : dnotify Building with SSL support ........... : yes (OpenSSL) Building with IPv6 support .......... : yes Building with pop3 server ........... : yes Building with mail delivery agent .. : yes Building with GSSAPI support ........ : no Building with user database modules . : static prefetch passwd passwd-file checkpassword ldap sql (modules) Building with password lookup modules : passwd passwd-file shadow pam checkpassword ldap sql (modules) Building with SQL drivers ............: pgsql # /usr/local/dovecot.rc27/sbin/dovecot -n # /usr/local/dovecot.rc27/etc/dovecot.conf base_dir: /var/run/dovecot/ log_path: /var/log/dovecot/dovecot.log protocols: imap imaps pop3 pop3s ssl_ca_file: /etc/ssl/certs/ca.crt ssl_cert_file(default): /etc/ssl/certs/imap.pem ssl_cert_file(imap): /etc/ssl/certs/imap.pem ssl_cert_file(pop3): /etc/ssl/certs/pop3.pem ssl_key_file(default): /etc/ssl/private/imap.key ssl_key_file(imap): /etc/ssl/private/imap.key ssl_key_file(pop3): /etc/ssl/private/pop3.key disable_plaintext_auth: no verbose_ssl: yes login_dir: /var/run/dovecot//login login_executable(default): /usr/local/dovecot.rc27/libexec/dovecot/imap-login login_executable(imap): /usr/local/dovecot.rc27/libexec/dovecot/imap-login login_executable(pop3): /usr/local/dovecot.rc27/libexec/dovecot/pop3-login login_log_format_elements: %p: user=<%u> method=%m rip=%r lip=%l %c verbose_proctitle: yes first_valid_uid: 1000 mail_location: maildir:%h/MailDir:CONTROL=/var/cache/dovecot/%i/control:INDEX=/var/cache/dovecot/%i/index mail_debug: yes dotlock_use_excl: yes maildir_copy_with_hardlinks: yes maildir_copy_preserve_filename: yes mail_drop_priv_before_exec: yes mail_executable(default): /usr/local/dovecot.rc27/libexec/dovecot/imap mail_executable(imap): /usr/local/dovecot.rc27/libexec/dovecot/imap mail_executable(pop3): /usr/local/dovecot.rc27/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_log zlib mail_plugins(imap): quota imap_quota mail_log zlib mail_plugins(pop3): quota mail_log mail_plugin_dir(default): /usr/local/dovecot.rc27/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/dovecot.rc27/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/dovecot.rc27/lib/dovecot/pop3 mail_log_prefix: %Us(%u) [%p]: mail_log_max_lines_per_sec: 30 pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %u pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): oe-ns-eoh namespace: type: private separator: . inbox: yes hidden: yes auth default: mechanisms: plain login cache_size: 10 username_chars: abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890 verbose: yes debug: yes passdb: driver: ldap args: /usr/local/dovecot.rc26/etc/dovecot-ldap.conf passdb: driver: shadow userdb: driver: prefetch userdb: driver: ldap args: /usr/local/dovecot.rc26/etc/dovecot-ldap.conf userdb: driver: passwd-file args: /etc/passwd socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 master: path: /var/run/dovecot/auth-master mode: 432 group: mail plugin: quota: fs Bye, - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBRf/E3i9SORjhbDpvAQI5oQgAhOYKF9Rw0Ezz8Fn22xLjm/EtXdsRgpPx 6kcBkT+v4gLc15ORtcCrWLmU0bG+an//2wQ33cAiH2X/veZXEdXtwNultq+HPM3/ MYnjvNhL+OkkN9wAGOLTaRFJUDofEiXQzeNiVhYelmUsT4NaLzVQT4tRiYrbIAvK sHlkZJayE3gTIc5SzAviWxPPEkcJO+0HpMrym9USYbFtWNMVEMSYl9pGTVp9YTa0 oMUGpge1H8Qw75VMuf9jHC2SU8Xe889pv9CiButE0cpkQSsQj7Gf4+rX5V3ts3xv vw8oAZdsc3Rzms3JrWRiAsQHomXeAW0VPZSi/DjIOXT/AshXvnLDQw== =FLRG -----END PGP SIGNATURE-----

2 9