October 2007 - dovecot

[Dovecot] two small low importance issues in 1.1b
by Adam McDougall 21 Oct '07

21 Oct '07

Item 1: I store the indexes and control in their own directory. I noticed in 1.1: 1192 ./private/indexes/.INBOX 32 ./private/indexes/.Trash 1228 ./private/indexes 4 ./private/control/.INBOX/.INBOX 84 ./private/control/.INBOX 4 ./private/control/.Trash/.Trash 12 ./private/control/.Trash 100 ./private/control 1332 ./private .INBOX/.INBOX and .Trash/.Trash are both empty. This was an empty directory, I logged in with thunderbird and looked at the inbox and Trash folders. Item 2: I wanted to try imap_logout_format, forgetting what params it uses and taking a stab at it, but it seems to be ignored? It shows up in dovecot -n. I made it 'bbytes' just to see if the intentional typo would come through, but no: imap_logout_format: bbytes=%i/%o username=%u dovecot: IMAP(mcdouga9): Disconnected: Logged out bytes=439/169567

2 2

[Dovecot] dovecot 1.1.beta3 crashes on NetBSD/sparc64 4.0_RC3 with Thunderbird 2.0.0.6
by Lloyd Parkes 21 Oct '07

21 Oct '07

Hi all, I'm getting core dumps from Dovecot when using Thunderbird as a client. Everything is fine when using the Apple Mail client. I've appended the debug dumps to this message so they are out of the way, but they should be read now. I think the problem might be that io->prev and io->next aren't reliably initialised in ioloop.c:io_add. The value that io->prev has when the program crashes is just completely bogus. It isn't aligned correctly, and it doesn't point to anywhere inside the process's memory map. I don't know what *io->next is, but it sure doesn't look like a struct io_file. It doesn't even look like a struct io. Part of the garbage in *io->next is the char sequence "/vol/home/lloyd/Maildir/new" which is part of my mail box. *io is not in the doubly linked list in current_io_look->io_files. I'm guessing that this object has been unlinked already. Some further digging (not shown here) shows that io->callback is notify_callback. It is set to NULL by the time imap crashes. Here are all the gory GDB details: Version: 1.1.beta3 OS: NetBSD 4.0_RC3/sparc64 (a 64 bit big endian architecture) Client: Mozilla Thunderbird 2.0.0.6 bt full: #0 0x00000000001e846c in io_file_unlink (io=0x32e2c0) at ioloop.c:54 No locals. #1 0x00000000001e8568 in io_remove (_io=0x3420a8) at ioloop.c:82 io_file = (struct io_file *) 0x32e2c0 io = (struct io *) 0x32e2c0 #2 0x000000000017a0a4 in index_mailbox_check_remove_all (ibox=0x331048) at index-mailbox-check.c:118 file = (struct index_notify_file *) 0x0 aio = (struct index_notify_io *) 0x3420a0 #3 0x0000000000139850 in maildir_notify_changes (box=0x331048) at maildir-storage.c:824 mbox = (struct maildir_mailbox *) 0x331048 #4 0x00000000001c0724 in mailbox_notify_changes (box=0x331048, min_interval=0, callback=0, context=0x0) at mail-storage.c:544 No locals. #5 0x00000000001c074c in mailbox_notify_changes_stop (box=0x331048) at mail-storage.c:550 No locals. #6 0x00000000001167fc in idle_finish (ctx=0x3380b0, done_ok=true, free_cmd=true) at cmd-idle.c:48 client = (struct client *) 0x334100 #7 0x0000000000116a04 in idle_client_input (ctx=0x3380b0) at cmd-idle.c:90 client = (struct client *) 0x334100 line = 0x33a075 "DONE" #8 0x00000000001ea2ac in io_loop_handler_run (ioloop=0x320080) at ioloop-kqueue.c:149 ctx = (struct ioloop_handler_context *) 0x32e080 events = (struct kevent *) 0x332000 event = (const struct kevent *) 0x332000 tv = {tv_sec = 0, tv_usec = 999441} ts = {tv_sec = 0, tv_nsec = 999441000} io = (struct io_file *) 0x320a80 events_count = 4 t_id = 2 msecs = 1000 ret = 1 i = 0 __PRETTY_FUNCTION__ = "io_loop_handler_run" #9 0x00000000001e8f28 in io_loop_run (ioloop=0x320080) at ioloop.c:303 No locals. #10 0x000000000012d710 in main (argc=1, argv=0xffffffffffffc978, envp=0xffffffffffffc988) at main.c:293 No locals. More GDB Info: (gdb) print *io $2 = {io = {condition = 0, callback = 0, context = 0x331048}, prev = 0x10000000d, next = 0x32e280, refcount = 0, fd = 0} (gdb) print *io->next $18 = {io = {condition = 0, callback = 0, context = 0x471a982100000000}, prev = 0x0, next = 0x2f766f6c2f686f6d, refcount = 1697606764, fd = 1870226479} (gdb) print *current_ioloop->io_files $35 = {io = {condition = IO_READ, callback = 0x1e9284 <event_callback>, context = 0x342080}, prev = 0x0, next = 0x320140, refcount = 1, fd = 11} (gdb) print *current_ioloop->io_files->next $36 = {io = {condition = IO_ERROR, callback = 0x12ca70 <log_error_callback>, context = 0x0}, prev = 0x320a00, next = 0x3200c0, refcount = 1, fd = 2} (gdb) print *current_ioloop->io_files->next->next $37 = {io = {condition = IO_READ, callback = 0x1ea6f0 <signal_read>, context = 0x0}, prev = 0x320140, next = 0x0, refcount = 1, fd = 5} (gdb) print io $1 = (struct io_file *) 0x32e2c0 (gdb) print *current_ioloop $2 = {prev = 0x0, io_files = 0x320a00, next_io_file = 0x0, timeouts = 0x320a40, handler_context = 0x32e080, notify_handler_context = 0x342080, running = 1}

3 2

[Dovecot] Problems with dovecot 1.1b3
by Adam McDougall 21 Oct '07

21 Oct '07

I am testing dovecot 1.1 betas on a testing system, hoping to take advantage of self repairing and NFS-capable indexes when 1.1 is sensible to put into production. I ran into a number of issues that I did not have with 1.0, and I think I'll just pick one or more at a time and report them rather than try to report every single issue at once, hoping that some of the fixes will solve more than one problem. I started with 1.1b2 just for record, but the issues seem to be present in 1.1b3 as well. I realized yesterday that using the acl plugin may be complicating my testing and results, so I have disabled it for now and am connecting with plaintext IMAP just so I can sniff connections and see what is happening. I am running it on FreeBSD 6.2-stable from Sep 19 2007. First my dovecot -n output: # 1.1.beta3: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/ssl.pem ssl_key_file: /usr/local/etc/apache2/ssl/ssl.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 0 mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: mail_log mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / prefix: mail/ location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private2/control:INDEX=%h/Maildir/dovecot/private2/indexes hidden: yes list: yes namespace: type: private separator: / inbox: yes list: yes auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: acl: vfile:/usr/local/etc/dovecot-acls The first thing I did was to try to update my dovecot.conf from 1.0 to work with 1.1. When I started dovecot 1.1 for the first time, I noticed it was listening to 143 but not 993, even though my config was designed for ssl/tls in 1.0 without any listen ports specified: # netstat -a | grep LISTEN | grep imap tcp4 0 0 *.imap *.* LISTEN So, I thought I would try specifying the ports in the config: protocol imap { listen = *:143 ssl_listen = *:993 # .. } Success! # netstat -a | grep LISTEN | grep imap tcp4 0 0 *.imaps *.* LISTEN tcp4 0 0 *.imap *.* LISTEN Initially I was then testing using ACLs, but I disabled it to simplify. One of the more basic problems I've seen is the behavior from "list = no/yes". For the sake of simplicity and example, I've disabled most of the additional public and private namespaces from my 1.0 config. I have some private namespaces with prefix "mail/" and "Mail/" that I wish to provide for users for compatability with clients set with those prefixes. In dovecot 1.0, this worked as expected: namespace: type: private separator: / prefix: mail/ hidden: yes But in 1.1b2 or b3, I never seem to get a list of the contents under mail/ when I tell mutt to default to servername/mail/ as the default folder path. Cases: hidden = yes list = yes a0006 LIST "" "mail" * LIST (\Noselect \HasChildren) "/" "mail" a0006 OK List completed. a0007 LIST "" "mail/%" a0007 OK List completed. mutt lists contents of only "../" and selecting it brings me to the imap root with the expected folders. When ../ is followed, I see all my folders as well as all my folders repeated under mail/ prefix. hidden = yes list = no a0006 LIST "" "mail" a0006 OK List completed. a0007 LIST "" "mail/%" a0007 OK List completed. mutt claims "No such folder" hidden = no list = yes a0006 LIST "" "mail" * LIST (\Noselect \HasChildren) "/" "mail" a0006 OK List completed. a0007 LIST "" "mail/%" a0007 OK List completed. mutt lists contents of only "../" and selecting it brings me to the imap root with the expected folders. When ../ is followed, I see all my folders as well as all my folders repeated under mail/ prefix.

2 6

[Dovecot] Problem with fts found against Dovecot hg; examples + trace attached
by Asheesh Laroia 21 Oct '07

21 Oct '07

I've been wanting to try out Squat for full-text indexing for a while now, so I finally gave it a whirl. I did a fresh pull from hg. I have it enabled, but it's taking *forever* to do a query. (Using alpine, I issued a Select command for all messages with some word anywhere, which presumably caused the Squat indexer to run.) I think Dovecot is infinite looping. I thought something was wrong when I attached strace to it for five minutes and saw no calls to read() or open(). I can reproduce the "select takes forever" on a tiny Maildir with 5 messages: http://paulproteus.acm.jhu.edu/bug-report/2007-10-21/broken.tar.gz I've also attached the *.in and *.out files that I think are related to this IMAP conversation. On a different IMAP conversation with the same "SEARCH takes forever" problem, I attached a gdb to the running Dovecot and got this backtrace: (gdb) bt #0 parse_next_body_to_boundary (ctx=0x85746e8, block_r=0xbf92325c) at message-parser.c:332 #1 0x080c1b7d in message_parser_parse_next_block (ctx=0x85746e8, block_r=0xbf92325c) at message-parser.c:717 #2 0xb7f582c2 in fts_mailbox_search_next_nonblock (ctx=0x811c0c0, mail=0x811d380, tryagain_r=0xbf9232bb) at fts-storage.c:175 #3 0x0805f3e6 in cmd_search_more (cmd=0x80feec0) at cmd-search.c:74 #4 0x0805f768 in cmd_search_more_callback (cmd=0x80feec0) at cmd-search.c:112 #5 0x080ce8eb in io_loop_handle_timeouts (ioloop=0x80fb250, update_run_now=true) at ioloop.c:264 #6 0x080cf309 in io_loop_handler_run (ioloop=0x80fb250) at ioloop-poll.c:159 #7 0x080ce7c8 in io_loop_run (ioloop=0x80fb250) at ioloop.c:303 #8 0x08068c9c in main (argc=135213400, argv=0xbf923464, envp=0xbf92346c) at main.c:293 So I do know that it's using the fts system. Every few minutes, I've interrupted it in gdb, and I have gotten these messages back from gdb: Program received signal SIGINT, Interrupt. parse_next_body_to_boundary (ctx=0x85746e8, block_r=0xbf92325c) at message-parser.c:332 332 if (data[i] == '\n') { (gdb) cont Continuing. [wait a few minutes] Program received signal SIGINT, Interrupt. parse_next_body_to_boundary (ctx=0x85746e8, block_r=0xbf92325c) at message-parser.c:331 331 for (; i < block_r->size; i++) { (gdb) cont Continuing. [wait a few minutes] Program received signal SIGINT, Interrupt. parse_next_body_to_boundary (ctx=0x85746e8, block_r=0xbf92325c) at message-parser.c:332 332 if (data[i] == '\n') { The ctx and block_r are the same at each point I interrupt it. This sounds like an infinite loop to me if it can't exit this for minutes and minutes. Furthermore, for the past couple of minutes, when I do this: (gdb) print block_r[0] $8 = {part = 0x80f3158, hdr = 0x0, data = 0x8240d08 "\n<DIV><BR></DIV>Perhaps, but it is important to note that this is a dissenting<BR>opinion, not part of the actual text of the Holt bill.<BR><BR><BR>--- In <A href=\"mailto:TrueVote@yahoogroups.com\">Tru"..., size = 8192} I always get the same response back. I've linked to a fairly small Maildir (five messages) that reproduces the problem for me. It's possible that only one of those messages is actually at fault; I haven't tried with any Maildir smaller than five messages. Is there something more I can do to help fix this problem? I'm happy to do anything reasonable further if you deem it useful. -- Asheesh. -- Bolub's Fourth Law of Computerdom: Project teams detest weekly progress reporting because it so vividly manifests their lack of progress.

2 1

[Dovecot] Multiple server sections for SSL
by Wolfram Schlich 21 Oct '07

21 Oct '07

Hi, I would like to achieve the following with my dovecot server: - listen von FQDN 'mail.foo.example.com'/IP 'A' for imaps and pop3s with an SSL certificate for 'mail.foo.example.com' - listen von FQDN 'mail.bar.example.com'/IP 'B' for imaps and pop3s with an SSL certificate for 'mail.bar.example.com' I stumbled over a posting by Timo made in 2004 that got my attention: http://osdir.com/ml/mail.imap.dovecot/2004-07/msg00133.html Now, what is the correct nesting/structure for the server, protocol, auth, dict and plugin section as well as any other (by default) "non-sectional" configuration options (like 'login_greeting')? I failed to find *any* documentation references on this topic :-( TIA. -- Regards, Wolfram Schlich <wschlich(a)gentoo.org> Gentoo Linux * http://dev.gentoo.org/~wschlich/

2 3

[Dovecot] Homedir Access without exposing whole Homedir.
by Dan Mahoney, System Admin 21 Oct '07

21 Oct '07

Hello All, Dovecot 1.0.3 I am coming from UW IMAP, and I'm finding for some reason that mail is getting stored in a variety of places (which I believe, is because by default UW imap allowed access to the entire home directory). All files are mbox. My default delivery location is ~/.mail Thus: * At least a couple of my users have mail in ~/INBOX, as well as ~/INBOX.drafts (not many users so I don't really care about this). * Several of my users have their mail folders in their home directory, AND in a mail folder. For example, my "test case" user, had the following .mailboxlist file: mail/Sent mail/Drafts mail/Spam mail/Trash Sent Items Drafts Deleted Items Junk E-mail Now, here's the question. I'd like to make it so that everyone has what they had before, but that any new folders are created in ~/mail/ (so that if I ever DO decide to convert, I know where to look). What I thought would be useful for this is make the default namespace "", but set it hidden. Then define a secondary namespace (mail) and use that. However, when I did this, and got a mailbox list, it still recursively listed every file in the homedir. Is there a simple way to do what I want? -Dan Mahoney -- "Station!" -Bill & Ted's Bogus Journey --------Dan Mahoney-------- Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC ICQ: 13735144 AIM: LarpGM Site: http://www.gushi.org ---------------------------

2 2

[Dovecot] [Fwd: Re: Namespace Question]
by Jeff Grossman 21 Oct '07

21 Oct '07

Timo Sirainen wrote: > On Fri, 2007-10-05 at 17:27 -0700, Jeff Grossman wrote: > >> Timo Sirainen wrote: >> >>> On Tue, 2007-10-02 at 18:12 -0700, Jeff Grossman wrote: >>> >>> >>>> So, what would >>>> have changed between version 1.0.5 and 1.1beta2 to somehow make the >>>> phone think all of the messages are older than 30 days? >>>> >>>> >>> Is it using SORT command? Apparently there are some bugs still in it. >>> http://dovecot.org/bugreport.html#sniffing >>> >>> >>> >> I don't think so. Here is the first .in file from the rawlog >> >> A6 NAMESPACE >> A7 SELECT "Inbox" >> A8 SEARCH SINCE 6-Sep-2007 >> > > OK, this finds the messages based on their INTERNALDATE. With maildir > that means the files' mtime. Are all the mtimes older than 6 Sep? If > not, try deleting dovecot.index.cache file to see if it helps. > > Out of 13 messages in my inbox, 12 of them are newer than 6 Sep. I deleted the dovecot.index.cache file and it still will not display any messages in my inbox when I have the setting to only display the most current 30 days worth of mail. Jeff Timo, Just FYI, the above problem is still not fixed with beta 3. Jeff

2 7

[Dovecot] v1.1 QUOTA and inbox quota
by Kristian Vlahovicek 21 Oct '07

21 Oct '07

My mail agent is postfix and by default it imposes the INBOX size of 50 meg. I would like my imap mail clients (I'm using Thunderbird) to "see" this and warn me when my inbox fills up. I tried doing it with the old quota system but wasn't successful. With the new quota rule system it looks like per-mailbox quotas are possible, however when I do this: quota = maildir quota_rule = INBOX:storage=50000 I get an error in the log: Sep 28 02:18:31 discworld dovecot: IMAP(kristian): open((null)) failed: Bad address Sep 28 02:18:31 discworld dovecot: IMAP(kristian): Internal quota calculation error Sep 28 02:20:04 discworld dovecot: IMAP(kristian): open((null)) failed: Bad address Sep 28 02:20:04 discworld dovecot: IMAP(kristian): open((null)) failed: Bad address is there any way to get the inbox quota to work properly? I guess many systems use postfix and have 50M inbox limit so it would really be helpful if this information is pushed somehow through imap to the client. thanks, K

2 1

Re: [Dovecot] Error: net_connect(/var/run/dovecot/auth-master) failed: Resource temporarily unavailable
by Maykel Moya 21 Oct '07

21 Oct '07

> El lun, 15-10-2007 a las 10:41 -0400, Charles Marcus escribió: > > On 10/15/2007, Maykel Moya (moya(a)infomed.sld.cu) wrote: > > > I'm using deliver (1.0.5) with Postfix (2.1.6). > > > > <snip> > > > > > Any hint? > > > > May not be the cause - but postfix 2.1.6 is - well - *ancient*... > > > > I'd make upgrading a priority, even it didn't fix the problem... > > I'd upgraded postfix to 2.4.5. Although not sure if it helps I also > raised auth_worker_max_count. Up to this moment I haven't had experience > the error again. Well, there it's, again. Postfix, by defaults, don't do more than 100 concurrent deliveries with the same transport, local in my case. I raised up auth_worker_max_count to 128, to accomodate 100 potential concurrent deliveries plus some pop3/imaps clients. The problem appears with certain frequency. There is no other info in the log besides the error notification. FWIW: I have auth_debug enabled. In normal operation I can see the LDAP queries: dovecot: Oct 16 09:46:14 Info: auth(default): ldap(foo,x.y.z.w): pass search: base=ou=users,dc=foo,dc=org scope=subtree filter=(uid=foo) fields=uid,userPassword,sldMailbox nevertheless, next to net_connect errors, the ldap related line is something like dovecot: Oct 16 09:46:14 Info: auth(default): ldap(foo,x.y,z,w): unknown user in both cases, the user don't exist in the directory. Don't know is this could be something with the same nature of[1]. My config -- base_dir = /var/run/dovecot-ldap # IO optimizations fsync_disable = yes dotlock_use_excl = yes maildir_copy_with_hardlinks = yes maildir_stat_dirs = no maildir_copy_preserve_filename = yes protocol lda { postmaster_address = bar(a)foor.org auth_socket_path = /var/run/dovecot/auth-master mail_plugins = quota # http://wiki.dovecot.org/MailLocation/LocalDisk fsync_disable = yes } auth default { mechanisms = plain auth_worker_max_count = 128 socket listen { master { path = /var/run/dovecot/auth-master mode = 0660 user = dovecot group = nusers } } passdb ldap { args = /etc/dovecot/dovecot-ldap.conf } passdb pam { } userdb ldap { args = /etc/dovecot/dovecot-ldap.conf } userdb prefetch { } userdb passwd { } user = root } -- Regards, maykel [1] http://www.dovecot.org/list/dovecot/2007-October/026069.html

2 1

[Dovecot] Errata: dovecot + ldap + virtual mailbox: building mail location path
by Claudio Strizzolo 21 Oct '07

21 Oct '07

Hi all again, I've just realized that I did a bad cut&paste in my previous mail, then I rewrite my request. I apologize. I am trying to set up my dovecot configuration using LDAP and virtual mailboxes. I succeeded in authenticating the users correctly with LDAP, but have problems with referencing the correct mail location directory. I manage several domains. The mailboxes are stored under separate directories for each domain: /mail/vmail/domains/writers.com/William.Shakespeare/ /mail/vmail/domains/writers.com/Jack.London/ /mail/vmail/domains/painters.com/Pablo.Picasso/ /mail/vmail/domains/painters.com/Vincent.VanGogh/ Now, I supposed I could build the path by setting the common part of the paths as mail_location into dovecot.conf: mail_location = maildir:/mail/vmail/domains/%h passdb ldap { args = /etc/dovecot-ldap.conf } userdb ldap { args = /etc/dovecot-ldap.conf } userdb prefetch { } and getting the variable part ("domain/user", i.e. writers.com/Jack.London) from the userdb in dovecot-ldap.conf: auth_bind = yes auth_bind_userdn = mail=%u,vd=%d,o=hosting,dc=example,dc=com ldap_version = 3 base = o=hosting,dc=example,dc=com deref = never scope = subtree user_attrs = mail,,mailbox,mail,, user_filter = (&(objectClass=VirtualMailAccount)(accountActive=TRUE)(mail=%u)) pass_attrs = mail=user,userPassword=password,mailbox=userdb_home,mail=userdb_uid,mail =userdb_mail pass_filter = (&(objectClass=VirtualMailAccount)(accountActive=TRUE)(mail=%u)) default_pass_scheme = CRYPT In the LDAP database the fields are something like: mail = Jack.London(a)writers.com mailbox = writers.com/Jack.London uid = Jack.London However, the above does not work. Here is an extract from the log: Oct 16 15:57:07 mymail.example.com dovecot: auth(default): ldap(Jack.London@writers.com,::ffff:127.0.0.1): user search: base=o=hosting,dc=example,dc=com scope=subtree filter=(&(objectClass=VirtualMailAccount)(accountActive=TRUE)(mail=Jack.London(a)writers.com)) fields=mail,mailbox,mail Oct 16 15:57:07 mymail.example.com dovecot: IMAP(Jack.London(a)writers.com): Ambiguous mail location setting, don't know what to do with it: writers.com/Jack.London/ (try prefixing it with mbox: or maildir:) Oct 16 15:57:07 mymail.example.com dovecot: IMAP(Jack.London(a)writers.com): Failed to create storage with data: writers.com/Jack.London Any hints? BTW, I am trying to prefetch, as you see, but I am not sure that works too. Thanks in advance Claudio

2 1