- dovecot - dovecot.org

[Dovecot] filtering out LIST cruft?
by Dan Price 16 Feb '07

16 Feb '07

In planning our migration, I've noticed that lots of users have bits of lint in their ~/mail and/or ~/Mail. For example-- for an mbox called "blah" we may see: blah blah.msf (from Thunderbird) .blah.summary (from... I'm not sure, dtmail maybe) .blah_SIMS_index0003 (from our old IMAP server). When the user goes to "subscribe" they are often overloaded with too many items to choose from-- because all of these files are selectable in the UI-- LIST happily returns them. Is there a way to get dovecot to skip over these, by regex? Could it optionally "taste" files to see if they look like mboxes? I looked at the wiki and the source but didn't find anything. In the absence of this feature, I wonder how other sites have handled this? Obviously I can do some cleanup during migration, but I find that those pesky msf files like to come back. -dp -- Daniel Price - Solaris Kernel Engineering - dp(a)eng.sun.com - blogs.sun.com/dp

2 1

[Dovecot] imap core dump with rc18
by Nicolas STRANSKY 16 Feb '07

16 Feb '07

Hi, I had a core dump while using rc18. Here are the backtraces: Jan 23 01:01:44 rouge dovecot: IMAP(user): file mail-index-view.c: line 386 (_view_lookup_uid_range): assertion failed: (*last_seq_r >= *first_seq_r) Jan 23 01:01:44 rouge dovecot: IMAP(user): Raw backtrace: [0x47f25b00000000] Jan 23 01:01:44 rouge dovecot: child 24319 (imap) killed with signal 6 Core was generated by `imap'. Program terminated with signal 6, Aborted. Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 #0 0x00002b3448083545 in raise () from /lib/libc.so.6 (gdb) bt full #0 0x00002b3448083545 in raise () from /lib/libc.so.6 No symbol table info available. #1 0x00002b3448084cce in abort () from /lib/libc.so.6 No symbol table info available. #2 0x0000000000465fd5 in i_set_failure_file () No symbol table info available. #3 0x000000000046585c in i_panic () No symbol table info available. #4 0x000000000044fff3 in mail_index_view_unref_maps () No symbol table info available. #5 0x00000000004505c7 in mail_index_lookup_uid_range () No symbol table info available. #6 0x000000000043be90 in index_storage_get_uids () No symbol table info available. #7 0x0000000000458d37 in mailbox_get_uids () No symbol table info available. #8 0x0000000000419eef in imap_messageset_parse () No symbol table info available. #9 0x000000000041aaae in imap_search_get_arg () No symbol table info available. #10 0x0000000000412e45 in cmd_fetch () No symbol table info available. #11 0x0000000000415910 in cmd_uid () No symbol table info available. #12 0x00000000004164a1 in _client_reset_command () No symbol table info available. #13 0x0000000000416430 in _client_reset_command () No symbol table info available. #14 0x0000000000416591 in _client_input () No symbol table info available. #15 0x000000000046b5bb in io_loop_handler_run () No symbol table info available. #16 0x000000000046ab4c in io_loop_run () No symbol table info available. ---Type <return> to continue, or q <return> to quit--- #17 0x000000000041dd1b in main () No symbol table info available. Regards, -- Nico Ce n'est pas en continuant de faire ce que l'on connaît que l'on pourra faire ce que l'on ne connaît pas

2 3

[Dovecot] severe performance problem (mail cache related?)
by Dan Price 16 Feb '07

16 Feb '07

Hi all, we're having a bad day with email :) I have a user who was complaining of poor performance today when opening mailboxes. Total time to open the box was about 1 minute. Upon truss'ing the imap process during an open, I saw the trace quoted below at the end of the email. As you can see, for each "FETCH" line of response, we were seeing dovecot do a ton of work-- mmaping anon, lots of munmaps(), etc. -- mmap() and munmap() are not free. The stack trace of at least one of the munmaps is captured below, as well-- deep inside of the mail caching code. Then, I moved aside the user's .imap directory, assuming that forcing dovecot to rebuild its indices and caches could help-- and sure enough the user reported that mailbox open times had dropped to 18 seconds. I truss'd that open, briefly, and it was just a ton of preads at high speed. So I don't know why this user was fine for a week, then suddenly hit this. This is a grave concern if I'm going to roll this out from 15 test users to 250 production users. Thanks for your help. PS: Raw logs are also attached below. -dp write(1, " * 1 0 7 9 5 F E T C".., 2050) = 2050 mmap64(0x00000000, 7356416, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFD800000 munmap(0xFE602000, 1048576) = 0 munmap(0xFE502000, 1048576) = 0 munmap(0xFE402000, 1048576) = 0 munmap(0xFE302000, 1048576) = 0 munmap(0xFE202000, 1048576) = 0 munmap(0xFE102000, 1048576) = 0 munmap(0xFE002000, 1048576) = 0 munmap(0xFE000000, 8192) = 0 pread64(9, " n c o m @ o s s 1 >\n\0".., 8192, 7340032) = 8192 mmap64(0x00000000, 7364608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFE800000 munmap(0xFDE04000, 1048576) = 0 munmap(0xFDD04000, 1048576) = 0 munmap(0xFDC04000, 1048576) = 0 munmap(0xFDB04000, 1048576) = 0 munmap(0xFDA04000, 1048576) = 0 munmap(0xFD904000, 1048576) = 0 munmap(0xFD804000, 1048576) = 0 munmap(0xFD802000, 8192) = 0 munmap(0xFD800000, 8192) = 0 pread64(9, " l @ p l a t i n u m >\n".., 8192, 7348224) = 8192 mmap64(0x00000000, 7372800, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFE000000 munmap(0xFEE06000, 1048576) = 0 munmap(0xFED06000, 1048576) = 0 munmap(0xFEC06000, 1048576) = 0 munmap(0xFEB06000, 1048576) = 0 munmap(0xFEA06000, 1048576) = 0 munmap(0xFE906000, 1048576) = 0 munmap(0xFE806000, 1048576) = 0 munmap(0xFE802000, 16384) = 0 munmap(0xFE800000, 8192) = 0 pread64(9, "\0\0\0 0\0\0\0\0 C c : ".., 8192, 7356416) = 8192 ---------- ---------- ---------- ---------- ---------- ---------- ---------- ff1c3e14 munmap (100000, bdc000, fe6de000, 2dc000, 0, 1ef0) + 8 00080bb4 file_cache_set_size (c03a0, 0, bdc110, 408, 0, 110) + e0 00080cec file_cache_read (0, bdbd08, bdbd08, 408, 0, 684a8) + a4 000594a0 mail_cache_map (191268, bdbd08, 408, ffbff274, 3, fffffffc) + ec 0005ae20 mail_cache_get_record (0, bdbd08, ffbff274, ffbff274, 0, 0) + 20 0005b00c mail_cache_foreach_rec (1959c8, ffbff2ec, 5b3d8, 0, b5760, 7ed8) + 10 0005b2bc mail_cache_foreach (1, 47ae, 5b3d8, 0, 0, 0) + b4 0005b47c mail_cache_field_exists (0, 47ae, 0, c0000000, b4f10, 88184) + 84 0005b64c mail_cache_lookup_field (1959c8, b5898, 47ae, 0, 191d48, 195560) + c 000531ac index_mail_get_fixed_field (195dd8, 0, 195ea0, 4, ffffffff, ffffffff) + 4c 000545e8 index_mail_set_seq (195dd8, 47ae, 19a1c8, b4b4f, 0, 47ae) + bc 00070dac mail_set_seq (195dd8, 47ae, 1, c2ce8, 0, 39c) + c 0005770c index_storage_search_next (1, 195dd8, 1, bd778, 5769c, c8ae0) + 70 00072024 mailbox_search_next (19a1c8, 195dd8, 3, 2, 0, c2b50) + 14 0002b3bc imap_fetch (c4e20, c4de8, ffbff748, 5, c2eb0, 80000000) + 6c 00026210 cmd_fetch (1, c2e88, 0, 75696400, 26054, 80000000) + 1bc 00029000 cmd_uid (0, c22f8, 0, 8, 2a134, c22f8) + 8c 00029b58 client_handle_input (c2b94, c2e80, b5760, bd778, 7ef8, 2000000) + 138 00029ad0 client_handle_input (0, 2f05e860, b5760, bd778, 45d3834d, b5000) + b0 00029cb0 _client_input (c2b50, c2b50, 26ee, 0, 1, 0) + 68 00086338 io_loop_handler_run (c0348, 0, 0, ffbffa14, 4c, 80000000) + 140 00085c14 io_loop_run (c0348, ff212cb0, 1, b573c, bae80, ff215dbc) + 34 00032064 main (ffbffc18, b4400, b5000, b573c, c1314, ff3a0180) + 3f8 00024b18 _start (0, 0, 0, 0, 0, 0) + 5c ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ [in] 3 select "Mail/OpenSolaris/discuss" 4 UID fetch 1:* (FLAGS) 5 IDLE DONE 6 close 7 logout ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ [out, taking 1+ minutes] * OK [RAWLOG TIMESTAMP] 2007-02-14 14:05:35 * FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $Forwarded Junk) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $Forwarded Junk \*)] Flags permitted. * 20462 EXISTS * 0 RECENT * OK [UIDVALIDITY 1159980013] UIDs valid * OK [UIDNEXT 20487] Predicted next UID 3 OK [READ-WRITE] Select completed. * 1 FETCH (FLAGS (\Seen) UID 1) * 2 FETCH (FLAGS (\Seen) UID 2) * 3 FETCH (FLAGS (\Seen) UID 3) * 4 FETCH (FLAGS (\Seen) UID 4) * 5 FETCH (FLAGS (\Seen) UID 5) * 6 FETCH (FLAGS (\Seen) UID 6) * 7 FETCH (FLAGS (\Seen) UID 7) * 8 FETCH (FLAGS (\Seen) UID 8) * 9 FETCH (FLAGS (\Seen) UID 9) * 10 FETCH (FLAGS (\Seen) UID 10) * 11 FETCH (FLAGS (\Seen) UID 11) * 12 FETCH (FLAGS (\Seen) UID 12) * 13 FETCH (FLAGS (\Seen) UID 13) * 14 FETCH (FLAGS (\Seen) UID 14) * 15 FETCH (FLAGS (\Seen) UID 15) * 16 FETCH (FLAGS (\Seen) UID 16) * 17 FETCH (FLAGS (\Seen) UID 17) * 18 FETCH (FLAGS (\Seen) UID 18) * 19 FETCH (FLAGS (\Seen) UID 19) * 20 FETCH (FLAGS (\Seen) UID 20) * 21 FETCH (FLAGS (\Seen) UID 21) * 22 FETCH (FLAGS (\Seen) UID 22) * 23 FETCH (FLAGS (\Seen) UID 23) [-----elided-----] * 20459 FETCH (FLAGS (\Seen) UID 20483) * 20460 FETCH (FLAGS (\Seen) UID 20484) * 20461 FETCH (FLAGS (\Seen) UID 20485) * 20462 FETCH (FLAGS (\Seen) UID 20486) 4 OK Fetch completed. + idling 5 OK Idle completed. 6 OK Close completed. * BYE Logging out 7 OK Logout completed. -- Daniel Price - Solaris Kernel Engineering - dp(a)eng.sun.com - blogs.sun.com/dp

3 12

[Dovecot] trash plugin sql
by Robert Schetterer 16 Feb '07

16 Feb '07

Hi @ll, with dovecot latest following http://wiki.dovecot.org/DovecotLDAPostfixAdminMySQL i have defined plugin { quota = maildir acl = vfile:/etc/dovecot/acls trash = /etc/dovecot/dovecot-trash.conf } Example dovecot-trash.conf file 1 Spam 2 Trash 3 Sent from wiki The Trash plugin. When saving a message would make user go over quota, this plugin automatically deletes the oldest mails from configured mailboxes until the message can be saved within quota limits. this dont work at my setup where quota is coming out of sql is there any trick to solve this? or did i missunderstand something -- Mit freundlichen Gruessen Best Regards Robert Schetterer https://www.schetterer.org Munich/Bavaria/Germany

3 6

Re: [Dovecot] repeat downloads of message headers in 1.0rc19
by Jim Garrison 16 Feb '07

16 Feb '07

I reported this against rc7 and rc8, and still see it occasionally. (I'm running the rc8 that comes with Fedora 5). It is always associated with this message in dovecot.log: dovecot: Oct 23 19:26:17 Error: POP3(user(a)domain.tld): mbox sync: UID inserted in the middle of mailbox /var/mail/vhosts/domain.tld/user (12 > 4, seq=2, idx_msgs=5) Once this happens, dovecot will refuse to clear out the mbox file when the user retrieves email via POP. Each time the user logs in to retrieve mail s/he gets everything in the mbox, which continues to grow. The only fix I've been able to find is to manually delete the mbox file. I'm using postfix as the MTA.

5 6

[Dovecot] Error message in logs
by Jay Chandler 16 Feb '07

16 Feb '07

Mbox format, running r22. Only happening to two users so far, happened on r19 as well. Feb 12 13:52:29 wurfel dovecot: imap-login: Login: user=<username>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Feb 12 13:52:30 wurfel dovecot: IMAP(username): file mbox-sync-rewrite.c: line 404 (mbox_sync_read_and_move): assertion failed: (need_space == (uoff_t)-mails[idx].space) -- Jay Chandler Network Administrator, Chapman University 714.628.7249 / chandler(a)chapman.edu Today's Excuse: Dyslexics retyping hosts file on servers

2 1

[Dovecot] Proxy with MasterUser + Authentication issues
by Cassidy B. Larson 16 Feb '07

16 Feb '07

Per Timo's direction, I decided to give the MasterUser a try in connecting my proxy to my destination server. This might allow for the use of 'secure password' for my clients that like to check that box by default. A couple of problems I'm running into running RC15: #1: My Proxy's SQL password_query line: password_query = SELECT a.clearpasswd AS password, v.storeIP AS host, CONCAT(v.userID,'*someuser') AS destuser, 'Y' AS nologin, 'Y' AS nodelay, 'Y' AS proxy, 'somepass' AS pass FROM iwmailsystem.virtmailbox AS v NATURAL JOIN authenticate.users AS a WHERE v.userID = '%u' and v.imap_active = 1 I'm wanting to authenticate the user on the proxy before I send it off via the MasterUser to the destination server. However, I show this in the logs: Info: auth(default): client in: AUTH 1 PLAIN service=IMAP lip= x.y.z.a rip=a.b.c.d resp=AGJ1dGNoQGluZm93ZXN0LmNvbQBzZWNyZXQ= Info: auth-worker(default): sql(user(a)host.com,a.b.c.d): query: SELECT a.clearpass wd AS password, v.storeIP AS host, CONCAT(v.userID,'*someuser') AS destuser, 'Y' AS nologin, 'Y' AS nodelay, 'Y' AS p roxy, 'somepass' AS pass FROM iwmailsystem.virtmailbox AS v NATURAL JOIN authenticate.users AS a WHERE v.userID = 'user(a)host.com' and v.imap_active = 1 Error: auth(default): file auth-request.c: line 207 (auth_request_save_cache): assertion failed: (extra_fields == NULL || (strstr(extra_fields, "\tpass=") == NULL && strncmp(extra_fields, "pass=", 5) != 0)) Error: child 54122 (auth) killed with signal 6 The Wiki showed I should return a 'pass' column. However, that's choking the query up in auth-request.c. Could it be the "\tpass" from auth-request.c? If I change user(a)host.com's password to 'someass' and remove the 'pass' column from the password_query, it authenticates, and proxy's over just fine. Am I doing something wrong? I want to authenticate the user locally on the proxy and pass off the 'master password' to the proxy, but using 'pass' doesnt work. 'password' column does. Doesnt help me though for my situation :) Problem #2: I'm wanting to put the masteruser/pass in the database for quick retrieval and updating. So I setup dovecot.conf as: passdb sql { # Path for SQL configuration file, see doc/dovecot-sql.conf for example args = /usr/local/etc/dovecot-sql.conf master=yes #pass=yes } passdb passwd { } and dovecot-sql.conf with: password_query = SELECT '%u' AS user, 'master-password' AS password Trying to connect locally with a "telnet localhost 143" via: 1 login user(a)host.com*someuser "somepass" Give me this in the detail log file: Info: auth(default): client in: AUTH 1 PLAIN service=IMAP lip= a.b.c.d rip=w.x.y.z resp=AGJ1dGNoQGluZm93ZXN0LmNvbSphZG1pbkBpbmZvd2VzdC5jb20AbWFzdGVyLXBhc3N3b3Jk Error: auth-worker(default): BUG: PASSV had invalid passdb ID Warning: Killed with signal 15 I am able to successfully authenticate with the master user/pass if I store it in a passdb file. However, I'd prefer to have it in the database for obvious reasons. Any help to the above would be appreciated. thanks, Cassidy

2 6

[Dovecot] Can't login in IMAP server
by Mário Gamito 16 Feb '07

16 Feb '07

Hi, I can't login in the IMAP server. This is the log from a telnet session to port 143: Feb 16 15:38:32 mail2 dovecot: auth(default): client in: AUTH 1 PLAIN service=IMAP secured lip=192.168.84.128 rip=192.168.84.128 resp=<hidden> Feb 16 15:38:32 mail2 dovecot: auth(default): checkpassword(gamito(a)mail2.telbit.pt,192.168.84.128): Executed: /usr/local/bin/auth_imap /usr/local/libexec/dovecot/checkpassword-reply Feb 16 15:38:32 mail2 dovecot: auth(default): checkpassword(gamito(a)mail2.telbit.pt,192.168.84.128): Received no input Feb 16 15:38:32 mail2 dovecot: auth(default): checkpassword: Child 15764 died with signal 11 Feb 16 15:38:34 mail2 dovecot: auth(default): client out: FAIL 1 user=gamito(a)mail2.telbit.pt temp Feb 16 15:38:40 mail2 dovecot: imap-login: Aborted login: user=<gamito(a)mail2.telbit.pt>, method=PLAIN, rip=192.168.84.128, lip=192.168.84.128, secured What is happening here ? Any help would be appreciated. Warm Regards, Mário Gamito

2 2

[Dovecot] Dovecot proxy question (hostname instead of ip)
by Wouter de Jong 16 Feb '07

16 Feb '07

Hello, We are planning to use the Proxy feature of Dovecot, but fail to implement it currently the way we would like to. Reason is that proxy insists to have an IP-address as host : Feb 15 15:43:24 proxy-test.widexs.nl dovecot: pop3-login: proxy(username): mspool01-test.widexs.nl is not a valid IP When I changed the query to force the IP-address of the above server, it works. Would it be possible to make this work, but more important ... Are there any reasons NOT to make this work ? :)) Kind regards, Wouter. -- WideXS http://www.widexs.nl Wouter de Jong System-Administrator Tel +31 (0)23 5698070 Fax +31 (0)23 5698099 Bijlmermeerstraat 62, 2131 HG HOOFDDORP, NL

3 3

[Dovecot] domain truncated
by Nemanja Miletic 16 Feb '07

16 Feb '07

Hi all, I have a problem with domain information being truncated from username during ldap search, so it can not be used in mail_location. This happens when user part of the username %n is used in pass_filter in passdb ldap search. When I use %u all username information remains and mailbox can be located. Here is my configuration. In dovecot.conf I have two locations one for system and one for virtual users: ... mail_location = maildir:/var/spool/mail/%n mail_location = maildir:/var/spool/mail/vdom/%d/%n ... Also two configurations for passdb: ... passdb ldap { args = /etc/dovecot-ldap.conf } passdb ldap { args = /etc/dovecot-ldap-virt.conf } ... Here info is searched in ldap: /etc/dovecot-ldap-virt.conf ... base = dc=%d,ou=Virtual Domains,dc=eunet,dc=yu pass_filter = (&(objectClass=posixAccount)(uid=%n)) ... And the line in log file stating that maildir can not be located: mkdir(/var/spool/mail/vdom//cone/cur) failed: Permission denied Is this a bug or I am doing something wrong. Thank You, Nemanja

3 4