June 2008 - dovecot - dovecot.org

[Dovecot] v1.2 development tree started
by Timo Sirainen 20 Jun '08

20 Jun '08

I merged all the new features and latest v1.1 changes under one tree: http://hg.dovecot.org/dovecot-1.2/ The changeset order is kind of weird though. The timestamps jump all around. Wonder if there's a way to reorder them so that it looks more sane.. The status of the new features: 1. CONDSTORE extension is probably the largest change. It adds new "modification sequences" for messages that increase whenever the message's metadata changes. I'll probably have to reimplement the way modseqs are calculated, because modseqs will be very useful when implementing replication and the current way just doesn't work with it. If modseq-supporting clients see the current modseqs and later the server gets upgraded to new modseqs, the clients will most likely break. So this change should be done for v1.2. 2. QRESYNC, ESEARCH, SEARCHRES and WITHIN extensions are fully implemented. QRESYNC might need some cleanups and optimizations though. 3. CONTEXT=SEARCH extension is fully implemented, but could use a few optimizations (caching PARTIAL and CONTEXT searches, maybe others too). 4. Virtual mailboxes should work fast after mailbox is opened. The initial opening could use several optimizations though. It could probably share some code with QRESYNC to avoid the full initial search (storing each backend's modseq to index header). Also if search parameters don't contain any dynamically changing data, there's no point in searching the old messages. The current design doesn't allow changing the search parameters or list of mailboxes, otherwise it breaks more or less badly. I guess I could add code to check if the dovecot-virtual file's mtime has changed and if so make it do a full resync. This anyway means that there's no way to support wildcard mailbox names (e.g. "all mailboxes"). But does anyone really want that (yet)? It'll anyway be faster/easier to implement once mailbox list indexes are implemented. I'll still have to add a new X-MAILBOX search parameter which can be used to test what the backend mailbox name is. This will be especially useful with INTHREAD extension. I guess it wouldn't hurt to have FETCH X-MAILBOX if someone wants it. 5. Thread indexes should work, but some optimizations missing and there are some small bugs left. X-REFERENCES2 thread algorithm works. 6. INTHREAD extension isn't started yet, but I'll start it soon. Hopefully won't be too tricky to get it working with virtual mailboxes and CONTEXT=SEARCH..

4 7

[Dovecot] sig11 in 1.1rc5 fts
by Adam McDougall 20 Jun '08

20 Jun '08

This happened from one user near noon on the 17th and 19th (today) of this month. From the backtrace it looks like they were searching, but I won't know for sure unless I need to ask them. Is this possibly fixed already? I just haven't upgraded dovecot in a while due to lack of problems. The sig11 happened a few dozen times, a few seconds apart each day. I have one coredump from each day, and the size was the same. This is a trace from only one. The other backtrace looks pretty much the same. Version: 1.1rc5 OS: FreeBSD 7.0-STABLE #0 0x000000004101bf11 in node_read_children (trie=0x40c5a800, node=0x40c5a800, level=1) at squat-trie.c:461 data = (const uint8_t *) 0x416b7f9a ";\031\030\023\031\0305\031\030O\031\030\033\031\0309\031\030#\031\030" end = (const uint8_t *) 0x416b7ffe "" child_chars = (const unsigned char *) 0x414dbd25 <Address 0x414dbd25 out of bounds> child = (struct squat_node *) 0x40c685f8 children = (struct squat_node *) 0x40c68040 node_offset = 1944868 i = 61 child_idx = 61 child_count = 89 base_offset = 1944314 num = 25 __PRETTY_FUNCTION__ = "node_read_children" #1 0x000000004101f17f in squat_trie_map (trie=0x40c5a800, building=false) at squat-trie.c:1518 file_lock = (struct file_lock *) 0x0 dotlock = (struct dotlock *) 0x0 changed = true ret = 0 #2 0x000000004101b4c9 in squat_trie_open (trie=0x40c5a800) at squat-trie.c:242 No locals. #3 0x000000004101f908 in squat_trie_get_last_uid (trie=0x40c5a800, last_uid_r=0x7fffffffce5c) at squat-trie.c:1725 No locals. #4 0x000000004101a79c in fts_backend_squat_get_last_uid (_backend=0x40c2b150, last_uid_r=0x7fffffffce5c) at fts-backend-squat.c:104 backend = (struct squat_fts_backend *) 0x40c2b150 #5 0x0000000040e0f822 in fts_backend_get_last_uid (backend=0x40c2b150, last_uid_r=0x7fffffffce5c) at fts-api.c:80 No locals. #6 0x0000000040e10bb1 in fts_build_init (fctx=0x40cee0f0) at fts-storage.c:177 t = (struct mailbox_transaction_context *) 0x40c2b1e0 backend = (struct fts_backend *) 0x40c2b150 ctx = (struct fts_storage_build_context *) 0x40c2b150 build = (struct fts_backend_build_context *) 0x40c7e120 seqset = {seq1 = 16, seq2 = 1, next = 0x0} last_uid = 0 last_uid_locked = 131072 __PRETTY_FUNCTION__ = "fts_build_init" #7 0x0000000040e11351 in fts_try_build_init (fctx=0x40cee0f0) at fts-storage.c:320 No locals. #8 0x0000000040e114a6 in fts_mailbox_search_init (t=0x40c2b1e0, charset=0x4ceca0 "UTF-8", args=0x40c7e0c0, sort_program=0x0) at fts-storage.c:355 fbox = (struct fts_mailbox *) 0x40cdc040 ctx = (struct mail_search_context *) 0x40c28bc0 fctx = (struct fts_search_context *) 0x40cee0f0 #9 0x00000000004714e1 in mailbox_search_init (t=0x40c2b1e0, charset=0x4ceca0 "UTF-8", args=0x40c7e0c0, sort_program=0x0) at mail-storage.c:621 No locals. #10 0x0000000000419f9c in imap_search_init (cmd=0x40c7e048, box=0x40c67048, charset=0x4ceca0 "UTF-8", sargs=0x40c7e0c0) at cmd-search.c:36 ctx = (struct imap_search_context *) 0x40c7e190 #11 0x000000000041a644 in cmd_search (cmd=0x40c7e048) at cmd-search.c:190 ctx = (struct imap_search_context *) 0x40c1e240 sargs = (struct mail_search_arg *) 0x40c7e0c0 args = (const struct imap_arg *) 0x40c84080 args_count = 4 error = 0x0 charset = 0x4ceca0 "UTF-8" #12 0x000000000041bd12 in cmd_uid (cmd=0x40c7e048) at cmd-uid.c:26 command = (struct command *) 0x40c1e2a0 cmd_name = 0x40c84170 "SEARCH" #13 0x000000000041d3a4 in client_command_input (cmd=0x40c7e048) at client.c:553 client = (struct client *) 0x40c36180 command = (struct command *) 0x2340c7e0b8 __PRETTY_FUNCTION__ = "client_command_input" #14 0x000000000041d5e1 in client_command_input (cmd=0x40c7e048) at client.c:602 client = (struct client *) 0x40c36180 command = (struct command *) 0x40c1e240 __PRETTY_FUNCTION__ = "client_command_input" #15 0x000000000041d723 in client_handle_next_command (client=0x40c36180, remove_io_r=0x7fffffffd0fd) at client.c:643 size = 40 #16 0x000000000041d769 in client_handle_input (client=0x40c36180) at client.c:653 ---Type <return> to continue, or q <return> to quit--- _data_stack_cur_id = 3 ret = false remove_io = false handled_commands = false #17 0x000000000041d904 in client_input (client=0x40c36180) at client.c:698 cmd = (struct client_command_context *) 0xffffd16c output = (struct ostream *) 0x40c7a0a8 bytes = 40 __PRETTY_FUNCTION__ = "client_input" #18 0x00000000004be449 in io_loop_handler_run (ioloop=0x40c19140) at ioloop-kqueue.c:149 ctx = (struct ioloop_handler_context *) 0x40c1be60 events = (struct kevent *) 0x40c35000 event = (const struct kevent *) 0x40c35000 tv = {tv_sec = 1799, tv_usec = 999471} ts = {tv_sec = 1799, tv_nsec = 999471000} io = (struct io_file *) 0x40c15740 events_count = 3 t_id = 2 ret = 1 i = 0 __PRETTY_FUNCTION__ = "io_loop_handler_run" #19 0x00000000004bd51b in io_loop_run (ioloop=0x40c19140) at ioloop.c:301 No locals. #20 0x0000000000428a19 in main (argc=3, argv=0x7fffffffd250, envp=0x7fffffffd270) at main.c:293 No locals.

2 2

Re: [Dovecot] While searching: Assertion failed (offset > = ctx- >input->v_offset)
by Timo Sirainen 20 Jun '08

20 Jun '08

On Fri, 2008-06-20 at 09:42 +0200, Diego Liziero wrote: > On 6/20/08, Patrick Nagel <patrick.nagel(a)star-group.net> wrote: > > Hi, > > [..] > > > > Now I get this with 1.1.rc10: > > > > Jun 20 11:53:53 stshamail1 dovecot: Panic: IMAP(username): file > > message-parser.c: line 770 (message_parser_parse_next_block): assertion > > failed: (ctx->input->eof || ctx->input->closed || ctx->input->stream_errno != 0) > > [..] > > > > Patrick. > > Timo, > here is an anonymized mbox file that causes it at every body search > (tested with rc12). Did you test it without index files? I couldn't reproduce the crash. Do you use 32bit or 64bit Dovecot?

3 4

[Dovecot] v1.1.rc12 released
by Timo Sirainen 20 Jun '08

20 Jun '08

http://dovecot.org/releases/1.1/rc/dovecot-1.1.rc12.tar.gz http://dovecot.org/releases/1.1/rc/dovecot-1.1.rc12.tar.gz.sig Sorry, one more RC because of the mbox+quota bug. v1.1.0 still planned for tomorrow. - mbox: Don't give "Can't find next message offset" warnings when plugin (e.g. quota) accesses the message being saved. - deliver: Settings inside protocol imap {} weren't ignored.

5 5

[Dovecot] While searching: Assertion failed (offset >= ctx->input->v_offset)
by Patrick Nagel 20 Jun '08

20 Jun '08

Hi, while a user is searching through his (1.2 GB, >150000) mails, I get a lot of these messages in the log, and the search (with Thunderbird) doesn't finish: Jun 19 14:35:45 stshamail1 dovecot: Panic: IMAP(username): file message-parser.c: line 620 (preparsed_parse_body_init): assertion failed: (offset >= ctx->input->v_offset) Jun 19 14:35:45 stshamail1 dovecot: IMAP(username): Raw backtrace: imap [0x80ce9f4] -> imap [0x80ce60c] -> imap [0x80c83f4] -> imap(message_parser_parse_next_block+0x1d) [0x80c802d] -> imap(message_search_msg+0x5f) [0x80c9a7f] -> imap [0x8098e0b] -> imap [0x809e42b] -> imap(mail_search_args_foreach+0x40) [0x809e540] -> imap(index_storage_search_next_nonblock+0x256) [0x80989f6] -> imap [0x805dc96] -> imap [0x805e0d6] -> imap(io_loop_handle_timeouts+0x80) [0x80d59c0] -> imap(io_loop_handler_run+0x82) [0x80d6522] -> imap(io_loop_run+0x28) [0x80d5828] -> imap(main+0x4ac) [0x806809c] -> /lib/libc.so.6(__libc_start_main+0xdc) [0x4003adec] -> imap [0x805a021] Jun 19 14:35:45 stshamail1 dovecot: child 30070 (imap) killed with signal 6 What's wrong here? How to proceed with debugging? Thanks, Patrick. -- STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779 PGP key: https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005

2 3

[Dovecot] v1.1.rc11 released
by Timo Sirainen 20 Jun '08

20 Jun '08

http://dovecot.org/releases/1.1/rc/dovecot-1.1.rc11.tar.gz http://dovecot.org/releases/1.1/rc/dovecot-1.1.rc11.tar.gz.sig This is the final v1.1 RC. Tomorrow I'll change only the version number, update the documentation and call it v1.1.0. But please test this anyway to make sure I didn't happen to break anything since rc10 :) - dovecot-uidlist is now recreated if it results in file shrinking over 25%. - Some other minor fixes

5 8

[Dovecot] Compile failure on Dovecot 1.1-rc12
by Erinn Looney-Triggs 20 Jun '08

20 Jun '08

I am getting this failure when trying to compile dovecot on Sparc Solaris 8: make[4]: Entering directory `/home/systems/looneytr/dovecot-1.1.rc12/src/lib' source='str-find.c' object='str-find.o' libtool=no \ DEPDIR=.deps depmode=gcc /bin/bash ../../depcomp \ gcc -DHAVE_CONFIG_H -I. -I../.. -I/usr/local/openssl/include -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -c str-find.c str-find.c:17: field `goodtab' has incomplete type make[4]: *** [str-find.o] Error 1 This problem has occurred on two separate systems. Have you run into this issue, or are you aware of this issue? Dovecot 1.0.14 builds just fine. Thanks, -Erinn

3 2

Re: [Dovecot] Compile failure on Dovecot 1.1-rc12
by Chris Wakelin 20 Jun '08

20 Jun '08

Erinn Looney-Triggs wrote: > Chris Wakelin wrote: > >> What version of gcc? What configure options? I had no problems with gcc >> 3.3.2 (well, a still a few compiler warnings; see >> http://www.dovecot.org/list/dovecot/2008-June/031350.html) >> >> Chris >> > > Hello, > > gcc --version 2.95.3 > > Had to set LDFLAGS and CPPFLAGS to include the openssl in > /usr/local/openssl. > > ./configure --prefix=/usr/local/dovecot --with-ssl=openssl > > I am guessing that it probably has something to do with the old version > of gcc. > > Thanks for your time, > -Erinn > No, *my* version is 'old' - yours is prehistoric :) I've just checked, and yes, gcc 2.95.2 breaks in exactly the same way for me! Best Wishes, Chris -- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin(a)reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094

1 0

[Dovecot] debugging dovecot and sieve
by Frank Church 19 Jun '08

19 Jun '08

I have install dovecot and sieve and it doesn't appear to be working. Below is my dovecot.conf Are there some debugging settings I can use to test if dovecot is linking with sieve properly. Nothing appears to show up in the logs start dovecot.conf ======================= cat /usr/local/etc/dovecot.conf base_dir = /var/run/dovecot/ auth_verbose=yes auth_debug=yes mail_debug=yes auth_debug_passwords=yes disable_plaintext_auth=no protocols = imap imaps pop3 pop3s protocol imap { mail_plugins = quota imap_quota imap_client_workarounds = outlook-idle delay-newmail } protocol pop3 { mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xx } protocol lda { postmaster_address = postmaster(a)abc.meripol.net mail_plugins = quota cmusieve log_path = /var/log/vmail/dovecot-deliver.log info_log_path = /var/log/vmail/dovecot-deliver.log } ## ## Logging ## # Log file to use for error messages, instead of sending them to syslog. # /dev/stderr can be used to log into stderr. log_path = /var/log/dovecot/error_log # Log file to use for informational and debug messages. # Default is the same as log_path. info_log_path = /var/log/dovecot/general_log # Prefix for each line written to log file. % codes are in strftime(3) # format. #log_timestamp = "%b %d %H:%M:%S " # Syslog facility to use if you're logging to syslog. Usually if you don't # want to use "mail", you'll use local0..local7. Also other standard # facilities are supported. #syslog_facility = mail #ssl_listen = ssl_disable = no ssl_cert_file = /etc/ssl/certs/dovecot.pem ssl_key_file = /etc/ssl/private/dovecot.pem #ssl_key_password = #ssl_ca_file = #ssl_verify_client_cert = no #ssl_parameters_regenerate = 168 # SSL ciphers to use #ssl_cipher_list = ALL:!LOW # Show protocol level SSL errors. #verbose_ssl = no #ssl_ca_file = /etc/pki/dovecot/certs/ca-bundle.crt mail_location = maildir:/var/vmail/%d/%u mail_extra_groups = mail first_valid_uid = 601 last_valid_uid = 601 maildir_copy_with_hardlinks = yes auth default { mechanisms = plain passdb sql { args = /etc/dovecot/sql.conf } userdb sql { args = /etc/dovecot/sql.conf } userdb prefetch { } user = nobody socket listen { master { # path = /var/run/dovecot/auth-master path = /usr/local/var/run/dovecot/auth-master mode = 0660 user = vmail group = mail } client { path = /var/spool/postfix/private/auth mode = 0660 user = postfix group = mail } } } dict { } plugin { quota = maildir:storage=10240:messages=1000 acl = vfile:/etc/dovecot/acls trash = /etc/dovecot/trash.conf sieve = /var/vmail/%d/%u/.dovecot.sieve } ===================== end dovecot.conf Thanks vfclists

2 1

[Dovecot] Can't find next message offset
by Dean Brooks 19 Jun '08

19 Jun '08

Hi, I am getting a lot of warnings since upgrading to 1.1rc10 (from rc4): mbox /home2/seta/Mail/Sent: Can't find next message offset for uid=12 Almost always, it is the Sent or Trash folders that are giving these. Users aren't experiencing any problems, but the errors continue to come. Is this something I should be concerned about? Here is output of "dovecot -n": # 1.1.rc10: /usr/local/dovecot/etc/dovecot.conf base_dir: /var/run/dovecot/ protocols: imap login_dir: /var/run/dovecot//login login_executable: /usr/local/dovecot/libexec/dovecot/imap-login login_user: daemon mail_max_userip_connections: 5 first_valid_uid: 100 last_valid_uid: 128000 mail_privileged_group: mail mail_location: mbox:~/Mail:INBOX=/var/mail/%u:INDEX=/var/indexes/local/%u mail_nfs_storage: yes mbox_write_locks: fcntl mbox_lock_timeout: 180 mbox_lazy_writes: no mail_plugins: quota imap_quota namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes auth default: mechanisms: plain login failure_delay: 5 debug: yes passdb: driver: shadow userdb: driver: passwd plugin: quota: fs -- Dean Brooks dean(a)iglou.com

2 2