- dovecot - dovecot.org

[dovecot] Message with lots of attachments breaks Dovecot
by Tim Hunt 21 Apr '03

21 Apr '03

I have a message with over a 1000 embedded attachments(!), and it breaks Dovecot. The issue appears to be buffer management. There is some logic in the function message_part_deserialize_part that seems to assume that the message part structure is in a contiguous array, but that doesn't seem to be the case. I haven't been able to figure out enough about the memory management in Dovecot to fix the problem. Below is a horrible workaround that just allocates a lot more memory for the message part array, so that it remains contiguous. *** mail-index-update.c.orig Wed Mar 26 10:41:55 2003 --- mail-index-update.c Sun Apr 20 16:53:45 2003 *************** *** 498,504 **** if (cache_fields & DATA_FIELD_MESSAGEPART) { t_push(); ! buf = buffer_create_dynamic(data_stack_pool, 2048, (size_t)-1); message_part_serialize(part, buf); --- 498,504 ---- if (cache_fields & DATA_FIELD_MESSAGEPART) { t_push(); ! buf = buffer_create_dynamic(data_stack_pool, 51200, (size_t)-1); message_part_serialize(part, buf);

2 1

[dovecot] Patch for zero content-length
by Tim Hunt 21 Apr '03

21 Apr '03

mbox-index.c uses a zero content length as a flag for an invalid Content-Length: header. Unfortunately zero is also a valid value for Content-Length. The patch below attempts to address the problem. *** mbox-index.h.orig Tue Apr 15 09:04:02 2003 --- mbox-index.h Sun Apr 20 11:42:12 2003 *************** *** 19,24 **** --- 19,25 ---- struct istream *input; uoff_t content_length; int set_read_limit; + unsigned int content_length_good:1; }; int mbox_set_syscall_error(struct mail_index *index, const char *function); *** mbox-index.c.orig Mon Mar 24 05:52:44 2003 --- mbox-index.c Sun Apr 20 12:41:16 2003 *************** *** 264,277 **** start_offset = ctx->input->v_offset; i_stream_set_read_limit(ctx->input, 0); ! end_offset = start_offset + ctx->content_length; ! if (ctx->content_length == 0 || !mbox_verify_end_of_body(ctx->input, end_offset)) { ! if (ctx->content_length != 0) i_stream_seek(ctx->input, start_offset); mbox_skip_message(ctx->input); end_offset = ctx->input->v_offset; ctx->content_length = end_offset - start_offset; } i_stream_seek(ctx->input, start_offset); --- 264,280 ---- start_offset = ctx->input->v_offset; i_stream_set_read_limit(ctx->input, 0); ! end_offset = start_offset; ! if ( ctx->content_length_good ) ! end_offset += ctx->content_length; ! if (!ctx->content_length_good || !mbox_verify_end_of_body(ctx->input, end_offset)) { ! if (ctx->content_length_good ) i_stream_seek(ctx->input, start_offset); mbox_skip_message(ctx->input); end_offset = ctx->input->v_offset; ctx->content_length = end_offset - start_offset; + ctx->content_length_good = TRUE; } i_stream_seek(ctx->input, start_offset); *************** *** 301,311 **** --- 304,316 ---- strcasecmp(hdr->name, "Content-Length") == 0) { /* manual parsing, so we can deal with uoff_t */ ctx->content_length = 0; + ctx->content_length_good = TRUE; for (i = 0; i < hdr->value_len; i++) { if (hdr->value[i] < '0' || hdr->value[i] > '9') { /* invalid */ ctx->content_length = 0; + ctx->content_length_good = FALSE; break; }

1 0

[dovecot] a few small bug
by Farkas Levente 18 Apr '03

18 Apr '03

during the ./autogen.sh there is a few problem: ----------------------------------------- configure.in: installing `./install-sh' configure.in: installing `./mkinstalldirs' configure.in: installing `./missing' Use of uninitialized value in concatenation (.) or string at /usr/bin/automake line 8449. : installing `./config.guess' Use of uninitialized value in concatenation (.) or string at /usr/bin/automake line 8449. : installing `./config.sub' configure.in:3: required file `./config.h.in' not found aclocal.m4:916: required file `./ltmain.sh' not found src/auth/Makefile.am: installing `./depcomp' configure.in:13: warning: AC_ARG_PROGRAM invoked multiple times autoheader: `config.h.in' is created ----------------------------------------- would it be possible to not use your own libtool in the cvs? in this case I can run autogen.sh during the rpm build and doesn't have to run manulay and patch the genereated makefile. ps. this is the patch I use: --- dovecot-1.0/configure.lfarkas 2003-03-20 15:53:04.000000000 +0100 +++ dovecot-1.0/configure 2003-03-20 15:54:37.000000000 +0100 @@ -8205,7 +8205,7 @@ LIBTOOL_DEPS="$ac_aux_dir/ltmain.sh" # Always use our own libtool. -LIBTOOL='$(SHELL) $(top_builddir)/libtool' +LIBTOOL='$(SHELL) libtool' # Prevent multiple expansion -- Levente "Si vis pacem para bellum!"

2 3

[dovecot] Corrupt mbox handling
by Kristian Hoffmann 17 Apr '03

17 Apr '03

I've found that if an mbox file has two consecutive 'From' delimiters, I get the error: pop3(user): Error indexing mbox file /var/spool/mail/user: LF not found where expected UofW apparently ignores the second 'From' silently. I've been trying to run dovecot in gdb to get more details on exactly what is happening, but I haven't quite mastered the art of attaching to the pop3 process between the login process completing and it exiting. Is there some trick to doing that? If I can't work around it in dovecot, I can just write something in perl to fix all the duplicate 'From' lines. Thanks, Kristian Hoffmann PC-InTOUCH/FireToWire System Administrator khoff(a)pc-intouch.com ---

2 1

[dovecot] Changing hierarchy separator to '/'
by Charlie Brady 17 Apr '03

17 Apr '03

3 4

[dovecot] PATCH Add support for kqueue in ioloop subsystem
by Dominic Marks 16 Apr '03

16 Apr '03

Hey, I noticed that there was an ioloop "module" (if we can call it that) for select and poll and decided to add one for kqueue (aka kevent) BSDs high performance descriptor multiplexing API. I haven't done any of the configure glue stuff but the code is complete and works well. kqueue is available on all recent versions of FreeBSD, NetBSD, OpenBSD and Darwin (and therefore MacOS X). I've tested both pop3 and imap with no problems. To get it working: (bodge method) > tar zxvf dovecot-0.99.8.1.tar.gz > cp ioloop-kevent.c dovecot-0.99.8.1/src/lib/ > cd dovecot-0.99.8.1/ > ./configure > vi config.h 1. replace IOLOOP_POLL (or IOLOOP_SELECT) with IOLOOP_KEVENT > vi src/lib/Makefile 1. find liblib_a_SOURCES add ioloop-kevent.c to it 2. find liblib_a_OBJECTS add ioloop-kevent.$(OBJEXT) to it > make There is a paper on kqueue which includes a performance evaulation here: http://people.freebsd.org/~jlemon/papers/kqueue.pdf Thanks, -- Dominic <dom at cus.org.uk> <d.marks at student.umist.ac.uk>

2 1

[dovecot] current cvs
by Farkas Levente 16 Apr '03

16 Apr '03

hi, so after upgrade to the current cvs all of our problem disappear:-) but we don't have time to test the ssl part... although after I compile and install the new dovecot I stop it I do a -------------------- find /home/*/Maildir -name '.imap*' -exec rm {} \; find /home/*/Maildir -name 'dovecot-uidlist' -exec rm {} \; find /home/*/Maildir -name 'msgid.cache' -exec rm {} \; find /home/*/Maildir -name '.customflags' -exec rm {}\; -------------------- just to be sure that we use a fresh version and no other problem can be caused by the old index or any other cached files. this was a BIG mistake!!! we've got about 300 users and about 100 online at the same time. after a two minutes break (while I reinstall and delete) every user check his mail which casue a load about 90!!! the cpu usage 1-2% but the avarage load was 90 (curently about 0.22)! the reason was IO. we have to stop imap and wait for a while... I don't know the exact reason bu I assume dovecot is regenerate everybody's index a other cache at the same time. is there any way to reduce the possible number of IO load? this would be useful if never happend again!:-) this happend around 4:40. the output of sar -b: -------------------- 03:00:00 PM tps rtps wtps bread/s bwrtn/s 03:10:00 PM 48.18 29.19 18.99 1456.53 771.29 03:20:00 PM 41.66 25.80 15.86 1054.88 830.05 03:30:00 PM 42.64 25.67 16.97 1223.03 759.77 03:40:00 PM 44.98 31.95 13.03 1959.15 467.40 03:50:00 PM 26.68 14.63 12.05 741.48 424.12 04:00:00 PM 25.80 11.70 14.10 516.21 450.04 04:10:00 PM 59.55 34.91 24.64 1755.07 939.11 04:20:00 PM 95.98 61.15 34.83 4453.24 1563.25 04:30:00 PM 95.08 55.45 39.63 2182.22 1591.56 04:40:00 PM 168.85 145.99 22.86 11155.73 1037.25 04:50:00 PM 135.98 106.50 29.48 7908.21 1604.76 05:00:00 PM 125.87 97.65 28.23 5876.03 1060.50 05:10:00 PM 80.83 62.92 17.90 4321.82 590.19 05:20:00 PM 102.04 77.65 24.40 4907.27 1076.92 05:30:00 PM 129.04 106.58 22.46 6387.18 883.13 05:40:00 PM 89.44 73.20 16.24 4905.64 782.94 05:50:00 PM 88.23 73.16 15.06 5539.93 965.34 06:00:00 PM 92.49 72.67 19.81 5486.25 692.18 06:10:00 PM 45.24 32.82 12.42 1711.35 618.31 06:20:00 PM 42.17 28.16 14.01 960.11 622.33 06:30:00 PM 29.90 20.24 9.66 1538.27 376.17 06:40:00 PM 25.89 12.14 13.74 796.73 833.25 06:50:00 PM 20.50 4.98 15.52 264.91 993.62 07:00:00 PM 15.08 7.57 7.52 453.41 282.63 -------------------- the output of sar -d -------------------- 04:10:00 PM dev22-3 0.00 0.00 04:10:00 PM dev33-0 1.76 109.55 04:10:00 PM dev34-0 8.51 919.94 04:20:00 PM dev3-0 8.56 183.93 04:20:00 PM dev22-2 67.62 3798.02 04:20:00 PM dev22-3 0.00 0.00 04:20:00 PM dev33-0 4.80 463.81 04:20:00 PM dev34-0 15.01 1570.73 04:30:00 PM dev3-0 4.40 126.63 04:30:00 PM dev22-2 77.02 2440.08 04:30:00 PM dev22-3 0.00 0.00 04:30:00 PM dev33-0 1.94 124.80 04:30:00 PM dev34-0 11.73 1082.28 04:40:00 PM dev3-0 4.78 124.56 04:40:00 PM dev22-2 137.27 10306.18 04:40:00 PM dev22-3 0.00 0.00 04:40:00 PM dev33-0 13.50 410.44 04:40:00 PM dev34-0 13.29 1351.80 04:50:00 PM dev3-0 6.33 194.27 04:50:00 PM dev22-2 111.50 7961.44 04:50:00 PM dev22-3 0.00 0.00 04:50:00 PM dev33-0 5.64 156.51 -------------------- -- Levente "Si vis pacem para bellum!"

2 5

[dovecot] Shadow password file
by Jaime Medrano 16 Apr '03

16 Apr '03

¿Is there any way to make dovecot to take the passwords from a file with a shadow password format? I've tried with the passwd-file auth plugin, but it complains that users have uid=0 Thanks in advance, Jaime Medrano

2 3

[dovecot] it seems to unusable
by Farkas Levente 16 Apr '03

16 Apr '03

hi, is there any way to turn off the whole indexing stuff? unfortunately the system getting totaly unusable beacuase of the high load and the mail index generation is so slow that every users call the helpdask that his mail is not working (what's more in this case just click on OE again and start another mailer process:-() we've got about 100k mails which is about 10-20GB! -- Levente "Si vis pacem para bellum!"

1 0

[dovecot] pop3 coredump
by Mark E. Mallett 16 Apr '03

16 Apr '03

Hi- I was able to get a coredump out of the new pop3 (from 0.99.9-test4) by attempting to simulate the execution environment and simply running the pop3 program. Running the older pop3 from 0.99.8.1 works fine, repeatedly: % /usr/local/libexec/dovecot/pop3 quit +OK Logging out. However running the new one: % /usr/local/libexec/dovecot/pop3.new pop3(user9): Error: Corrupted index file /users/48/user9/Maildir/.INBOX/.imap.index: index.next_uid (4539) > uidlist.next_uid (4000) Segmentation fault (core dumped) It doesn't always dump core even with this message. The error message is not always the same, but it's usually what's above. Simply repeating the invocation a number of times I also saw: pop3(user9): Error: Corrupted index file (in-memory index for /users/48/user9/Maildir): Filename mismatch for UID 1: 1050440153.17005.mercury.mv.net vs 1050437948.5418.mercury.mv.net Here's some gdb output corresponding to the coredump above. -mm- Script started on Tue Apr 15 16:51:28 2003 % gdb pop3/pop3 pop3.core ... Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/local/lib/libiconv.so.2...done. Reading symbols from /shlib/libgcc.so.1...done. Reading symbols from /shlib/libc.so.2...done. Reading symbols from /shlib/ld-bsdi.so...done. #0 0x0 in ?? () (gdb) bt #0 0x0 in ?? () #1 0x8053439 in index_storage_init (storage=0x808f000, box=0x8080658, index=0x808c200, name=0x807660e "INBOX", readonly=0, fast=0) at index-storage.c:264 #2 0x804b8e0 in maildir_open (storage=0x808f000, name=0x807660e "INBOX", readonly=0, fast=0) at maildir-storage.c:239 #3 0x804ba20 in maildir_open_mailbox (storage=0x808f000, name=0x80765b1 "INBOX", readonly=0, fast=0) at maildir-storage.c:288 #4 0x804b1c7 in main_init () at main.c:98 #5 0x804b269 in main (argc=1, argv=0x8047d3c, envp=0x8047d44) at main.c:132 #6 0x8049cae in __start () (gdb) up 1 #1 0x8053439 in index_storage_init (storage=0x808f000, box=0x8080658, index=0x808c200, name=0x807660e "INBOX", readonly=0, fast=0) at index-storage.c:264 264 storage->callbacks->notify_no(&ibox->box, (gdb) l 259 if (!index->open(index, flags)) 260 break; 261 262 if (INDEX_IS_IN_MEMORY(index) && 263 storage->index_dir != NULL) { 264 storage->callbacks->notify_no(&ibox->box, 265 "Couldn't use index files", 266 storage->callback_context); 267 } 268 } (gdb) p index $1 = (struct mail_index *) 0x808c200 (gdb) p *index $2 = {open = 0x8054590 <maildir_index_open>, free = 0x8054abc <maildir_index_free>, set_lock = 0x805c874 <mail_index_set_lock>, try_lock = 0x805c88c <mail_index_try_lock>, set_lock_notify_callback = 0x805c8a4 <mail_index_set_lock_notify_callback>, rebuild = 0x8054f20 <maildir_index_rebuild>, fsck = 0x805de50 <mail_index_fsck>, sync_and_lock = 0x8055db4 <maildir_index_sync>, get_header = 0x805c928 <mail_index_get_header>, lookup = 0x805c968 <mail_index_lookup>, next = 0x805cac0 <mail_index_next>, lookup_uid_range = 0x805cb48 <mail_index_lookup_uid_range>, lookup_field = 0x805cc58 <mail_index_lookup_field>, lookup_field_raw = 0x805ccac <mail_index_lookup_field_raw>, cache_fields_later = 0x805cd70 <mail_index_cache_fields_later>, open_mail = 0x8054de0 <maildir_open_mail>, get_internal_date = 0x8054afc <maildir_get_internal_date>, expunge = 0x805d078 <mail_index_expunge>, update_flags = 0x8054b6c <maildir_index_update_flags>, append_begin = 0x805d3ec <mail_index_append_begin>, append_end = 0x805d4e4 <mail_index_append_end>, append_abort = 0x805d590 <mail_index_append_abort>, update_begin = 0x805f9b0 <mail_index_update_begin>, update_end = 0x805ff68 <mail_index_update_end>, update_abort = 0x8060050 <mail_index_update_abort>, update_field = 0x806022c <mail_index_update_field>, update_field_raw = 0x806025c <mail_index_update_field_raw>, get_last_error = 0x805d618 <mail_index_get_last_error>, get_last_error_text = 0x805d678 <mail_index_get_last_error_text>, data = 0x80821a0, tree = 0x80900c0, modifylog = 0x808d200, custom_flags = 0x808d100, dir = 0x8090000 "/users/48/user9/Maildir/.INBOX", filepath = 0x8090040 "(in-memory index for /users/48/user9/Maildir)", mailbox_path = 0x8082160 "/users/48/user9/Maildir", custom_flags_dir = 0x0, default_cache_fields = 0, never_cache_fields = 0, indexid = 1050439124, sync_id = 6, excl_lock_counter = 1, mbox_fd = 0, mbox_stream = 0x0, mbox_lock_type = MAIL_LOCK_UNLOCK, mbox_dotlock = { dev = 0, ino = 0, mtime = 0}, mbox_lock_counter = 0, mbox_sync_counter = 0, mbox_size = 0, mbox_dev = 0, mbox_ino = 0, uidlist_dev = 18875392, uidlist_ino = 1835546, maildir_lock_fd = -1, fd = -1, error = 0x0, mmap_base = 0x20073000, mmap_used_length = 264, mmap_full_length = 1608, header = 0x20073000, lock_type = MAIL_LOCK_EXCLUSIVE, file_sync_stamp = 1050438868, first_recent_uid = 1, lock_notify_cb = 0x8053238 <lock_notify>, lock_notify_context = 0x808c400, set_flags = 0, set_cache_fields = 0, anon_mmap = 1, opened = 0, rebuilding = 0, mail_read_mmaped = 0, inconsistent = 0, nodiskspace = 0, index_lock_timeout = 0, allow_new_custom_flags = 0, mailbox_readonly = 0, mailbox_lock_timeout = 0} (gdb) p storage $3 = (struct mail_storage *) 0x808f000 (gdb) p *storage $5 = {name = 0x8076805 "maildir", hierarchy_sep = 46 '.', create = 0x804b2c0 <maildir_create>, free = 0x804b428 <maildir_free>, autodetect = 0x804b480 <maildir_autodetect>, set_callbacks = 0x80535e4 <index_storage_set_callbacks>, open_mailbox = 0x804b940 <maildir_open_mailbox>, create_mailbox = 0x804ba5c <maildir_create_mailbox>, delete_mailbox = 0x804bb08 <maildir_delete_mailbox>, rename_mailbox = 0x804bea4 <maildir_rename_mailbox>, list_mailbox_init = 0x804c624 <maildir_list_mailbox_init>, list_mailbox_deinit = 0x804c800 <maildir_list_mailbox_deinit>, list_mailbox_next = 0x804cc48 <maildir_list_mailbox_next>, set_subscribed = 0x8064594 <subsfile_set_subscribed>, get_mailbox_name_status = 0x804bffc <maildir_get_mailbox_name_status>, get_last_error = 0x805455c <mail_storage_get_last_error>, dir = 0x8082100 "/users/48/user9/Maildir", inbox_file = 0x0, index_dir = 0x8082120 "/users/48/user9/Maildir", user = 0x808e050 "user9", error = 0x0, callbacks = 0x8082140, callback_context = 0x0, syntax_error = 0} (gdb) p ibox $6 = (struct index_mailbox *) 0x808c400 (gdb) p *ibox $7 = {box = {name = 0x808e070 "INBOX", storage = 0x808f000, close = 0x805359c <index_storage_close>, get_status = 0x8052ee4 <index_storage_get_status>, sync = 0x8053b94 <index_storage_sync>, auto_sync = 0x804c0c8 <maildir_storage_auto_sync>, expunge = 0x804fa64 <index_storage_expunge>, update_flags = 0x8053d58 <index_storage_update_flags>, copy = 0x804c3e4 <maildir_storage_copy>, fetch_init = 0x804fb00 <index_storage_fetch_init>, fetch_deinit = 0x804fbe8 <index_storage_fetch_deinit>, fetch_next = 0x804fc70 <index_storage_fetch_next>, fetch_uid = 0x804fd5c <index_storage_fetch_uid>, fetch_seq = 0x804fdc0 <index_storage_fetch_seq>, search_get_sorting = 0x8052af4 <index_storage_search_get_sorting>, search_init = 0x8052b08 <index_storage_search_init>, search_deinit = 0x8052be0 <index_storage_search_deinit>, search_next = 0x8052ca0 <index_storage_search_next>, save_init = 0x804cf4c <maildir_storage_save_init>, save_deinit = 0x804cfe0 <maildir_storage_save_deinit>, save_next = 0x804ce04 <maildir_storage_save_next>, is_inconsistency_error = 0x8054578 <mail_storage_is_inconsistency_error>, readonly = 0, allow_custom_flags = 0, inconsistent = 0}, expunge_locked = 0, index = 0x808c200, autosync_to = 0x0, autosync_files = 0x0, autosync_type = MAILBOX_SYNC_NONE, sync_last_check = 0, min_newmail_notify_interval = 0, fetch_mail = {mail = {seq = 0, uid = 0, seen_updated = 0, get_flags = 0, get_parts = 0, get_received_date = 0, get_date = 0, get_size = 0, get_header = 0, get_address = 0, get_first_mailbox = 0, get_stream = 0, get_special = 0}, data = {flags = { flags = 0, custom_flags = 0x0, custom_flags_count = 0}, received_date = 0, size = 0, sent_time = 0, sent_timezone = 0, headers = 0x0, parts = 0x0, envelope = 0x0, body = 0x0, bodystructure = 0x0, envelope_data = 0x0, rec = 0x0, stream = 0x0, hdr_size = {physical_size = 0, virtual_size = 0, lines = 0}, body_size = { physical_size = 0, virtual_size = 0, lines = 0}, parse_header = 0, save_sent_time = 0, save_envelope = 0, hdr_size_set = 0, body_size_set = 0}, pool = 0x0, ibox = 0x0, header_buf = 0x0, wanted_fields = 0, wanted_headers = 0x0}, synced_messages_count = 0, next_lock_notify = 1050439154, last_notify_type = MAIL_LOCK_NOTIFY_MAILBOX_ABORT, sent_diskspace_warning = 0} (gdb) quit % exit Script done on Tue Apr 15 16:53:06 2003

2 3