August 2009 - dovecot

[Dovecot] sieve/managesieve and spam filtering
by Charles Sprickman 11 Aug '09

11 Aug '09

Hello all, I've got a test environment setup in preparation for a move from qmail/vpopmail/courier to postfix/padmin/dovecot. I have a number of questions that seem to span multiple pieces of software, and this is one of them... Our policy with spam filtering is that a user should be able to turn it off (ie: not put tagged spam into a spam folder, but deliver to their inbox) if they want to. Currently qmailadmin and a custom squirrelmail plugin give people the option to do this by dumping a .qmail file in their directory that calls a common maildrop script that will deliver spam to the spam folder. It looks like I can emulate this with sieve and something that can speak to Dovecot's managesieve server. Where I'm stuck is that if I want users to be able to do other custom filtering using sieve, how do I go about making sure that my common spam delivery rule does not get clobbered? This really has me a bit stumped. I see there's a global sieverc that can be included, but I need something along the lines of a per-user include that brings in the spam filtering rule that will "stick" until the user explicitly deletes it. Any ideas? Thanks, Charles ___ Charles Sprickman NetEng/SysAdmin Bway.net - New York's Best Internet - www.bway.net spork(a)bway.net - 212.655.9344

2 2

[Dovecot] Index cache file problems in Dovecot 1.2.0
by Chris Wakelin 11 Aug '09

11 Aug '09

I've been seeing lots of index cache file errors (using mbox on Solaris 8 sparc 64-bit, but with 32-bit Dovecot) since I switched my account to Dovecot 1.2.0 (we're still on 1.0.15 mostly, but I'm hoping to upgrade to 1.1.17+ this summer, or 1.2.x if it's stable enough). e.g. Error: Corrupted index cache file <path>/.imap/INBOX/dovecot.index.cache: field index too large (47 >= 25) I've tried renaming/removing the .imap/INBOX directory, but the "Corrupted index cache file" errors return within a day or so. This morning, Thunderbird got itself in a mess retrying something that caused multiple core dumps: > Jul 14 08:52:53 dovecot: Error: child 17694 (imap) killed with signal 10 (core dumped) > Jul 14 08:52:54 dovecot: Error: child 17697 (imap) killed with signal 10 (core dumped) > Jul 14 08:52:55 dovecot: Error: child 17700 (imap) killed with signal 10 (core dumped) Here's a backtrace > #0 mail_cache_get_record (cache=0x124e28, offset=289, rec_r=0xffbeeb6c) > at mail-cache-lookup.c:30 > 30 if (rec->size < sizeof(*rec)) { > (gdb) bt > #0 mail_cache_get_record (cache=0x124e28, offset=289, rec_r=0xffbeeb6c) > at mail-cache-lookup.c:30 > #1 0x84e2c in mail_cache_lookup_iter_next_record (ctx=0xffbeeb60) > at mail-cache-lookup.c:186 > #2 0x84f14 in mail_cache_lookup_iter_next (ctx=0xffbeeb60, field_r=0xffbeeb50) > at mail-cache-lookup.c:219 > #3 0x85650 in mail_cache_lookup_headers_real (view=0x128510, dest=0x129728, > seq=11416, field_idxs=0xffbeec74, fields_count=2) > at mail-cache-lookup.c:500 > #4 0x85848 in mail_cache_lookup_headers (view=0x128510, dest=0x129728, > seq=11416, field_idxs=0xffbeec74, fields_count=2) > at mail-cache-lookup.c:555 > #5 0x6ba94 in index_mail_get_raw_headers (mail=0x128f20, > field=0xff142ee8 "Message-ID", value_r=0xffbeecfc) > at index-mail-headers.c:614 > #6 0x6c084 in index_mail_get_first_header (_mail=0x128f20, > field=0xff142ee8 "Message-ID", decode_to_utf8=false, value_r=0xffbeede4) > at index-mail-headers.c:771 > #7 0x7893c in mail_get_first_header (mail=0x128f20, > field=0xff142ee8 "Message-ID", value_r=0xffbeede4) at mail.c:118 > #8 0xff1419a0 in mail_log_add_hdr (mail=0x128f20, str=0x10edc8, > name=0xff142d68 "msgid", header=0xff142ee8 "Message-ID") > at mail-log-plugin.c:277 > #9 0xff141c5c in mail_log_action (dest_trans=0x128f20, mail=0x128f20, > event=MAIL_LOG_EVENT_DELETE, data=0x0) at mail-log-plugin.c:318 > #10 0xff141e5c in mail_log_mail_update_flags (_mail=0x128f20, > modify_type=MODIFY_ADD, flags=12) at mail-log-plugin.c:376 > #11 0x78af8 in mail_update_flags (mail=0x128f20, modify_type=MODIFY_ADD, > flags=12) at mail.c:184 > #12 0x2ab3c in cmd_store (cmd=0x11bec8) at cmd-store.c:192 > #13 0x2b5f8 in cmd_uid (cmd=0x11bec8) at cmd-uid.c:27 > #14 0x2ccc4 in client_command_input (cmd=0x11bec8) at client.c:611 > #15 0x2cb68 in client_command_input (cmd=0x11bec8) at client.c:660 > #16 0x2cd4c in client_handle_next_command (client=0x11bc40, > remove_io_r=0xffbef23f) at client.c:701 > #17 0x2ce40 in client_handle_input (client=0x11bc40) at client.c:713 > #18 0x2cf78 in client_input (client=0x11bc40) at client.c:752 > #19 0xc6c04 in io_loop_handler_run (ioloop=0x1186d8) at ioloop-poll.c:203 > #20 0xc63f4 in io_loop_run (ioloop=0x1186d8) at ioloop.c:338 > #21 0x369f4 in main (argc=1107968, argv=0xffbef484, envp=0xffbef494) > at main.c:323 I didn't have these problems with 1.1.x. I guess I ought to get out the imaptest tool and see if I can reproduce this reliably ... Best Wishes, Chris -- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin(a)reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094

2 3

[Dovecot] % in Usernames (dovecot 1.1.18)
by Michal Hlavinka 11 Aug '09

11 Aug '09

Hi, I'm forwarding feature request from one Fedora user: <snip> Shortly before suicide after migration to dbmail/postfix from Eudora Mailserver because we use % in Usernames as fallback and Apple-Mail does no Plaintext- Auth if CRAM-MD% was used before i installed dovecot as proxy BUT it allows no % in Username Please could be the following patch included? --- dovecot-1.1.18/src/master/master-settings.c 2009-07-27 03:56:32.000000000 +0200 +++ dovecot-1.1.18-patched/src/master/master-settings.c 2009-08-10 22:57:09.728065279 +0200 @@ -310,7 +310,7 @@ MEMBER(executable) PKG_LIBEXECDIR"/dovecot-auth", MEMBER(user) "root", MEMBER(chroot) "", - MEMBER(username_chars) "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@", + MEMBER(username_chars) "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@%", MEMBER(username_translation) "", MEMBER(username_format) "", MEMBER(master_user_separator) "", </snip> Would it be possible or is there any reason why this is a wrong idea? Regards, Michal Hlavinka

2 1

[Dovecot] dovecot-1.2.3 (managesieve) crash with backtrace
by Ralf Hildebrandt 11 Aug '09

11 Aug '09

>From the log: Aug 11 09:07:23 postamt dovecot: IMAP(zensy): Panic: file mail-index-transaction-view.c: line 106 (tview_apply_flag_updates): assertion failed: (map->hdr.record_size <= tview->record_size) Aug 11 09:07:23 postamt dovecot: IMAP(zensy): Raw backtrace: imap [0x80f0411] -> imap [0x80f0482] -> imap [0x80efe29] -> imap [0x80c839b] -> imap [0x80cea95] -> imap [0x80c85cf] -> imap(mail_index_lookup_ext_full+0x36) [0x80cda46] -> imap(mail_cache_lookup_cur_offset+0x49) [0x80bbad9] -> imap [0x80bbb5d] -> imap(mail_cache_lookup_iter_init+0x55) [0x80bbc05] -> imap(mail_cache_field_exists+0xaf) [0x80bc51f] -> imap(mail_cache_field_want_add+0x48) [0x80bcd98] -> imap(index_mail_parse_header_init+0x1c3) [0x80a4a33] -> imap(index_mail_parse_headers+0x60) [0x80a4c40] -> imap [0x80a500f] -> imap(index_mail_get_first_header+0x22) [0x80a5372] -> imap(mail_get_first_header+0x25) [0x80b0a85] -> /usr/local/lib/dovecot/imap/lib20_mail_log_plugin.so [0xb7e72bfb] -> /usr/local/lib/dovecot/imap/lib20_mail_log_plugin.so [0xb7e72f95] -> /usr/local/lib/dovecot/imap/lib20_mail_log_plugin.so [0xb7e73494] -> imap(cmd_store+0x475) [0x8065e95] -> imap [0x806769c] -> imap [0x8067743] -> imap(client_handle_input+0x2d) [0x806789d] -> im Aug 11 09:07:23 postamt dovecot: IMAP(zensy): io_loop_handler_run+0xe0) [0x80f9000] -> imap(io_loop_run+0x20) [0x80f8470] -> imap(main+0x5d2) [0x8070b52] Aug 11 09:07:23 postamt dovecot: dovecot: child 16960 (imap) killed with signal 6 (core dumped) Backtrace: GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu"... Reading symbols from /lib/i686/cmov/libdl.so.2...Reading symbols from /usr/lib/debug/lib/i686/cmov/libdl-2.9.so...done. done. Loaded symbols for /lib/i686/cmov/libdl.so.2 Reading symbols from /lib/i686/cmov/librt.so.1...Reading symbols from /usr/lib/debug/lib/i686/cmov/librt-2.9.so...done. done. Loaded symbols for /lib/i686/cmov/librt.so.1 Reading symbols from /lib/i686/cmov/libc.so.6...Reading symbols from /usr/lib/debug/lib/i686/cmov/libc-2.9.so...done. done. Loaded symbols for /lib/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...Reading symbols from /usr/lib/debug/lib/ld-2.9.so...done. done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/i686/cmov/libpthread.so.0...Reading symbols from /usr/lib/debug/lib/i686/cmov/libpthread-2.9.so...done. done. Loaded symbols for /lib/i686/cmov/libpthread.so.0 Reading symbols from /usr/local/lib/dovecot/imap/lib10_quota_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib10_quota_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib11_trash_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib11_trash_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib20_autocreate_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib20_autocreate_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib20_fts_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib20_fts_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib20_mail_log_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib20_mail_log_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so Reading symbols from /usr/lib/libz.so.1...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /lib/libbz2.so.1.0...done. Loaded symbols for /lib/libbz2.so.1.0 Reading symbols from /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Core was generated by `imap [zensy 141.42.4.251]'. Program terminated with signal 6, Aborted. [New process 16960] #0 0xb801d424 in __kernel_vsyscall () #0 0xb801d424 in __kernel_vsyscall () No symbol table info available. #1 0xb7ecf3d0 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 resultvar = <value optimized out> pid = -1207963660 selftid = 16960 #2 0xb7ed2a85 in *__GI_abort () at abort.c:88 act = {__sigaction_handler = {sa_handler = 0x940d238, sa_sigaction = 0x940d238}, sa_mask = {__val = {3214463400, 135192458, 3087001312, 0, 3214463520, 3214463508, 512, 0, 0, 3214463584, 3087251056, 134566650, 1131, 0, 0, 0, 1, 135265534, 512, 1, 3214463508, 155243032, 69, 3214463636, 3214463480, 135194054, 155244195, 3214463508, 3214463512, 1132, 155243064, 3086442259}}, sa_flags = 1132, sa_restorer = 0x81059a9 <write_full+57>} sigs = {__val = {32, 0 <repeats 31 times>}} #3 0x080f0425 in default_fatal_finish (type=<value optimized out>, status=0) at failures.c:160 backtrace = 0x940ca08 "imap [0x80f0411] -> imap [0x80f0482] -> imap [0x80efe29] -> imap [0x80c839b] -> imap [0x80cea95] -> imap [0x80c85cf] -> imap(mail_index_lookup_ext_full+0x36) [0x80cda46] -> imap(mail_cache_lookup_cur_"... #4 0x080f0482 in i_internal_fatal_handler (type=LOG_TYPE_PANIC, status=0, fmt=0x81084ec "file %s: line %d (%s): assertion failed: (%s)", args=0xbf98d2f4 "�X\021\bj") at failures.c:440 No locals. #5 0x080efe29 in i_panic ( format=0x81084ec "file %s: line %d (%s): assertion failed: (%s)") at failures.c:207 No locals. #6 0x080c839b in tview_lookup_full (view=0x9425230, seq=1, map_r=0xbf98d3f8, expunged_r=0xbf98d3c7) at mail-index-transaction-view.c:106 rec = (const struct mail_index_record *) 0x9434838 #7 0x080cea95 in view_lookup_ext_full (view=0x9425230, seq=1, ext_id=4, map_r=0xbf98d3f8, data_r=0xbf98d3f4, expunged_r=0xbf98d3c7) at mail-index-view.c:414 rec = <value optimized out> #8 0x080c85cf in tview_lookup_ext_full (view=0x9425230, seq=1, ext_id=4, map_r=0xbf98d3f8, data_r=0xbf98d3f4, expunged_r=0xbf98d3c7) at mail-index-transaction-view.c:408 ext_buf = (const union array__seq_array *) 0x0 data = <value optimized out> idx = <value optimized out> __PRETTY_FUNCTION__ = "tview_lookup_ext_full" #9 0x080cda46 in mail_index_lookup_ext_full (view=0x6, seq=1, ext_id=4, map_r=0xbf98d3f8, data_r=0xbf98d3f4, expunged_r=0x0) at mail-index-view.c:596 expunged = false #10 0x080bbad9 in mail_cache_lookup_cur_offset (view=0x9425230, seq=1, reset_id_r=0xbf98d42c) at mail-cache-lookup.c:65 cache = (struct mail_cache *) 0x94233b8 map = <value optimized out> data = (const void *) 0x4 offset = <value optimized out> #11 0x080bbb5d in mail_cache_lookup_offset (cache=0x94233b8, view=0x6, seq=16960, offset_r=0xbf98d498) at mail-cache-lookup.c:87 offset = <value optimized out> reset_id = 155333560 ret = <value optimized out> #12 0x080bbc05 in mail_cache_lookup_iter_init (view=0x9425410, seq=1, ctx_r=0xbf98d480) at mail-cache-lookup.c:154 ret = <value optimized out> #13 0x080bc51f in mail_cache_field_exists (view=0x9425410, seq=1, field=12) at mail-cache-lookup.c:292 __PRETTY_FUNCTION__ = "mail_cache_field_exists" #14 0x080bcd98 in mail_cache_field_want_add (ctx=0x9427af0, seq=1, field_idx=12) at mail-cache-transaction.c:1009 decision = MAIL_CACHE_DECISION_NO #15 0x080a4a33 in index_mail_parse_header_init (mail=0x94264b8, _headers=0x94319b0) at index-mail-headers.c:189 _data_stack_cur_id = 5 i = <value optimized out> field_idx = <value optimized out> __PRETTY_FUNCTION__ = "index_mail_parse_header_init" #16 0x080a4c40 in index_mail_parse_headers (mail=0x94264b8, headers=0x94319b0) at index-mail-headers.c:431 input = (struct istream *) 0x9423900 old_offset = 0 #17 0x080a500f in index_mail_get_raw_headers (mail=0x94264b8, field=0xb7e736f4 "Message-ID", value_r=0xbf98d608) at index-mail-headers.c:625 headers = {0xb7e736f4 "Message-ID", 0x0} value = <value optimized out> headers_ctx = (struct mailbox_header_lookup_ctx *) 0x0 data = <value optimized out> field_idx = 12 dest = (string_t *) 0x942cbf8 i = <value optimized out> len = <value optimized out> ret = <value optimized out> __PRETTY_FUNCTION__ = "index_mail_get_raw_headers" #18 0x080a5372 in index_mail_get_first_header (_mail=0x94264b8, field=0xb7e736f4 "Message-ID", decode_to_utf8=false, value_r=0xbf98d65c) at index-mail-headers.c:771 list = (const char * const *) 0x108 ret = 155239272 i = 0 #19 0x080b0a85 in mail_get_first_header (mail=0x94264b8, field=0xb7e736f4 "Message-ID", value_r=0xbf98d65c) at mail.c:118 No locals. #20 0xb7e72bfb in mail_log_add_hdr (mail=0x0, str=0x940c368, name=0xb7e736ff "msgid", header=0xb7e736f4 "Message-ID") at mail-log-plugin.c:277 value = 0xb7e72bdc "\201ô\035" #21 0xb7e72f95 in mail_log_action (dest_trans=<value optimized out>, mail=0x94264b8, event=MAIL_LOG_EVENT_DELETE, data=0x0) at mail-log-plugin.c:318 size = <value optimized out> str = (string_t *) 0x940c368 #22 0xb7e73494 in mail_log_mail_update_flags (_mail=0x94264b8, modify_type=MODIFY_ADD, flags=MAIL_DELETED) at mail-log-plugin.c:376 _data_stack_cur_id = 16960 lmail = <value optimized out> old_flags = <value optimized out> new_flags = <value optimized out> #23 0x08065e95 in cmd_store (cmd=0x9419688) at cmd-store.c:192 client = (struct client *) 0x94193f8 args = (const struct imap_arg *) 0x941e718 search_args = (struct mail_search_args *) 0x0 search_ctx = (struct mail_search_context *) 0x9427890 t = (struct mailbox_transaction_context *) 0x9427a50 mail = (struct mail *) 0x94264b8 ctx = {cmd = 0x9419688, max_modseq = 18446744073709551615, flags = MAIL_DELETED, keywords = 0x0, modify_type = MODIFY_ADD, silent = false} modified_set = {arr = {buffer = 0x9427b90, element_size = 8}, v = 0x9427b90, v_modifiable = 0x9427b90} uids = {arr = {buffer = 0x941e788, element_size = 3214464872}, v = 0x941e788, v_modifiable = 0x941e788} flags = <value optimized out> imap_sync_flags = <value optimized out> reply = <value optimized out> tagged_reply = <value optimized out> str = <value optimized out> ret = <value optimized out> #24 0x0806769c in client_command_input (cmd=0x9419688) at client.c:611 client = (struct client *) 0x94193f8 command = <value optimized out> __PRETTY_FUNCTION__ = "client_command_input" #25 0x08067743 in client_command_input (cmd=0x9419688) at client.c:660 client = (struct client *) 0x94193f8 command = <value optimized out> __PRETTY_FUNCTION__ = "client_command_input" #26 0x0806789d in client_handle_input (client=0x94193f8) at client.c:701 _data_stack_cur_id = 3 ret = false remove_io = <value optimized out> handled_commands = false __PRETTY_FUNCTION__ = "client_handle_input" #27 0x080681df in client_input (client=0x94193f8) at client.c:752 cmd = <value optimized out> output = (struct ostream *) 0x94195ac bytes = <value optimized out> __PRETTY_FUNCTION__ = "client_input" #28 0x080f9000 in io_loop_handler_run (ioloop=0x94149b0) at ioloop-epoll.c:208 ctx = (struct ioloop_handler_context *) 0x9414ab8 event = (const struct epoll_event *) 0x9414af8 list = (struct io_list *) 0x9419630 io = (struct io_file *) 0x9419608 tv = {tv_sec = 1, tv_usec = 575794} t_id = 2 msecs = <value optimized out> ret = 1 i = 0 j = 0 call = <value optimized out> #29 0x080f8470 in io_loop_run (ioloop=0x94149b0) at ioloop.c:338 No locals. #30 0x08070b52 in main (argc=) at main.c:327 No locals. $1 = {prev = 0x0, next = 0x0, client = 0x94193f8, pool = 0x9419670, tag = 0x9419700 "A004", name = 0x9419710 "UID STORE", args = 0x9419720 "1:* +FLAGS (\\Deleted)", cmd_flags = COMMAND_FLAG_BREAKS_SEQS, func = 0x8065a20 <cmd_store>, context = 0x0, module_contexts = {arr = {buffer = 0x94196c8, element_size = 4}, v = 0x94196c8, v_modifiable = 0x94196c8}, parser = 0x941e678, state = CLIENT_COMMAND_STATE_WAIT_INPUT, sync = 0x0, uid = 1, cancel = 0, param_error = 0, search_save_result = 0, temp_executed = 0} (gdb) quit -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

1 0

[Dovecot] Problems with Dovecot 1.2.3 + IMAP (SSL) + iPhone OS 3.0
by Sascha Scandella 11 Aug '09

11 Aug '09

Hello Has anyone else problems with Dovecot 1.2.3? For clients such as Thunderbird, Outlook 2007 everything works perfect. When I use an iPhone the child process is killed: Aug 11 08:08:44 imap-login: Info: Login: user=<username>, method=PLAIN,rip=1.2.3.4, lip=4.3.2.1, TLS Aug 11 08:08:44 dovecot: Error: child 5382 (imap) killed with signal 11 (core dumps disabled) Currently I have downgraded again to the working 1.2. I wonder if this problem is known or anyone is looking for it.

2 1

[Dovecot] RFC: Storing indexes in a (remote) database to increase dovecot scalability
by Mark Zealey 10 Aug '09

10 Aug '09

Hi all, This is just an idea I had over the weekend; I was wondering if instead of storing the indexes/caches on disk, it would be possible to have an option to store them in a remote database (eg mysql). There are several issues that we currently have with scaling dovecot, and I think that if it could store indexes in an external database we could alleviate most of these issues. In no particular order: 1) One major issue we have with dovecot is the number of iops it requires to keep its files in sync on our big disk arrays. If this were a mysql database, we'd have better ways of keeping these in sync, for example switching the speed at which it flushes the updates to disk, growth using partitioning/sharding, read-only slave instances etc, specific ssd storage for indexes/caches. I know that we could create a separate ssd disk array and export that over nfs too, but see issue (2) below. If indexes were stored in a mysql database we could choose to have a different reliability for the indexes (ie if the database server crashes we have 1 second of index changes lost, whereas for the email itself we can ensure it is always kept in sync on our other disk arrays). 2) nfs locking issues. No matter how hard we try, we always find there are some issues with concurrency over nfs. It really doesn't seem designed to handle these issues very well. I've not tried with nfsv4 and I believe there are some improvements there, but at the same time nfs wasn't really designed to store databases and have heavy concurrency. A proper database server (eg mysql, postgres) on the other hand thrives in this type of environment, especially when coupled with a transactional storage backend such as innodb which includes support for deadlock handling etc. 3) As we store all of our mail systems over nfs, effectively each export is a single-threaded connection to the server. No matter how fast the central nfs server you can realistically not go above 10k nfs ops/sec on the fastest of local networks due to latency issues. With mysql you could have multiple parallel connections to the central database store (1 per process?) which would mean more nfs ops available for actually serving the files. 4) Caching. Much more tuneable on a database server than in a filesystem. Eg in mysql there are loads of options to tune the various innodb/query/key caches, on nfs there are very few even on advanced netapps etc I know some of you store all your indexes on a single server locally & then store the emails on a remote central filestore. Whilst this is a good idea, it doesn't scale very well in that if the one server fails then everything has to be reindexed as clients login which causes incredible load as dovecot has to re-read most of the messages (in pop3 mode at least) in order to work out size counts etc. In my experience this reindexing would take out our email system for the best part of a day. I'm sure we could set something up with drbd mirroring or a central san array with ssd's for the index data, but this intrinsically does not scale with as much ease as a replicating database cluster could. There are also failover issues which can be very easily solved with a database system (eg mysql with an active/passive dual-master setup). Also, I know that most of these points refer to nfs; and I know that there are alternative methods (eg san + gfs or some other scalable filesystem) however most of these are relatively new technologies whereas nfs is pretty much the standard for remote file storage; certainly I'd consider it a lot more reliable when compared to the likes of gfs or lustre. Your typical large scale mail setup would be over nfs and this is the sort of environment that I've run into these scaling issues in. I've heard that gfs also doesn't work very well with 'hot spots' which indexes tend to be, although I've no first hand experience of this. So, although this post only really mentions nfs & mysql (the two technologies that I'm most familiar with) I think this is a much more general problem so I'm not advocating the use of either technology apart from them being most people's first choices for ease of use. What I'm really proposing is a way to separate the two different types of data that dovecot uses. Indexes&caches contain small frequent transactions; and whilst these are relatively important, dropping the past few seconds of these transactions doesn't seriously matter. On the other hand emails are larger, infrequently changed & it's important they are kept fully in sync - if we accept a message it should be guaranteed that we don't loose it. At the end of the day it seems to me that a filesystem is the ideal place to store emails, whereas a database server would be the ideal place to store indexes/caches. Obviously for speed of setup, simply storing everything on a filesystem is easy so I'm not suggesting that we abandon this route, but for truly large infrastructures I think being able to use a database system would have significant advantages in each of these four areas. Any thoughts? Mark -- Mark Zealey -- Platform Architect Product Development * Webfusion 123-reg.co.uk, webfusion.co.uk, donhost.co.uk, supanames.co.uk This mail is subject to http://www.gxn.net/disclaimer

2 1

[Dovecot] Dovecot-1.1.15 panics
by Mike Abbott 10 Aug '09

10 Aug '09

Dovecot-1.1.15 assert-fails frequently. Attached are some errors from the logs, and a stack trace from the common panic. Hope you can fix these soon. Thanks. 0 libSystem.B.dylib 0x00007fff8825e606 __kill + 10 1 libSystem.B.dylib 0x00007fff882e1c92 abort + 83 2 imap 0x0000000100069e1b default_fatal_finish + 70 3 imap 0x0000000100069e69 i_syslog_fatal_handler + 0 4 imap 0x00000001000695c8 i_info + 0 5 imap 0x000000010001913f maildir_uidlist_iter_next_rec + 210 6 imap 0x0000000100019177 maildir_uidlist_iter_next + 38 7 imap 0x0000000100018abb maildir_sync_index + 2019 8 imap 0x0000000100017788 maildir_sync_context + 662 9 imap 0x000000010001797c maildir_storage_sync_init + 104 10 imap 0x000000010000bbe3 imap_sync_init + 132 11 imap 0x000000010000be35 cmd_sync_delayed + 477 12 imap 0x0000000100006692 client_handle_input + 419 13 imap 0x000000010000676c client_input + 148 14 imap 0x0000000100070e60 io_loop_handler_run + 272 15 imap 0x0000000100070225 io_loop_run + 55 16 imap 0x000000010000d746 main + 1640 17 imap 0x0000000100000b84 start + 52

3 8

Re: [Dovecot] Mail archive
by fernando＠dfcom.com.br 10 Aug '09

10 Aug '09

I would like 'some process' to store old mails at a cheap storage. I know how to do it with symlinks, but I don´t know if it is the best option. So, I´m asking if dovecot improves it somehow. Fernando fernando at dfcom.com.br schrieb: > Hi, > > I´m using dovecot at our storages, accounts is distributed among many > storages (not entire domain), all of them with compression (when message > > 30Kb), nightly crontab script. > > Even though compression and many storages, we always have problems with > disk space and need to migrate accounts. > > Have you ever implemented any archive solution working with dovecot ? > > Regards, > Fernando > you might give more info what kind of "archive" you exactly mean meanwhile study http://wiki.dovecot.org/HowTo/ReadOnlyArchive http://wiki.dovecot.org/Plugins/Zlib http://wiki.dovecot.org/Plugins/Expire Alternative dbox directory expiration -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria

2 1

[Dovecot] Mail archive
by fernando＠dfcom.com.br 10 Aug '09

10 Aug '09

Hi, I´m using dovecot at our storages, accounts is distributed among many storages (not entire domain), all of them with compression (when message > 30Kb), nightly crontab script. Even though compression and many storages, we always have problems with disk space and need to migrate accounts. Have you ever implemented any archive solution working with dovecot ? Regards, Fernando

2 1

[Dovecot] inconsistency with expire-tool and expire dict
by LEVAI Daniel 10 Aug '09

10 Aug '09

Hi! Here is the problem: passdb: daniell:*::::::user=daniell2 userdb: daniell2::uid:gid:gecos:home:: dovecot.conf: plugin { expire = SA.* 1 # (There are SA.HAM and SA.SPAM directories) } When copying a message to eg. the SA.HAM directory, then dovecot inserts this into my expires table: dovecot=# select * from expires ; username | mailbox | expire_stamp ---------------------+---------+-------------- daniell | SA.HAM | 1249976454 ^^^^^^^ wrong username This causes an error when running expire-tool (after changing expire_stamp): # /usr/local/sbin/dovecot --exec-mail ext /usr/local/libexec/dovecot/expire- tool --test Info: User lookup failed: daniell Info: daniell/SA.HAM: no messages left If I change the username field in the expires table... : # UPDATE expires SET username = 'daniell2'; UPDATE 1 ... expire-tool is fine: # /usr/local/sbin/dovecot --exec-mail ext /usr/local/libexec/dovecot/expire- tool --test Info: daniell2/SA.HAM: timestamp 1249880093 (Mon Aug 10 06:54:53 2009) -> 1249976454 (Tue Aug 11 09:40:54 2009) Daniel -- LÉVAI Dániel PGP key ID = 0x4AC0A4B1 Key fingerprint = D037 03B9 C12D D338 4412 2D83 1373 917A 4AC0 A4B1

1 1