August 2010 - dovecot

Re: [Dovecot] (Single instance) attachment storage
by Damon Atkins 27 Aug '10

27 Aug '10

Some interesting reading on SHA256 checksum http://blogs.sun.com/bonwick/entry/zfs_dedup http://blogs.sun.com/darren/entry/improving_zfs_dedup_performance_via

1 0

[Dovecot] Single instance storage - testing please
by Timo Sirainen 27 Aug '10

27 Aug '10

http://hg.dovecot.org/dovecot-2.0-sis contains the code for it. Otherwise it's the latest (as of writing this) dovecot-2.0 hg tree. Please test if you're interested in SIS. :) Once there's at least some testing, I'll probably add this to v2.0.x since very little of this new code is used when SIS is disabled (which is the default of course). SIS works pretty much like explained in http://dovecot.org/list/dovecot/2010-July/050832.html and http://dovecot.org/list/dovecot/2010-July/050992.html Two things I'm not yet entirely sure about: 1. What hash algorithm to use? Currently it's hard coded to SHA1. Besides more CPU usage, the other potential problem with larger hashes is that they also generate larger filenames. The filenames are currently hex-encoded, but to save space they could be changed to some kind of modified-base64 (base64 uses '/' chars, so it can't be regular base64). Example filename lengths: hex modified-base64 SHA1 73 50 SHA256 97 66 SHA512 161 109 Yet another possibility would be to use SHA256/SHA512 and just truncate the hash length to less number of bits. 2. Should I add support for trusting hash uniqueness and to avoid disk I/O generated by the byte-by-byte comparison? It could still first check that the file sizes match. Usage ----- You can enable SIS for sdbox and mdbox: mail_attachment_dir = /var/attachments Just setting the above enables "instant SIS", where byte-by-byte comparison is done immediately during saving mails. Alternative is to leave the comparing later by setting: mail_attachment_fs = sis-queue /var/attachments/queue:posix This does no deduplication itself yet. To do that you'll need a nighty (or whatever) run, which calls: doveadm sis deduplicate /var/attachments /var/attachments/queue There's also a feature to easily find all attachments based on a hash. For example: % sha1sum foo 351641b73feb7cf7e87e5a8c3ca9a37d7b21e525 foo % doveadm sis find /var/attachments 351641b73feb7cf7e87e5a8c3ca9a37d7b21e525 /var/attachments/35/16/351641b73feb7cf7e87e5a8c3ca9a37d7b21e525-e13a841f28ba764c123b00008c4a11c1 /var/attachments/35/16/351641b73feb7cf7e87e5a8c3ca9a37d7b21e525-1d3b940628ba764c0b3b00008c4a11c1 If you want to save attachments to a separate files without SIS (e.g. you want to use your filesystems deduplication), set: mail_attachment_fs = posix By default only attachments larger than 128 kB are written to attachment storage. You can change it from: mail_attachment_min_size = 128k It's also possible to create a plugin that adds further restrictions to when the attachment is saved separately. This might be useful to reduce disk seeks for attachments that are typically shown inline by clients/webmail. You can do this by overriding mailbox.save_is_attachment() method. If you want to distribute attachments to multiple filesystems, just create /var/attachments/[0-9a-f][0-9a-f] as symlinks pointing to whatever mount paths you want.

3 8

[Dovecot] Panic, rawtrace, log, etc
by Brandon Lamb 27 Aug '10

27 Aug '10

While running some doveadm expunge -A mailbox Spam savedbefore 30d and doveadm purge -A after 2010-08-26 16:20:51 auth: Error: net_accept() failed: Too many open files 2010-08-26 16:20:51 master: Warning: service(auth): process_limit reached, client connections are being dropped 2010-08-26 16:21:21 auth: Panic: file auth-request-handler.c: line 85 (auth_request_handler_unref): assertion failed: (hash_table_count(handler->requests) == 0) 2010-08-26 16:21:21 auth: Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x39c1a) [0x7fe41f5f5c1a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x39c6a) [0x7fe41f5f5c6a] -> /usr/local/lib/dovecot/libdovecot.so.0(i_error+0) [0x7fe41f5d3026] -> dovecot/auth [8 wait, 0 passdb, 0 userdb]() [0x412153] -> dovecot/auth [8 wait, 0 passdb, 0 userdb]() [0x40c3ad] -> dovecot/auth [8 wait, 0 passdb, 0 userdb](auth_client_connections_deinit+0x41) [0x40c461] -> dovecot/auth [8 wait, 0 passdb, 0 userdb](main+0x2ea) [0x4168ca] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7fe41e60ac4d] -> dovecot/auth [8 wait, 0 passdb, 0 userdb]() [0x40b1f9] 2010-08-26 16:21:21 master: Error: service(auth): child 14880 killed with signal 6 (core dumps disabled) 2010-08-26 16:21:21 auth: Error: Master requested auth for nonexistent client 14875 2010-08-26 16:21:21 auth: Error: Master requested auth for nonexistent client 14873 2010-08-26 16:21:21 auth: Error: Master requested auth for nonexistent client 14873 2010-08-26 16:21:21 auth: Error: Master requested auth for client 14874 with invalid cookie 2010-08-26 16:21:21 auth: Error: Master requested auth for client 14874 with invalid cookie 2010-08-26 16:21:21 auth: Error: Master requested auth for client 14875 with invalid cookie 2010-08-26 16:21:21 auth: Error: Master requested auth for client 14876 with invalid cookie 2010-08-26 16:21:21 auth: Error: Master requested auth for client 14875 with invalid cookie

2 3

[Dovecot] Configuration Update From v1.x To v2.0.1-2
by Carlos Mennens 26 Aug '10

26 Aug '10

I just upgraded my Dovecot IMAP package on my mail server and when I restarted the service, I got the following warnings... doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:12: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:14: passdb pam {} has been replaced by passdb { driver=pam } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:16: userdb passwd {} has been replaced by userdb { driver=passwd } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:18: auth_user has been replaced by service auth { user } The list was much longer but I was able to self resolve them. I was wondering if someone here can assist me in converting my 'dovecot.conf' file? Below is how my dovecot.conf file appears: protocols = imap disable_plaintext_auth = yes log_timestamp = "%b %d %H:%M:%S " mail_location = maildir:~/mail auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep } auth default { mechanisms = plain login passdb pam { } userdb passwd { } user = root socket listen { client { path = /var/run/dovecot/auth-client user = postfix group = postfix mode = 0660 } } } ***doveconf -n*** [root@mail /]# doveconf -n # 2.0.1: /etc/dovecot/dovecot.conf doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:12: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:14: passdb pam {} has been replaced by passdb { driver=pam } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:16: userdb passwd {} has been replaced by userdb { driver=passwd } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:18: auth_user has been replaced by service auth { user } # OS: Linux 2.6.35 x86_64 auth_mechanisms = plain login mail_location = maildir:~/mail passdb { driver = pam } protocols = imap service auth { unix_listener auth-client { group = postfix mode = 0660 user = postfix } user = root } userdb { driver = passwd } protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep }

4 7

[Dovecot] How to pipe to external program
by Egbert 26 Aug '10

26 Aug '10

Hi, Second try... I have mailsystem with virtual users only (/home/vmail/domain/user). I need to integrate RT3.x (a ticketing system). I have installed the RT-client but I need to send the mail to an extrnal program. Rt wants me to insert: |/usr/bin/rt-mailgate --queue bla --action bla --url http://bla.bal/ In /etc/postfix/aliases. That does not work for me becase my local delivery agent is dovecot. The Sieve plugin seems not capable to forward to a external program. What is the right way to achieve that mail to rt@mydomain is sent to the pipe? TIA Egbert Jan

2 1

Re: [Dovecot] Investigating suspected cache corruption
by Maria Elinger 26 Aug '10

26 Aug '10

Hi David and all, I´ve read the mail you posted in the Dovecot list. We´re are having this same problem in a site which is very similar to yours. We also migrated from Courier to Dovecot nearly 9 months ago, and every now and then users complain about slowness in the imap connection. We fix it by removing the index files in the user's maildir. We use Horde as webmail, but the problem appears with any imap or pop connection, so we feel that the problem is in Dovecot. Did you find any solution or workaround at your site?, it would be very useful if you gave us a hint on this. Thanks in advance, María Elinger [Dovecot] Investigating suspected cache corruption David Warden Wed, 14 Oct 2009 13:18:17 Hi All, I'm running Dovecot 1.1.11 for a site of about 7000 active users with a Squirrelmail frontend. We migrated from Courier (big shout to everyone who worked on the wonderful part of the Dovecot wiki that deals with Courier migration) and I have been extremely happy with the performance gains Dovecot has given us. Not only is the IMAP part of Squirrelmail faster, but our same hardware is under far less stress than it was before the migration. To date, the only problem we have experienced is a seemingly random but significant slowdown in Squirrelmail's loading of the messages in a user's Inbox. Each time, I have been able to fix the problem by deleting the user's dovecot.index, dovecot.index.cache and dovecot.index.log files in the user's INDEX directory for their dovecot Inbox. I tried having users experiencing this problem log in to a test Squirrelmail box with mail_debug on, but I didn't see anything out of the ordinary and they report the same slowness on the test server. I don't see anything but normal login/logout IMAP lines in the production dovecot logs. I have hesitated a bit in posting to the list since this seems to be happening to a small percentage of users and it technically may be a problem with Squirrelmail, but the fact that I'm able to fix the problem by nuking Dovecot's index files has me concerned - does anyone have any suggestions for troubleshooting? My first idea when I get another one of these is to do a telnet IMAP session to Dovecot on one of the Squirrelmail boxes and see if I can get it to slow down when retrieving messages from the Inbox. We are using Maildir over NFS with the control/index files stored on NFS as well. We're doing FS quota over NFS using rquotad to query the NFS server for each user's quota. Here's the output of dovecot -n: # 1.1.11: /etc/dovecot.conf Warning: fd limit 1024 is lower than what Dovecot can use under full load (more than 1536). Either grow the limit or change login_max_processes_count and max_mail_processes settings # OS: Linux 2.6.18-128.1.1.0.1.el5 x86_64 Red Hat Enterprise Linux Server release 5.3 (Tikanga) login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_process_per_connection: no login_processes_count: 10 login_max_processes_count: 512 verbose_proctitle: yes first_valid_uid: 0 mail_location: maildir:~/Maildir:CONTROL=/Mail/mailhome/ %u/.dovecot:INDEX=/Mail/mailhome/%u/.dovecot mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_drop_priv_before_exec: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): imap_logout_format(default): bytes(in/out)=%i/%o imap_logout_format(imap): bytes(in/out)=%i/%o imap_logout_format(pop3): bytes=%i/%o pop3_uidl_format(default): %08Xu%08Xv pop3_uidl_format(imap): %08Xu%08Xv pop3_uidl_format(pop3): UID%u-%v pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh pop3_logout_format(default): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_logout_format(imap): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_logout_format(pop3): top(count/bytes)=%t/%p, retr(count/bytes)=%r/ %b, del(deleted/pre-deleted)=%d/%m, mailbox-size(bytes)=%s namespace: type: private prefix: INBOX. inbox: yes list: yes subscriptions: yes auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd plugin: quota: fs:Mail Quota:user Many thanks, David Warden

2 2

[Dovecot] 1.2.13 QRESYNC crash.
by David Woodhouse 26 Aug '10

26 Aug '10

Aug 18 22:07:31 twosheds IMAP(dwmw2): : Panic: file mail-index-transaction.c: line 637 (mail_index_transaction_lookup): assertion failed: (seq >= t->first_new_seq && seq <= t->last_new_seq) * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS] Logged in as dwmw2 0 enable condstore qresync * ENABLED CONDSTORE QRESYNC 0 OK Enabled. A00131 SELECT lists.bluez (QRESYNC (1154090296 1861 1:* (1,120,1578,2064,2226,2280,2298 1,120,12037,12523,12685,12739,12757))) Program received signal SIGABRT, Aborted. 0x00000034ace326c5 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-9.fc12.x86_64 glibc-2.11.2-1.x86_64 keyutils-libs-1.2-6.fc12.x86_64 krb5-libs-1.7.1-9.fc12.x86_64 libcom_err-1.41.9-8.fc12.x86_64 libgcc-4.4.4-10.fc12.x86_64 libselinux-2.0.90-5.fc12.x86_64 nss-softokn-freebl-3.12.6-2.fc12.1.x86_64 openldap-2.4.19-4.fc12.x86_64 openssl-1.0.0a-1.fc12.x86_64 zlib-1.2.3-23.fc12.x86_64 (gdb) bt #0 0x00000034ace326c5 in raise () from /lib64/libc.so.6 #1 0x00000034ace33ea5 in abort () from /lib64/libc.so.6 #2 0x00000000004a2698 in default_fatal_finish (type=<value optimized out>, status=0) at failures.c:160 #3 0x00000000004a273b in i_syslog_fatal_handler (type=LOG_TYPE_PANIC, status=0, fmt=<value optimized out>, args=<value optimized out>) at failures.c:315 #4 0x00000000004a1d86 in i_panic (format=0x6 <Address 0x6 out of bounds>) at failures.c:207 #5 0x00000000004787f0 in mail_index_transaction_lookup ( t=<value optimized out>, seq=<value optimized out>) at mail-index-transaction.c:637 #6 0x000000000047be45 in tview_lookup_uid (view=<value optimized out>, seq=<value optimized out>, uid_r=<value optimized out>) at mail-index-transaction-view.c:162 #7 0x00000000004580fb in index_mail_set_seq (_mail=0x7282b8, seq=1578) at index-mail.c:1229 #8 0x0000000000423eb4 in expunges_drop_known (ctx=0x70a628) at imap-fetch.c:178 #9 get_expunges_fallback (ctx=0x70a628) at imap-fetch.c:269 #10 imap_fetch_send_vanished (ctx=0x70a628) at imap-fetch.c:291 #11 imap_fetch_begin (ctx=0x70a628) at imap-fetch.c:314 #12 0x000000000041e8e2 in select_qresync (cmd=<value optimized out>, ---Type <return> to continue, or q <return> to quit--- readonly=<value optimized out>) at cmd-select.c:243 #13 select_open (cmd=<value optimized out>, readonly=<value optimized out>) at cmd-select.c:320 #14 cmd_select_full (cmd=<value optimized out>, readonly=<value optimized out>) at cmd-select.c:382 #15 0x0000000000420ead in client_command_input (cmd=0x70a458) at client.c:612 #16 0x0000000000420f8d in client_command_input (cmd=0x70a458) at client.c:661 #17 0x00000000004211b5 in client_handle_next_command (client=0x709ff0) at client.c:702 #18 client_handle_input (client=0x709ff0) at client.c:714 #19 0x0000000000421a5c in client_input (client=0x709ff0) at client.c:753 #20 0x00000000004aa7cd in io_loop_handler_run (ioloop=<value optimized out>) at ioloop-epoll.c:208 #21 0x00000000004a9c38 in io_loop_run (ioloop=0x707cc0) at ioloop.c:335 #22 0x0000000000429726 in main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at main.c:327 It's a public mailing list folder, so I've uploaded it to http://david.woodhou.se/bluez-maildir.tar.gz Filed as Red Hat bug #625907: https://bugzilla.redhat.com/show_bug.cgi?id=625207 -- dwmw2

3 10

[Dovecot] New problem with migration to 2.0
by Carsten Laun-De Lellis 26 Aug '10

26 Aug '10

Hi guys After fixing my master_auth problem i have one error message left and i don't know what to do. The message from dovecot -n is: service auth { unix_listener /var/spool/postfix/private/auth { group = vmail mode = 0666 user = vmail } unix_listener auth-master { mode = 0666 } } My conf file looks like userdb { driver=static args = uid=10000 gid=10000 home=/mail/%d/%n allow_all_users=yes } As you can see pretty simple, but for me it looks like that the driver=static won't be accepted. Does anyone know which driver to use here ? Mit freundlichem Gruß Carsten Laun-De Lellis Dipl.-Ing. Elektrotechnik Certified Information Systems Auditor (CISA) Hauptstrasse 13 D-67705 Trippstadt Phone: +49 (6306) 992140 Mobile: +49 (151) 27530865 email: carsten.laun-delellis(a)delellis.net

2 4

[Dovecot] migration to dovecot 2.0
by Carsten Laun-De Lellis 26 Aug '10

26 Aug '10

Hi everyone I am not the dovecot guru and i would appreciate if anyone could help me on my problems. I am running a mail server on dovecot 1.2 and want to migrate to dovecot 2. I have edited the conf file and now using the new keywords and got everything fixed but the following error message still exists, when i try to start dovecot. doveconf: Fatal: Error in configuration file /usr/local/etc/dovecot/dovecot.conf line 1036: Unknown setting: auth_socket doveconf: Error: managesieve-login: dump-capability process returned 89 doveconf: Fatal: Error in configuration file ./dovecot.conf line 1036: Unknown setting: auth_socket My conf file looks like auth_socket listen { master { # Master socket provides access to userdb information. It's typically # used to give Dovecot's local delivery agent access to userdb so it # can find mailbox locations. path = /var/run/dovecot/auth-master mode = 0600 # Default user/group is the one who started dovecot-auth (root) user = vmail #group = } client { # The client socket is generally safe to export to everyone. Typical use # is to export it to your SMTP server so it can do SMTP AUTH lookups # using it. path = /var/spool/postfix/private/auth # path = /var/run/dovecot/auth-client mode = 0660 user = postfix group = postfix } } I would appreciate any hints and help. Mit freundlichem Gruß Carsten Laun-De Lellis Dipl.-Ing. Elektrotechnik Certified Information Systems Auditor (CISA) Hauptstrasse 13 D-67705 Trippstadt Phone: +49 (6306) 992140 Mobile: +49 (151) 27530865 email: carsten.laun-delellis(a)delellis.net

2 2

[Dovecot] Does Dovecot support the use of the INDEX directive for mdbox?
by William Blunn 26 Aug '10

26 Aug '10

Does Dovecot support the use of the INDEX directive for mdbox? e.g. mail_location = mdbox:~/mdbox:INDEX=/var/indexes/%u Bill

2 1