November 2007 - dovecot

[Dovecot] dovecot: pipe() failed: Too many open files
by Ralf Hildebrandt 12 Nov '07

12 Nov '07

I'm encountering lots of Oct 31 11:14:25 postamt dovecot: pipe() failed: Too many open files errors around midday ever since the latest upgrade to 1.0.7. The errors appear in pairs like this one: Oct 31 11:20:01 postamt dovecot: pipe() failed: Too many open files Oct 31 11:20:01 postamt dovecot: imap-login: Internal login failure: user=<username>, method=PLAIN, rip=193.175.174.239, lip=141.42.4.250, TLS What gives? -- Ralf Hildebrandt (Ralf.Hildebrandt(a)charite.de) plonk(a)charite.de Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155 http://www.arschkrebs.de We're using Postfix - it generally seems to take the approach that the message that arrives is the message that gets delivered, without any extra bits being added. -- Chris Stratford

2 3

[Dovecot] v2.0 / Master/config rewrite
by Timo Sirainen 12 Nov '07

12 Nov '07

If anyone's interested, I updated my master/config rewrite to latest Dovecot v1.1 tree and added it to hg: http://hg.dovecot.org/dovecot-master-rewrite/ It probably doesn't work, but with the exception of plugins it compiles anyway. :) I'm thinking that maybe after v1.1.0 I could continue with the rewrite and create v2.0 after all instead of v1.2. Just look at how much cleaner deliver.c is :) http://hg.dovecot.org/dovecot-master-rewrite/file/59ebbfcf21cb/src/deliver/… vs http://hg.dovecot.org/dovecot/file/eb4f1fbeecda/src/deliver/deliver.c Another thing with v2.0 I was thinking about was to change index file formats to be more efficient.

2 1

[Dovecot] some mail accesses in dovecot 1.1 stall for a bit (in progress)
by Adam McDougall 11 Nov '07

11 Nov '07

I noticed this in an earlier beta and took a note, and I still see it sometimes in 1.1b4, but it is only occasional so I have not tracked down yet what dovecot is doing. Just a FYI for the list incase others see it and to say that its an issue I tend to track down that I did not see in 1.0, and its been low on my list until now. Sometimes when I login to dovecot and read a message from an imap maildir folder, the message contents do not load for at least 10 seconds, sometimes closer to a minute, but after that first message loads, the rest seem fine in that mailbox. The time seems to vary some, usually longer rather than shorter.

2 9

[Dovecot] imap search/content-transfer-encoding assertion failed
by Andrew Garner 11 Nov '07

11 Nov '07

When performing an imap search, I hit the following assertion failure: imap(vmail): Panic: file unichar.c: line 128 (uni_ucs4_to_utf8_c): assertion failed: (chr <= 0x40000000) imap(vmail): Error: Raw backtrace: /usr/lib/dovecot/imap [0x80c7f7f] -> /usr/lib/dovecot/imap [0x80c7d6a] -> /usr/lib/dovecot/imap [0x80d8616] -> /usr/lib/dovecot/imap(uni_utf8_to_decomposed_titlecase+0x9f) [0x80d892f] -> /usr/lib/dovecot/imap(message_decoder_decode_next_block+0x679) [0x80c46e9] -> /usr/lib/dovecot/imap(message_search_more+0x3e) [0x80c2e8e] -> /usr/lib/dovecot/imap(message_search_msg+0x72) [0x80c3012] -> /usr/lib/dovecot/imap [0x8093f47] -> /usr/lib/dovecot/imap [0x80b7719] -> /usr/lib/dovecot/imap(mail_search_args_foreach+0x32) [0x80b77c2] -> /usr/lib/dovecot/imap(index_storage_search_next_nonblock+0x243) [0x8093af3] -> /usr/lib/dovecot/imap [0x805cdc7] -> /usr/lib/dovecot/imap [0x805d138] -> /usr/lib/dovecot/imap(io_loop_handle_timeouts+0x112) [0x80ce892] -> /usr/lib/dovecot/imap(io_loop_handler_run+0x66) [0x80cf316] -> /usr/lib/dovecot/imap(io_loop_run+0x28) [0x80ce768] -> /usr/lib/dovecot/imap(main+0x4ab) [0x806675b] -> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xc8) [0xb7e68ea8] -> /usr/lib/dovecot/imap [0x8059141] My quick analysis: This seemed to be caused by a single message in this mailbox. I think dovecot is expanding base64 encoded binary data and then inevitably failing on the utf8 conversion of the decoded random binary data. The mime headers for this message are: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------080305000603070709090808" ... This is a multi-part message in MIME format. --------------080305000603070709090808 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit <text content> ... --------------080305000603070709090808 Content-Type: application/x-zip-compressed; name="xxxxxxxx.zip" Content-Transfer-Encoding: base64 Content-Disposition: inline; filename="xxxxxxxxxxxx.zip" <base64 encoded zip file> ... --------------080305000603070709090808-- # dovecot -n # 1.1.beta6: /etc/dovecot/dovecot.conf protocols: imap pop3 listen: 127.0.0.1 disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(pop3): no first_valid_uid: 89 mail_location: maildir:~/Maildir mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_executable(default): /usr/lib/dovecot/rawlog /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/rawlog /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/rawlog /usr/lib/dovecot/pop3 mail_plugins(default): fts fts_lucene mail_plugins(imap): fts fts_lucene mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 mail_log_prefix: %Us(%u[%r]): mail_log_max_lines_per_sec: 100 pop3_uidl_format(default): %08Xu%08Xv pop3_uidl_format(imap): %08Xu%08Xv pop3_uidl_format(pop3): %f auth default: mechanisms: plain cram-md5 verbose: yes debug: yes debug_passwords: yes passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 384 user: vmail group: vmail plugin: fts: lucene

2 1

[Dovecot] assertion failed: (mailbox_list_is_valid_existing_name(_list, name))
by Adam McDougall 11 Nov '07

11 Nov '07

Do you need a gdb backtrace for this one? I'm not sure why all of a sudden this started happening, its probably due to me adding a folder somewhere? not sure. It happened when I went to open my folder subscriptions in thunderbird. It was taking longer than usual (call this phase 1) and I noticed this in the logs: (cant remember if this was phase 1 or 2 but the error is probably the same) Nov 11 00:38:38 boomhauer dovecot: imap-login: Login: user=<mcdouga9>, method=PLAIN, rip=208.53.102.126, lip=35.9.37.190, TLS Nov 11 00:38:39 boomhauer dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.boomhauer.3050.af3db3ac545170a7) failed: Operation not permitted Nov 11 00:38:44 boomhauer dovecot: IMAP(mcdouga9): file mailbox-list-maildir.c: line 186 (maildir_list_get_path): assertion failed: (mailbox_list_is_valid_existing_name(_list, name)) Nov 11 00:38:44 boomhauer dovecot: child 3050 (imap) killed with signal 6 Nov 11 00:38:45 boomhauer dovecot: imap-login: Login: user=<mcdouga9>, method=PLAIN, rip=208.53.102.126, lip=35.9.37.190, TLS Nov 11 00:38:45 boomhauer dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.boomhauer.3052.5538be060c2f9c7d) failed: Operation not permitted Nov 11 00:38:46 boomhauer dovecot: IMAP(mcdouga9): file mailbox-list-maildir.c: line 186 (maildir_list_get_path): assertion failed: (mailbox_list_is_valid_existing_name(_list, name)) Nov 11 00:38:46 boomhauer dovecot: child 3052 (imap) killed with signal 6 When I cancelled the subscription window, "phase 2" is the same thing but happening more rapidly. I traced what tbird is doing and can reproduce it with: ? OK Logged in. a list "" "#shared/decs/%/%" Connection closed by foreign host. Doing a list on #shared/decs/% works though.

2 2

[Dovecot] Outlook can't see namespace folders
by James 10 Nov '07

10 Nov '07

Hi guys i understand that you can't subscribe to the Prefix names of namespace folders right now with dovecot. But for instance if i user outlook express and it can't see the prefix name then it can't see the subfolders of that "prefix folder" either. So even though it can detect and see the subfolders it won't display them cause it can't see the "parent" Is there any fix for this? it's also kind of annoying in thunderbird but at least you can still see the subfolders. TIA James Here's my config # /etc/dovecot/dovecot.conf protocols = imap imaps pop3 pop3s log_timestamp = "%Y-%m-%d %H:%M:%S " first_valid_uid = 100 login_user=vmail mail_extra_groups = mail mail_debug = yes pop3_uidl_format = %08Xu%08Xv mail_location = maildir:/home/vmail/domains/%n/Maildir #default_mail_env = maildir:/home/vmail/domains/%n/Maildir auth default { mechanisms = plain passdb ldap { args = /etc/dovecot/dovecot-ldap.conf } userdb ldap { args = /etc/dovecot/dovecot-ldap.conf } } # maildir namespace private { separator = / prefix = location = maildir:/home/vmail/domains/%n/Maildir inbox = yes hidden = no } # mbox namespace private { separator = / prefix = "Mbox/" location = mbox:/home/vmail/domains/%n/Mbox inbox = no hidden = no } #public spam folder namespace public { separator = / prefix = /Spam/ location = maildir:/home/vmail/domains/Spam/Maildir/ hidden = no } ssl_disable = no ssl_cert_file = /etc/ssl/certs/dovecot.pem ssl_key_file = /etc/ssl/private/dovecot.pem ssl_verify_client_cert = no ssl_cipher_list = ALL:!LOW:HIGH:MEDIUM:+SSLv2:RSA

3 10

[Dovecot] Request for variable unique to each server?
by Adam McDougall 10 Nov '07

10 Nov '07

I've been looking for a variable I can use in my dovecot.conf within the INDEX= setting so I can have one index dir per imap server on NFS. I've been looking at http://wiki.dovecot.org/Variables but no variable seems to contain something like server hostname, and I've been hunting for the variable expansion code in dovecot but haven't been able to find it. %l (local IP) won't help me because when using our load balancer, %l is the same IP on all of my servers. %r is close to useful but having a local webmail server on each server means I'd end up with 127.0.0.1 being shared (also I would end up with one directory per unique user/clientIP combo). Is there a variable that holds the server hostname or something else unique per server, or could one be added, or could someone point me at the code? Thanks.

2 2

[Dovecot] UTF-8 in dovecot-sieve ?
by Arthur Pétry 10 Nov '07

10 Nov '07

Hi, Reading http://tools.ietf.org/html/rfc3028#section-2.7.2 I thought that out-of-the-box dovecot can filter messages by matching header in UTF-8. But when I tested it, it failed : (I'm using 1.0.5 with sieve plugin) Here is my .dovecot.sieve file : ---------- require ["fileinto", "imapflags"]; if allof(header :contains "subject" "filtre") { addflag "test_1"; } if allof(header :contains "subject" "là") { addflag "test_2"; } if allof(header :comparator "i;ascii-casemap" :contains "subject" "hé") { addflag "test_3"; } ---------- Only the first of the three rules do work. When I try to match a subject string with accent encoded in ISO-8859 like "hélà" : "=?ISO-8859-1?Q?h=E9l=E0?=" there is no match Am I doing something wrong ?

2 1

[Dovecot] 1.0beta7 still has SEARCH TEXT not search the FROM
by Asheesh Laroia 10 Nov '07

10 Nov '07

I just tried out 1.0beta7 and erased all my indexes and with fts and fts_squat enabled, and when I SEARCH TEXT in Dovecot, it does *not* find messages where the thing I SEARCH for appear only in the From: header. Timo, can you confirm or deny that you still see this behavior? I did clear my indexes before doing it. -- Asheesh. -- Man is the measure of all things. -- Protagoras

2 2

[Dovecot] Index corruption ?
by Geoffroy Desvernay 09 Nov '07

09 Nov '07

Hi, We are using dovecot for some times (a little before 1.0), for IMAP and (a little for the moment) deliver. We have ~20000 logins/day, Maildirs are over NFS (with indexes too, for the moment...) Up to 1.0.5, we sometime have this error in maillog: ________________________________________________________________________ Nov 1 13:55:41 tic dovecot: IMAP(login): file mail-index-view-sync.c: line 666 (mail_index_view_sync_end): assertion failed: (view->log_file_offset >= view->map->hdr.log_file_int_offset) ________________________________________________________________________ I didn't find any bits on google, nor on this list's archive... # gdb (without debug, I may do it if required) ________________________________________________________________________ Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/libiconv.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x182ad51f in kill () from /lib/libc.so.6 (gdb) bt #0 0x182ad51f in kill () from /lib/libc.so.6 #1 0x182ad4be in raise () from /lib/libc.so.6 #2 0x182ac0e8 in abort () from /lib/libc.so.6 #3 0x080a85da in i_set_failure_syslog () #4 0x080a80e1 in data_stack_deinit () #5 0x08092c97 in mail_index_view_sync_end () #6 0x0808229e in index_mailbox_sync_next () #7 0x0000001c in ?? () #8 0xbfbfe7e8 in ?? () #9 0x080d3140 in ?? () #10 0x080d8044 in ?? () #11 0x0000001c in ?? () #12 0xbfbfe828 in ?? () #13 0x08060761 in imap_sync_deinit () Previous frame inner to this frame (corrupt stack?) ____________________________________________________________________ dovecot -n # 1.0.5: /usr/local/etc/dovecot.conf ssl_ca_file: /etc/ssl/cacert.pem ssl_cert_file: /etc/ssl/private/imaps.ec-marseille.fr.pem ssl_key_file: /etc/ssl/private/imaps.ec-marseille.fr.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login login_greeting: IMAP Centrale ready. login_greeting_capability: yes login_process_size: 64 login_processes_count: 20 login_max_processes_count: 512 valid_chroot_dirs: /users:/home max_mail_processes: 2048 verbose_proctitle: yes first_valid_uid: 1000 last_valid_uid: 55000 first_valid_gid: 999 last_valid_gid: 55000 mail_extra_groups: mail mail_location: maildir:%h/Maildir/ mmap_disable: yes lock_method: flock maildir_stat_dirs: yes maildir_copy_with_hardlinks: yes mail_drop_priv_before_exec: yes mail_executable: /usr/local/libexec/dovecot/imap_convert imap_client_workarounds: delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep auth default: verbose: yes worker_max_count: 60 passdb: driver: pam args: imap passdb: driver: ldap args: /usr/local/etc/dovecot-ldap.conf userdb: driver: passwd userdb: driver: ldap args: /usr/local/etc/dovecot-uldap.conf socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 384 plugin: quota: fs ___________________________________________________________________ # dovecot-ldap.conf (dovecot-uldap.conf is a symlink) ___________________________________________________________________ hosts = ldapr.serv.int:389 ldap:389 dn = <our binddn> dnpass = <our pass> auth_bind = yes ldap_version = 3 base = ou=People,<our_basedn> deref = never scope = onelevel user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid,uid=system_user user_filter = (&(objectClass=inetLocalMailRecipient)(|(uid=%u)(mail=%u)(mail=%n@<our_primary_domain>))) pass_attrs = uid=user pass_filter = (&(objectClass=inetLocalMailRecipient)(|(uid=%u)(mail=%u)(mail=%n@<our_primary_domain>))) ____________________________________________________________________ -- ___________________________________________________________ / Geoffroy DESVERNAY | \ /\ `Service info` | Tel: (+33|0)4 91 05 45 24 /\ \/ Ecole Centrale de Marseille | Fax: (+33|0)4 91 05 45 98 \/ \ (ex-EGIM) | Mail: dgeo(a)ec-marseille.fr / -----------------------------------------------------------

3 2