- dovecot - dovecot.org

[Dovecot] got too little data ??
by Benoit Branciard 12 Dec '07

12 Dec '07

I just switched in a hurry one of our mailbox servers from UW-imapd to Dovecot in an attempt to address performance and stability problems. This server hosts ~5500 accounts, ~2000 are daily used, ~600 simultaneous IMAP connexions in rush hours, and lots of POP ones. Some accounts may be accessed simultaneously with IMAP and POP. The system is : - Debian Sarge (x86 with amd64 kernel) - MBOX files mailboxes - indexes migrated out of filesystem quotas - Sendmail MTA - Procmail local delivery agent (not easily replaceable due to extensive use of procmailrc filters) - Both Procmail and Dovecot use [dotlock, fcntl] as mailbox locking scheme - No other processes have access to mailboxes in normal use - Dovecot 1.0.8 hand-compiled from tarball The good new is that it seems to perform better than UW-imapd. The bad new is that we get lots or errors like this in log: IMAP(username): FETCH for mailbox INBOX UID 23862 got too little data: 3186 vs 3206 Such "accidents" happen about 4-5 times a day, and does not correct themselves, even with a mailbox expunge or reopen. However manual deletion of the faulty mailbox index seems to clear the problem, until another one occurs. On such errors some mail clients (Thunderbird) retry operation in infinite loop, so log gets really cluttered. In imap-fetch-body.c it is stated : "We shouldn't really ever get here. One reason is if mail was deleted from NFS server while we were reading it. Another is some temporary disk error." Both reasons do not seem to apply in our case: mail partition is not NFS-mounted, and disks (a RAID5 array) seem healthy. Furthermore, affected accounts are not over quota. So may we have found a bug ? Or may we have a locking problem ? Or an index handling one ? I tried to disable mbox_lazy_writes without success. I will try to disable disk indexes totally, which should help since problems seem index-related; but performance may suffer. Any clues appreciated... Dovecot -n says : # 1.0.8: /usr/local/etc/dovecot.conf protocols: imap imaps pop3 pop3s listen: [::] ssl_cert_file: /usr/local/etc/ssl/private/our-certificate.pem ssl_key_file: /usr/local/etc/ssl/private/our-certificate.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login first_valid_uid: 200 first_valid_gid: 200 mail_location: mbox:~/mail:INBOX=~/mail/INBOX:INDEX=/espace/dovecot/indexes/%u fsync_disable: yes mbox_lazy_writes: no mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 imap_client_workarounds(default): outlook-idle tb-extra-mailbox-sep delay-newmail imap_client_workarounds(imap): outlook-idle tb-extra-mailbox-sep delay-newmail imap_client_workarounds(pop3): outlook-idle pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xv%08Xu pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: passdb: driver: pam args: * userdb: driver: passwd plugin: quota: fs -- Ce message a ete verifie par MailScanner pour des virus ou des polluriels et rien de suspect n'a ete trouve.

4 6

[Dovecot] Dovecot 1.08 seems very slow
by Marc Perkel 12 Dec '07

12 Dec '07

I know I'm late reporting this but 1.08 seems really slow. It seems like it is as slow as some of the early RCs were. Wish I had more info. Using Maildir format.

2 1

[Dovecot] 1.0.8 with quota 1.0.8 patch - undefined symbol: quota_set
by adam_jh＠hotmail.com 12 Dec '07

12 Dec '07

I get the following error after patching the 1.0.8 src.rpm with the quota-rewrite-1.0.8.diff [root@test dovecot]# /etc/rc.d/init.d/dovecot start Starting Dovecot Imap: Edlopen(/usr/lib/dovecot/imap/lib02_imap_quota_plugin.so) failed: /usr/lib/dovecot/imap/lib02_imap_quota_plugin.so: undefined symbol: quota_set FCouldn't load required plugins Error: imap dump-capability process returned 89 Fatal: Invalid configuration in /etc/dovecot.conf [FAILED] I have checked the new quota config changes. The src rpm is taken from http://atrpms.net/dist/el4/dovecot/. Using the same process without the patch, the rpm that's built and installed works (or at least starts). A process of elimination revealed it doesn't understand the imap_quota aspect of the quota imap_quota configuration line. Removing that it works, but I kind of need it! Can anyone help? Thanks, adam To patch it I did: Patch103: dovecot-1.0.beta2-mkcert-permissions.patch Patch105: dovecot-1.0.rc7-mkcert-paths.patch # quota patch Patch106: quota-rewrite-1.0.8.diff and %patch103 -p1 -b .mkcert-permissions %patch105 -p1 -b .mkcert-paths %patch106 -p1 -b .quota Since I'm using centos to match RHEL 4 [rpmbuild@test SPECS]$ cat /etc/redhat-release Red Hat Enterprise Linux WS release 4 (Nahant) I also added to my .rpmmacros for the rpmbuild user I have %bcond_with() %{expand:%%{?_with_%{1}:%%global with_%{1} 1}} %bcond_without() %{expand:%%{!?_without_%{1}:%%global without_%{1} 1}} %with() %{expand:%%{?with_%{1}:1}%%{!?with_%{1}:0}} %without() %{expand:%%{?without_%{1}:0}%%{!?without_%{1}:1}} then rpm -e dovecot su rpmbuild ... rpmbuild -ba postfix.spec rpm -Uvh ...dovecot.rpm [root@test dovecot]# /usr/sbin/postfix -n # 1.0.8: /etc/dovecot.conf login_dir: /var/run/dovecot-login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_processes_count: 1 first_valid_uid: 505 last_valid_uid: 505 mail_extra_groups: mail mail_location: maildir:/var/vmail/%d/%n maildir_copy_with_hardlinks: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): outlook-idle pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: user: nobody passdb: driver: sql args: /etc/dovecot-mysql.conf userdb: driver: sql args: /etc/dovecot-mysql.conf socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: mail master: path: /var/run/dovecot/auth-master mode: 432 user: vmail group: mail plugin: quota: maildir

2 3

[Dovecot] Fishing attempt locking up dovecot
by Jerry Yeager 12 Dec '07

12 Dec '07

On Dec 11, 2007, at 5:58 PM, dovecot-request(a)dovecot.org wrote: > > > Message: 10 > Date: Tue, 11 Dec 2007 15:58:16 -0700 > From: Patrick Milvich <patrick(a)milvich.com> > Subject: [Dovecot] Fishing attempt locking up dovecot > To: dovecot(a)dovecot.org > Message-ID: <8C5CE5FE-BD2F-40C4-8A36-A4CD8BD533DB(a)milvich.com> > Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes > > I've mentioned this before but only heard from one other person who > has experienced this, but it's becoming a pretty serious issue. > > The situation: > A spammer sets a bot on a fishing attempt to gain email addresses, > causing numerous login processes to spawn and suck up all available > resources. > > The problem: > Obviously this can act like a dos attack, but the real issue is after > the spammer stops (by virtue of being added to our firewall blacklist, > being caught and shut down by their isp, or otherwise), dovecot > doesn't seem to relinquish the resources, causing "too many files > open" errors for normal usage. > > stuff cut out > > End of dovecot Digest, Vol 56, Issue 33 > *************************************** Will the following be of any help to you? (it is a patch for Postfix 2.4.nn) It would seem that the type of fishing expedition you mention would fall into the bit described below (lots of errors). While it will not directly solve the "out of resources" Dovecot problem, it may limit the up-front damage, followed with a CRON script running every twenty minutes or so that scans the last line of the mail log for the 'too many files open' error and upon finding it runs a version of the killall imap-login processes. ftp://postfix.mirrors.pair.com/index.html Postfix 2.4 patch (PGP signature ) to add stress-adaptive behavior to the SMTP server. When some mail flood keeps all server ports busy, this feature can be used to quickly drop connections from clients that make errors, and to reduce the time that Postfix waits for a client command. This may delay some legitimate deliveries, but it will allow you to still keep some mail flowing. After the mail flood ends, Postfix reverts to its normal behavior.

1 0

[Dovecot] dovecot deliver errors
by Andrew Garner 12 Dec '07

12 Dec '07

I think someone else mentioned this problem earlier on the list, and I've recently started seeing the following error show up: file index-transaction.c: line 54 (index_transaction_finish_rollback): assertion failed: (t->mail_ref_count == 0) Raw backtrace: /usr/lib/dovecot/deliver(i_syslog_fatal_handler+0x2b) [0x80bff0b] -> /usr/lib/dovecot/deliver [0x80bfd1a] -> /usr/lib/dovecot/deliver [0x808f783] -> /usr/lib/dovecot/deliver(index_transaction_rollback+0x1a) [0x808f5aa] -> /usr/lib/dovecot/modules/lda//lib10_quota_plugin.so [0xb7e6d013] -> /usr/lib/dovecot/deliver(deliver_save+0x202) [0x8059812] -> /usr/lib/dovecot/modules/lda//lib90_cmusieve_plugin.so [0xb7e4bb1d] -> /usr/lib/dovecot/modules/lda//lib90_cmusieve_plugin.so [0xb7e5bdbc] -> /usr/lib/dovecot/modules/lda//lib90_cmusieve_plugin.so(sieve_execute_bytecode+0x1c4) [0xb7e5c7d4] -> /usr/lib/dovecot/modules/lda//lib90_cmusieve_plugin.so(cmu_sieve_run+0x30c) [0xb7e4c7bc] -> /usr/lib/dovecot/modules/lda//lib90_cmusieve_plugin.so [0xb7e4a81f] -> /usr/lib/dovecot/deliver(main+0x10c0) [0x805a990] -> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xc8) [0xb7e86ea8] -> /usr/lib/dovecot/deliver [0x80588e1] With beta11 I started running into this problem (using postfix's smtp-source messages): postfix/pipe[19560]: 85372294: to=<example(a)example.com>, relay=dovecot, delay=20385, delays=20384/0.01/0/0.19, dsn=5.3.0, status=bounced (Command died with signal 11: "/usr/ lib/dovecot/deliver") Program received signal SIGSEGV, Segmentation fault. 0x080d017a in str_sanitize_append (dest=0x80ee2f8, src=0x0, max_len=80) at str-sanitize.c:11 11 str-sanitize.c: No such file or directory. in str-sanitize.c (gdb) #0 0x080d017a in str_sanitize_append (dest=0x80ee2f8, src=0x0, max_len=80) at str-sanitize.c:11 p = <value optimized out> #1 0x080d02ce in str_sanitize (src=0x0, max_len=80) at str-sanitize.c:35 str = (string_t *) 0x80ee2f8 #2 0x080594f4 in deliver_log (mail=0x8107a48, fmt=0x80d349b "saved mail to %s") at deliver.c:120 args = 0xbfb087d8 "\200<DA>\016\b" str = (string_t *) 0x80ee120 #3 0x0805979f in deliver_save (namespaces=0x80f7a90, storage_r=0xbfb088e8, mailbox=0x80ded98 "INBOX", mail=0x8107a48, flags=0, keywords=0x0) at deliver.c:222 box = (struct mailbox *) 0x810be58 t = (struct mailbox_transaction_context *) 0x0 kw = (struct mail_keywords *) 0x0 error = MAIL_ERROR_NONE mailbox_name = 0x80eda80 "INBOX" ret = 0 #4 0x0805ac15 in main (argc=5, argv=Cannot access memory at address 0x4 ) at deliver.c:954 error_string = <value optimized out> error = <value optimized out> ret = <value optimized out> config_path = 0x80d34ba "/etc/dovecot/dovecot.conf" envelope_sender = 0x80ed030 "example(a)foo.net" mailbox = 0x80ded98 "INBOX" auth_socket = <value optimized out> home = <value optimized out> destaddr = 0xbfb09f9c "example(a)example.com" user = 0xbfb09f9c "example(a)example.com" error = <value optimized out> extra_fields = {arr = {buffer = 0x80ed080, element_size = 4}, v = 0x80ed080, v_modifiable = 0x80ed080} ns = (struct mail_namespace *) 0x80f7a90 raw_ns = (struct mail_namespace *) 0x80f7ab8 storage = (struct mail_storage *) 0x80f7e98 box = (struct mailbox *) 0x8102768 input = (struct istream *) 0x80f8670 t = (struct mailbox_transaction_context *) 0x8106be8 headers_ctx = (struct mailbox_header_lookup_ctx *) 0x8106508 mail = (struct mail *) 0x8107a48 process_euid = 89 namespace_pool = <value optimized out> stderr_rejection = false keep_environment = false user_auth = true i = 63 ret = <value optimized out> # dovecot -n # 1.1.beta11: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ protocols: imap imaps pop3 pop3s login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login verbose_proctitle: yes mail_uid: 89 mail_gid: 89 mail_location: maildir:~/Maildir mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_executable(default): /usr/lib/dovecot/rawlog /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/rawlog /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugins(default): fts fts_squat mail_plugins(imap): fts fts_squat mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/modules/imap/ mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap/ mail_plugin_dir(pop3): /usr/lib/dovecot/modules/ imap_client_workarounds(default): delay-newmail imap_client_workarounds(imap): delay-newmail imap_client_workarounds(pop3): pop3_uidl_format(default): %08Xu%08Xv pop3_uidl_format(imap): %08Xu%08Xv pop3_uidl_format(pop3): %f pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: mechanisms: login plain cram-md5 digest-md5 username_format: %Lu verbose: yes debug: yes passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf socket: type: listen client: path: /var/spool/postfix/private/auth user: postfix group: postfix master: path: /var/run/dovecot/auth-master user: vmail group: vmail plugin: quota: maildir quota_rule: *:storage=102400:messages=30000 fts: squat

2 1

[Dovecot] dovecot.index only gets updated in Root maildir Folder
by Florian Franzeck 11 Dec '07

11 Dec '07

version 1.0.rc15 (Newest Version available in Debian 4.0, newer version in testing tree requires new version libc...) I got a procmail rule that moves mail from some senders to another folder, e.g. .archive.facebook/ . This works fine, the message appear in .archive.facebook/new if i "ls" that directory by shell. But dovecot.index files don't get updated, hence IDLE doesn't tell the client new mail arrived. this only happens if i click on the folder in thunderbird. like this, not even clicking on Thunderbird "Check mail" button shows me new Mail arrived. Is this a common error in this version? would there any benefit if i self compiled a newer version regarding this issue? Thanx Config: log_timestamp: %Y-%m-%d %H:%M:%S ssl_listen: * login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login mail_extra_groups: mail mail_location: maildir:~/Maildir auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master:

1 0

[Dovecot] v1.0.9 released
by Timo Sirainen 11 Dec '07

11 Dec '07

http://dovecot.org/releases/1.0/dovecot-1.0.9.tar.gz http://dovecot.org/releases/1.0/dovecot-1.0.9.tar.gz.sig + Maildir: Don't wait on dovecot-uidlist.lock when we just want to find out a new filename for the message. - mbox: v1.0.8 changes sometimes caused FETCH to fail with "got too little data", disconnecting the client. - Fixed a memory leak when FETCHing message header/body multiple times within a command (e.g. BODY[1] BODY[2]) - IMAP: Partial body fetching was still slow with mboxes

1 0

[Dovecot] Using global und per-user sieve scripts at the same time
by Matthias Kellermann 11 Dec '07

11 Dec '07

Hi list, I'm trying to get a global sieve mailfilter with per-user sieve scripts to work. At the "lda"-section in the dovecot.conf file I have: global_script_path = /usr/local/etc/sieve/spam /usr/local/etc/sieve/spam: require "fileinto"; if header :contains ["Subject"] ["***Spam***"] { fileinto "Spam"; stop; } which will work fine delivering Spam Mails in a Spam dir for every user. If a user creates his own .dovecot.sieve file the global script will not work anymore. How can I enable global and local per-user sieve scripts at once? Is this possible? Do you know any workaround so all users will have the same (Spam-)sieve filters beside there own (global templates or so)? Matthias

5 11

[Dovecot] squat also not working in beta11
by Joe Wong 10 Dec '07

10 Dec '07

Hi Timo, Just tried squat in beta11. I also get the same error in the log: dovecot: Dec 10 14:21:17 Error: IMAP(joewong99:joew.outblaze.com): Corrupted squat uidlist file /mailfs/4/22/3/joewon g99:joew.outblaze.com@joew_outblaze_com/Maildir/dovecot.index.search.uids: Broken uidlists There is not change in my configuration file. # 1.1.beta11: /usr/local/etc/dovecot.conf log_path: /var/log/dovecot-1.1.log info_log_path: /var/log/dovecot-1.1.log protocols: imap ssl_disable: yes login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login login_user: nobody verbose_proctitle: yes mail_location: maildir:~/Maildir mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_drop_priv_before_exec: yes mail_plugins: fts fts_squat auth default: user: mdrop username_chars: abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@: username_translation: @: verbose: yes debug: yes debug_passwords: yes passdb: driver: sql args: /usr/local/etc/dovecot-sql.conf userdb: driver: sql args: /usr/local/etc/dovecot-sql.conf plugin: fts: squat --

2 5

Re: [Dovecot] New Mailbox Format Plugin Help
by Sacha Bernstein 10 Dec '07

10 Dec '07

On 12/8/07, Timo Sirainen <tss(a)iki.fi> wrote: > On Sat, 2007-12-08 at 10:11 -0500, Sacha Bernstein wrote: > > I would like to develop a new mailbox format for a very specialized > > application. Can anyone point me in the appropriate direction to get > > started writing a plugin? > > > > More detail for anyone interested: > > I need to store mail in a format that's quickly accessible and > > modifiable to an external process. This is for a very low-volume mail > > system (averaging less than 50 messages stored at any one time) that > > needs very good indexing. I need to create new mailboxes on the fly > > and perform fast searches across all the mailboxes. I intend to store > > all the messages in an SQL database. Any help getting started > > interfacing with dovecot (other than reading through _all_ the source > > code) would be greatly appreciated. > > v1.1 makes this a lot easier, so start with it instead of v1.0. You can > probably get started with my existing SQL backend code: > http://dovecot.org/list/dovecot/2007-November/026632.html > > The main idea is to implement APIs in lib-storage/mail-storage-private.h > and lib-storage/mailbox-list-private.h. Most of these calls can be > implemented by calling generic index_*() functions in > lib-storage/index/. Thanks, I'll get started with this. I'm sure I'll have more questions once I get further into it. If anyone's interested (especially any amateur radio operators that may be on this list), you can see what my plans are for the SQL storage backend for dovecot at w2sjb.org. Basically, the short version is a linux gateway to a radio-based email system for use in emergencies. Of course it has many other uses, but that's what I'm going to be using it for :-) Thanks, Sacha

1 0