December 2007 - dovecot

Re: [Dovecot] antispam integration (dovecot antispam plugin): chroot & no way to pass `--user=' to dspam?
by Johannes Berg 13 Dec '07

13 Dec '07

> I want have dspam statistics to be per-user, not global. And I like > idea to learn filter by moving messages from/to SPAM folder, it seems to > be natural and much better, than Web UI (I don't want to use dspam Web > UI at all). > > Is here any way to resolve two problems: > > (1) Run dspam in chrooted environment > (2) pass `--user=${USER}' parameter to dspam. No idea. Passing --user is easy, there's an example in the readme file for the antispam plugin. The former I have no idea about, I guess all it needs is access to the mysql db. > Oh, and even more: I think, that it will be useful to have `socket' > backend for spam-plugin, which allows to send messages to two different > configured sockets (TCP or UNIX domain) -- one for `learn as spam' and > other for `learn as ham'. It allows to run in chrooted environment > easily, and solve problem with exec-dspam-on-every-message. Dunno. Why are you running it chrooted anyway? If you write such a backend I can add it to my tree :) johannes

1 0

[Dovecot] SYN flooding / pipe() failed: Too many open files
by Frank Elsner 13 Dec '07

13 Dec '07

Hello *, I've the following problem with dovecot 1.0.7: /var/log/messages shows | Dec 13 13:48:27 mailbox kernel: possible SYN flooding on port 143. Sending cookies. and /var/log/maillog shows (Please note: nearly the same time): | Dec 13 13:48:28 mailbox dovecot: pipe() failed: Too many open files This leads to an unresponsive mail service and requires a dovecot restart. What can be done to prevent this? BTW, this is my dovecot -n: # 1.0.7: /usr/local/dovecot/etc/dovecot.conf base_dir: /var/run/dovecot/ protocols: imap imaps pop3 pop3s ssl_cert_file: /usr/local/dovecot/etc/mailbox.pem ssl_key_file: /usr/local/dovecot/etc/mailbox.key disable_plaintext_auth: no login_dir: /var/run/dovecot//login login_executable(default): /usr/local/dovecot/libexec/dovecot/imap-login login_executable(imap): /usr/local/dovecot/libexec/dovecot/imap-login login_executable(pop3): /usr/local/dovecot/libexec/dovecot/pop3-login login_greeting: TU-Berlin.DE POP3-/IMAP Service ready login_log_format_elements: %u %r %c login_processes_count: 24 login_max_processes_count: 512 verbose_proctitle: yes mail_location: maildir:/var/Mail/Boxes/%1u/%u mailbox_idle_check_interval: 15 maildir_copy_with_hardlinks: yes maildir_copy_preserve_filename: yes mail_executable(default): /usr/local/dovecot/libexec/dovecot/imap mail_executable(imap): /usr/local/dovecot/libexec/dovecot/imap mail_executable(pop3): /usr/local/dovecot/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/local/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/dovecot/lib/dovecot/pop3 mail_log_prefix: %Ls(%u): imap_client_workarounds(default): delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep imap_client_workarounds(pop3): outlook-idle pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: mechanisms: plain login anonymous count: 32 passdb: driver: ldap args: /usr/local/dovecot/etc/dovecot-ldap.conf userdb: driver: static args: uid=777 gid=12 plugin: quota: maildir:ignore=Trash Kind regards ______________________________________________________________ Frank Elsner / c/o Technische Universitaet Berlin | ____________/ tubIT [BD], Sekr. E-N 50 | | Einsteinufer 17 | | Phone: +49 30 314 23897 D-10587 Berlin | | eMail: Frank.Elsner(a)TU-Berlin.DE Germany __________________| |_________________________________________________________| in dubio pro exim

3 2

[Dovecot] How to copy Spammail for learning out of Postfix and Dovecot (vmail)
by Sven Schmidt 13 Dec '07

13 Dec '07

Hi! I am using Postfix 2.3.8 and Dovecot 1.0.rc15. Virtual Domains and Mailboxes are configured. I tried to copy spammy Mail with Procmail, but Procmail only sees local mails. How to configure it to get a copy of spammy mails in a mbox or mail-dir?

3 4

[Dovecot] Dovecot Antispam Problem (Using mail sending backend)
by Justin Refice 13 Dec '07

13 Dec '07

I'm trying to setup DSPAM training using the mail-sending backend of the Antispam filter, unfortunately, things don't appear to be working. I've compiled it against the debian etch dovecot libraries, and the plugin itself builds, and doesn't report any errors when running. As a matter of fact, it doesn't report ANYTHING when running, except for the initialization code (DEBUG=syslog): imap: antispam: plugin initialising imap: antispam: "trash" is trash folder imap: antispam: "Trash" is trash folder imap: antispam: "Deleted Items" is trash folder imap: antispam: "spam" is spam folder imap: antispam: "Spam" is spam folder imap: antispam: "SPAM" is spam folder imap: antispam: "junk" is spam folder imap: antispam: "Junk" is spam folder imap: antispam: "Junk Mail" is spam folder imap: antispam: no unsure folders imap: antispam: mail backend spam address dspam-add(a)xxx.yyy imap: antispam: mail backend not-spam address dspam-fp(a)xxx.yyy imap: antispam: mail backend sendmail /usr/sbin/sendmail imap: antispam: mail backend tmpdir /tmp .... And that's it. I never hear from the plugin again, regardless of how many times I create/delete the SPAM folder, or how many times I move/copy into/out of the folder. The plugin is basically silent. I don't see the mails being sent/recieved, so I'm assuming that's it isn't doing anything at all. I can manually use the training addresses, so I know they're working. Any help would be appreciated. -Justin R.

5 9

[Dovecot] got too little data ??
by Benoit Branciard 12 Dec '07

12 Dec '07

I just switched in a hurry one of our mailbox servers from UW-imapd to Dovecot in an attempt to address performance and stability problems. This server hosts ~5500 accounts, ~2000 are daily used, ~600 simultaneous IMAP connexions in rush hours, and lots of POP ones. Some accounts may be accessed simultaneously with IMAP and POP. The system is : - Debian Sarge (x86 with amd64 kernel) - MBOX files mailboxes - indexes migrated out of filesystem quotas - Sendmail MTA - Procmail local delivery agent (not easily replaceable due to extensive use of procmailrc filters) - Both Procmail and Dovecot use [dotlock, fcntl] as mailbox locking scheme - No other processes have access to mailboxes in normal use - Dovecot 1.0.8 hand-compiled from tarball The good new is that it seems to perform better than UW-imapd. The bad new is that we get lots or errors like this in log: IMAP(username): FETCH for mailbox INBOX UID 23862 got too little data: 3186 vs 3206 Such "accidents" happen about 4-5 times a day, and does not correct themselves, even with a mailbox expunge or reopen. However manual deletion of the faulty mailbox index seems to clear the problem, until another one occurs. On such errors some mail clients (Thunderbird) retry operation in infinite loop, so log gets really cluttered. In imap-fetch-body.c it is stated : "We shouldn't really ever get here. One reason is if mail was deleted from NFS server while we were reading it. Another is some temporary disk error." Both reasons do not seem to apply in our case: mail partition is not NFS-mounted, and disks (a RAID5 array) seem healthy. Furthermore, affected accounts are not over quota. So may we have found a bug ? Or may we have a locking problem ? Or an index handling one ? I tried to disable mbox_lazy_writes without success. I will try to disable disk indexes totally, which should help since problems seem index-related; but performance may suffer. Any clues appreciated... Dovecot -n says : # 1.0.8: /usr/local/etc/dovecot.conf protocols: imap imaps pop3 pop3s listen: [::] ssl_cert_file: /usr/local/etc/ssl/private/our-certificate.pem ssl_key_file: /usr/local/etc/ssl/private/our-certificate.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login first_valid_uid: 200 first_valid_gid: 200 mail_location: mbox:~/mail:INBOX=~/mail/INBOX:INDEX=/espace/dovecot/indexes/%u fsync_disable: yes mbox_lazy_writes: no mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 imap_client_workarounds(default): outlook-idle tb-extra-mailbox-sep delay-newmail imap_client_workarounds(imap): outlook-idle tb-extra-mailbox-sep delay-newmail imap_client_workarounds(pop3): outlook-idle pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xv%08Xu pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: passdb: driver: pam args: * userdb: driver: passwd plugin: quota: fs -- Ce message a ete verifie par MailScanner pour des virus ou des polluriels et rien de suspect n'a ete trouve.

4 6

[Dovecot] Dovecot 1.08 seems very slow
by Marc Perkel 12 Dec '07

12 Dec '07

I know I'm late reporting this but 1.08 seems really slow. It seems like it is as slow as some of the early RCs were. Wish I had more info. Using Maildir format.

2 1

[Dovecot] 1.0.8 with quota 1.0.8 patch - undefined symbol: quota_set
by adam_jh＠hotmail.com 12 Dec '07

12 Dec '07

I get the following error after patching the 1.0.8 src.rpm with the quota-rewrite-1.0.8.diff [root@test dovecot]# /etc/rc.d/init.d/dovecot start Starting Dovecot Imap: Edlopen(/usr/lib/dovecot/imap/lib02_imap_quota_plugin.so) failed: /usr/lib/dovecot/imap/lib02_imap_quota_plugin.so: undefined symbol: quota_set FCouldn't load required plugins Error: imap dump-capability process returned 89 Fatal: Invalid configuration in /etc/dovecot.conf [FAILED] I have checked the new quota config changes. The src rpm is taken from http://atrpms.net/dist/el4/dovecot/. Using the same process without the patch, the rpm that's built and installed works (or at least starts). A process of elimination revealed it doesn't understand the imap_quota aspect of the quota imap_quota configuration line. Removing that it works, but I kind of need it! Can anyone help? Thanks, adam To patch it I did: Patch103: dovecot-1.0.beta2-mkcert-permissions.patch Patch105: dovecot-1.0.rc7-mkcert-paths.patch # quota patch Patch106: quota-rewrite-1.0.8.diff and %patch103 -p1 -b .mkcert-permissions %patch105 -p1 -b .mkcert-paths %patch106 -p1 -b .quota Since I'm using centos to match RHEL 4 [rpmbuild@test SPECS]$ cat /etc/redhat-release Red Hat Enterprise Linux WS release 4 (Nahant) I also added to my .rpmmacros for the rpmbuild user I have %bcond_with() %{expand:%%{?_with_%{1}:%%global with_%{1} 1}} %bcond_without() %{expand:%%{!?_without_%{1}:%%global without_%{1} 1}} %with() %{expand:%%{?with_%{1}:1}%%{!?with_%{1}:0}} %without() %{expand:%%{?without_%{1}:0}%%{!?without_%{1}:1}} then rpm -e dovecot su rpmbuild ... rpmbuild -ba postfix.spec rpm -Uvh ...dovecot.rpm [root@test dovecot]# /usr/sbin/postfix -n # 1.0.8: /etc/dovecot.conf login_dir: /var/run/dovecot-login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_processes_count: 1 first_valid_uid: 505 last_valid_uid: 505 mail_extra_groups: mail mail_location: maildir:/var/vmail/%d/%n maildir_copy_with_hardlinks: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): outlook-idle pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: user: nobody passdb: driver: sql args: /etc/dovecot-mysql.conf userdb: driver: sql args: /etc/dovecot-mysql.conf socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: mail master: path: /var/run/dovecot/auth-master mode: 432 user: vmail group: mail plugin: quota: maildir

2 3

[Dovecot] Fishing attempt locking up dovecot
by Jerry Yeager 12 Dec '07

12 Dec '07

On Dec 11, 2007, at 5:58 PM, dovecot-request(a)dovecot.org wrote: > > > Message: 10 > Date: Tue, 11 Dec 2007 15:58:16 -0700 > From: Patrick Milvich <patrick(a)milvich.com> > Subject: [Dovecot] Fishing attempt locking up dovecot > To: dovecot(a)dovecot.org > Message-ID: <8C5CE5FE-BD2F-40C4-8A36-A4CD8BD533DB(a)milvich.com> > Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes > > I've mentioned this before but only heard from one other person who > has experienced this, but it's becoming a pretty serious issue. > > The situation: > A spammer sets a bot on a fishing attempt to gain email addresses, > causing numerous login processes to spawn and suck up all available > resources. > > The problem: > Obviously this can act like a dos attack, but the real issue is after > the spammer stops (by virtue of being added to our firewall blacklist, > being caught and shut down by their isp, or otherwise), dovecot > doesn't seem to relinquish the resources, causing "too many files > open" errors for normal usage. > > stuff cut out > > End of dovecot Digest, Vol 56, Issue 33 > *************************************** Will the following be of any help to you? (it is a patch for Postfix 2.4.nn) It would seem that the type of fishing expedition you mention would fall into the bit described below (lots of errors). While it will not directly solve the "out of resources" Dovecot problem, it may limit the up-front damage, followed with a CRON script running every twenty minutes or so that scans the last line of the mail log for the 'too many files open' error and upon finding it runs a version of the killall imap-login processes. ftp://postfix.mirrors.pair.com/index.html Postfix 2.4 patch (PGP signature ) to add stress-adaptive behavior to the SMTP server. When some mail flood keeps all server ports busy, this feature can be used to quickly drop connections from clients that make errors, and to reduce the time that Postfix waits for a client command. This may delay some legitimate deliveries, but it will allow you to still keep some mail flowing. After the mail flood ends, Postfix reverts to its normal behavior.

1 0

[Dovecot] dovecot deliver errors
by Andrew Garner 12 Dec '07

12 Dec '07

I think someone else mentioned this problem earlier on the list, and I've recently started seeing the following error show up: file index-transaction.c: line 54 (index_transaction_finish_rollback): assertion failed: (t->mail_ref_count == 0) Raw backtrace: /usr/lib/dovecot/deliver(i_syslog_fatal_handler+0x2b) [0x80bff0b] -> /usr/lib/dovecot/deliver [0x80bfd1a] -> /usr/lib/dovecot/deliver [0x808f783] -> /usr/lib/dovecot/deliver(index_transaction_rollback+0x1a) [0x808f5aa] -> /usr/lib/dovecot/modules/lda//lib10_quota_plugin.so [0xb7e6d013] -> /usr/lib/dovecot/deliver(deliver_save+0x202) [0x8059812] -> /usr/lib/dovecot/modules/lda//lib90_cmusieve_plugin.so [0xb7e4bb1d] -> /usr/lib/dovecot/modules/lda//lib90_cmusieve_plugin.so [0xb7e5bdbc] -> /usr/lib/dovecot/modules/lda//lib90_cmusieve_plugin.so(sieve_execute_bytecode+0x1c4) [0xb7e5c7d4] -> /usr/lib/dovecot/modules/lda//lib90_cmusieve_plugin.so(cmu_sieve_run+0x30c) [0xb7e4c7bc] -> /usr/lib/dovecot/modules/lda//lib90_cmusieve_plugin.so [0xb7e4a81f] -> /usr/lib/dovecot/deliver(main+0x10c0) [0x805a990] -> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xc8) [0xb7e86ea8] -> /usr/lib/dovecot/deliver [0x80588e1] With beta11 I started running into this problem (using postfix's smtp-source messages): postfix/pipe[19560]: 85372294: to=<example(a)example.com>, relay=dovecot, delay=20385, delays=20384/0.01/0/0.19, dsn=5.3.0, status=bounced (Command died with signal 11: "/usr/ lib/dovecot/deliver") Program received signal SIGSEGV, Segmentation fault. 0x080d017a in str_sanitize_append (dest=0x80ee2f8, src=0x0, max_len=80) at str-sanitize.c:11 11 str-sanitize.c: No such file or directory. in str-sanitize.c (gdb) #0 0x080d017a in str_sanitize_append (dest=0x80ee2f8, src=0x0, max_len=80) at str-sanitize.c:11 p = <value optimized out> #1 0x080d02ce in str_sanitize (src=0x0, max_len=80) at str-sanitize.c:35 str = (string_t *) 0x80ee2f8 #2 0x080594f4 in deliver_log (mail=0x8107a48, fmt=0x80d349b "saved mail to %s") at deliver.c:120 args = 0xbfb087d8 "\200<DA>\016\b" str = (string_t *) 0x80ee120 #3 0x0805979f in deliver_save (namespaces=0x80f7a90, storage_r=0xbfb088e8, mailbox=0x80ded98 "INBOX", mail=0x8107a48, flags=0, keywords=0x0) at deliver.c:222 box = (struct mailbox *) 0x810be58 t = (struct mailbox_transaction_context *) 0x0 kw = (struct mail_keywords *) 0x0 error = MAIL_ERROR_NONE mailbox_name = 0x80eda80 "INBOX" ret = 0 #4 0x0805ac15 in main (argc=5, argv=Cannot access memory at address 0x4 ) at deliver.c:954 error_string = <value optimized out> error = <value optimized out> ret = <value optimized out> config_path = 0x80d34ba "/etc/dovecot/dovecot.conf" envelope_sender = 0x80ed030 "example(a)foo.net" mailbox = 0x80ded98 "INBOX" auth_socket = <value optimized out> home = <value optimized out> destaddr = 0xbfb09f9c "example(a)example.com" user = 0xbfb09f9c "example(a)example.com" error = <value optimized out> extra_fields = {arr = {buffer = 0x80ed080, element_size = 4}, v = 0x80ed080, v_modifiable = 0x80ed080} ns = (struct mail_namespace *) 0x80f7a90 raw_ns = (struct mail_namespace *) 0x80f7ab8 storage = (struct mail_storage *) 0x80f7e98 box = (struct mailbox *) 0x8102768 input = (struct istream *) 0x80f8670 t = (struct mailbox_transaction_context *) 0x8106be8 headers_ctx = (struct mailbox_header_lookup_ctx *) 0x8106508 mail = (struct mail *) 0x8107a48 process_euid = 89 namespace_pool = <value optimized out> stderr_rejection = false keep_environment = false user_auth = true i = 63 ret = <value optimized out> # dovecot -n # 1.1.beta11: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ protocols: imap imaps pop3 pop3s login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login verbose_proctitle: yes mail_uid: 89 mail_gid: 89 mail_location: maildir:~/Maildir mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_executable(default): /usr/lib/dovecot/rawlog /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/rawlog /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugins(default): fts fts_squat mail_plugins(imap): fts fts_squat mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/modules/imap/ mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap/ mail_plugin_dir(pop3): /usr/lib/dovecot/modules/ imap_client_workarounds(default): delay-newmail imap_client_workarounds(imap): delay-newmail imap_client_workarounds(pop3): pop3_uidl_format(default): %08Xu%08Xv pop3_uidl_format(imap): %08Xu%08Xv pop3_uidl_format(pop3): %f pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: mechanisms: login plain cram-md5 digest-md5 username_format: %Lu verbose: yes debug: yes passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf socket: type: listen client: path: /var/spool/postfix/private/auth user: postfix group: postfix master: path: /var/run/dovecot/auth-master user: vmail group: vmail plugin: quota: maildir quota_rule: *:storage=102400:messages=30000 fts: squat

2 1

[Dovecot] dovecot.index only gets updated in Root maildir Folder
by Florian Franzeck 11 Dec '07

11 Dec '07

version 1.0.rc15 (Newest Version available in Debian 4.0, newer version in testing tree requires new version libc...) I got a procmail rule that moves mail from some senders to another folder, e.g. .archive.facebook/ . This works fine, the message appear in .archive.facebook/new if i "ls" that directory by shell. But dovecot.index files don't get updated, hence IDLE doesn't tell the client new mail arrived. this only happens if i click on the folder in thunderbird. like this, not even clicking on Thunderbird "Check mail" button shows me new Mail arrived. Is this a common error in this version? would there any benefit if i self compiled a newer version regarding this issue? Thanx Config: log_timestamp: %Y-%m-%d %H:%M:%S ssl_listen: * login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login mail_extra_groups: mail mail_location: maildir:~/Maildir auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master:

1 0