November 2008 - dovecot

Re: [Dovecot] [dspam-users] mysql fetch row error
by Marko Weber 15 Nov '08

15 Nov '08

Hi Steve, MTA = Postfix I have amavis - spamassassin - dspam installed. i think spamassassin scnas after dspam, cuase in the mail headers dspam comes at first. I use Dovecot as Imap Server OS = Gentoo Linux i put the config of dspam here = http://rafb.net/p/VCX8XO97.html if u need more confs just tell me , i can paste then marko Steve schrieb: > Hallo Marko > > This is not normal. DSPAM does not add the encoding. It must be something else you are using/doing to get that tagged with the encoding in front of every DSPAM tag. And I don't see the DSPAM signature in your header. This is not normal. If you have the other DSPAM header lines, then you should have the result as well. > > What MTA are you using? > How is the integration between MTA and DSPAM? > Do you have other filters running before and after DSPAM? > What IMAP server are you using? > What OS are you using? > Could you post your DSPAM config? > > > (Grüsse aus Zürich) > > // Steve > > -------- Original-Nachricht -------- > >> Datum: Sat, 15 Nov 2008 10:37:38 +0100 >> Von: Marko Weber <weber(a)zackbummfertig.de> >> An: "dspam-users(a)lists.nuclearelephant.com" <dspam-users(a)lists.nuclearelephant.com> >> Betreff: Re: [dspam-users] mysql fetch row error >> > > >> Hello Steve this is the "full" Header of a spam mail where the training >> also fails. >> withe the mysql fetch error............. >> >> marko >> >> >> Return-Path: <RoxieastronomerMyrick(a)fightingbobfest.org> >> X-Original-To: weber(a)salonwebserver.de >> Delivered-To: weber(a)salonwebserver.de >> Received: from localhost (localhost [127.0.0.1]) >> by salonwebserver.de (Postfix) with ESMTP id 02C8613BC09B >> for <weber(a)salonwebserver.de>; Fri, 14 Nov 2008 21:38:09 +0100 >> (CET) >> X-DSPAM-Result: =?iso-8859-1?Q?Spam=0D?= >> X-DSPAM-Confidence: =?iso-8859-1?Q?0.7177=0D?= >> X-DSPAM-Probability: =?iso-8859-1?Q?1.0000=0D?= >> ?iso-8859-1?Q?491de1b0254959695846858=0D?= >> X-DSPAM-Factors: 15, >> X-Virus-Scanned: amavisd-new at salonwebserver.de >> X-Spam-Flag: NO >> X-Spam-Score: 2.415 >> X-Spam-Level: ** >> X-Spam-Status: No, score=2.415 tagged_above=-1000 required=2.5 >> tests=[DNS_FROM_SECURITYSAGE=0.127, RCVD_IN_BL_SPAMCOP_NET=2.188, >> RDNS_DYNAMIC=0.1] >> Received: from salonwebserver.de ([127.0.0.1]) >> by localhost (kraftwerk1.salonwebserver.de [127.0.0.1]) >> (amavisd-new, port 10024) >> with ESMTP id nOQQilMNvp-o for <weber(a)salonwebserver.de>; >> Fri, 14 Nov 2008 21:38:07 +0100 (CET) >> Received: from zackbummfertig.de >> (static.243.42.46.78.clients.your-server.de [78.46.42.243]) >> (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) >> (No client certificate requested) >> by salonwebserver.de (Postfix) with ESMTPS id E86BD13BC078 >> for <weber(a)salonwebserver.de>; Fri, 14 Nov 2008 21:38:06 +0100 >> (CET) >> Received: from localhost (localhost [127.0.0.1]) >> by zackbummfertig.de (Postfix) with ESMTP id BC4BDECC067 >> for <weber(a)salonwebserver.de>; Fri, 14 Nov 2008 21:37:46 +0100 >> (CET) >> X-Virus-Scanned: amavisd-new at zackbummfertig.de >> Received: from zackbummfertig.de ([127.0.0.1]) >> by localhost (weberweb.zackbummfertig.de [127.0.0.1]) >> (amavisd-new, port 10024) >> with ESMTP id xr6ayW0lyDGC for <weber(a)salonwebserver.de>; >> Fri, 14 Nov 2008 21:37:43 +0100 (CET) >> Received: from rachel22b12579 (unknown [78.146.113.186]) >> by zackbummfertig.de (Postfix) with SMTP id 3E83CECC040 >> for <tbmarcom(a)acidbunker.de>; Fri, 14 Nov 2008 21:37:43 +0100 >> (CET) >> Date: Fri, 14 Nov 2008 20:38:01 +0000 >> From: Chrystal Huddleston <RoxieastronomerMyrick(a)fightingbobfest.org> >> Subject: gambling slot machines >> To: tbmarcom(a)acidbunker.de >> Message-id: <7686026460.20081111151378(a)fightingbobfest.org> >> Organization: fightingbobfest.org >> MIME-version: 1.0 >> Content-type: text/plain; charset=us-ascii >> Content-transfer-encoding: 7bit >> X-Priority: 3 (Normal) >> >> Steve schrieb: >> >>> Can you post the full header of the message in question? >>> >>> >>> -------- Original-Nachricht -------- >>> >>> >>>> Datum: Fri, 14 Nov 2008 23:18:39 +0100 >>>> Von: Marko Weber <weber(a)zackbummfertig.de> >>>> An: "dspam-users(a)lists.nuclearelephant.com" >>>> >> <dspam-users(a)lists.nuclearelephant.com> >> >>>> Betreff: Re: [dspam-users] mysql fetch row error >>>> >>>> >>> >>> >>>> Hi Steve, >>>> this sed thing dont worked for me...... sorry >>>> how i can get dspam working without this mysql fetch error ? >>>> u have any hints ? >>>> >>>> marko >>>> >>>> >>>> Steve schrieb: >>>> >>>> >>>>> -------- Original-Nachricht -------- >>>>> >>>>> >>>>> >>>>>> Datum: Fri, 14 Nov 2008 18:11:55 +0100 >>>>>> Von: Marko Weber <weber(a)zackbummfertig.de> >>>>>> An: "dspam-users(a)lists.nuclearelephant.com" >>>>>> >>>>>> >>>> <dspam-users(a)lists.nuclearelephant.com> >>>> >>>> >>>>>> Betreff: [dspam-users] mysql fetch row error >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>>> Hello to all in the Mailinglist, >>>>>> >>>>>> i installed dspam on our server and it works so far that incoming >>>>>> >> mails >> >>>>>> are tagged by dspam. >>>>>> i installed "dovecot-antispam" for retraining. >>>>>> when in try to retrain a mail i get this in the dspam.debug log = >>>>>> >>>>>> 9056: [11/14/2008 15:23:01] mysql_fetch_row() failed in >>>>>> >>>>>> >>>> _ds_get_signature >>>> >>>> >>>>>> in the /var/log/messages log i get this = >>>>>> >>>>>> Nov 14 15:29:25 kraftwerk1 imap: antispam: /usr/bin/dspam >>>>>> >>>>>> >>>> --source=error >>>> >>>> >>>>>> --class=spam --signature==?iso-8859-1?Q?491d76cb23581804284693=0D?= >>>>>> >> ... >> >>>>>> Nov 14 15:29:25 kraftwerk1 dspam[9189]: Signature retrieval for >>>>>> '=?iso-8859-1?Q?491d76cb23581804284693=0D?=' failed >>>>>> Nov 14 15:29:25 kraftwerk1 dspam[9189]: Unable to find a valid >>>>>> signature. Aborting. >>>>>> Nov 14 15:29:25 kraftwerk1 dspam[9189]: process_message returned >>>>>> >> error >> >>>>>> -5. dropping message. >>>>>> >>>>>> >>>>>> but when i search for the signatur 491d76cb23581804284693 in the >>>>>> >> mysql >> >>>>>> database i can find it there. >>>>>> >>>>>> what is this ?iso-8859-1?Q? , is this causing the error ? >>>>>> >>>>>> >>>>>> >>>>>> >>>>> This means, that the plugin developer used the signature from the body >>>>> >>>>> >>>> instead of the one found in the header. You can verify yourself what >>>> >> the >> >>>> proper signature would be. Just issue this small one liner in sed: >>>> >>>> >>>>> sed -e '/^$/q' -ne 's/^X\-DSPAM\-Signature: >>>>> >>>>> >>>> $\([1-9][0-9]*,$\{0,1\}$[a-f0-9]\{20,32\}$\)$/\1/p' >>>> >> /path/to/your/message >> >>>> >>>> >>>>> This should print out the proper signature DSPAM should receive from >>>>> >> the >> >>>>> >>>>> >>>> plugin. The iso-8859-1 part indicates to me that the message is taken >>>> >> from >> >>>> a iso8859-1 encoded message body. >>>> >>>> >>>>> >>>>> >>>>> >>>>>> do any1 need my dspam.conf , then i can paste it on web. >>>>>> >>>>>> thank for helping. >>>>>> >>>>>> marko >>>>>> >>>>>> >>>>>> >>>>>> >>>>> Steve >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > >

2 1

[Dovecot] Mailbox Hashing
by Justin Krejci 14 Nov '08

14 Nov '08

First off, the website documentation is really good for Dovecot but while reading I was not able to find anything pertaining to inbox hashing for Maildirs. I saw plenty about hashing the directories that the user mailboxes live in but nothing about specifically hashing an individual user's inbox directory itself. Is there any method for hashing the inbox automatically after say 5,000 messages are stored? Example $Maildir/in/0/message0 $Maildir/in/0/message1 $Maildir/in/0/message2 . $Maildir/in/0/message4999 $Maildir/in/1/message5000 $Maildir/in/1/message5001 etc I am not currently using Dovecot but am interested to know if this is available or does running with 20,000+ messages in a single inbox not affect the performance much? I have looked into other file system tuning techniques such as enabling ext3 dir_index or using ReiserFS (maybe not ReiserFS anymore). There will likely be 15,000 to 20,000 accounts spread out on one or more servers using a 6-drive RAID10 setup. Most accounts are not expected to have high message quantities but there will be lots of concurrent connections via pop and imap (and webmail imap). Any suggestions or feedback would be appreciated.

3 6

Re: [Dovecot] Install Configuring and run.
by Marko Weber 14 Nov '08

14 Nov '08

Hello Thorsten, i got "dovecot-antispam" working now. also moving mails now fully work. but retraining dont seems to work fully. emails that are delivered to mailbox are scanned with user "clamav" 28799: [11/13/2008 23:26:25] DSPAM Instance Startup 28799: [11/13/2008 23:26:25] input args: /usr/bin/dspam --stdout --deliver=spam,innocent --mode=tum --tokenizer=chained,noise --enable-signature-headers --user clamav 28799: [11/13/2008 23:26:25] pass-thru args: --tokenizer=chained,noise --enable-signature-headers 28799: [11/13/2008 23:26:25] processing user clamav 28799: [11/13/2008 23:26:25] uid = 105, euid = 105, gid = 449, egid = 1000 28799: [11/13/2008 23:26:25] loading preferences for user clamav 28799: [11/13/2008 23:26:25] Loading preferences for uid 1 i dont have a maildrop filter ready yet that move spam tagged mails into SPAM folder. all is delivered to "incoming". when i now move the mail to SPAM; this is done by user "postfix" 28917: [11/13/2008 23:40:17] DSPAM Instance Startup 28917: [11/13/2008 23:40:17] input args: /usr/bin/dspam --source=error --class=spam --signature==?iso-8859-1?Q?1,491930ac25229173225906=0D?= 28917: [11/13/2008 23:40:17] pass-thru args: 28917: [11/13/2008 23:40:17] processing user postfix 28917: [11/13/2008 23:40:17] uid = 207, euid = 207, gid = 207, egid = 1000 28917: [11/13/2008 23:40:17] loading preferences for user postfix 28917: [11/13/2008 23:40:17] Loading preferences for uid 6 28917: [11/13/2008 23:40:17] Loading preferences for uid 0 28917: [11/13/2008 23:40:17] Loading preferences for uid 0 28917: [11/13/2008 23:40:17] default preferences empty. reverting to dspam.conf preferences. 28917: [11/13/2008 23:40:17] Loading preferences from dspam.conf 28917: [11/13/2008 23:40:17] using /var/spool/dspam/opt-in/local/postfix.dspam as path 28917: [11/13/2008 23:40:17] using /var/spool/dspam/opt-out/local/postfix.nodspam as path 28917: [11/13/2008 23:40:17] sedation level set to: 0 28917: [11/13/2008 23:40:17] DSPAM Instance Shutdown. Exit Code: 0 Now when i move the mail again into incoming (also done by user postfix), the mail dont get tagged with "innocent" , it is still tagged "spam". How do i get it work all under "clamav" and why are the removed mails from spam to incoming not tagged with "inncoent" ? (do you need any logs?) thank you marko Thorsten Vollmer schrieb: > On Thu, 2008-11-13 at 21:30 +0100, Marko Weber wrote: > >> Sorry to ask, >> so if dovecot-antispam-20080829 dont work for me you mean i should >> install the latest from webpage ? >> > > Yes, dovecot-antispam-1.1 should be sufficient. I already sent you the > link to an updated ebuild. Please be more proactive. > > Thorsten > >

2 6

[Dovecot] Folder aliases
by Daniel L. Miller 14 Nov '08

14 Nov '08

I don't if this has been discussed before - or already implemented - but it would be great if I could define one or more "aliases" for a folder. A particular example might be "Sent" where different (badly-written) clients might have hard-coded "Sent" folder locations like "Sent", "Sent Items", "Sent Mail", "Mail Delivered", etc. - and I'd love to get those consolidated. It would also be nice to show a different set of folders based on the client - or if that's not possible (if the client doesn't identify itself as part of the login sequence) then somehow via login name - maybe by an extension? So where my usual login might be "user(a)domain.com", and that would continue to be supported as the default login, I could now have "user(a)domain.com+thunderbird", "user(a)domain.com+mobile", etc.? -- Daniel

6 10

Re: [Dovecot] Install Configuring and run.
by Marko Weber 13 Nov '08

13 Nov '08

Sorry to ask, so if dovecot-antispam-20080829 dont work for me you mean i should install the latest from webpage ? Thorsten Vollmer schrieb: > On Thu, 2008-11-13 at 00:24 +0100, Marko Weber wrote: > >> Hello Thorsten thank you for taking time. >> I recompiled with -debug, but i also had to set -syslog >> otherwise gentoo dont compile. it says then , "useflag syslog enabled >> needs flag debug also set" >> >> we proceed further in the list then ? >> > > Yes, once you have new results. > > If dovecot-antispam-20080829 does not yet work for you, please try the > latest version and re-enable debugging. > > Thorsten > >

1 0

[Dovecot] New Dovecot server buildout review
by David Jonas 13 Nov '08

13 Nov '08

Hello Everyone, I have the opportunity to do a new mail system build out for about 20k email boxes over a few hundred domains. I already run a system like this, but this time I can do it right :). My thought on storage is to have a 12 drive JBOD system attached via SAS to a 3ware 9690SA raid controller. Drives would be RAID'd (10 or 5) to appear as one for message storage. Dovecot would run on that box and deliver directly from postfix. POP and IMAP connections would come in directly (no proxy). Postfix head boxes in front of this do anti-virus and spamassassin, so this box won't be doing any heavy crunching, just delivery and serving of messages. Does this sound reasonable? Is there details that I'm missing? An obvious bottleneck I'm overlooking? Is anyone running a similar config with success? It all sounds very standard, but I haven't run something like this before. Thanks for the help! David

1 0

[Dovecot] Moving mail between folders
by Ilo Lorusso 13 Nov '08

13 Nov '08

Hi, Ive just installed Dovecot and postfix and Ive created a whole bunch of folders and started moving my mail around. Ive noticed that it copies the mail to the folder I wish to move the mail to and leave the original message waiting to be purged in the original folder Is there a way to have dovecot delete it straight after it has been moved? as I don't think users are going to take well to this change? any help would be appreciated Thanks, Ilo.

3 2

[Dovecot] dovecot pop3 performance, what size installation
by Brandon Lamb 13 Nov '08

13 Nov '08

Hello, I was wondering if anyone could tell me their results and maybe help me get an idea how big our mail system installation and usage is. For dovecot pop3 and nfs backend, I am curious how many machines I should expect to need. Here are my stats for what I can tell. 5-7 connections per second average, 8-12 connections per second peak. 30-50 constant connections between 3 servers (8-20+) on each server. We are servicing about 14,000+ email accounts. These are just the pop3 login stats, a lot of users use both or imap through our webmail. My question I guess is whether this is on the small side, medium or larger side mail setup, and should I be thinking that only 1 or 2 servers should be able to handle the load. Right now I am splitting it between 3. Should a single quad core box be able to handle this? Not sure if this is a simple question or not. Im not including my dovecot -n just because this is more of a generic question.

1 0

[Dovecot] removing uid, index and cache files
by Lars Uhlmann 13 Nov '08

13 Nov '08

Since a couple of days we're facing a problem with dovecot/MUA and multiple appeareances of single mails. The mail is physical existant only once but outlook shows it twice or more times. Then after some minutes these multiple "virtual" mails just disappear. Because this happens with different MUAs (Outlook and Thunderbid) I assume dovecot causes the problem. All mail are received by »getmail« from outside via POP3 and delivered to the users local Maildir. This was working without problems for over a year now. Is it save to remove to following files (will dovecot rebuild them) from every users Maildir? dovecot.index dovecot.index.cache dovecot.index.log dovecot.index.log.2 dovecot-uidlist Thahks in advance, Lars

3 4

[Dovecot] Sieve authentication / directory issue after upgrade to 1.0.13.
by Drew Calcott 12 Nov '08

12 Nov '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi everyone, I have come across a problem after upgrading from 1.0.rc17 to 1.0.13 (debian builds 1.0.13 is from etch-backports). I initially upgraded as 1.0.rc17 isn't compiled with regex support, so vacation messages that were uploaded via horde were being rejected. (Everything else was working fine, just messages that required regex functions were failing). Horde is uploading the filters via pysieved, which is then in turn placing them in the directory of: - --------- /shared/spool/active/%d/%0.1u/%1.1u/%n/sieve/ingo - --------- At least, it was previously. Since upgrading, dovecot has changed the way it passes authentication details to pysieved and the domain is no longer included in the username, so the domain cannot be parsed and pysieved fails as it is unable to strip a value for %d. (as a quick aside, this isn't an issue with single domain servers as there is a "defaultdomain" option that will force the value for %d, however, since we have 9 domains to deal with, this isn't going to cut it). Prior to upgrade, authentication was passed as thus (= are a new line in case the formatting dies) - -------- ======= > 'AUTH\t1\tPLAIN\tservice=pysieved\tresp=YWNhbD* restofstring * ====== < OK\t1\tuser=acal030(a)sit.auckland.ac.nz\tuser=acal030\t\n' ===== Finished SASL authentication : {'username':'acal030(a)sit.auckland.ac.nz', 'result': 'OK'} ===== Plugin returned home :'/shared/spool/active/sit.auckland.ac.nz/a/c/acal030/' = Authenticated user acal030(a)sit.auckland.ac.nz - -------- Both user and user@domain were being passed and it was a matter of simple nagging of the pysieved guys to update their code to strip the required information. However, the new version is passing auth like this: - -------- ======= > 'AUTH\t1\tPLAIN\tservice=pysieved\tresp=YWNhbD* restofstring * ======= < 'OK\t1\tuser=acal030\n' ===== Finished SASL authentication : {'username': 'acal030', 'result': 'OK'} ===== Plugin returned home : '/shared/spool/active//a/c/acal030/' - ------- The only change to the dovecot config I have made is to add "allow_all_users=yes" to userdb static as the new version was a lot stricter about the passdb pam lookup not containing the user data (and failing completely as a result). In the dovecot logs themselves, the following lines were from the earlier build: - ------ Info: auth(default): client out: OK 1 user=acal030(a)sit.auckland.ac.nz user=acal030 Info: auth(default): master in: REQUEST 243 23994 1 Info: auth(default): master out: USER 243 acal030(a)sit.auckland.ac.nz home=/blah, mail=/blah etc. - ------ Compared to this from the new one: - ------ Info: auth(default): auth(acal030(a)sit.auckland.ac.nz,130.216.39.182): username changed acal030(a)sit.auckland.ac.nz -> acal030 Info: auth(default): client out: OK 1 user=acal030 Info: auth(default): prefetch(acal030,130.216.39.182): success Info: auth(default): master out: USER 2 acal030 home=/blah, mail=/blah etc. - ------ I am rather at the end of my tether with this, unfortunately. :( I have browsed through many wiki pages looking for a possible solution or config variable I may have missed, but I really am coming up dry. Cleaned up dovecot.conf is at: http://pastebin.com/m7f4303af Should anyone want strace logs or whatever, I'm more than happy to make with them. Thanks in advance for any assistance. Regards, - --- Drew Calcott Science IT University of Auckland (p) +64 9 373 7599 x84269 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkaWWQACgkQD+yXTWfduLFGmACfW3XXNFLu8zhHUz8FeU+QqgmX pg4An3SdPh0OyWHGBe9k/qWzjqFRJmBl =fIMA -----END PGP SIGNATURE-----

1 1