i set "mail_debug = yes" in dovecot.conf, restartet dovecot and tried to move a mail out of the spamfolder with thunderbird to trigger the retraining via the dovecot-module. this of course still didn't work. then i checked mail.err, mail.info, mail.warn, mail.log and dspam.debug but it didn't write more lines than before in the logs. more ideas ?

stefan

Peter Fern schrieb:

IIRC when you turn mail debugging on in dovecot the plugin should send the output of the dspam call to the maillog, should tell you what's wrong then...

Stefan Schmidt wrote:

...

hello,

i added the dovecot-dspam-plugin [1] to my dovecot-installation. when trying to move a mail out of or into the predefined spamfolder with thunderbird a messagebox shows : "dspam failed" and the message remains in its original position. i am using virtual mail users contained in a postgresql-database [2]. normal imap access and dspam training from the commandline works fine. a posting concerning probably the same issue has been posted on the dspam-mailinglist [3], but there haven't been any helpful replies so far. furthermore i suspect the plugin is missing the --user option when calling dspam, since this seems to be mandatory [4,5], but unfortunately adding this option in the plugin doesn't change its behavior. i am working on a debian system and currently using dovecot 1.0.beta3-3, dspam 3.6.4-2+b1 and the dspam-plugin "for dovecot beta 2 and 3".

1. calling dspam from the commandline with the same parameters as in the (slightly modified) plugin works fine

:/# su virtual :/$ /usr/bin/dspam --user retrain --source=error --stdout --class=spam --signature=4,4434846e237406491211187 --debug

-> /var/log/dspam.debug

No QuarantineAgent option found. Using standard quarantine. DSPAM Instance Startup input args: /usr/bin/dspam --debug --user retrain --source=error --stdout --class=spam --signature=4,4434846e237406491211187 pass-thru args: processing user retrain uid = 0, euid = 0, gid = 0, egid = 101 loading preferences for user retrain Loading preferences for uid 7 Loading preferences for uid 0 Loading preferences for uid 0 default preferences empty. reverting to dspam.conf preferences. Loading preferences from dspam.conf using /var/spool/dspam/opt-in/local/retrain.dspam as path using /var/spool/dspam/opt-out/local/retrain.nodspam as path sedation level set to: 0 loading preferences for user some_virtual_account@mydomain.com Loading preferences for uid 4 Loading preferences for uid 0 Loading preferences for uid 0 default preferences empty. reverting to dspam.conf preferences. Loading preferences from dspam.conf processing signature. length: 1344 reversing 112 tokens Control: [10 10] [11 9] Delta: [1 -1] reclassifying iteration 1 result: 0 libdspam returned probability of 1.000000 message result: SPAM DSPAM Instance Shutdown. Exit Code: 0

2. when dspam is called via the (slightly modified) dovecot-dspam-plugin it aborts (the unmodified version aborts the same way)

-> /var/log/mail.err

dovecot: imap-login: Login: user=<someuser@mydomain.com>, method=plain, rip=123.123.123.123, lip=123.123.123.123 imap: /usr/bin/dspam --user retrain --source=error --stdout --class=spam --signature=4,4434846e237406491211187 dspam[25157]: Unable to determine the destination user dspam[25157]: DSPAM agent misconfigured: aborting

-> /etc/dspam/dspam.conf [extract]

Home /var/spool/dspam StorageDriver /usr/lib/dspam/libpgsql_drv.so TrustedDeliveryAgent "/usr/bin/procmail" OnFail error Trust root Trust dspam Trust mail Trust mailnull Trust smmsp Trust daemon Trust virtual TrainingMode teft TestConditionalTraining on Feature chained Feature whitelist Algorithm graham burton PValue graham Preference "spamAction=deliver" Preference "signatureLocation=headers" AllowOverride trainingMode AllowOverride spamAction spamSubject AllowOverride statisticalSedation AllowOverride enableBNR AllowOverride enableWhitelist AllowOverride signatureLocation AllowOverride showFactors AllowOverride optIn optOut AllowOverride whitelistThreshold HashRecMax 98317 HashAutoExtend on HashMaxExtents 0 HashExtentSize 49157 HashMaxSeek 100 HashConnectionCache 10 Notifications off PurgeSignatures 14 PurgeNeutral 90 PurgeUnused 90 PurgeHapaxes 30 PurgeHits1S 15 PurgeHits1I 15 LocalMX 127.0.0.1 SystemLog on UserLog on Opt out ProcessorBias on Include /etc/dspam/dspam.d/

-> /etc/dspam/dspam.d/pgsql.conf [extract]

PgSQLServer 127.0.0.1 PgSQLPort 5432 PgSQLUser dspam PgSQLPass mypassword PgSQLDb libdspam7drvpgsql PgSQLVirtualTable dspam_virtual_uids PgSQLVirtualUIDField uid PgSQLVirtualUsernameField username

-> dspam-headers from samples message

X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu Apr 6 03:32:07 2006 X-DSPAM-Confidence: 0.7524 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 44346f97188755315134984

ciao, stefan

[1] http://johannes.sipsolutions.net/Projects/dovecot-dspam-integration [2] http://www.gjdv.at/snippets/linux/virtual_mail_hosting [3] http://www.nuclearelephant.com/projects/dspam/dspam-users/2872.html [4] http://www.nuclearelephant.com/projects/dspam/dspam-users/2852.html [5] http://www.nuclearelephant.com/projects/dspam/dspam-users/2850.html

• the maildirs are stored in /var/vmail/domain.tld/userid
• all the directories/files belong to user 'virtual'
• the virtual users are stored in a database vmail
• in the database all virtual users have the uid/gid of the user 'virtual'
• mail delivery via postfix and other dovecot-actions work fine

to be on the safe side i added virtual as a trusted user in the dspam-configuration, but that didn't change anything either (should have been a different error-message anyway).

stefan

Peter Fern schrieb:

Hmmm, so in your virtual system all users are uid virtual?

Stefan Schmidt wrote:

...

i set "mail_debug = yes" in dovecot.conf, restartet dovecot and tried to move a mail out of the spamfolder with thunderbird to trigger the retraining via the dovecot-module. this of course still didn't work. then i checked mail.err, mail.info, mail.warn, mail.log and dspam.debug but it didn't write more lines than before in the logs. more ideas ?

stefan

Peter Fern schrieb:

...

IIRC when you turn mail debugging on in dovecot the plugin should send the output of the dspam call to the maillog, should tell you what's wrong then...

Stefan Schmidt wrote:

...

hello,

i added the dovecot-dspam-plugin [1] to my dovecot-installation. when trying to move a mail out of or into the predefined spamfolder with thunderbird a messagebox shows : "dspam failed" and the message remains in its original position. i am using virtual mail users contained in a postgresql-database [2]. normal imap access and dspam training from the commandline works fine. a posting concerning probably the same issue has been posted on the dspam-mailinglist [3], but there haven't been any helpful replies so far. furthermore i suspect the plugin is missing the --user option when calling dspam, since this seems to be mandatory [4,5], but unfortunately adding this option in the plugin doesn't change its behavior. i am working on a debian system and currently using dovecot 1.0.beta3-3, dspam 3.6.4-2+b1 and the dspam-plugin "for dovecot beta 2 and 3". [...]

hello,

...

1. calling dspam from the commandline with the same parameters as in the (slightly modified) plugin works fine

:/# su virtual :/$ /usr/bin/dspam --user retrain --source=error --stdout --class=spam --signature=4,4434846e237406491211187 --debug

That user looks odd. Is that intentional?

yes, if the "--user" option is missing dspam refuses to launch, but it is only a dummy user. when dspam is called it switches to user belonging to the signature.

I would guess that some environment is different, but I don't know. dovecot is running as user 'virtual' in that setup?

no, dovecot is running as user dovecot. but to my understanding it switches to the userid of the virtual user accounts before reading/writing data to the maildirs.

Personally, I'm still running an ancient version of dspam with the plugin. Maybe I'll run into the same problems once I managed to upgrade.

please inform me if you stumble across any solutions :-)

stefan

Does the user 'retrain' exist in the virtual_uids table? yes, there is a user retrain in the the table dspam_virtual_uids of my dspam database libdspam7drvpgsql. Also, you really shouldn't need to specify --user to run dspam, the uid is stored with the signature data, and I'm running without it... i would love to it this way :-) what error do you get without --user? This error also appears to usually be generated by a problem with the LDA which is somewhat odd since retraining shouldn't attempt to deliver the message. without the "--user"-option dspam throws the same error as dovecot does when called either with or without it and writes nothing to the dspam.debug-log.

Unable to determine the destination user DSPAM agent misconfigured: aborting

called with the "--user"-option via commandline the dspam.debug-log says

No QuarantineAgent option found. Using standard quarantine. DSPAM Instance Startup input args: /usr/bin/dspam --debug --user retrain --source=error --stdout --class=innocent --signature=4,443a49e9305501142080570 pass-thru args: processing user retrain uid = 0, euid = 0, gid = 0, egid = 101 loading preferences for user retrain Loading preferences for uid 7 Loading preferences for uid 0 Loading preferences for uid 0 default preferences empty. reverting to dspam.conf preferences. Loading preferences from dspam.conf using /var/spool/dspam/opt-in/local/retrain.dspam as path using /var/spool/dspam/opt-out/local/retrain.nodspam as path sedation level set to: 0 loading preferences for user the_virtual_user_belonging_to_the_signature@domain.tld Loading preferences for uid 4 Loading preferences for uid 0 Loading preferences for uid 0 default preferences empty. reverting to dspam.conf preferences. Loading preferences from dspam.conf processing signature. length: 10944 reversing 912 tokens Control: [10 10] [9 11] Delta: [-1 1] reclassifying iteration 1 result: 0 [graham] [0.010000] E: 7346274736306913280 (3frq, 0s, 3i) [burton] [0.010000] E: 7346274736306913280 (3frq, 0s, 3i) [...] [burton] [0.010000] E: 9113948169156146802 (1frq, 0s, 3i) [burton] [0.010000] E: 15732536091729293487 (1frq, 0s, 3i) Graham-Bayesian Probability: 0.000000 Samples: 15 Burton-Bayesian Probability: 0.000000 Samples: 27 no factors specified; using default Result Confidence: 0.99 libdspam returned probability of 0.000000 message result: NOT SPAM DSPAM Instance Shutdown. Exit Code: 0

stefan

hello,

yes, all my virtual users in the vmail-database have the uid of the user 'virtual'. anyway, i seem to have found the reason why it didn't work. as ryan pointed out:

"Even though dspam can determine the user from the signature, it still needs to startup as a valid user. If the user is notspecified on the command line with the --user parameter, dspam uses the user that invoked dspam"

okay, so far so good. i had already added the user virtual to my dspam-database but for the sake of it i tried again, but again no success.

"Unable to determine the destination user. DSPAM agent misconfigured: aborting"

same thing when trying to call dspam without the "--user foo" option logged in as user 'virtual' on the commandline. oddly enough when logged in with my "normal" system account i do not need the "--user foo" option and a user with my name is automatically created in the dspam database (fits together with ryans statement). however i did not go in researching on this behaviour, so let's for now settle on the fact that on my current configuration i need the "--user foo" option to call dspam with the user 'virtual' for whatever reason that is ;-)

thus i inserted a user 'retrain' in my database by feeding dspam some spam for this user. (could have probably used 'virtual' as a dspam-dummyuser, but i wanted to be on the safe side). now i inserted the parameter "--user retrain" and "--debug" into the plugin-source, compiled it and copied it the dovecot plugin directory.

execl ("/usr/bin/dspam", "dspam", "--user retrain","--debug","--source=error", "--stdout", class_arg, sign_arg, NULL);

when trying to move mails into or out of the spam folder this results in the well known above error message, although - as pointed out before - executing the same command with the user virtual from the command line works fine. to debug this i substituted the executable with a little shell script logging the command line arguments and the user that called the script, but it didn't shed any light. somehow i got the idea to seperate the command line argument and it's value in the execl-statement.

execl ("/usr/bin/dspam", "dspam", "--user","retrain","--debug","--source=error", "--stdout", class_arg, sign_arg, NULL);

compiled it, copied it to the dovecot plugin directory and now it magically worked ! i again substituted the dspam-executable with my debugging shell script to check what was different, but as far as i could see the results in my logfile were exactly the same(!) (didn't use a hex-editor though, so there could be a remote possibility of a blank not being a blank or something like that). further checking my logs, it seems as if dspam is now working correctly.

No QuarantineAgent option found. Using standard quarantine. DSPAM Instance Startup input args: dspam --user retrain --debug --source=error --stdout --class=spam --signature=4,443a49e9305501142080570 pass-thru args: processing user retrain uid = 107, euid = 107, gid = 107, egid = 101 loading preferences for user retrain Loading preferences for uid 7 Loading preferences for uid 0 Loading preferences for uid 0 default preferences empty. reverting to dspam.conf preferences. Loading preferences from dspam.conf using /var/spool/dspam/opt-in/local/retrain.dspam as path using /var/spool/dspam/opt-out/local/retrain.nodspam as path sedation level set to: 0 loading preferences for user the_virtual_user_of_the_signature@mydomain.tld Loading preferences for uid 4 Loading preferences for uid 0 Loading preferences for uid 0 default preferences empty. reverting to dspam.conf preferences. Loading preferences from dspam.conf processing signature. length: 10944 reversing 912 tokens Control: [10 10] [11 9] Delta: [1 -1] reclassifying iteration 1 result: 0 libdspam returned probability of 1.000000 message result: SPAM DSPAM Instance Shutdown. Exit Code: 0

so maybe the execl-statement doesn't allow any blanks in it's arguments

• could this possibly be ?

resumé: for most users the plugin should work just fine if they insert the user that dspam is called as into the dspam-database, but in some special cases the "--user foo"-option seems to needed (for whatever reason). i hope this thread is useful to some other potential dovecot-dspam-plugin-user :-)

stefan

Johannes Berg schrieb:

On Tue, 2006-04-11 at 15:12 +0200, Stefan Schmidt wrote:

...

no, dovecot is running as user dovecot. but to my understanding it switches to the userid of the virtual user accounts before reading/writing data to the maildirs.

Right. But is that the 'virtual' user for all virtual users?

johannes