[Dovecot] Deferred emails on alias emails
Hi,
I am currently making a fresh install of email server. I am having some issue with dovecot-antispam (cf. other emails), and noticed that I was having issues on aliases, which apparently where not forwarded to the actual SMTP server after passing through dspam. Still don't figure why but temporarily disable the dspam filter, which anyway is not useful in the current state of this installation. Now the email to an alias are well forwarded to the real finale server. But I still have many emails which were deferred in the last days and I cannot have them decrease.
# ls -l /var/spool/postfix/defer/* | wc -l 297
If I run "postqueue -f" to flush this all, I can see the number down, but fastly comes back to 297. Reading the logs, they are passed to dovecot, which was probably the issue before I disabled dspam. It was trying to deliver a message after filtering it, but in the case of aliases, of course, it cannot because the account is not local (so it does not find the account and defers the email forever)! So it should give it back to postfix so that this one contacts the aliased server (gmail.com in the example below).
Feb 4 13:33:18 ks34186 postfix/lmtp[31100]: 82769869FA: to=< someemail@gmail.com>, orig to=<someemail@mydomain.tld>, relay=mail.mydomain.tld[/var/run/dspam/dspam.sock] conn use=28, delay=142296, delays=142277/19/0/0.41, dsn=4.3.0, status=deferred (host mail.mydomain.tld[/var/run/dspam/dspam.sock] said: 421 4.3.0 <someemail@gmail.com> Auto-Whitelisted (in reply to end of DATA command)) Feb 4 13:33:18 ks34186 dovecot: auth(default): master in: USER 1 someemail@gmail.com service=deliver Feb 4 13:33:18 ks34186 dovecot: auth(default): passwd(someemail@gmail.com): lookup Feb 4 13:33:18 ks34186 dovecot: auth(default): passwd(someemail@gmail.com): unknown user Feb 4 13:33:18 ks34186 dovecot: auth-worker(default): sql( someemail@gmail.com): SELECT maildir as home, 1004 as uid, 1004 as gid FROM mailbox WHERE username = 'someemail@gmail.com' AND active = '1' Feb 4 13:33:18 ks34186 dovecot: auth-worker(default): sql( someemail@gmail.com): Unknown user Feb 4 13:33:18 ks34186 dovecot: auth(default): master out: NOTFOUND 1
So is there a way please to flush this whole spool so that it now uses the new method (which is stop using dspam until I figure how to use it) and so that I can forward these emails to their recipient? Thanks.
Jehan
P.S.: this is maybe more a postfix issue than a dovecot issue though... but you probably all know well postfix as well.
Hi,
I finally found my solution so I give it here if ever it interestes anyone. I needed to use "postsuper -r ALL" in order to requeue the whole message list and they were then using the new postfix configuration.
Note though that if you have a lot lot of spooled messages, the "ALL" is maybe not a good idea if they are mainly for a single server. In my case, most of these messages were forwarded to the gmail servers, and as there were many spams unfortunately in these emails forwarded (and it's not my server's purpose to remove the spams when forwarding), all requeued and sent in the following minutes, I saw in my logs that gmail has temporarily blocked my server with a human readable message from Google in my logs. Bye.
Jehan
Anyway now it works.
On Wed, Feb 4, 2009 at 1:46 PM, Jehan Pagès <jehan.marmottard@gmail.com>wrote:
Hi,
I am currently making a fresh install of email server. I am having some issue with dovecot-antispam (cf. other emails), and noticed that I was having issues on aliases, which apparently where not forwarded to the actual SMTP server after passing through dspam. Still don't figure why but temporarily disable the dspam filter, which anyway is not useful in the current state of this installation. Now the email to an alias are well forwarded to the real finale server. But I still have many emails which were deferred in the last days and I cannot have them decrease.
# ls -l /var/spool/postfix/defer/* | wc -l 297
If I run "postqueue -f" to flush this all, I can see the number down, but fastly comes back to 297. Reading the logs, they are passed to dovecot, which was probably the issue before I disabled dspam. It was trying to deliver a message after filtering it, but in the case of aliases, of course, it cannot because the account is not local (so it does not find the account and defers the email forever)! So it should give it back to postfix so that this one contacts the aliased server (gmail.com in the example below).
Feb 4 13:33:18 ks34186 postfix/lmtp[31100]: 82769869FA: to=< someemail@gmail.com>, orig to=<someemail@mydomain.tld>, relay=mail.mydomain.tld[/var/run/dspam/dspam.sock] conn use=28, delay=142296, delays=142277/19/0/0.41, dsn=4.3.0, status=deferred (host mail.mydomain.tld[/var/run/dspam/dspam.sock] said: 421 4.3.0 <someemail@gmail.com> Auto-Whitelisted (in reply to end of DATA command)) Feb 4 13:33:18 ks34186 dovecot: auth(default): master in: USER 1 someemail@gmail.com service=deliver Feb 4 13:33:18 ks34186 dovecot: auth(default): passwd(someemail@gmail.com): lookup Feb 4 13:33:18 ks34186 dovecot: auth(default): passwd(someemail@gmail.com): unknown user Feb 4 13:33:18 ks34186 dovecot: auth-worker(default): sql( someemail@gmail.com): SELECT maildir as home, 1004 as uid, 1004 as gid FROM mailbox WHERE username = 'someemail@gmail.com' AND active = '1' Feb 4 13:33:18 ks34186 dovecot: auth-worker(default): sql( someemail@gmail.com): Unknown user Feb 4 13:33:18 ks34186 dovecot: auth(default): master out: NOTFOUND 1
So is there a way please to flush this whole spool so that it now uses the new method (which is stop using dspam until I figure how to use it) and so that I can forward these emails to their recipient? Thanks.
Jehan
P.S.: this is maybe more a postfix issue than a dovecot issue though... but you probably all know well postfix as well.
On 2/5/2009, Jehan Pagès (jehan.marmottard@gmail.com) wrote:
and as there were many spams unfortunately in these emails forwarded (and it's not my server's purpose to remove the spams when forwarding), all requeued and sent in the following minutes, I saw in my logs that gmail has temporarily blocked my server with a human readable message from Google in my logs. Bye.
You should make it your servers business to do just that, otherwise you risk getting blocked (like you said you just did), when your server blindly forwards a ton of spam to other servers that DO care.
--
Best regards,
Charles
Hi,
I don't agree. If antispam systems were really 100% sure to never have any false positive, I would agree (having false negative is less an issue). But we all know that such system does not exist yet (and probably never will IMHO, even though efficiency may increase to close to 100%, as it is still humans behind emails, we will never be able to trust fully a machine to class them). For such reason, I cannot do such a thing as blocking spams for users (I could tag them, but not block them) without giving possibility to check the blocked email. As it is an alias, then it is more difficult to manage (you can provide such system to local users, but when someone makes a redirection from one of his emails to another, this is usually to deal all of one's emails with the finale account). I don't think any email provider does "just block" spams.
For what happened here, the issue was that I have had an issue of the emails for aliases which were blocked in my spool for days. Then I managed to deblock them, but they have been all sent on the row (so this looked suspicious by the gmail server, which is normal). And as for many accounts, they receive a lot of spam (I receive a hundred everyday myself because of the age of my address). On normal use, they are simply received slowly during the day.
That's unfortunate, but I cannot do what you say for these reasons. This is somehow a question of network neutrality ( http://dig.csail.mit.edu/breadcrumbs/node/144 ). But don't make me tell what I don't: I would love to be able to block spam and if a secure antispam system appears, then I would use it. I don't argue that spam is potentially interesting and that we should allow it for people loving it (who?!) of course; the neutrality is only lost when the antispam fails and caught non-spam as well, because it is then a regulation system which controls people's personal data and would improve net experience of many at the price of data loss for others.
As a conclusion: email hosting is not my business, I am a developper, not an admin, and this server is only for my personal use and of a few friends... (you guess I wouldn't have such issue if it was a professional server). So there will never be tons of spam redirected to other servers (and only one guy in my server has such a redirection anyway, moreover my smtp servers is not accepting email relaying from unknown users). Regards,
Jehan
On Thu, Feb 5, 2009 at 12:47 PM, Charles Marcus <CMarcus@media-brokers.com>wrote:
On 2/5/2009, Jehan Pagès (jehan.marmottard@gmail.com) wrote:
and as there were many spams unfortunately in these emails forwarded (and it's not my server's purpose to remove the spams when forwarding), all requeued and sent in the following minutes, I saw in my logs that gmail has temporarily blocked my server with a human readable message from Google in my logs. Bye.
You should make it your servers business to do just that, otherwise you risk getting blocked (like you said you just did), when your server blindly forwards a ton of spam to other servers that DO care.
--
Best regards,
Charles
On 2/5/2009, Jehan Pagès (jehan.marmottard@gmail.com) wrote:
That's unfortunate, but I cannot do what you say for these reasons. This is somehow a question of network neutrality ( http://dig.csail.mit.edu/breadcrumbs/node/144 ).
Not sure what net neutrality has to do with this... unless you are syaing you are operating an ISP service?
But don't make me tell what I don't: I would love to be able to block spam and if a secure antispam system appears, then I would use it. I don't argue that spam is potentially interesting and that we should allow it for people loving it (who?!) of course; the neutrality is only lost when the antispam fails and caught non-spam as well, because it is then a regulation system which controls people's personal data and would improve net experience of many at the price of data loss for others.
Even for ISPs, there are a number of ways to dramatically reduce spam with pretty much zero false positives...
ASSP (and now ASSPS) is a great tool...
http://apps.sourceforge.net/mediawiki/assp/
Forwarding OBVIOUS spam is simply passing on what should be your problem to others.
--
Best regards,
Charles
Hi,
On Thu, Feb 5, 2009 at 3:14 PM, Charles Marcus <CMarcus@media-brokers.com>wrote:
Not sure what net neutrality has to do with this... unless you are syaing you are operating an ISP service?
As for I, net neutrality is not about ISP only, but also for service providers (even if not professional like I, just for the fun). Being a neutral service means not trying to control data. Hence providing email service implies that I must do my possible so that the user receives all one's data. Spam is not data, only annoyance, so if I can remove it safely, it is OK. But expected emails are data, and if there is a risk to lose it (which is still better than "using" it, for instance for commercial use, or said "security", but still is bad), then it is not neutral. It is becoming a "judge" on my users' data (deciding alone what is good or not... and maybe making mistakes). This is my opinion at least.
Even for ISPs, there are a number of ways to dramatically reduce spam with pretty much zero false positives...
I read some documents about such methodology and was planning to probably implement whitelister ( http://blog.madism.org/index.php/2006/03/25/79-debianorg-and-spam The concept looks interesting because it is not just a stupid email blocker as using RBL which can blocks a lot of real email. Just have a look to bad email provider as hotmail or Yahoo! on which I met many people having issues of unreceived emails. These are what I call pretty unneutral service which decides that such or that small server is probably bad, without knowing it) on my server...
But first I need to have my server fully functional, and currently I am first fighting with dspam. I make stuffs in the right order and whitelister (or other systems) is the next step. ;-)
ASSP (and now ASSPS) is a great tool...
I will have a look at this, thanks. Just the website on such a complicated topic does not say much about the methodology. :-)
Forwarding OBVIOUS spam is simply passing on what should be your problem to others.
I am right when you are 100% sure, once again. In any other cases, it is a question of point of view as I said. Many users (I included) would consider that it is not the service provider issue to decide what is good or bad for me. Regards,
Jehan
on 2-5-2009 6:47 AM Jehan Pagès spake the following:
Hi,
On Thu, Feb 5, 2009 at 3:14 PM, Charles Marcus <CMarcus@media-brokers.com>wrote:
Not sure what net neutrality has to do with this... unless you are syaing you are operating an ISP service?
As for I, net neutrality is not about ISP only, but also for service providers (even if not professional like I, just for the fun). Being a neutral service means not trying to control data. Hence providing email service implies that I must do my possible so that the user receives all one's data. Spam is not data, only annoyance, so if I can remove it safely, it is OK. But expected emails are data, and if there is a risk to lose it (which is still better than "using" it, for instance for commercial use, or said "security", but still is bad), then it is not neutral. It is becoming a "judge" on my users' data (deciding alone what is good or not... and maybe making mistakes). This is my opinion at least.
Even for ISPs, there are a number of ways to dramatically reduce spam with pretty much zero false positives...
I read some documents about such methodology and was planning to probably implement whitelister ( http://blog.madism.org/index.php/2006/03/25/79-debianorg-and-spam The concept looks interesting because it is not just a stupid email blocker as using RBL which can blocks a lot of real email. Just have a look to bad email provider as hotmail or Yahoo! on which I met many people having issues of unreceived emails. These are what I call pretty unneutral service which decides that such or that small server is probably bad, without knowing it) on my server...
But first I need to have my server fully functional, and currently I am first fighting with dspam. I make stuffs in the right order and whitelister (or other systems) is the next step. ;-)
ASSP (and now ASSPS) is a great tool...
I will have a look at this, thanks. Just the website on such a complicated topic does not say much about the methodology. :-)
Forwarding OBVIOUS spam is simply passing on what should be your problem to others.
I am right when you are 100% sure, once again. In any other cases, it is a question of point of view as I said. Many users (I included) would consider that it is not the service provider issue to decide what is good or bad for me. Regards,
Jehan
But if you relay any spam and get blacklisted for it, you will be the one that suffers. The blacklists don't care what your point of view is or if you generated the spam or just relayed it.
Then everyone on your server will suffer for the actions of another.
That is why ISP's try and block as much spam as they can.
Society is full of entities that decide what is good for you or not.
The Government decides that driving too fast is dangerous, so they set speed limits.
Someone might get some tainted food from a manufacturer, and they are stopped from shipping anymore goods until it can be determined what happened.
Drinking and driving is dangerous to others, so you are not allowed to do it.
Spam is a bane to the normal flow of e-mail, and is considered by some as theft of service since you are stealing someone elses paid for bandwidth to send the junk to people that don't want it anyway.
I stop a lot of spam before it ever enters or leaves my servers. It is just good practice, no matter how Libertarian my users might be.
Your servers are yours, and if someone wants to use your server to relay mail, they should have to also follow good practice. If your server sends me junk, it will get blacklisted. I will not look at the messages and see if they came from someone else, and most other systems won't either. You forward it, you are considered just as guilty.
If your friend robs a bank, and you are "just driving the car" you still go to jail with him, don't you? The police won't let you go because it isn't "your problem".
Rant over!
-- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
Hi,
On Thu, Feb 5, 2009 at 7:37 PM, Scott Silva <ssilva@sgvwater.com> wrote:
But if you relay any spam and get blacklisted for it, you will be the one that suffers. The blacklists don't care what your point of view is or if you generated the spam or just relayed it.
Then everyone on your server will suffer for the actions of another.
That is why ISP's try and block as much spam as they can.
Society is full of entities that decide what is good for you or not.
Yes that's true and for many of them, I really don't agree that it is "the way to do". In some dictatorial country, they would say that sending an email criticizing your governement is bas (is it spam?! Maybe for these governments), and then you would go to jail. This is not because "things are this way" that they are the right way.
The Government decides that driving too fast is dangerous, so they set speed limits.
Someone might get some tainted food from a manufacturer, and they are stopped from shipping anymore goods until it can be determined what happened.
Drinking and driving is dangerous to others, so you are not allowed to do it.
Spam is a bane to the normal flow of e-mail, and is considered by some as theft of service since you are stealing someone elses paid for bandwidth to send the junk to people that don't want it anyway.
Really, I try, but I cannot see the parallel between just forwarding messages (as I was asked to when setting this alias!) and getting other people's live in danger.
I stop a lot of spam before it ever enters or leaves my servers. It is just good practice, no matter how Libertarian my users might be.
Your servers are yours, and if someone wants to use your server to relay mail, they should have to also follow good practice. If your server sends me junk, it will get blacklisted. I will not look at the messages and see if they came from someone else, and most other systems won't either. You forward it, you are considered just as guilty.
I won't forward spam to you, I already told you. This is just an email alias for a friend. It is just a email translation for a single external email which has been explicitely asked. My smtp server is not an open relay for any organized spammer who would like to forwards hundreds of spams to the world... Nobody can use my server for spamming the world, so neither your server.
If your friend robs a bank, and you are "just driving the car" you still go to jail with him, don't you? The police won't let you go because it isn't "your problem".
No because you participate actively to the robbery. But let's just say that you are a taxi that passes by and you don't know that the guy is a robber and has a bank's money in his bag because you are not here to investigate on all your customers nor to look in their bag. As long as the guy don't ask you to drive fast over speed limitation or to try to flee the cops, then you have no reason to be suspicious, do you? Then you just drive your customer and leaves him his intimacy. If ever the cops ask to stop your car (then the taximan just stops), you should not be sent to jail. And really you should have nothing to be blamed for in such situation. You provide a service: getting people from one place to another, and you just do it when you are asked to.
The reason of this all is that this is not your job to spy on anyone, trying to guess if they do bad things, or whatever. This is among the worse that could happen in a society and results in a society of denunciation, where you cannot trust anyone (not even your family or friends), where everyone spies each other, etc.
Of course all of this change if you are sure that the guy is a robber (for instance, you saw him rob the bank with his gun for instance). Then you would not drive him (but not because this is the law for my own, but just because it is bad) of course. That's the same for spam. If ever you are sure, of course, don't send. Still as I say again and again: the problem is to be sure! If you have even a slight uncertaincy, then what should you do? Refuse him the lift? Maybe he was running because his wife is going to have her baby and he needs to go back fast. Or he is going to miss his fly or his job meeting... You don't know and that's not because you think this guy "look strange" for instance that you should refuse him in your car. This leads also to bad behavior (if one can accept this, then one can accept to refuse people on "liking their face or not"...).
Rant over!
Anyway I have the impression that we may never agree... I don't know if we should continue this discussion on this list. ;-)
Jehan
On 2/5/2009, Jehan Pagès (jehan.marmottard@gmail.com) wrote:
I won't forward spam to you, I already told you. This is just an email alias for a friend. It is just a email translation for a single external email which has been explicitely asked. My smtp server is not an open relay for any organized spammer who would like to forwards hundreds of spams to the world... Nobody can use my server for spamming the world, so neither your server.
Um.. did you or did you not say:
"In my case, most of these messages were forwarded to the gmail servers, and as there were many spams unfortunately in these emails forwarded (and it's not my server's purpose to remove the spams when forwarding),"
Did you mis-speak?
"... as there were many spams..." means that your server DID forward 'many spams' - and you even said it got your server temporarily blacklisted...
Its your server, do what you want... just don't be surprised when people respond accordingly.
--
Best regards,
Charles
Hi,
On Thu, Feb 5, 2009 at 9:33 PM, Charles Marcus <CMarcus@media-brokers.com>wrote:
Um.. did you or did you not say:
"In my case, most of these messages were forwarded to the gmail servers, and as there were many spams unfortunately in these emails forwarded (and it's not my server's purpose to remove the spams when forwarding),"
Did you mis-speak?
"... as there were many spams..." means that your server DID forward 'many spams' - and you even said it got your server temporarily blacklisted...
Yes but the issue was only this one time because I sent in one single row several days of emails on an account which exists for years and years (so unfortunately my friend has now many spams in several days, as well as many real excepted emails, just as I) because they were blocked before on my misconfigured new email server. That's not going to happen every day that I will suddenly forward all the emails of several days for my friend.
Its your server, do what you want... just don't be surprised when people respond accordingly.
Thanks for the concern, but don't worry, I take all the necessary actions. I just don't trust spam systems because they are not reliable, except for finale destination when they can be checked (that's why I install dspam for locale users using IMAP), that's all. But I am looking for other more reliable methods (cf. for instance the whitelister program...). Regards,
Jehan
On 2/5/2009, Jehan Pagès (jehan.marmottard@gmail.com) wrote:
Yes but the issue was only this one time because I sent in one single row several days of emails on an account which exists for years and years (so unfortunately my friend has now many spams in several days, as well as many real excepted emails, just as I) because they were blocked before on my misconfigured new email server. That's not going to happen every day that I will suddenly forward all the emails of several days for my friend.
Doesn't really matter.
If servers see you sending enough even OCCASIONAL spam, you will EVENTUALLY get on blacklists - and some of those are not easy to get off of.
AGAIN... it is your server, do what you want, but don't be surprised when other admins act accordingly and BLOCK you.
I think, like Zed, this thread is dead.
--
Best regards,
Charles
participants (3)
-
Charles Marcus
-
Jehan Pagès
-
Scott Silva