On 03-05-13 18:49, lst_hoe02@kwsoft.de wrote:

Zitat von Timo Sirainen <tss@iki.fi>:

...

GMail doesn't delete mails when POP3 client issues a DELE command for it. Instead they just become invisible for future POP3 sessions, but they still exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot:

• Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
• When DELEting a message, add this flag to the message.
• When listing messages, skip all messages that have this flag.
• Also hide this flag from IMAP clients(?)

Thoughts? Probably 20 lines of new code.

Hmm, this would lead to every increasing mailboxes for POP3 only users. What about this

• Use some setting to express that POP3 deleted mail should be IMAP mark-as-deleted

This conflicts with (many) IMAP user agents that don't use mark-as-deleted but have some version of 'move to trash folder' instead. I do agree that it's probably the Right Way (tm) :)

• Skip messages marked as deleted with POP3, maybe even as default?

With this on IMAP access you clearly see what is going on. Also if you delete Messages by IMAP they won't get downloaded by POP3. This would also fit with mail providers doing an "IMAP" expunge/purge on a fixed timebase.

Kind regards, Tom

Zitat von Timo Sirainen <tss@iki.fi>:

On 3.5.2013, at 20.30, Tom Hendrikx <tom@whyscream.net> wrote:

...

On 03-05-13 18:49, lst_hoe02@kwsoft.de wrote:

...

Zitat von Timo Sirainen <tss@iki.fi>:

...

GMail doesn't delete mails when POP3 client issues a DELE command for it. Instead they just become invisible for future POP3 sessions, but they still exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot:

• Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
• When DELEting a message, add this flag to the message.
• When listing messages, skip all messages that have this flag.
• Also hide this flag from IMAP clients(?)

Thoughts? Probably 20 lines of new code.

Hmm, this would lead to every increasing mailboxes for POP3 only users. What about this

• Use some setting to express that POP3 deleted mail should be IMAP mark-as-deleted

This conflicts with (many) IMAP user agents that don't use mark-as-deleted but have some version of 'move to trash folder' instead. I do agree that it's probably the Right Way (tm) :)

Depends on what purpose this feature is going to be used for. Many
POP3 users don't actually want to delete the mails, they just want
to download them, and their client just happens to delete the mails
as well. They might still want to use webmail occasionally to find
any old mails. For that purpose \Deleted flag wouldn't be ideal.
But, with this setting it would be possible as well:
pop3_deleted_flag = \Deleted (although that would need a few more
lines of code).

But for simply leave mail on server with POP3 most MUAs have this
feature anyway today, no? So at least for Thunderbird and Outlook this
couldn't be the only use-case. Most requests i have seen where because
of

• Misguided user using POP3 on different devices with the same account
  --> Better use IMAP
• People unable to throw away anything because they are unable to
  decide what is important

For the second case you will end up with multi-gigabyte mailboxes on
the server with mails no one need anymore. That's why i would prefer
\Deleted to flag the mail as "not as important anymore" (can be delted
if needed) in spirit of the waste basket on desktops.

...

...

• Skip messages marked as deleted with POP3, maybe even as default?

With this on IMAP access you clearly see what is going on. Also if you delete Messages by IMAP they won't get downloaded by POP3. This would also fit with mail providers doing an "IMAP" expunge/purge on a fixed timebase.

If mail is expunged from IMAP, it also is expunged from POP3 in any case.

Yes, that's the intention after all IMHO. You could set your IMAP
client to expunge on leave or let the mails still around. If the
provider have some policy to say expunge overy month you have no ever
growing mailbox and give the provider a hint what can be deleted. For
archive purpose you still can use IMAP folders.

Regards

Andreas

On 5/3/2013 11:36 AM, Timo Sirainen wrote:

On 3.5.2013, at 19.55, Robert Schetterer <rs@sys4.de> wrote:

...

Am 03.05.2013 18:14, schrieb Timo Sirainen:

...

GMail doesn't delete mails when POP3 client issues a DELE command for it. Instead they just become invisible for future POP3 sessions, but they still exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot:

• Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
• When DELEting a message, add this flag to the message.
• When listing messages, skip all messages that have this flag.

My vote, do it , if it doesnt break anything else, but a wiki help should be offered too.

...

• Also hide this flag from IMAP clients(?)

wouldnt do this, that may end in user confusion

Just about zero IMAP clients make IMAP keywords visible to users, so I'm not really sure which way to go here. I'm more thinking of it as an internal POP3 state which doesn't really need to be visible to IMAP clients, but then again I guess it wouldn't really hurt either.

Be careful about adding features just for the sake of adding features. Dovecot is already a powerful, but quite complex. This would make it even more complex and non-standard. POP and IMAP clients do not expect this behavior from the server. Also, I have never heard anyone request this feature.

Google has already admitted that they do not delete email, even when those emails are deleted from the Trash. Google states that emails may be archived for a minimum of 2 months past when they are deleted - emails, may, however, be saved forever. You do not know. Google has lots of storage and their business is marketing, which means selling people's personal information. Google is large enough that they are under scrutiny and have to obey EU privacy laws. However, adding this feature to Dovecot could put smaller email operators in violation of those laws. Not deleting emails when the client says to, is effectively archiving, and data that is archived is regulated by EU privacy laws. Lots of legal problems have arisen as a result of this.

http://arstechnica.com/tech-policy/2011/12/patriot-act-and-privacy-laws-take...

http://uk.news.yahoo.com/eu-judges-hear-arguments-google-test-privacy-case-1...

http://www.afterdawn.com/news/article.cfm/2012/03/02/google_in_breach_of_eu_...

The problem is already solved by the POP client. Most have a setting that allow emails to remain on the server for a period of time. The default is generally 5 days, but can be set to any value desired. Apple Mail, Thunderbird and outlook all have this feature. Most smartphones also have this feature, including the iPhone, Blackberry and Android devices, and even though the settings are generally not easily accessible, it defaults to being turned on.

So in conclusion, in my experience, this feature is not one that has been requested, it makes Dovecot more complex and behave in ways not expected by the client, and it may have privacy and legal problems. I would recommend not doing it.

I think it would be better to concentrate on features that people ask for. As awful as exchange is, the reason people give me why they use it over a normal email solution is for the groupware features, which standard POP and IMAP email is still lacking.

Dem

On Fri, May 3, 2013 at 3:49 PM, Timo Sirainen <tss@iki.fi> wrote:

I'll add a comment to the setting that enabling it may violate (EU) privacy laws.

Please consider a stronger, broader statement saying that enabling it may violate user's expectations of privacy.

I'm not in a position to cite chapter and voice on this now but folks curious about the practical aspects of these considerations under US law might find why many prefer POP to IMAP by looking at the recommendations by, for example, The Electronic Frontier Foundation (EFF) at < https://ssd.eff.org/>.

On 5/3/2013 7:38 PM, Noel Butler wrote:

Incidentally, the last time I read the pop3 RFC, admittedly some decade or so ago (and yeah it's likely been updated since?) I can not recall there ever being a "MUST" or "SHOULD" when it comes to deleting messages (it might have been deliberately omitted) apart from the server MUST NOT delete messages that are not marked for deletion.

FYI: In addition to the significant privacy and legal concerns, adding and enabling this option makes the server non-compliant. The section under "The UPDATE State" is clear about the behavior of the server:

"The POP3 server removes all messages marked as deleted from the maildrop"

Also note that I am only referring to the public version of the server. Whatever Timo wants to do for a special version for his client is up to him. It is he and his client who are legally responsible for their actions. Similarly, if some third party wants to take the source and modify it for themselves to make it non-compliant, then that is their business. They are legally responsible for their actions and have to answer for those actions if they end up violating any laws. I do not feel that an option like this should be in the public version of the server, however, even if it is off by default.

Dem

RFC 1939 "Post Office Protocol - Version 3"

Under section 5 "The TRANSACTION State", Pages 7-8.

   DELE msg

      Arguments:
          a message-number (required) which may NOT refer to a
          message marked as deleted

      Restrictions:
          may only be given in the TRANSACTION state

      Discussion:
          The POP3 server marks the message as deleted.  Any future
          reference to the message-number associated with the message
          in a POP3 command generates an error.  The POP3 server does
          not actually delete the message until the POP3 session
          enters the UPDATE state.

Under section 6 "The UPDATE State", Page 9.

When the client issues the QUIT command from the TRANSACTION state,
the POP3 session enters the UPDATE state.

   QUIT

      Arguments: none

      Restrictions: none

      Discussion:
          The POP3 server removes all messages marked as deleted
          from the maildrop and replies as to the status of this
          operation.

Timo Sirainen wrote:

Kelsey already mentioned that lazy_expunge plugin has similar issues, and nobody has mentioned privacy concerns related to that.. I guess there should be something about it in the wiki. Also mdbox format doesn't physically delete the data until doveadm purge is run (if ever). So this isn't the first such feature.

Some of our users are using pop3 collectors to fetch (and delete) the mails, while other users are using mobile phones and might want to use this feature.

If the pop3_delete_flag option is globally enabled, will the pop3 user be able to delete any pop3 mails and not be billed for the storage? Our POP3 customers really don't want to pay for emails which they don't see and don't want to keep on the server, but the pop3 collector will not connect via IMAP to the server to finally expunge the mails.

So this rises a quota/storage accounting question for pop3-only users.

Regards Daniel

https://plus.google.com/103021802792276734820

On Sat, 2013-05-04 at 05:29 -0700, Professa Dementia wrote:

On 5/3/2013 7:38 PM, Noel Butler wrote:

...

Incidentally, the last time I read the pop3 RFC, admittedly some decade or so ago (and yeah it's likely been updated since?) I can not recall there ever being a "MUST" or "SHOULD" when it comes to deleting messages (it might have been deliberately omitted) apart from the server MUST NOT delete messages that are not marked for deletion.

FYI: In addition to the significant privacy and legal concerns, adding

Hardly see it as that, I can hardly see ISP's etc using this option, especially since it still counts towards users quota, I'm sure the U.S. Govt would love this feature in there out of control power trips to spy on people, but in all honesty, it is likely to be used mostly, in the business world because your emails in work, are not yours, they are the property of your employers, and that's so in many EU jurisdictions as well.

and enabling this option makes the server non-compliant. The section under "The UPDATE State" is clear about the behavior of the server:

"The POP3 server removes all messages marked as deleted from the maildrop"

Did you read what I actually said? and in your quotation you have even verified it, RFC's are very clear about compliance instructions, by use of the words SHOULD, MUST, MUST NOT ... etc

There is no such word in the RFC relating to deleting marked deleted messages, as you have also even quoted, therefore, Timo's proposed flags do not breach compliancy.

They are legally responsible for their actions and have to answer for those actions if they end up violating any laws. I do not

Of course, it's why it is not being a default action.

feel that an option like this should be in the public version of the server, however, even if it is off by default.

I can not understand why you are so outraged by this, if you feel so strongly, just don't enable the option, or is that too simple...

On Mon, 2013-05-06 at 15:39 +0200, Axel Luttgens wrote:

Hmmm... Let's consider the RFC's part related to, for example, the TOP command:

If the POP3 server issues a positive response, then the response given is multi-line. After the initial +OK, the POP3 server sends the headers of the message, the blank line separating the headers from the body, and then the number of lines of the indicated message's body, being careful to byte-stuff the termination character (as with all multi-line responses).

So, no MUST keyword there. Would this mean that a server sending garbage after a positive response is a compliant one?

Technically? Yes it would (doesn't mean it's right or wrong), RFC are updated, if one disagrees with the wording, one is always welcome to contact the author recommending a change.

On Tue, 2013-05-07 at 09:25 +0200, Axel Luttgens wrote:

Reading RFCs is kind of an art.

That we certainly agree on :)

Let's have a look at RFC 2119:

Authors who follow these guidelines should incorporate this phrase near the beginning of their document:

  The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
  NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and
  "OPTIONAL" in this document are to be interpreted as described in
  RFC 2119.

So, if you want to build your reasoning on those keywords for establishing compliancy or lack thereof, you should restrict yourself to RFCs posterior to RFC 2119 *and* coming with above phrase.

But that leaves earlier RFC's open, once again, to individual interpretation, since there is no specific type of "direction". I'm sure there are plenty of later RFC's that do not include "directions" because they are meant to be a guide. Remember, the POP3 RFC does include a "direction" elsewhere in it (as I previously mentioned), the fact it does not include a "direction" in relation to deletion or marked deleted messages, is open IMHO to be interpreted as being "your choice".

But then, §6 indicates that those keywords are to be used sparingly, not as a general mean to convey compliancy.

Perhaps because it should be up to individual implementers to decide how their software/systems are setup, unless something may be rather detrimental - I fail to see Timo's proposal as detrimental since it is not configured as default.

Ultimately the choice is ours, it is like everything server/network-ish, if we do not want a feature, we do not build it in, or enable the config (file) option to use that feature (kind of why I was disappointed that Timo removed the ability to disable many auth functions like we could do in 1.x series).

It's also like everything else with responsibility to running services, in each of our own countries, laws differ, we need to be aware of those laws (and of any country you host content in) with regards to what can or can not be done, either outright, or with provision (eg: clear statement of data retention in your T&C's or privacy policy etc).

Cheers Noel

On 2013-05-03 15:44, Timo Sirainen wrote:

I didn't ask what their main reason for this was, but for me it would be: "Oops, I accidentally configured my new email client as POP3 instead of IMAP, and now it deleted everything from my INBOX." With lazy_expunge the user would have to explicitly go and undelete the mails, and it would also undelete those mails that were intentionally deleted. With this feature nothing at all would go wrong on IMAP/webmail side.

Ah, that makes more sense now. We're only using lazy_expunge for POP3 but with a namespace visible to IMAP so a POP3 user can restore a message using webmail/IMAP if needed.

-K

On 5/3/2013 3:44 PM, Timo Sirainen wrote:

...

On 4.5.2013, at 1.27, Kelsey Cummings <kgc@corp.sonic.net> wrote:

...

On 2013-05-03 09:14, Timo Sirainen wrote:

...

GMail doesn't delete mails when POP3 client issues a DELE command for it. Instead they just become invisible for future POP3 sessions, but they still exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot:

How does the usage case by your large customer differ from that allowed by the lazy_expunge plugin?

I didn't ask what their main reason for this was, but for me it would be: "Oops, I accidentally configured my new email client as POP3 instead of IMAP, and now it deleted everything from my INBOX." With lazy_expunge the user would have to explicitly go and undelete the mails, and it would also undelete those mails that were intentionally deleted. With this feature nothing at all would go wrong on IMAP/webmail side.

I agree with AJAX. This seems to be a matter of convenience and features versus privacy rights. Do the desires of the mail handling organization outweigh the privacy needs of individuals. This is a long standing argument. Ajax said that a service enabling this feature should make a strong, broad and clear statement about it, and I agree. I'm all in favor of

On 05/03/2013 10:13 PM, Professa Dementia wrote: transparency. And customer service.

I am glad that this was brought up. History is littered with inventions and creations that were designed for one purpose, but misused for another.

It seems this mod was designed to deal with stupid users who are unable to set up their email correctly, and the IT departments who are too lazy to manage the situation properly. I think this attempt to make the software idiot proof will fail, however.

Writing extra code to provide better service is lazy? You're thinking about a corporate environment. That's not the only use case, and usually not the largest deployment size either.

There is a saying the goes something like "You cannot make anything idiot proof because idiots are so ingenious."

We're trying to mitigate problems, not make the world perfect.

If someone is worried about end users setting up POP accidentally and deleting emails, then firewall ports 110 and 995. Simple solution. Problem solved with no inadvertent introduction of privacy and legal violations.

Why firewall only ports 110 and 995? We can also just shut down the entire mail service and this way we have no problems at all.

What worries me, is that as an end user, I now have no idea if this "feature" is turned on or not. When I specify that an email be deleted from the server, I expect that it is *deleted*. I feel that a feature like this is ripe for abuse.

Email is ripe for abuse. I can read every single email on the mail server. That's just wrong. The alternative is to let users manage their own private keys... yeah right. We just agreed that users are stupid, didn't we? Emails are not private from the eyes of anyone accessing the server, that's fundamentally the case, I do not need Timo's help if I want to commit identify theft, or to disobey a user's DELE command. I can mirror / archive every single email in an infinite number of ways, thanks to the beauty of the UNIX philosophy. And if what I wanted was to illegally hold on to user data, as opposed to doing what my users want me to do - which would be the case in this discussion, then the more sensible way to archive everything is at the MTA level. And if I want to do that without telling anyone, who is to stop me? Users implicitly trust me, and I ought to be nice enough and, as Ajax said, transparent. That's very important.

Is there any way for the end user to know that this feature is turned on? What if a hacker got access to the server and changed the value of this setting? As pointed out by AJAX, POP3 comes with an expectation of privacy. There should be some way that the end user gets notified that his deleted POP emails are not actually deleted.

An attacker breaking into the system becomes equal to me in his powers, see above.

If Timo wants to add these features to private copies of the software for specific organizations, that is a matter between him, his client and the law. However, I do not feel it belongs in the mainstream release.

Dem

On Mon, 2013-05-06 at 11:45 -0700, Kelsey Cummings wrote:

While I see the point you're trying to make, I don't think it is valid.
On our servers the deleted message could exist in filesystem snapshots, disk->disk backups and on tape.

That's what most people forget - until they accidentally delete that one email they need :)

Le 4 mai 2013 à 00:44, Timo Sirainen a écrit :

I didn't ask what their main reason for this was, but for me it would be: "Oops, I accidentally configured my new email client as POP3 instead of IMAP, and now it deleted everything from my INBOX." With lazy_expunge the user would have to explicitly go and undelete the mails, and it would also undelete those mails that were intentionally deleted. With this feature nothing at all would go wrong on IMAP/webmail side.

Hello Timo,

I'll sure appear, once again, to be a bit dumb, but I'm somewhat uncomfortable with that "POP3 deletion as a flag" option.

As others already have noticed, this is liable to lead to confusion in the user's mind as well as to privacy concerns.

More specifically, a POP server is supposed to remove the messages marked as deleted when entering its update state. Unless I'm wrong, beside technical problems hindering that deletion, this is a mandatory behavior. Put in other words, if configured with the "POP3 deletion as a flag" option, the POP server won't be compliant anymore and users may feel betrayed. At the very least, this should probably come with some kind of experimental capability issued by the server in its reply to a CAPA command.

On the other hand:

Le 3 mai 2013 à 21:49, Timo Sirainen a écrit :

[...] And this feature was requested by a large customer of ours, so it is needed.. I was mainly wondering about details.

It is true that if an ISP wants such a feature, well...

But then, the problem appears because the ISP allows both POP and IMAP access to mail stores, and POP users should thus be treated as first class citizen too.

Let's suppose that I occasionally use an IMAP client that issues an EXPUNGE after I've inadvertently marked a lot of messages for deletion. Next time I've access to my usual computer with its POP client, I don't find those messages anymore. Wouldn't this need some kind of "IMAP expunge as flag" option as well?

OK, probably splitting hairs... ;-)

Axel

On 4.5.2013, at 13.32, Odhiambo Washington <odhiambo@gmail.com> wrote:

What happens to quotas in places where it is enforced? Where does the undeleted mail go? Does it still count towards the quota?

Good point. It does count towards quota. Worth mentioning also in the setting docs. The mail isn't really "deleted" from the IMAP point of view, it simply gets added that $POP3Deleted flag, so "undelete" is simply removing that flag with IMAP protocol.

Also if wanted this could be combined with cronjob scripts and such to delete old mails, e.g.:

doveadm expunge -A savedsince 1week keyword $POP3Deleted

On 3 May 2013 19:14, Timo Sirainen <tss@iki.fi> wrote:

...

GMail doesn't delete mails when POP3 client issues a DELE command for it. Instead they just become invisible for future POP3 sessions, but they still exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot:

• Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
• When DELEting a message, add this flag to the message.
• When listing messages, skip all messages that have this flag.
• Also hide this flag from IMAP clients(?)

Thoughts? Probably 20 lines of new code.

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 "I can't hear you -- I'm using the scrambler."

On 5/5/2013 11:20 AM, Oscar del Rio wrote:

On 03/05/2013 12:14 PM, Timo Sirainen wrote:

...

GMail doesn't delete mails when POP3 client issues a DELE command for it. Instead they just become invisible for future POP3 sessions, but they still exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot:

• Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
• When DELEting a message, add this flag to the message.
• When listing messages, skip all messages that have this flag.
• Also hide this flag from IMAP clients(?)

Thoughts? Probably 20 lines of new code.

Outlook live.com webmail has a special (virtual) folder called Deleted/POP and the following option:

"If you use POP to download Outlook messages to another program, that program could make it so you can't read your messages on Outlook. (For example, this might happen if you use Mac Mail or Mozilla Thunderbird.) program is set to "delete messages from the server," we'll simply move

• Don't let another program delete messages from Outlook. (If your other

them to a special POP folder. They won't be deleted.)

• Do what my other program says—if it says to delete messages, then delete them. "

This is fine. The user is informed and makes a choice about the behavior of the server. The problem I have with the proposed changes by Timo is that the user is *not* informed and has no choice. The user may expect that the server behaves in a standard manner, but it may not, based on decisions of the system admin. The user has no way of knowing this behavior - only the system admin knows.

Dem

On 05/05/2013 03:29 PM, Timo Sirainen wrote:

On 5.5.2013, at 22.06, Professa Dementia <professa@dementianati.com> wrote:

...

This is fine. The user is informed and makes a choice about the behavior of the server. The problem I have with the proposed changes by Timo is that the user is *not* informed and has no choice. The user may expect that the server behaves in a standard manner, but it may not, based on decisions of the system admin. The user has no way of knowing this behavior - only the system admin knows. Nothing prevents admin from creating a per-user setting in (e.g.) webmail where this behavior is enabled/disabled.

I was just going to say that - dovecot is not a website or a GUI. But, how can this be applied per user? Can you elaborate? This might be a stupid question, I'm sorry :D

On 05/05/2013 03:40 PM, Timo Sirainen wrote:

On 5.5.2013, at 22.32, Gedalya <gedalya@gedalya.net> wrote:

...

On 05/05/2013 03:29 PM, Timo Sirainen wrote:

...

On 5.5.2013, at 22.06, Professa Dementia <professa@dementianati.com> wrote:

...

This is fine. The user is informed and makes a choice about the behavior of the server. The problem I have with the proposed changes by Timo is that the user is *not* informed and has no choice. The user may expect that the server behaves in a standard manner, but it may not, based on decisions of the system admin. The user has no way of knowing this behavior - only the system admin knows. Nothing prevents admin from creating a per-user setting in (e.g.) webmail where this behavior is enabled/disabled.

I was just going to say that - dovecot is not a website or a GUI. But, how can this be applied per user? Can you elaborate? This might be a stupid question, I'm sorry :D http://wiki2.dovecot.org/UserDatabase/ExtraFields

Haha, I'm using extra fields actually, I guess I never properly read that page to understand what it can really do :D Thanks Timo.