-----Original Message----- From: dovecot-bounces@dovecot.org [mailto:dovecot- bounces@dovecot.org] On Behalf Of Michael M Slusarz Quoting Simon Brereton <simon.brereton@buongiorno.com>:

...

...

-----Original Message----- From: dovecot-bounces@dovecot.org [mailto:dovecot- bounces@dovecot.org] On Behalf Of Simon Brereton

...

-----Original Message----- From: Timo Sirainen [mailto:tss@iki.fi] On Fri, 2011-09-09 at 13:07 -0400, Simon Brereton wrote:

...

I have a server that's been running Courier for about 6 years and in all that time I think I've only ever had 1 issues where an entire mail box was repopped by a webmail client. However, since moving to a new server and dovecot 4 weeks ago, I've now had the webmail client repop this account 4 times (there are about 230 mails in the account).

Is there a setting I need to tighten to prevent/remedy this? I have no idea if it's happening on other accounts, but this is one that I see. The format is maildir. There has been no changes to the webmail client.

dovecot -n output would have been nice. Also do you see anything in error logs?

Ah. My apologies of course. Here it is..

mail:~# dovecot -n

1.2.15: /etc/dovecot/dovecot.conf

OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.2 ext3

log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s ssl_ca_file: /etc/ssl/keys/rhodes-ca.crt ssl_cert_file: /etc/ssl/keys/mail.domain.net.crt ssl_key_file: /etc/ssl/private/mail.domain.net.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login mail_privileged_group: mailsystem mail_location: maildir:/var/spool/mail/virtual/%d/%n maildir_very_dirty_syncs: yes mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh lda: postmaster_address: postmaster@domain.net mail_plugins: quota log_path: info_log_path: deliver_log_format: msgid=%m: %f: %$ auth default: mechanisms: plain login user: mailsystem verbose: yes passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: prefetch userdb: driver: static args: uid=999 gid=115 home=/var/spool/mail/virtual/%d/%n allow_all_users=yes socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: mailsystem master: path: /var/run/dovecot/auth-master mode: 432 user: mailsystem group: mailsystem plugin: quota: maildir

Could you make dovecot -n munge the certificate and postmaster email addresses? I'm not comfortable with that floating on the internet..

The only thing I have in the logs is 2 sessions where mail was popped (note, it doesn't even add up to the 183 messages in the mail box). But those sessions are vastly longer than the regular ones (tens of minutes compared to a few seconds). Since both IPs are on the back- bone, that's quite a while to download 100 mails (none of which are over

Sep 11 21:36:25 mail dovecot: pop3-login: Login: user=<user@domain.com>, method=PLAIN, rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 11 21:36:34 mail dovecot: POP3(user@domain.com): Disconnected: Logged out top=0/0, retr=0/0, del=0/183, size=14025971 Sep 11 21:43:44 mail dovecot: pop3-login: Login: user=<user@domain.com>, method=PLAIN, rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 11 21:44:54 mail dovecot: POP3(user@domain.com): Disconnected: Logged out top=0/0, retr=0/0, del=0/183, size=14025971 Sep 11 21:52:31 mail dovecot: pop3-login: Login: user=<user@domain.com>, method=PLAIN, rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 11 22:56:01 mail dovecot: POP3(user@domain.com): Disconnected: Logged out top=0/0, retr=100/9182678, del=0/183, size=14025971 Sep 11 23:08:58 mail dovecot: pop3-login: Login: user=<user@domain.com>, method=PLAIN, rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 11 23:37:57 mail dovecot: POP3(user@domain.com): Disconnected: Logged out top=0/0, retr=75/4748674, del=0/183, size=14025971 Sep 12 00:04:11 mail dovecot: pop3-login: Login: user=<user@domain.com>, method=PLAIN, rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 12 00:04:26 mail dovecot: POP3(user@domain.com): Disconnected: Logged out top=0/0, retr=0/0, del=0/183, size=14025971 Sep 12 00:07:40 mail dovecot: pop3-login: Login: user=<user@domain.com>, method=PLAIN, rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 12 00:07:53 mail dovecot: POP3(user@domain.com): Disconnected: Logged out top=0/0, retr=0/0, del=0/183, size=14025971

...

If you're using the default pop3_uidl_format it'll rely on IMAP UIDs to stay the same, and I guess it's possible that due to some other problem they change (that should be logged as an error/warning though).

You could try setting pop3_uidl_format=%f, but it will cause everyone to redownload mails. With newer Dovecot versions you could set pop3_save_uidl=yes and when you think everyone's downloaded mails once you can safely change the pop3_uidl_format.

Sorry, I'm very new to dovecot and I'm not sure I understand. I presume because neither of those keys are in the dovecot -n output that they are as the defaults, yes? The account is indeed accessed by IMAP as well (from a mobile device mostly), but I don't see anything fishy there either. How could I see if the IMAP UIDs have changed?

Sep 11 21:20:32 mail dovecot: IMAP(user@domain.com): Connection closed bytes=1095/8292

Sep 11 21:26:03 mail dovecot: imap-login: Login: user=<user@domain.com>, method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 11 22:11:20 mail dovecot: IMAP(user@domain.com): Disconnected for inactivity bytes=725/5638 Sep 11 22:17:10 mail dovecot: imap-login: Login: user=<user@domain.com>, method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 11 23:12:06 mail dovecot: IMAP(user@domain.com): Disconnected for inactivity bytes=1471/11025 Sep 11 23:23:22 mail dovecot: imap- login: Login: user=<user@domain.com>, method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 11 23:52:52 mail dovecot: IMAP(user@domain.com): Connection closed bytes=1841/13679 Sep 12 00:08:47 mail dovecot: imap-login: Login: user=<user@domain.com>, method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 12 01:19:05 mail dovecot: imap-login: Login: user=<user@domain.com>, method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 12 02:57:01 mail dovecot: IMAP(user@domain.com): Connection closed bytes=2713/60026 Sep 12 02:57:01 mail dovecot: IMAP(user@domain.com): Connection closed bytes=2688/18635

There are no errors or warnings in the mail log (I have one shared log file for postfix, amavis and dovecot). Reading the notes for pop3_save_uidl it doesn't seem to be a dangerous option - should I turn that on? Why will it force everyone to redownload mails (there's nothing about it on the wiki)?

Thanks!

Simon

Any help would be appreciated.

What do you mean by "repopped"? You mean downloading the entire data of the messages from the POP3 server? This is expected behavior when using a stateless (e.g. webmail) client. Kind of the whole reason you don't use POP3 in the first place.

Michael - I use a spam filtering service, that uses Horde as the web-front end. Essentially, it pops all my mail accounts (that allow popping) one of which is the one I control and is now running Dovecot - though was previously running Courier. Until now, mails that the service has popped once have never been repopped. That is, I assume that when Horde does a RETR on the account it knows what it has already popped and what it new and only retrieves the new mails. Right now though, it's redownloaded them all 5 or 6 times in 4 weeks.

I don't think this is a Horde issue (since that hasn't changed), which is why I didn't post there. Horde continues to be a fantastic project.

From my limited knowledge (meaning I didn't understand the rest of your mail :) I suspect that Dovecot is doing something with the IDs that Courier wasn't doing and that's causing Horde to see those old mails as new every now and again.

Simon

Although caching can help. e.g., Here's what the first connection to the server looks like (this is using IMP 5 on a mailstore with 82 messages):

S (1315951197.4976): +OK Dovecot ready. C (1315951197.513): [AUTH PLAIN Command - username: slusarz] S (1315951197.5319): +OK Logged in. C (1315951197.5325): STAT S (1315951197.5328): +OK 82 482351 C (1315951197.5348): UIDL S (1315951197.5354): +OK S (1315951197.5354): 1 000000014935d409 S (1315951197.5354): 2 000000024935d409 S (1315951197.5354): 3 000000114935d409 [...] S (1315951197.5363): 82 000000824935d409 S (1315951197.5363): . C (1315951197.9582): TOP 1 0 S (1315951198.0411): From user@domain.com Thu Jun 22 11:16:26 2006 [...] S (1315951198.0416): . [...] C (1315951199.0607): LIST S (1315951199.061): +OK 82 messages: S (1315951199.061): 1 118630 [...] S (1315951199.0619): .

We need to grab all headers so we can build the envelope information (needed to produce the mailbox listing). And the LIST command grabs the size information (also used in the mailbox listing).

But remember that the full headers will need to be redownloaded *EVERY* time you reload the page unless some sort of caching is enabled in the client. That's just the nature of POP3. (IMAP has the same sort of issues - if the stateless client does not cache, the envelope information must be downloaded on every access. However, with IMAP, the network traffic is reduced - you can download only the needed information, not all header text - and IMAP servers have the ability to cache this information behind the scenes due to the abstraction of the API.).

This is where caching is pretty much essential on the webmail side. If caching is enabled, the best-case scenario is that the the webmail server only needs to grab the list of UIDLs on every POP3 server access going forward - if the UIDL list has not changed, we know the mailbox hasn't changed and the cached information is still valid. (CONDSTORE/QRESYNC extensions for IMAP make this synchronization check even more efficient in IMAP)

michael