[Dovecot] Mails repopping

[Simon Brereton]

> -----Original Message-----
> From: dovecot-bounces at dovecot.org [mailto:dovecot-
> bounces at dovecot.org] On Behalf Of Michael M Slusarz
> Quoting Simon Brereton <simon.brereton at buongiorno.com>:
> 
> >> -----Original Message-----
> >> From: dovecot-bounces at dovecot.org [mailto:dovecot-
> >> bounces at dovecot.org] On Behalf Of Simon Brereton
> >> > -----Original Message-----
> >> > From: Timo Sirainen [mailto:tss at iki.fi] On Fri, 2011-09-09 at
> 13:07
> >> > -0400, Simon Brereton wrote:
> >> >
> >> > > I have a server that's been running Courier for about 6 years
> and
> >> > in
> >> > > all that time I think I've only ever had 1 issues where an
> entire
> >> > mail
> >> > > box was repopped by a webmail client.  However, since moving
> to a
> >> > new
> >> > > server and dovecot 4 weeks ago, I've now had the webmail
> client
> >> > repop
> >> > > this account 4 times (there are about 230 mails in the
> account).
> >> > >
> >> > > Is there a setting I need to tighten to prevent/remedy this?
> I
> >> > have
> >> > > no idea if it's happening on other accounts, but this is one
> that
> >> I
> >> > > see.  The format is maildir.  There has been no changes to the
> >> > webmail
> >> > > client.
> >> >
> >> > dovecot -n output would have been nice. Also do you see anything
> in
> >> > error logs?
> >>
> >> Ah.  My apologies of course.  Here it is..
> >>
> >> mail:~# dovecot -n
> >> # 1.2.15: /etc/dovecot/dovecot.conf
> >> # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.2 ext3
> >> log_timestamp: %Y-%m-%d %H:%M:%S
> >> protocols: imap imaps pop3 pop3s
> >> ssl_ca_file: /etc/ssl/keys/rhodes-ca.crt
> >> ssl_cert_file: /etc/ssl/keys/mail.domain.net.crt
> >> ssl_key_file: /etc/ssl/private/mail.domain.net.key
> >> disable_plaintext_auth: no
> >> login_dir: /var/run/dovecot/login
> >> login_executable(default): /usr/lib/dovecot/imap-login
> >> login_executable(imap): /usr/lib/dovecot/imap-login
> >> login_executable(pop3): /usr/lib/dovecot/pop3-login
> >> mail_privileged_group: mailsystem
> >> mail_location: maildir:/var/spool/mail/virtual/%d/%n
> >> maildir_very_dirty_syncs: yes
> >> mbox_write_locks: fcntl dotlock
> >> mail_executable(default): /usr/lib/dovecot/imap
> >> mail_executable(imap): /usr/lib/dovecot/imap
> >> mail_executable(pop3): /usr/lib/dovecot/pop3
> >> mail_plugins(default): quota imap_quota
> >> mail_plugins(imap): quota imap_quota
> >> mail_plugins(pop3): quota
> >> mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
> >> mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
> >> mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
> >> imap_client_workarounds(default): outlook-idle delay-newmail
> >> imap_client_workarounds(imap): outlook-idle delay-newmail
> >> imap_client_workarounds(pop3):
> >> pop3_client_workarounds(default):
> >> pop3_client_workarounds(imap):
> >> pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
> >> lda:
> >>   postmaster_address: postmaster at domain.net
> >>   mail_plugins: quota
> >>   log_path:
> >>   info_log_path:
> >>   deliver_log_format: msgid=%m: %f: %$ auth default:
> >>   mechanisms: plain login
> >>   user: mailsystem
> >>   verbose: yes
> >>   passdb:
> >>     driver: sql
> >>     args: /etc/dovecot/dovecot-sql.conf
> >>   userdb:
> >>     driver: prefetch
> >>   userdb:
> >>     driver: static
> >>     args: uid=999 gid=115 home=/var/spool/mail/virtual/%d/%n
> >> allow_all_users=yes
> >>   socket:
> >>     type: listen
> >>     client:
> >>       path: /var/spool/postfix/private/auth
> >>       mode: 432
> >>       user: postfix
> >>       group: mailsystem
> >>     master:
> >>       path: /var/run/dovecot/auth-master
> >>       mode: 432
> >>       user: mailsystem
> >>       group: mailsystem
> >> plugin:
> >>   quota: maildir
> >>
> >> Could you make dovecot -n munge the certificate and postmaster
> email
> >> addresses?  I'm not comfortable with that floating on the
> internet..
> >>
> >> The only thing I have in the logs is 2 sessions where mail was
> popped
> >> (note, it doesn't even add up to the 183 messages in the mail
> box).
> >> But those sessions are vastly longer than the regular ones (tens
> of
> >> minutes compared to a few seconds).  Since both IPs are on the
> back-
> >> bone, that's quite a while to download 100 mails (none of which
> are
> >> over
> >>
> >> Sep 11 21:36:25 mail dovecot: pop3-login: Login:
> >> user=<user at domain.com>, method=PLAIN, rip=64.88.168.84,
> >> lip=83.170.65.xxx, TLS Sep 11 21:36:34 mail dovecot:
> >> POP3(user at domain.com): Disconnected: Logged out top=0/0, retr=0/0,
> >> del=0/183, size=14025971 Sep 11 21:43:44 mail dovecot: pop3-login:
> >> Login: user=<user at domain.com>, method=PLAIN, rip=64.88.168.84,
> >> lip=83.170.65.xxx, TLS Sep 11 21:44:54 mail dovecot:
> >> POP3(user at domain.com): Disconnected: Logged out top=0/0, retr=0/0,
> >> del=0/183, size=14025971 Sep 11 21:52:31 mail dovecot: pop3-login:
> >> Login: user=<user at domain.com>, method=PLAIN, rip=64.88.168.84,
> >> lip=83.170.65.xxx, TLS Sep 11 22:56:01 mail dovecot:
> >> POP3(user at domain.com): Disconnected: Logged out top=0/0,
> >> retr=100/9182678, del=0/183, size=14025971 Sep 11 23:08:58 mail
> >> dovecot: pop3-login: Login: user=<user at domain.com>, method=PLAIN,
> >> rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 11 23:37:57 mail
> >> dovecot: POP3(user at domain.com): Disconnected: Logged out top=0/0,
> >> retr=75/4748674, del=0/183, size=14025971 Sep 12 00:04:11 mail
> >> dovecot: pop3-login: Login: user=<user at domain.com>, method=PLAIN,
> >> rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 12 00:04:26 mail
> >> dovecot: POP3(user at domain.com): Disconnected: Logged out top=0/0,
> >> retr=0/0, del=0/183, size=14025971 Sep 12 00:07:40 mail dovecot:
> >> pop3-login: Login: user=<user at domain.com>, method=PLAIN,
> >> rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 12 00:07:53 mail
> >> dovecot: POP3(user at domain.com): Disconnected: Logged out top=0/0,
> >> retr=0/0, del=0/183, size=14025971
> >>
> >>
> >> > If you're using the default pop3_uidl_format it'll rely on IMAP
> >> UIDs
> >> > to stay the same, and I guess it's possible that due to some
> other
> >> > problem they change (that should be logged as an error/warning
> >> > though).
> >> >
> >> > You could try setting pop3_uidl_format=%f, but it will cause
> >> everyone
> >> > to redownload mails. With newer Dovecot versions you could set
> >> > pop3_save_uidl=yes and when you think everyone's downloaded
> mails
> >> once
> >> > you can safely change the pop3_uidl_format.
> >>
> >> Sorry, I'm very new to dovecot and I'm not sure I understand.  I
> >> presume because neither of those keys are in the dovecot -n output
> >> that they are as the defaults, yes?  The account is indeed
> accessed
> >> by IMAP as well (from a mobile device mostly), but I don't see
> >> anything fishy there either.  How could I see if the IMAP UIDs
> have
> >> changed?
> >>
> >> Sep 11 21:20:32 mail dovecot: IMAP(user at domain.com): Connection
> >> closed bytes=1095/8292
> >>
> >> Sep 11 21:26:03 mail dovecot: imap-login: Login:
> >> user=<user at domain.com>, method=PLAIN, rip=174.252.83.244,
> >> lip=83.170.65.xxx, TLS Sep 11 22:11:20 mail dovecot:
> >> IMAP(user at domain.com): Disconnected for inactivity bytes=725/5638
> Sep
> >> 11 22:17:10 mail dovecot: imap-login: Login:
> user=<user at domain.com>,
> >> method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 11
> >> 23:12:06 mail dovecot: IMAP(user at domain.com): Disconnected for
> >> inactivity bytes=1471/11025 Sep 11 23:23:22 mail dovecot: imap-
> login:
> >> Login: user=<user at domain.com>, method=PLAIN, rip=174.252.83.244,
> >> lip=83.170.65.xxx, TLS Sep 11 23:52:52 mail dovecot:
> >> IMAP(user at domain.com): Connection closed bytes=1841/13679 Sep 12
> >> 00:08:47 mail dovecot: imap-login: Login: user=<user at domain.com>,
> >> method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 12
> >> 01:19:05 mail dovecot: imap-login: Login: user=<user at domain.com>,
> >> method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 12
> >> 02:57:01 mail dovecot: IMAP(user at domain.com): Connection closed
> >> bytes=2713/60026 Sep 12 02:57:01 mail dovecot:
> IMAP(user at domain.com):
> >> Connection closed bytes=2688/18635
> >>
> >>
> >> There are no errors or warnings in the mail log (I have one shared
> >> log file for postfix, amavis and dovecot).  Reading the notes for
> >> pop3_save_uidl it doesn't seem to be a dangerous option  - should
> I
> >> turn that on?  Why will it force everyone to redownload mails
> >> (there's nothing about it on the wiki)?
> >>
> >> Thanks!
> >>
> >> Simon
> >
> > Any help would be appreciated.
> 
> What do you mean by "repopped"?  You mean downloading the entire data
> of the messages from the POP3 server?  This is expected behavior when
> using a stateless (e.g. webmail) client.  Kind of the whole reason
> you don't use POP3 in the first place.

Michael - I use a spam filtering service, that uses Horde as the web-front end.  Essentially, it pops all my mail accounts (that allow popping) one of which is the one I control and is now running Dovecot - though was previously running Courier.  Until now, mails that the service has popped once have never been repopped.  That is, I assume that when Horde does a RETR on the account it knows what it has already popped and what it new and only retrieves the new mails.  Right now though, it's redownloaded them all 5 or 6 times in 4 weeks.

I don't think this is a Horde issue (since that hasn't changed), which is why I didn't post there.  Horde continues to be a fantastic project.

>From my limited knowledge (meaning I didn't understand the rest of your mail :) I suspect that Dovecot is doing something with the IDs that Courier wasn't doing and that's causing Horde to see those old mails as new every now and again.

Simon



> Although caching can help.  e.g., Here's what the first connection to
> the server looks like (this is using IMP 5 on a mailstore with 82
> messages):
> 
> S (1315951197.4976): +OK Dovecot ready.
> C (1315951197.513): [AUTH PLAIN Command - username: slusarz] S
> (1315951197.5319): +OK Logged in.
> C (1315951197.5325): STAT
> S (1315951197.5328): +OK 82 482351
> C (1315951197.5348): UIDL
> S (1315951197.5354): +OK
> S (1315951197.5354): 1 000000014935d409
> S (1315951197.5354): 2 000000024935d409
> S (1315951197.5354): 3 000000114935d409
> [...]
> S (1315951197.5363): 82 000000824935d409 S (1315951197.5363): .
> C (1315951197.9582): TOP 1 0
> S (1315951198.0411): From user at domain.com  Thu Jun 22 11:16:26 2006
> [...] S (1315951198.0416): .
> [...]
> C (1315951199.0607): LIST
> S (1315951199.061): +OK 82 messages:
> S (1315951199.061): 1 118630
> [...]
> S (1315951199.0619): .
> 
> We need to grab all headers so we can build the envelope information
> (needed to produce the mailbox listing).  And the LIST command grabs
> the size information (also used in the mailbox listing).
> 
> But remember that the full headers will need to be redownloaded
> *EVERY* time you reload the page unless some sort of caching is
> enabled in the client.  That's just the nature of POP3.  (IMAP has
> the same sort of issues - if the stateless client does not cache, the
> envelope information must be downloaded on every access.  However,
> with IMAP, the network traffic is reduced - you can download only the
> needed information, not all header text - and IMAP servers have the
> ability to cache this information behind the scenes due to the
> abstraction of the API.).
> 
> This is where caching is pretty much essential on the webmail side.
> If caching is enabled, the best-case scenario is that the the webmail
> server only needs to grab the list of UIDLs on every POP3 server
> access going forward - if the UIDL list has not changed, we know the
> mailbox hasn't changed and the cached information is still valid.
> (CONDSTORE/QRESYNC extensions for IMAP make this synchronization
> check even more efficient in IMAP)
> 
> michael