[Dovecot] Mails repopping
Michael M Slusarz
slusarz at curecanti.org
Wed Sep 14 01:15:36 EEST 2011
Quoting Simon Brereton <simon.brereton at buongiorno.com>:
>> -----Original Message-----
>> From: dovecot-bounces at dovecot.org [mailto:dovecot-
>> bounces at dovecot.org] On Behalf Of Simon Brereton
>> > -----Original Message-----
>> > From: Timo Sirainen [mailto:tss at iki.fi] On Fri, 2011-09-09 at 13:07
>> > -0400, Simon Brereton wrote:
>> >
>> > > I have a server that's been running Courier for about 6 years and
>> > in
>> > > all that time I think I've only ever had 1 issues where an entire
>> > mail
>> > > box was repopped by a webmail client. However, since moving to a
>> > new
>> > > server and dovecot 4 weeks ago, I've now had the webmail client
>> > repop
>> > > this account 4 times (there are about 230 mails in the account).
>> > >
>> > > Is there a setting I need to tighten to prevent/remedy this? I
>> > have
>> > > no idea if it's happening on other accounts, but this is one that
>> I
>> > > see. The format is maildir. There has been no changes to the
>> > webmail
>> > > client.
>> >
>> > dovecot -n output would have been nice. Also do you see anything in
>> > error logs?
>>
>> Ah. My apologies of course. Here it is..
>>
>> mail:~# dovecot -n
>> # 1.2.15: /etc/dovecot/dovecot.conf
>> # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.2 ext3
>> log_timestamp: %Y-%m-%d %H:%M:%S
>> protocols: imap imaps pop3 pop3s
>> ssl_ca_file: /etc/ssl/keys/rhodes-ca.crt
>> ssl_cert_file: /etc/ssl/keys/mail.domain.net.crt
>> ssl_key_file: /etc/ssl/private/mail.domain.net.key
>> disable_plaintext_auth: no
>> login_dir: /var/run/dovecot/login
>> login_executable(default): /usr/lib/dovecot/imap-login
>> login_executable(imap): /usr/lib/dovecot/imap-login
>> login_executable(pop3): /usr/lib/dovecot/pop3-login
>> mail_privileged_group: mailsystem
>> mail_location: maildir:/var/spool/mail/virtual/%d/%n
>> maildir_very_dirty_syncs: yes
>> mbox_write_locks: fcntl dotlock
>> mail_executable(default): /usr/lib/dovecot/imap
>> mail_executable(imap): /usr/lib/dovecot/imap
>> mail_executable(pop3): /usr/lib/dovecot/pop3
>> mail_plugins(default): quota imap_quota
>> mail_plugins(imap): quota imap_quota
>> mail_plugins(pop3): quota
>> mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
>> mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
>> mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
>> imap_client_workarounds(default): outlook-idle delay-newmail
>> imap_client_workarounds(imap): outlook-idle delay-newmail
>> imap_client_workarounds(pop3):
>> pop3_client_workarounds(default):
>> pop3_client_workarounds(imap):
>> pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
>> lda:
>> postmaster_address: postmaster at domain.net
>> mail_plugins: quota
>> log_path:
>> info_log_path:
>> deliver_log_format: msgid=%m: %f: %$
>> auth default:
>> mechanisms: plain login
>> user: mailsystem
>> verbose: yes
>> passdb:
>> driver: sql
>> args: /etc/dovecot/dovecot-sql.conf
>> userdb:
>> driver: prefetch
>> userdb:
>> driver: static
>> args: uid=999 gid=115 home=/var/spool/mail/virtual/%d/%n
>> allow_all_users=yes
>> socket:
>> type: listen
>> client:
>> path: /var/spool/postfix/private/auth
>> mode: 432
>> user: postfix
>> group: mailsystem
>> master:
>> path: /var/run/dovecot/auth-master
>> mode: 432
>> user: mailsystem
>> group: mailsystem
>> plugin:
>> quota: maildir
>>
>> Could you make dovecot -n munge the certificate and postmaster email
>> addresses? I'm not comfortable with that floating on the internet..
>>
>> The only thing I have in the logs is 2 sessions where mail was popped
>> (note, it doesn't even add up to the 183 messages in the mail box).
>> But those sessions are vastly longer than the regular ones (tens of
>> minutes compared to a few seconds). Since both IPs are on the back-
>> bone, that's quite a while to download 100 mails (none of which are
>> over
>>
>> Sep 11 21:36:25 mail dovecot: pop3-login: Login:
>> user=<user at domain.com>, method=PLAIN, rip=64.88.168.84,
>> lip=83.170.65.xxx, TLS Sep 11 21:36:34 mail dovecot:
>> POP3(user at domain.com): Disconnected: Logged out top=0/0, retr=0/0,
>> del=0/183, size=14025971 Sep 11 21:43:44 mail dovecot: pop3-login:
>> Login: user=<user at domain.com>, method=PLAIN, rip=64.88.168.84,
>> lip=83.170.65.xxx, TLS Sep 11 21:44:54 mail dovecot:
>> POP3(user at domain.com): Disconnected: Logged out top=0/0, retr=0/0,
>> del=0/183, size=14025971 Sep 11 21:52:31 mail dovecot: pop3-login:
>> Login: user=<user at domain.com>, method=PLAIN, rip=64.88.168.84,
>> lip=83.170.65.xxx, TLS Sep 11 22:56:01 mail dovecot:
>> POP3(user at domain.com): Disconnected: Logged out top=0/0,
>> retr=100/9182678, del=0/183, size=14025971 Sep 11 23:08:58 mail
>> dovecot: pop3-login: Login: user=<user at domain.com>, method=PLAIN,
>> rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 11 23:37:57 mail
>> dovecot: POP3(user at domain.com): Disconnected: Logged out top=0/0,
>> retr=75/4748674, del=0/183, size=14025971 Sep 12 00:04:11 mail
>> dovecot: pop3-login: Login: user=<user at domain.com>, method=PLAIN,
>> rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 12 00:04:26 mail
>> dovecot: POP3(user at domain.com): Disconnected: Logged out top=0/0,
>> retr=0/0, del=0/183, size=14025971 Sep 12 00:07:40 mail dovecot:
>> pop3-login: Login: user=<user at domain.com>, method=PLAIN,
>> rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 12 00:07:53 mail
>> dovecot: POP3(user at domain.com): Disconnected: Logged out top=0/0,
>> retr=0/0, del=0/183, size=14025971
>>
>>
>> > If you're using the default pop3_uidl_format it'll rely on IMAP
>> UIDs
>> > to stay the same, and I guess it's possible that due to some other
>> > problem they change (that should be logged as an error/warning
>> > though).
>> >
>> > You could try setting pop3_uidl_format=%f, but it will cause
>> everyone
>> > to redownload mails. With newer Dovecot versions you could set
>> > pop3_save_uidl=yes and when you think everyone's downloaded mails
>> once
>> > you can safely change the pop3_uidl_format.
>>
>> Sorry, I'm very new to dovecot and I'm not sure I understand. I
>> presume because neither of those keys are in the dovecot -n output
>> that they are as the defaults, yes? The account is indeed accessed
>> by IMAP as well (from a mobile device mostly), but I don't see
>> anything fishy there either. How could I see if the IMAP UIDs have
>> changed?
>>
>> Sep 11 21:20:32 mail dovecot: IMAP(user at domain.com): Connection
>> closed bytes=1095/8292
>>
>> Sep 11 21:26:03 mail dovecot: imap-login: Login:
>> user=<user at domain.com>, method=PLAIN, rip=174.252.83.244,
>> lip=83.170.65.xxx, TLS Sep 11 22:11:20 mail dovecot:
>> IMAP(user at domain.com): Disconnected for inactivity bytes=725/5638 Sep
>> 11 22:17:10 mail dovecot: imap-login: Login: user=<user at domain.com>,
>> method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 11
>> 23:12:06 mail dovecot: IMAP(user at domain.com): Disconnected for
>> inactivity bytes=1471/11025 Sep 11 23:23:22 mail dovecot: imap-login:
>> Login: user=<user at domain.com>, method=PLAIN, rip=174.252.83.244,
>> lip=83.170.65.xxx, TLS Sep 11 23:52:52 mail dovecot:
>> IMAP(user at domain.com): Connection closed bytes=1841/13679 Sep 12
>> 00:08:47 mail dovecot: imap-login: Login: user=<user at domain.com>,
>> method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 12
>> 01:19:05 mail dovecot: imap-login: Login: user=<user at domain.com>,
>> method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 12
>> 02:57:01 mail dovecot: IMAP(user at domain.com): Connection closed
>> bytes=2713/60026 Sep 12 02:57:01 mail dovecot: IMAP(user at domain.com):
>> Connection closed bytes=2688/18635
>>
>>
>> There are no errors or warnings in the mail log (I have one shared
>> log file for postfix, amavis and dovecot). Reading the notes for
>> pop3_save_uidl it doesn't seem to be a dangerous option - should I
>> turn that on? Why will it force everyone to redownload mails
>> (there's nothing about it on the wiki)?
>>
>> Thanks!
>>
>> Simon
>
> Any help would be appreciated.
What do you mean by "repopped"? You mean downloading the entire data
of the messages from the POP3 server? This is expected behavior when
using a stateless (e.g. webmail) client. Kind of the whole reason you
don't use POP3 in the first place.
Although caching can help. e.g., Here's what the first connection to
the server looks like (this is using IMP 5 on a mailstore with 82
messages):
S (1315951197.4976): +OK Dovecot ready.
C (1315951197.513): [AUTH PLAIN Command - username: slusarz]
S (1315951197.5319): +OK Logged in.
C (1315951197.5325): STAT
S (1315951197.5328): +OK 82 482351
C (1315951197.5348): UIDL
S (1315951197.5354): +OK
S (1315951197.5354): 1 000000014935d409
S (1315951197.5354): 2 000000024935d409
S (1315951197.5354): 3 000000114935d409
[...]
S (1315951197.5363): 82 000000824935d409
S (1315951197.5363): .
C (1315951197.9582): TOP 1 0
S (1315951198.0411): From user at domain.com Thu Jun 22 11:16:26 2006
[...]
S (1315951198.0416): .
[...]
C (1315951199.0607): LIST
S (1315951199.061): +OK 82 messages:
S (1315951199.061): 1 118630
[...]
S (1315951199.0619): .
We need to grab all headers so we can build the envelope information
(needed to produce the mailbox listing). And the LIST command grabs
the size information (also used in the mailbox listing).
But remember that the full headers will need to be redownloaded
*EVERY* time you reload the page unless some sort of caching is
enabled in the client. That's just the nature of POP3. (IMAP has the
same sort of issues - if the stateless client does not cache, the
envelope information must be downloaded on every access. However,
with IMAP, the network traffic is reduced - you can download only the
needed information, not all header text - and IMAP servers have the
ability to cache this information behind the scenes due to the
abstraction of the API.).
This is where caching is pretty much essential on the webmail side.
If caching is enabled, the best-case scenario is that the the webmail
server only needs to grab the list of UIDLs on every POP3 server
access going forward - if the UIDL list has not changed, we know the
mailbox hasn't changed and the cached information is still valid.
(CONDSTORE/QRESYNC extensions for IMAP make this synchronization check
even more efficient in IMAP)
michael
More information about the dovecot
mailing list