Dovecot/K9/Thunderbird IMAP interaction issues: replicated messages, constant reloading (TB) & missing new messages (K-9)
Version Info: FreeBSD 9.2-RELEASE #0 r266046 AMD64, Dovecot 2.2.13, dovecot-pigeonhole-0.4.3, amavisd-new-2.8.0_2,1, clamav-0.98.4, postfix29-2.9.9_1,1, postfixadmin-2.3.6, spamassassin-3.4.0_12 (zpool data and output of dovecot -n at bottom of post)
I suspect these are related problems and further that they are related to some trivial newb error I've made in my configuration, but searching descriptions of the symptoms has not yet guided me to a solution.
I recently migrated from Courier to Dovecot on new hardware and imap synced a 16GB mailstore (du -hs) divided up into annual directories with about 15,000 messages each. I mention the size because it seems an errant Remove Duplicate Messages (Alternate) 0.3.9 acting on what seem to be false duplicates in Thunderbird reduced the inbox size from 7500-ish messages to 1300 (DOH!) and may have solved my K-9 problem described below. My old Courier server still has the vast majority of these messages and I will resync someday and then try to file them away more expediently than my previously somewhat lazy management style to keep the inbox small.
In thunderbird, I get constant downloading of messages. Every time I switch to the inbox from another folder, the messages start redownloading. This sometimes, but not always, seems to create duplicate messages, but I'm now fairly sure they are only created in the client and not on the server. Searching through various forums, it appears this has been a problem for Thunderbird for some time, but it wasn't for me on Courier. As posts about similar errors seem to have petered out over the last year without any clear guidance on how to fix it that I could find, I expect there's a simple fix I just haven't found yet, either in TB configuration or in Dovecot. Any hints? I tried Thunderfix to remove the .msf files and will be redownloading my mail over the thin pipe out to Iraq for the next 6 months (DOH!). One artifact is that when I restart thunderbird, my inbox is reported as having about 130 messages in it, then thunderbird starts redownloading headers and eventually gets back up to the 1297 currently listed both in the TB pane and by ls | wc -l in /cur. Is it possible there is a time out that needs to be extended? In TB, I've set (among other less apparently relevant options) mailnews.tcptimeout;1000 mail.db.idle_limit;30000000
Also, with Thunderbird, I get duplication of messages in the client. I suspect this is related to the previous redownload error, but there is some indication in some posts that it is potentially related to a dirty flag. I've had this problem before with Thunderbird and Courier, but it went away after a TB update. It seems to have reappeared with Dovecot. The TB client hasn't been updated during the transition, so some interaction issue seems more likely than a pure TB bug. Likely TB's fault as this is a well known, but poorly diagnosed TB issue (deleting duplicates regularly seems to be the solution, but my inbox would double in size every few hours on the LAN. Now that I'm remote over a 2mbps link, it has slowed way down: bonus!).
"Expunged message reappeared, giving a new UID" errors appear a lot. This might well be related to the last error. This seems to happen with TB client connections, but I'm not sure what action triggers it. I am not replicating. I tried renaming 10-director.conf to 10-director.conf.disabled and got no complaints from Dovecot (as I'm not using replication, it shouldn't) but that indicates that this Expunged message reappeared error is probably of a different source than, for example, Simon Fraser's debug effort.
K9 was only showing a subset of my messages and generally not the new ones. They'd appear in the alert properly, but weren't visible in the client and sometimes they'd be visible and then disappear later. I imagined there could a correlation between TB checking mail and perhaps moving messages from /new to /cur and K9 losing track, but the potentially tragic "duplicate" removal may have reduced my inbox to a size that k9 can handle as K9 inbox display seems normal now as far as I can tell. When I resync (assuming I can), I sort of dread this problem reappearing. K9 worked as expected against Courier with the same inbox.
These apparently difficult to diagnose problems are occasionally suggested as related to namespace configuration issues and thunderbird is reported to have issues with some separator options. I vaguely suspect this, but I'm pretty sure my namespace config is default and should be well tested. I haven't, for example, explicitly stated the default "/" but left it commented out. I increased the connection limit count per user*IP to 20 as TB seems to be parallel downloading a lot of folders. This has neither improved nor degraded performance in any way I can see and is probably irrelevant.
Any hints greatly appreciated.
Filesystem/Zpool data: NAME PROPERTY VALUE SOURCE zroot size 400G - zroot capacity 28% - zroot altroot - default zroot health ONLINE - zroot guid 1061309925223582150 default zroot version - default zroot bootfs zroot local zroot delegation on default zroot autoreplace off default zroot cachefile - default zroot failmode wait default zroot listsnapshots off default zroot autoexpand off default zroot dedupditto 0 default zroot dedupratio 1.10x - zroot free 286G - zroot allocated 114G - zroot readonly off - zroot comment - default zroot expandsize 0 - zroot freeing 0 default zroot feature@async_destroy enabled local zroot feature@empty_bpobj active local zroot feature@lz4_compress active local
server # dovecot -n # 2.2.13: /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 9.2-RELEASE amd64 auth_default_realm = mydomain.com auth_mechanisms = plain login auth_realms = mydomain.com mydomain2.org disable_plaintext_auth = no first_valid_gid = 5000 first_valid_uid = 5000 hostname = mydomain.com imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags imap_idle_notify_interval = 29 mins last_valid_gid = 5000 last_valid_uid = 5000 lda_mailbox_autocreate = yes listen = * mail_gid = 5000 mail_location = maildir:~/:CONTROL=/var/no-quota/%u mail_plugins = quota mail_log notify mail_temp_dir = /var/tmp mail_uid = 5000 maildir_broken_filename_sizes = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate spamtest spamtestplus editheader mbox_lazy_writes = no namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { expire = Trash mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = maildir:User quota quota_exceeded_message = Storage quota for this account has been exceeded, please try again later. quota_rule = *:storage=50G quota_rule2 = Trash:storage=+30%% quota_rule3 = Sent:storage=+30%% quota_warning = storage=90%% quota-warning 90 %u quota_warning2 = storage=75%% quota-warning 75 %u sieve = ~/.dovecot.sieve sieve_before = /usr/local/etc/dovecot/sieve/ sieve_dir = ~/sieve sieve_extensions = +spamtest +spamtestplus +relational +comparator-i;ascii-numeric +editheader } postmaster_address = gessel@mydomain.com protocols = imap lmtp sieve sieve sendmail_path = /usr/local/sbin/sendmail service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 vsz_limit = 128 M } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } ssl = required ssl_cert =
Am 15.07.2014 03:12, schrieb David Gessel:
- In thunderbird, I get constant downloading of messages. Every time I switch to the inbox from another folder, the messages start redownloading. This sometimes, but not always, seems to create duplicate messages, but I'm now fairly sure they are only created in the client and not on the server. Searching through various forums, it appears this has been a problem for Thunderbird for some time, but it wasn't for me on Courier. As posts about similar errors seem to have petered out over the last year without any clear guidance on how to fix it that I could find, I expect there's a simple fix I just haven't found yet, either in TB configuration or in Dovecot. Any hints? I tried Thunderfix to remove the .msf files and will be redownloading my mail over the thin pipe out to Iraq for the next 6 months (DOH!). One artifact is that when I restart thunderbird, my inbox is reported as having about 130 messages in it, then thunderbird starts redownloading headers and eventually gets back up to the 1297 currently listed both in the TB pane and by ls | wc -l in /cur. Is it possible there is a time out that needs to be extended? In TB, I've set (among other less apparently relevant options) mailnews.tcptimeout;1000 mail.db.idle_limit;30000000
How old is your Thunderbird profile. Have you tried creating a new one and see if the problem persists?
Thunderfix should not be needed by the way. Just right click on the folder => Properties => Repair
server # dovecot -n # 2.2.13: /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 9.2-RELEASE amd64 auth_default_realm = mydomain.com auth_mechanisms = plain login auth_realms = mydomain.com mydomain2.org disable_plaintext_auth = no first_valid_gid = 5000 first_valid_uid = 5000 hostname = mydomain.com imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags imap_idle_notify_interval = 29 mins last_valid_gid = 5000 last_valid_uid = 5000 lda_mailbox_autocreate = yes listen = * mail_gid = 5000 mail_location = maildir:~/:CONTROL=/var/no-quota/%u mail_plugins = quota mail_log notify mail_temp_dir = /var/tmp mail_uid = 5000 maildir_broken_filename_sizes = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate spamtest spamtestplus editheader mbox_lazy_writes = no namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { expire = Trash mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = maildir:User quota quota_exceeded_message = Storage quota for this account has been exceeded, please try again later. quota_rule = *:storage=50G quota_rule2 = Trash:storage=+30%% quota_rule3 = Sent:storage=+30%% quota_warning = storage=90%% quota-warning 90 %u quota_warning2 = storage=75%% quota-warning 75 %u sieve = ~/.dovecot.sieve sieve_before = /usr/local/etc/dovecot/sieve/ sieve_dir = ~/sieve sieve_extensions = +spamtest +spamtestplus +relational +comparator-i;ascii-numeric +editheader } postmaster_address = gessel@mydomain.com protocols = imap lmtp sieve sieve sendmail_path = /usr/local/sbin/sendmail service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 vsz_limit = 128 M } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } ssl = required ssl_cert =
I don't know if it makes a difference for you as you are not using "LAYOUT=fs" but try to disable those imap_client_workarounds.
-- Alex JOST
-------- Original Message -------- Subject: Re: Dovecot/K9/Thunderbird IMAP interaction issues: replicated messages, constant reloading (TB) & missing new messages (K-9) From: Alex JOST jost+lists@dimejo.at To: dovecot@dovecot.org Date: Tue Jul 15 2014 13:30:31 GMT+0300 (Arabic Standard Time)
Am 15.07.2014 03:12, schrieb David Gessel:
- In thunderbird, I get constant downloading of messages. Every time I switch to the inbox from another folder, the messages start redownloading. This sometimes, but not always, seems to create duplicate messages, but I'm now fairly sure they are only created in the client and not on the server. Searching through various forums, it appears this has been a problem for Thunderbird for some time, but it wasn't for me on Courier. As posts about similar errors seem to have petered out over the last year without any clear guidance on how to fix it that I could find, I expect there's a simple fix I just haven't found yet, either in TB configuration or in Dovecot. Any hints? I tried Thunderfix to remove the .msf files and will be redownloading my mail over the thin pipe out to Iraq for the next 6 months (DOH!). One artifact is that when I restart thunderbird, my inbox is reported as having about 130 messages in it, then thunderbird starts redownloading headers and eventually gets back up to the 1297 currently listed both in the TB pane and by ls | wc -l in /cur. Is it possible there is a time out that needs to be extended? In TB, I've set (among other less apparently relevant options) mailnews.tcptimeout;1000 mail.db.idle_limit;30000000
How old is your Thunderbird profile. Have you tried creating a new one and see if the problem persists?
It is a new profile created with the transition from Cyrus to Dovecot about a week ago, but I used IMAPSync to move all my mail from the ancient Cyrus server to the shiny new Dovecot server. I then deleted the old profile.
Thunderfix should not be needed by the way. Just right click on the folder => Properties => Repair
I did see that, but first thought there was something magic about "thunderfix" then decided to use it anyway because I have annual sent/recieved archives from 2000 (and one for pre-2000) so that would have been a lotta right-clicking.
server # dovecot -n # 2.2.13: /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 9.2-RELEASE amd64 auth_default_realm = mydomain.com auth_mechanisms = plain login auth_realms = mydomain.com mydomain2.org disable_plaintext_auth = no first_valid_gid = 5000 first_valid_uid = 5000 hostname = mydomain.com imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags imap_idle_notify_interval = 29 mins last_valid_gid = 5000 last_valid_uid = 5000 lda_mailbox_autocreate = yes listen = * mail_gid = 5000 mail_location = maildir:~/:CONTROL=/var/no-quota/%u mail_plugins = quota mail_log notify mail_temp_dir = /var/tmp mail_uid = 5000 maildir_broken_filename_sizes = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate spamtest spamtestplus editheader mbox_lazy_writes = no namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { expire = Trash mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = maildir:User quota quota_exceeded_message = Storage quota for this account has been exceeded, please try again later. quota_rule = *:storage=50G quota_rule2 = Trash:storage=+30%% quota_rule3 = Sent:storage=+30%% quota_warning = storage=90%% quota-warning 90 %u quota_warning2 = storage=75%% quota-warning 75 %u sieve = ~/.dovecot.sieve sieve_before = /usr/local/etc/dovecot/sieve/ sieve_dir = ~/sieve sieve_extensions = +spamtest +spamtestplus +relational +comparator-i;ascii-numeric +editheader } postmaster_address = gessel@mydomain.com protocols = imap lmtp sieve sieve sendmail_path = /usr/local/sbin/sendmail service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 vsz_limit = 128 M } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } ssl = required ssl_cert =
ssl_key =
I don't know if it makes a difference for you as you are not using "LAYOUT=fs" but try to disable those imap_client_workarounds.
Good catch, thank you. (I used LAYOUT=fs at first, but found behavior a little anomalous. I forgot to disable the workarounds).
This may have stopped the problem of redownloading the entire mailbox (on restart I got downloading 1 of 8 instead of 1 of 170... 30 of 1330 in the status pane) . My other mailboxes still need to sync and it might take a few days to verify that things are stable. I did get 81 immediate "Warning: Maildir /mail/mydomain.com/gessel//.Bulkmail: Expunged message reappeared, giving a new UID" warnings on connect, but if this is the problem, it might stop after all the directories are resynced.
I'll update with further data as it comes in.
I am also somewhat suspect of QRSYNC from this post: http://dovecot.org/pipermail/dovecot/2011-August/130687.html - though this is from a long time ago and it seems a little surprising that bugs on this seemingly useful capability would persist. In any event (if useful information), my capability enumeration reads:
# telnet localhost 143
- OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. a login user pass a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE COMPRESS=DEFLATE QUOTA] Logged in
I did try setting mail.server.default.use_condstore;false in Thunderbird, but that didn't change the behavior of reloading the entire inbox on startup so I've changed that back as that wasn't the desired result.
-------- Original Message -------- Subject: Re: Dovecot/K9/Thunderbird IMAP interaction issues: replicated messages, constant reloading (TB) & missing new messages (K-9) From: David Gessel gessel@blackrosetech.com To: Alex JOST jost+lists@dimejo.at, dovecot@dovecot.org Date: Tue Jul 15 2014 15:08:31 GMT+0300 (Arabic Standard Time)
-------- Original Message -------- Subject: Re: Dovecot/K9/Thunderbird IMAP interaction issues: replicated messages, constant reloading (TB) & missing new messages (K-9) From: Alex JOST jost+lists@dimejo.at To: dovecot@dovecot.org Date: Tue Jul 15 2014 13:30:31 GMT+0300 (Arabic Standard Time)
Am 15.07.2014 03:12, schrieb David Gessel:
- In thunderbird, I get constant downloading of messages. Every time I switch to the inbox from another folder, the messages start redownloading. This sometimes, but not always, seems to create duplicate messages, but I'm now fairly sure they are only created in the client and not on the server. Searching through various forums, it appears this has been a problem for Thunderbird for some time, but it wasn't for me on Courier. As posts about similar errors seem to have petered out over the last year without any clear guidance on how to fix it that I could find, I expect there's a simple fix I just haven't found yet, either in TB configuration or in Dovecot. Any hints? I tried Thunderfix to remove the .msf files and will be redownloading my mail over the thin pipe out to Iraq for the next 6 months (DOH!). One artifact is that when I restart thunderbird, my inbox is reported as having about 130 messages in it, then thunderbird starts redownloading headers and eventually gets back up to the 1297 currently listed both in the TB pane and by ls | wc -l in /cur. Is it possible there is a time out that needs to be extended? In TB, I've set (among other less apparently relevant options) mailnews.tcptimeout;1000 mail.db.idle_limit;30000000
How old is your Thunderbird profile. Have you tried creating a new one and see if the problem persists?
It is a new profile created with the transition from Cyrus to Dovecot about a week ago, but I used IMAPSync to move all my mail from the ancient Cyrus server to the shiny new Dovecot server. I then deleted the old profile.
Thunderfix should not be needed by the way. Just right click on the folder => Properties => Repair
I did see that, but first thought there was something magic about "thunderfix" then decided to use it anyway because I have annual sent/recieved archives from 2000 (and one for pre-2000) so that would have been a lotta right-clicking.
server # dovecot -n # 2.2.13: /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 9.2-RELEASE amd64 auth_default_realm = mydomain.com auth_mechanisms = plain login auth_realms = mydomain.com mydomain2.org disable_plaintext_auth = no first_valid_gid = 5000 first_valid_uid = 5000 hostname = mydomain.com imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags imap_idle_notify_interval = 29 mins last_valid_gid = 5000 last_valid_uid = 5000 lda_mailbox_autocreate = yes listen = * mail_gid = 5000 mail_location = maildir:~/:CONTROL=/var/no-quota/%u mail_plugins = quota mail_log notify mail_temp_dir = /var/tmp mail_uid = 5000 maildir_broken_filename_sizes = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate spamtest spamtestplus editheader mbox_lazy_writes = no namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { expire = Trash mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = maildir:User quota quota_exceeded_message = Storage quota for this account has been exceeded, please try again later. quota_rule = *:storage=50G quota_rule2 = Trash:storage=+30%% quota_rule3 = Sent:storage=+30%% quota_warning = storage=90%% quota-warning 90 %u quota_warning2 = storage=75%% quota-warning 75 %u sieve = ~/.dovecot.sieve sieve_before = /usr/local/etc/dovecot/sieve/ sieve_dir = ~/sieve sieve_extensions = +spamtest +spamtestplus +relational +comparator-i;ascii-numeric +editheader } postmaster_address = gessel@mydomain.com protocols = imap lmtp sieve sieve sendmail_path = /usr/local/sbin/sendmail service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 vsz_limit = 128 M } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } ssl = required ssl_cert =
ssl_key =
I don't know if it makes a difference for you as you are not using "LAYOUT=fs" but try to disable those imap_client_workarounds.
Good catch, thank you. (I used LAYOUT=fs at first, but found behavior a little anomalous. I forgot to disable the workarounds).
This may have stopped the problem of redownloading the entire mailbox (on restart I got downloading 1 of 8 instead of 1 of 170... 30 of 1330 in the status pane) . My other mailboxes still need to sync and it might take a few days to verify that things are stable. I did get 81 immediate "Warning: Maildir /mail/mydomain.com/gessel//.Bulkmail: Expunged message reappeared, giving a new UID" warnings on connect, but if this is the problem, it might stop after all the directories are resynced.
I'll update with further data as it comes in.
I am also somewhat suspect of QRSYNC from this post: http://dovecot.org/pipermail/dovecot/2011-August/130687.html - though this is from a long time ago and it seems a little surprising that bugs on this seemingly useful capability would persist. In any event (if useful information), my capability enumeration reads:
# telnet localhost 143
- OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. a login user pass a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE COMPRESS=DEFLATE QUOTA] Logged in
I did try setting mail.server.default.use_condstore;false in Thunderbird, but that didn't change the behavior of reloading the entire inbox on startup so I've changed that back as that wasn't the desired result.
Disabling the imap_client_workarounds in TB ended up neither helping nor harming. The first sync was promising but as soon as that was finished (it can take a while to download 8 messages to Iraq) it started over re-downloading the entire mailbox. I don't seem to be generating duplicate messages, but I am redownloading every message over and over. My TB activity manager is filled with paused "bringing folder XXX up to date" messages.
Is there any indication of current problems with CONDSTORE/QRESYNC since 2011 or so?
I've set: imap_capability = IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE COMPRESS=DEFLATE QUOTA
and will report if this yields any changes
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 16 Jul 2014, David Gessel wrote:
Disabling the imap_client_workarounds in TB ended up neither helping nor harming. The first sync was promising but as soon as that was finished (it can take a while to download 8 messages to Iraq) it started over re-downloading the entire mailbox. I don't seem to be generating duplicate messages, but I am redownloading every message over and over. My TB activity manager is filled with paused "bringing folder XXX up to date" messages.
Maybe you should check: http://wiki2.dovecot.org/Debugging/Rawlog and see if the IMAP commands make sense. Check if IMAP validity / UIDs changing ... .
If "it can take a while", maybe the connection gets interrupted or corrupted and some client is trying to work around this.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iQEVAwUBU8YPEXz1H7kL/d9rAQJ/jggAqJUsxAUFRCcQWtOHXj9X+nE9QU3ck/pR hMdzFtv6BZjAOC9eZrefD80Md6aUa1d35RzvE2kLG3D8mnkx6hLVETemT70I1KqA EniRIn6svb3jP2H32n0DdIIPHEtmC+Wf3k/HK2yhH6RAlZu2GNJhbuQ8Bf9kvATS pyauaQ+qHirPzo8MjAv90wBmyrgN3PHZ7+4kE2P6W9th9sHFgW3YOB0BuPqnBLvy Jb/qM1BHqz/Yt+T9qCDgCzPknm5RR+iwHPTcjF2Vz/GKz/8WWM0bJniRqFs4g1a6 9UWffK8FO5QC2oI5KpD6KOkJSKhUAs54Qj9+VuxesUC6Bm7eLfJ4CA== =AH5A -----END PGP SIGNATURE-----
Am 2014-07-15 14:08, schrieb David Gessel:
Good catch, thank you. (I used LAYOUT=fs at first, but found behavior a little anomalous. I forgot to disable the workarounds).
This may have stopped the problem of redownloading the entire mailbox (on restart I got downloading 1 of 8 instead of 1 of 170... 30 of 1330 in the status pane) . My other mailboxes still need to sync and it might take a few days to verify that things are stable. I did get 81 immediate "Warning: Maildir /mail/mydomain.com/gessel//.Bulkmail: Expunged message reappeared, giving a new UID" warnings on connect, but if this is the problem, it might stop after all the directories are resynced.
I'll update with further data as it comes in.
I am also somewhat suspect of QRSYNC from this post: http://dovecot.org/pipermail/dovecot/2011-August/130687.html - though this is from a long time ago and it seems a little surprising that bugs on this seemingly useful capability would persist. In any event (if useful information), my capability enumeration reads:
# telnet localhost 143
- OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. a login user pass a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE COMPRESS=DEFLATE QUOTA] Logged in
I did try setting mail.server.default.use_condstore;false in Thunderbird, but that didn't change the behavior of reloading the entire inbox on startup so I've changed that back as that wasn't the desired result.
There are too many possible reasons for your problem. You should really start to locate the source of your problem by eliminating 1 possibility after the other instead of tweaking some (hidden) settings in Thunderbird.
- Find the Message-ID of one of your duplicates and grep for it on the server
- Use a local client (like Mutt or Roundcube) on the server to access the mailbox
- Use a different client than Thunderbird on your PC
-- Alex JOST
participants (3)
-
Alex JOST
-
David Gessel
-
Steffen Kaiser