[Dovecot] Outlook 2003 and Thunderbird IMAP + getting new headers
Hi,
We run 1.1.3 and I use Outlook to connect to my IMAP box and a couple of times per day (randomly) when I click certain folders IMAP will start retrieving ALL headers for that folder again ('retrieving new headers')!
I don't understand this. What triggers IMAP to do so, why not just retrieve only the NEW headers?
Is it Dovecot or Outlook (probably) and what can be done about this? I already use the Dovecot Outlook workarounds.
I have also gotten word that this behavior happens with Thunderbird.
This is really a problem causing a lot of load on our storage (I am not the only user).
Cheers,
Jan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Jan,
Jan van den Berg wrote:
We run 1.1.3 and I use Outlook to connect to my IMAP box and a couple of times per day (randomly) when I click certain folders IMAP will start retrieving ALL headers for that folder again ('retrieving new headers')!
I don't understand this. What triggers IMAP to do so, why not just retrieve only the NEW headers?
Is it Dovecot or Outlook (probably) and what can be done about this? I already use the Dovecot Outlook workarounds.
I have also gotten word that this behavior happens with Thunderbird.
This is really a problem causing a lot of load on our storage (I am not the only user).
Please give us some more information. Output of dovecot -n would be a good
start. Also check the logs for dovecot reporting problems (and/or switch on
debug logging).
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFIx4zX7yMg/OiDoAURAtWVAKCiFcHQGzyGoF3NtRzqlH78JGVPIgCeKeOV v9ixInWS+RWG0gcMFtg5Lk8= =1uNp -----END PGP SIGNATURE-----
Hi,
Here's the dovecot -n output:
# 1.1.3: /usr/local/dovecot/etc/imap-solcon.conf base_dir: /var/run/imap-solcon/ syslog_facility: local7 protocols: imap listen: *:143 ssl_disable: yes disable_plaintext_auth: no login_dir: /var/run/imap-solcon/login login_executable: /usr/local/dovecot/libexec/dovecot/imap-login login_greeting: Solcon MX. login_process_per_connection: no max_mail_processes: 1024 mail_location: maildir:/var/spool/mail/%1u/%1.1u/%u mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes maildir_stat_dirs: yes mail_executable: /usr/local/dovecot/libexec/dovecot/imap mail_plugins: quota imap_quota mail_plugin_dir: /usr/local/dovecot/lib/dovecot/imap imap_client_workarounds: outlook-idle delay-newmail auth default: mechanisms: plain digest-md5 cram-md5 login cache_size: 8192 cache_negative_ttl: 0 executable: /usr/local/dovecot/libexec/dovecot/dovecot-auth user: exim passdb: driver: sql args: /usr/local/dovecot/etc/imap-solcon-sql-auth.conf userdb: driver: prefetch plugin: quota: maildir quota_rule2: Prullenbak:storage=25M
I am a bit hesitative for debugging on production I will try and see if can set up some sort of test env. to debug.
Cheers,
Jan
-----Oorspronkelijk bericht----- Van: Patrick Nagel [mailto:patrick.nagel@star-group.net] Verzonden: woensdag 10 september 2008 11:01 Aan: Jan van den Berg CC: 'Dovecot Mailing List' Onderwerp: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + getting new headers
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Jan,
We run 1.1.3 and I use Outlook to connect to my IMAP box and a couple of times per day (randomly) when I click certain folders IMAP will start retrieving ALL headers for that folder again ('retrieving new headers')!
I don't understand this. What triggers IMAP to do so, why not just retrieve only the NEW headers?
Is it Dovecot or Outlook (probably) and what can be done about this? I already use the Dovecot Outlook workarounds.
I have also gotten word that this behavior happens with Thunderbird.
This is really a problem causing a lot of load on our storage (I am not
Jan van den Berg wrote: the
only user).
Please give us some more information. Output of dovecot -n would be a good
start. Also check the logs for dovecot reporting problems (and/or switch on
debug logging).
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFIx4zX7yMg/OiDoAURAtWVAKCiFcHQGzyGoF3NtRzqlH78JGVPIgCeKeOV v9ixInWS+RWG0gcMFtg5Lk8= =1uNp -----END PGP SIGNATURE-----
Hi,
I've done some debugging and attached are (parts) of 2 strace files.
correct.txt. This is when I click on a folder in Outlook and everything is OK (no headers are retrieved etc.)
wrong.txt. This is when I click on a folder in Outlook and it starts retrieving all the headers again.
Im not sure what to look for in these files.
Cheers,
Jan
-----Oorspronkelijk bericht----- Van: dovecot-bounces+jan.vandenberg=isp.solcon.nl@dovecot.org [mailto:dovecot-bounces+jan.vandenberg=isp.solcon.nl@dovecot.org] Namens Jan van den Berg Verzonden: woensdag 10 september 2008 11:23 Aan: 'Dovecot Mailing List' CC: 'Patrick Nagel' Onderwerp: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + getting newheaders
Hi,
Here's the dovecot -n output:
# 1.1.3: /usr/local/dovecot/etc/imap-solcon.conf base_dir: /var/run/imap-solcon/ syslog_facility: local7 protocols: imap listen: *:143 ssl_disable: yes disable_plaintext_auth: no login_dir: /var/run/imap-solcon/login login_executable: /usr/local/dovecot/libexec/dovecot/imap-login login_greeting: Solcon MX. login_process_per_connection: no max_mail_processes: 1024 mail_location: maildir:/var/spool/mail/%1u/%1.1u/%u mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes maildir_stat_dirs: yes mail_executable: /usr/local/dovecot/libexec/dovecot/imap mail_plugins: quota imap_quota mail_plugin_dir: /usr/local/dovecot/lib/dovecot/imap imap_client_workarounds: outlook-idle delay-newmail auth default: mechanisms: plain digest-md5 cram-md5 login cache_size: 8192 cache_negative_ttl: 0 executable: /usr/local/dovecot/libexec/dovecot/dovecot-auth user: exim passdb: driver: sql args: /usr/local/dovecot/etc/imap-solcon-sql-auth.conf userdb: driver: prefetch plugin: quota: maildir quota_rule2: Prullenbak:storage=25M
I am a bit hesitative for debugging on production I will try and see if can set up some sort of test env. to debug.
Cheers,
Jan
-----Oorspronkelijk bericht----- Van: Patrick Nagel [mailto:patrick.nagel@star-group.net] Verzonden: woensdag 10 september 2008 11:01 Aan: Jan van den Berg CC: 'Dovecot Mailing List' Onderwerp: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + getting new headers
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Jan,
We run 1.1.3 and I use Outlook to connect to my IMAP box and a couple of times per day (randomly) when I click certain folders IMAP will start retrieving ALL headers for that folder again ('retrieving new headers')!
I don't understand this. What triggers IMAP to do so, why not just retrieve only the NEW headers?
Is it Dovecot or Outlook (probably) and what can be done about this? I already use the Dovecot Outlook workarounds.
I have also gotten word that this behavior happens with Thunderbird.
This is really a problem causing a lot of load on our storage (I am not
Jan van den Berg wrote: the
only user).
Please give us some more information. Output of dovecot -n would be a good
start. Also check the logs for dovecot reporting problems (and/or switch on
debug logging).
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFIx4zX7yMg/OiDoAURAtWVAKCiFcHQGzyGoF3NtRzqlH78JGVPIgCeKeOV v9ixInWS+RWG0gcMFtg5Lk8= =1uNp -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jan van den Berg wrote:
I've done some debugging and attached are (parts) of 2 strace files.
correct.txt. This is when I click on a folder in Outlook and everything is OK (no headers are retrieved etc.)
wrong.txt. This is when I click on a folder in Outlook and it starts retrieving all the headers again.
Im not sure what to look for in these files.
Cheers,
Jan
Hm ok, maybe somebody can read something out of the strace output, I can't. The dovecot log output would have been more helpful / readable.
Maybe the dovecot.index and/or dovecot-uidlist file in those mailbox folders got corrupted/deleted? But I really know too little about all that... I think dovecot's log output (see http://wiki.dovecot.org/Logging) would really be a great help in figuring out what's going on.
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFIx56F7yMg/OiDoAURAo3VAJwMjqzL0QVR8CVoTkIEVaiK6cB6igCfY5+Q s/UqI42myf3Zto+LtWwFgqQ= =PNFJ -----END PGP SIGNATURE-----
Hi,
Well I enabled verbose debugs and all but there isn't anything interesting in those logs...
Cheers,
Jan
-----Oorspronkelijk bericht----- Van: dovecot-bounces+jan.vandenberg=isp.solcon.nl@dovecot.org [mailto:dovecot-bounces+jan.vandenberg=isp.solcon.nl@dovecot.org] Namens Patrick Nagel Verzonden: woensdag 10 september 2008 12:17 Aan: Jan van den Berg CC: 'Dovecot Mailing List' Onderwerp: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + getting newheaders
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jan van den Berg wrote:
I've done some debugging and attached are (parts) of 2 strace files.
correct.txt. This is when I click on a folder in Outlook and everything is OK (no headers are retrieved etc.)
wrong.txt. This is when I click on a folder in Outlook and it starts retrieving all the headers again.
Im not sure what to look for in these files.
Cheers,
Jan
Hm ok, maybe somebody can read something out of the strace output, I can't. The dovecot log output would have been more helpful / readable.
Maybe the dovecot.index and/or dovecot-uidlist file in those mailbox folders got corrupted/deleted? But I really know too little about all that... I think dovecot's log output (see http://wiki.dovecot.org/Logging) would really be a great help in figuring out what's going on.
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFIx56F7yMg/OiDoAURAo3VAJwMjqzL0QVR8CVoTkIEVaiK6cB6igCfY5+Q s/UqI42myf3Zto+LtWwFgqQ= =PNFJ -----END PGP SIGNATURE-----
On Wed, 2008-09-10 at 10:29 +0200, Jan van den Berg wrote:
We run 1.1.3 and I use Outlook to connect to my IMAP box and a couple of times per day (randomly) when I click certain folders IMAP will start retrieving ALL headers for that folder again ('retrieving new headers')!
Typically that means that the mailbox's UIDVALIDITY has changed. Or the messages get assigned new UIDs for some reason (e.g. Dovecot sees that they're gone, later sees them again). You could verify these by using e.g. rawlog (http://wiki.dovecot.org/Debugging/Rawlog) and seeing if the UIDVALIDITY value after SELECT command changes and if the returned UIDs get changed.
Do you do anything on the server with dovecot-uidlist files? Or dovecot.index* files? Like deleting them once in a while as someone was doing in a cron job? :)
Hi,
I dont do anything with the dovecot-uidlist.
This is part of what the rawlog looks like (with a folder that started getting new headers):
tbgy SELECT "My Folders.Klanten" rty3 UID FETCH 1668:* (UID FLAGS RFC822.SIZE BODY.PEEK[HEADER] INTERNALDATE) 9d54 UID FETCH 1:1667 (UID FLAGS) 0ntr UID FETCH 1667 (UID FLAGS BODY.PEEK[] INTERNALDATE) znim IDLE DONE 4ogq UID FETCH 1668 (UID FLAGS BODY.PEEK[] INTERNALDATE) 5131 IDLE
How should I interpret this? I see 1668 and 1667?
Cheers,
Jan
-----Oorspronkelijk bericht----- Van: dovecot-bounces+jan.vandenberg=isp.solcon.nl@dovecot.org [mailto:dovecot-bounces+jan.vandenberg=isp.solcon.nl@dovecot.org] Namens Timo Sirainen Verzonden: woensdag 10 september 2008 20:45 Aan: Jan van den Berg CC: 'Dovecot Mailing List' Onderwerp: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + gettingnew headers
On Wed, 2008-09-10 at 10:29 +0200, Jan van den Berg wrote:
We run 1.1.3 and I use Outlook to connect to my IMAP box and a couple of times per day (randomly) when I click certain folders IMAP will start retrieving ALL headers for that folder again ('retrieving new headers')!
Typically that means that the mailbox's UIDVALIDITY has changed. Or the messages get assigned new UIDs for some reason (e.g. Dovecot sees that they're gone, later sees them again). You could verify these by using e.g. rawlog (http://wiki.dovecot.org/Debugging/Rawlog) and seeing if the UIDVALIDITY value after SELECT command changes and if the returned UIDs get changed.
Do you do anything on the server with dovecot-uidlist files? Or dovecot.index* files? Like deleting them once in a while as someone was doing in a cron job? :)
Im still trying to figure this problem. Because of other issues (non-dovecot related but application related) we had to revert back yesterday to 1.1rc3. This version has been running fine for 6 months.
But now I get quite a few of these errors in my logs I can't really explain them:
Log synchronization error at seq=2,offset=164 for /var/spool/mail/j/a/jan.vandenberg.isp/.Sentbox/dovecot.index: ext reset: invalid record size
This is something that happens since we reverted back:
Number of times this happens in our logs: dovecot:2679 (so far, today) dovecot.1:3184 (yesterday) dovecot.2:0 (tuesday) dovecot.3:0 (etc.) dovecot.4:5 dovecot.5:7 dovecot.6:0 dovecot.7:1
We only ran 1.1.3 for a day or so (with zero of these errors occuring). Before that with 1.1rc3 we also had few of these errors. But now a couple of 1000 a day. It is just a small percent of our total log file: but should I be worried and what is it?
Cheers,
Jan
----- Original Message ----- From: "Jan van den Berg" jan.vandenberg@isp.solcon.nl To: "'Dovecot Mailing List'" dovecot@dovecot.org Sent: Thursday, September 11, 2008 11:57 AM Subject: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + gettingnew headers
Hi,
I dont do anything with the dovecot-uidlist.
This is part of what the rawlog looks like (with a folder that started getting new headers):
tbgy SELECT "My Folders.Klanten" rty3 UID FETCH 1668:* (UID FLAGS RFC822.SIZE BODY.PEEK[HEADER] INTERNALDATE) 9d54 UID FETCH 1:1667 (UID FLAGS) 0ntr UID FETCH 1667 (UID FLAGS BODY.PEEK[] INTERNALDATE) znim IDLE DONE 4ogq UID FETCH 1668 (UID FLAGS BODY.PEEK[] INTERNALDATE) 5131 IDLE
How should I interpret this? I see 1668 and 1667?
Cheers,
Jan
-----Oorspronkelijk bericht----- Van: dovecot-bounces+jan.vandenberg=isp.solcon.nl@dovecot.org [mailto:dovecot-bounces+jan.vandenberg=isp.solcon.nl@dovecot.org] Namens Timo Sirainen Verzonden: woensdag 10 september 2008 20:45 Aan: Jan van den Berg CC: 'Dovecot Mailing List' Onderwerp: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + gettingnew headers
On Wed, 2008-09-10 at 10:29 +0200, Jan van den Berg wrote:
We run 1.1.3 and I use Outlook to connect to my IMAP box and a couple of times per day (randomly) when I click certain folders IMAP will start retrieving ALL headers for that folder again ('retrieving new headers')!
Typically that means that the mailbox's UIDVALIDITY has changed. Or the messages get assigned new UIDs for some reason (e.g. Dovecot sees that they're gone, later sees them again). You could verify these by using e.g. rawlog (http://wiki.dovecot.org/Debugging/Rawlog) and seeing if the UIDVALIDITY value after SELECT command changes and if the returned UIDs get changed.
Do you do anything on the server with dovecot-uidlist files? Or dovecot.index* files? Like deleting them once in a while as someone was doing in a cron job? :)
On Thu, 2008-09-11 at 20:45 +0200, Jan van den Berg wrote:
Im still trying to figure this problem. Because of other issues (non-dovecot related but application related) we had to revert back yesterday to 1.1rc3. This version has been running fine for 6 months.
But now I get quite a few of these errors in my logs I can't really explain them:
Log synchronization error at seq=2,offset=164 for /var/spool/mail/j/a/jan.vandenberg.isp/.Sentbox/dovecot.index: ext reset: invalid record size
It's seeing records in dovecot.index.log written by 1.1.3 that it doesn't really understand. They should shrink over time. Or you could delete dovecot.index* files and let them be rebuilt.
On Thu, 2008-09-11 at 20:45 +0200, Jan van den Berg wrote:
Im still trying to figure this problem. Because of other issues (non-dovecot related but application related) we had to revert back yesterday to 1.1rc3. This version has been running fine for 6 months.
So Outlook hadn't been ever redownloading the headers with rc3? Let me know if downgrading back to it now stops doing that again. I can't really think of any reason what could have changed between 1.1.rc3 and 1.1.3 that would cause this.
Hi,
1.1rc3 also shows this behaviour; I actually upgraded to 1.1.3 to try and solve this..
We had to revert back because of an internally developed IMAP webclient that all of sudden couldn't show attachments anymore with 1.1.3. So we reverted back: and all was fine. So that is more likely a application bug than a Dovecot one (probably with Content-disposition: inline vs. attachment). We're still investigating. But that is beside this point.
So as well 1.1rc3 as 1.1.3 has the problem of getting new headers randomly (at least so it seems). It might be NFS related because after getting new headers for a folder I can see the dovecot.index files and uidlist are recreated and a couple .nfs* files are in the dir.... However I can't explain it (all NFS related Dovecot optimizations are on).
Cheers,
Jan ----- Original Message ----- From: "Timo Sirainen" tss@iki.fi To: "Jan van den Berg" jan.vandenberg@isp.solcon.nl Cc: "'Dovecot Mailing List'" dovecot@dovecot.org Sent: Thursday, September 11, 2008 9:11 PM Subject: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + gettingnew headers
On Thu, 2008-09-11 at 21:24 +0200, Jan van den Berg wrote:
So as well 1.1rc3 as 1.1.3 has the problem of getting new headers randomly (at least so it seems). It might be NFS related because after getting new headers for a folder I can see the dovecot.index files and uidlist are recreated and a couple .nfs* files are in the dir.... However I can't explain it (all NFS related Dovecot optimizations are on).
Do you have backups of the dovecot.index* and dovecot-uidlist files? It would be useful to know what the contents of dovecot-uidlist were before and after a redownload. That would show if the UIDVALIDITY changed or only the UIDs.
I can get those files next Monday without problem and compare them. So I will get back to that. I also posted some part of the rawlog earlier in the thread; dont know if it of any use
Thanks,
Jan
----- Original Message ----- From: "Timo Sirainen" tss@iki.fi To: "Jan van den Berg" jan.vandenberg@isp.solcon.nl Cc: "Dovecot Mailing List" dovecot@dovecot.org Sent: Thursday, September 11, 2008 9:31 PM Subject: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + gettingnew headers
I can get those files next Monday without problem and compare them. So I will get back to that. I also posted some part of the rawlog earlier in the thread; dont know if it of any use
Thanks,
Jan
----- Original Message ----- From: "Timo Sirainen" tss@iki.fi To: "Jan van den Berg" jan.vandenberg@isp.solcon.nl Cc: "Dovecot Mailing List" dovecot@dovecot.org Sent: Thursday, September 11, 2008 9:31 PM Subject: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + gettingnew headers
I can get those files next Monday without problem and compare them. So I will get back to that. I also posted some part of the rawlog earlier in the thread; dont know if it of any use
Thanks,
Jan
----- Original Message ----- From: "Timo Sirainen" tss@iki.fi To: "Jan van den Berg" jan.vandenberg@isp.solcon.nl Cc: "Dovecot Mailing List" dovecot@dovecot.org Sent: Thursday, September 11, 2008 9:31 PM Subject: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + gettingnew headers
I was gonna get back on this.
Short answer: it's pretty much solved.
Long answer: there was indeed a cron that did 'something' with the dovecot-uidlist files. It didn't touch the cache files or anything. I disabled this script and now everything works fine...
Cheers,
Jan -----Oorspronkelijk bericht----- Van: dovecot-bounces+jan.vandenberg=isp.solcon.nl@dovecot.org [mailto:dovecot-bounces+jan.vandenberg=isp.solcon.nl@dovecot.org] Namens Jan van den Berg Verzonden: donderdag 11 september 2008 21:53 Aan: Dovecot Mailing List Onderwerp: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + gettingnew headers
I can get those files next Monday without problem and compare them. So I will get back to that. I also posted some part of the rawlog earlier in the thread; dont know if it
of any use
Thanks,
Jan
----- Original Message ----- From: "Timo Sirainen" tss@iki.fi To: "Jan van den Berg" jan.vandenberg@isp.solcon.nl Cc: "Dovecot Mailing List" dovecot@dovecot.org Sent: Thursday, September 11, 2008 9:31 PM Subject: Re: [Dovecot] Outlook 2003 and Thunderbird IMAP + gettingnew headers
participants (3)
-
Jan van den Berg
-
Patrick Nagel
-
Timo Sirainen