[Dovecot] All mail in mbox disappears when using Outlook
I thought I was imaging it but... I have a friend (honestly) that likes to clear his inbox every month by copying older into a folder (e.g. June-2009). Every month he complains that all the mail in the new folder has disappeared. This has happened three times now, and he's sort-of-right. I now believe it's a bug rather than him.
The setup is Dovecot 1.1.3 on FreeBSD 7.0 and the client is Outlook 2003 SP3 (using IMAP).
What you end up with is a folder called June-2009 with no messages visible. Outlook reckons it's completely empty - size 0. I don't know how he gets to this point, but it can't be anything that bizarre - just copying messages. It's happened three months in a row. However, he can't see the contents and I can't see the contents using Outlook. Strangely, Eudora and Squirrel have no problem.
Corrupt mbox file? Probably not. If you copy the problem one, the copy works fine. The really weird thing is that if you rename the problem file (using mv) it STILL DOESN'T work. I've considered the possibility that it's got a bad lock on it, but if so, how come Squirrel and Eudora have no problem?
I've searched as far as I can for any known problems using Outlook.
What next?
Thanks, Frank.
====================================================== # 1.1.3: /usr/local/etc/dovecot.conf log_path: /var/log/dovecot protocols: imap imaps pop3 pop3s ssl_cert_file: /etc/mail/certs/dovecot-cert.pem ssl_key_file: /etc/mail/certs/dovecot-private.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(pop3): no verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_privileged_group: mail mail_location: mbox:~/mail/:INBOX=/var/mail/%u maildir_copy_preserve_filename: yes mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_process_size: 512 mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 imap_client_workarounds(default): delay-newmail netscape-eoh tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail netscape-eoh tb-extra-mailbox-sep imap_client_workarounds(pop3): pop3_enable_last(default): no pop3_enable_last(imap): no pop3_enable_last(pop3): yes pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: mechanisms: plain login username_format: %Ln process_size: 512 passdb: driver: pam args: session=yes dovecot userdb: driver: passwd args: blocking=yes socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 master: path: /var/run/dovecot/auth-master mode: 384
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Frank Leonhardt said the following on 01/08/09 12:00:
I've searched as far as I can for any known problems using Outlook.
What about the size of the mailbox?
Each version of Outlook has its own issue regarding the maximum size of PST files.
Ciao, luigi
/ +--[Luigi Rosa]-- \
There are three possibilities: Pioneer's solar panel has turned away from the sun; there's a large meteor blocking transmission; or someone loaded Star Trek 3.2 into our video processor. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkp0PpkACgkQ3kWu7Tfl6ZRJMQCfSxIGWTz9NTGUslKHeBlDoJ8q fXwAoJkJapjpCyfUVRrQTGGi9zmE2n9Q =nckU -----END PGP SIGNATURE-----
This was sent on Saturday but didn't make it to the list (possibly a problem my end).
Incidentally, I've since restarted Dovecot to see if that helped. It didn't.
Luigi Rosa wrote on 01 August 2009 14:10
Frank Leonhardt said the following on 01/08/09 12:00:
I've searched as far as I can for any known problems using Outlook.
What about the size of the mailbox?
Each version of Outlook has its own issue regarding the maximum size of PST files.
Good guess knowing Outlook, and a good question as I'd forgot to mention it, but these are IMAP mailboxes, and they're therefore 'cached' in individual .pst files for each box. Originally I'd suspected it was a size issue with Dovecot as it worked when I split the files, but I now know that simply copying the files is enough. FWIW the problem mbox files are ~100M on the server - nothing exceptional.
Incidentally, having had much experience of Outlook, 2Gb is a good working assumption (32-bit signed addressing). This was the official limit in earlier versions (pre-Unicode), but if you reached it, big trouble. Personally I never let them get to more than 1.5Gb. Outlook 2003 is an interesting case - standard PST files are Unicode but those used for caching IMAP are ASCII, so you still have the limit to worry about. I believe the physical limit was removed in Outlook 2007 - the upper limit is now set in the registry at a default of ~20Gb - but creating bigger files is still bad news as you'll either wreck a Microsoft filing system somewhere or need to import the file into an older version of Outlook (or data recover tool).
On Sat, 2009-08-01 at 11:00 +0100, Frank Leonhardt wrote:
Corrupt mbox file? Probably not. If you copy the problem one, the copy works fine. The really weird thing is that if you rename the problem file (using mv) it STILL DOESN'T work. I've considered the possibility that it's got a bad lock on it, but if so, how come Squirrel and Eudora have no problem?
If it works in other clients, I can't think of how this could be Dovecot's fault. But I guess it's possible that Outlook gets confused by something that Dovecot sends. I've no idea, and I can't test it myself..
Timo Sirainen wrote on 03 August 2009 19:46:
On Sat, 2009-08-01 at 11:00 +0100, Frank Leonhardt wrote:
Corrupt mbox file? Probably not. If you copy the problem one, the copy works fine. The really weird thing is that if you rename the problem file (using mv) it STILL DOESN'T work. I've considered the possibility that it's got a bad lock on it, but if so, how come Squirrel and Eudora have no problem?
If it works in other clients, I can't think of how this could be Dovecot's fault. But I guess it's possible that Outlook gets confused by something that Dovecot sends. I've no idea, and I can't test it myself..
Yes - it's VERY strange, and very repeatable. Both client machines are running Outlook 2003 SP3 (Version 11.8217.8221), one on XP and the other on Vista. Dovecot is running on FreeBSD 7.0-RELEASE #0, which is a bit unusual.
It's the Vista machine that broke it - I've never had any problems with the XP box (in relation to this, anyway!)
I'm currently compiling the latest FreeBSD port of Dovecot (1.1.16) to see what happens (and rebuilding what looks like the entire planet in the process - 5+ hours and still building).
Any suggestions about where I could look for a clue will be followed up immediately. I'm new to Dovecot, but not BSD. My next step will be to put a network analyser on it.
Thanks, Frank.
On Mon, 2009-08-03 at 20:03 +0100, Frank Leonhardt wrote:
Any suggestions about where I could look for a clue will be followed up immediately. I'm new to Dovecot, but not BSD. My next step will be to put a network analyser on it.
Looking at the imap traffic could help, especially if you can reproduce "works" and "doesn't work" cases. You could use also http://wiki.dovecot.org/Debugging/Rawlog
Timo Sirainen wrote on 03 August 2009 20:11:
Any suggestions about where I could look for a clue will be followed up immediately. I'm new to Dovecot, but not BSD. My next step will be to
On Mon, 2009-08-03 at 20:03 +0100, Frank Leonhardt wrote: put a
network analyser on it.
Looking at the imap traffic could help, especially if you can reproduce "works" and "doesn't work" cases. You could use also http://wiki.dovecot.org/Debugging/Rawlog
I saw that a few weeks back and thought I'd give it a go - the analyser looks less scarey ;-)
I've had some progress. The problem was the same on 1.1.16. However, if I rename the mbox file to something completely different (rather than adding a suffix as I had done before) it all suddenly starts to work. The old file was called "inbox090630". It doesn't like "inbox090630-1" but it's okay about "Inbox-09-06-30". Moving the file name back makes everything disappear again. Unsubscribing and subscribing alone makes no difference.
It's obviously looking like an internal Outlook problem. I'll check it out with other IMAP servers and see if I can confirm it. However, I'll let Microsoft fix it themselves.
Thanks for all your help!
Further to this, Outlook 2000 has the same problem as Outlook 2003 - if the folder name takes the form 'inbox123456' then it will display as empty in Outlook (although other IMAP clients have no trouble).
I'd have difficulty believing this if I wasn't seeing it myself.
Frank Leonhardt wrote:
Timo Sirainen wrote on 03 August 2009 20:11:
On Mon, 2009-08-03 at 20:03 +0100, Frank Leonhardt wrote:
Any suggestions about where I could look for a clue will be followed up immediately. I'm new to Dovecot, but not BSD. My next step will be to
put a
network analyser on it.
Looking at the imap traffic could help, especially if you can reproduce "works" and "doesn't work" cases. You could use also http://wiki.dovecot.org/Debugging/Rawlog
I saw that a few weeks back and thought I'd give it a go - the analyser looks less scarey ;-)
I've had some progress. The problem was the same on 1.1.16. However, if I rename the mbox file to something completely different (rather than adding a suffix as I had done before) it all suddenly starts to work. The old file was called "inbox090630". It doesn't like "inbox090630-1" but it's okay about "Inbox-09-06-30". Moving the file name back makes everything disappear again. Unsubscribing and subscribing alone makes no difference.
It's obviously looking like an internal Outlook problem. I'll check it out with other IMAP servers and see if I can confirm it. However, I'll let Microsoft fix it themselves.
Thanks for all your help!
This sounds like an issue I've noticed with Outlook myself, in that
whenever it sees
a mail folder starting with the word "inbox" (whether it is "inboX" or
"iNbOX" or
"inbox-dfsjaf"), Outlook always looks for "Inbox" or "Inbox-dfsjaf" (it
uppercases the
I and lowercases the "nbox" before passing the request to the backend).
Thus when
I was converting my users from UW imap mboxes to Dovecot IMAP, I took
care in
my conversion script to rename folders on their behalf if they start
with "inbox"
so they are "Inbox" instead. Also, if you have a mailbox called
"Inbox", Thunderbird
will not see it since it is hidden by the real inbox.
Workarounds in perl: # If a folder is named Inbox, thunderbird will not see it. Rename. $target =~ s/^inbox$/Inbox-renamed-by-migrate/ig;
# If a folder starts with inbox (any case, eg. INbOX) Outlook won't see it properly. # Outlook expects Inbox(something) or bust. $target =~ s/^inbox(.*)$/Inbox$1/ig;
Adam McDougall wrote on 07 August 2009 15:01
Frank Leonhardt wrote:
Timo Sirainen wrote on 03 August 2009 20:11:
On Mon, 2009-08-03 at 20:03 +0100, Frank Leonhardt wrote:
Any suggestions about where I could look for a clue will be followed up immediately. I'm new to Dovecot, but not BSD. My next step will be to
put a
network analyser on it.
Looking at the imap traffic could help, especially if you can reproduce "works" and "doesn't work" cases. You could use also http://wiki.dovecot.org/Debugging/Rawlog
I saw that a few weeks back and thought I'd give it a go - the analyser looks less scarey ;-)
I've had some progress. The problem was the same on 1.1.16. However, if I rename the mbox file to something completely different (rather than adding a suffix as I had done before) it all suddenly starts to work. The old file was called "inbox090630". It doesn't like "inbox090630-1" but it's okay about "Inbox-09-06-30". Moving the file name back makes everything disappear again. Unsubscribing and subscribing alone makes no difference.
It's obviously looking like an internal Outlook problem. I'll check it out with other IMAP servers and see if I can confirm it. However, I'll let Microsoft fix it themselves.
Thanks for all your help!
This sounds like an issue I've noticed with Outlook myself, in that whenever it sees a mail folder starting with the word "inbox" (whether it is "inboX" or "iNbOX" or "inbox-dfsjaf"), Outlook always looks for "Inbox" or "Inbox-dfsjaf" (it uppercases the I and lowercases the "nbox" before passing the request to the backend). Thus when I was converting my users from UW imap mboxes to Dovecot IMAP, I took care in my conversion script to rename folders on their behalf if they start with "inbox" so they are "Inbox" instead. Also, if you have a mailbox called "Inbox", Thunderbird will not see it since it is hidden by the real inbox.
Workarounds in perl: # If a folder is named Inbox, thunderbird will not see it. Rename. $target =~ s/^inbox$/Inbox-renamed-by-migrate/ig;
# If a folder starts with inbox (any case, eg. INbOX) Outlookwon't see it properly. # Outlook expects Inbox(something) or bust. $target =~ s/^inbox(.*)$/Inbox$1/ig;
Thanks - very interesting (and stupid!). I'm glad you've confirmed it's not my imagination. What WAS MS thinking?
I've reproduced the problem with Outlook 2000 and 2003 so far. Anyone tried 2007?
Is there a general case-sensitivity problem with other folder names?
I'm taking a look at commands-util.c and RCF3501 Section 5.1, which has some interesting things to say about folders called *exactly* inbox (case insensitive).
Regards, Frank
participants (4)
-
Adam McDougall
-
Frank Leonhardt
-
Luigi Rosa
-
Timo Sirainen