[Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2
I have a situation where I need to migrate e-mails from Outlook 2011 (Mac) to Apple Mail. Having looked at all options, I have resorted to the, perhaps, most difficult way: Create folders on the IMAP server, copy e-mails into them from Outlook, connect Apple Mail and do the reverse. However, I have hit a wall.
I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage.
In my first attempt, I have 1792 messages in the "Sent Items" folder for Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent Items folder, but the process dies at some point. When that happens, Outlook pops a screen saying "IMAP session state is inconsistent, please relogin".
Dovecot says: Mar 16 13:30:26 jaribu dovecot: master: Warning: Killed with signal 15 (by pid=72242 uid=0 code=kill) Mar 16 13:30:29 master: Info: Dovecot v2.1.2 starting up Mar 16 13:31:15 auth-worker(72594): Info: mysql(localhost): Connected to database exim4u Mar 16 13:31:15 imap-login: Info: Login: user=<wash@kictanet.or.ke>, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72595 Mar 16 13:31:15 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:15 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:15 imap(wash@kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:15 imap(wash@kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:15 imap(wash@kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:15 imap(wash@kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:15 imap(wash@kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid= -1 Mar 16 13:31:30 imap-login: Info: Login: user=<wash@kictanet.or.ke>, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72646 Mar 16 13:31:30 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:30 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:30 imap(wash@kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:30 imap(wash@kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:30 imap(wash@kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:30 imap(wash@kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:30 imap(wash@kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid= -1 Mar 16 13:31:30 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:30 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:30 imap(wash@kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:30 imap(wash@kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:30 imap(wash@kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:30 imap(wash@kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:30 imap(wash@kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid=-1 Mar 16 13:31:30 imap-login: Info: Login: user=<wash@kictanet.or.ke>, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72647 Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4 (around offset=894): msg header has bad magic value Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Warning: mdbox /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage: rebuilding indexes Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Error: /var/spool/virtual/kictanet.or.ke/wash/mdbox/mailboxes/SentItems/dbox-Mails/dovecot.index reset, view is now inconsistent Mar 16 13:33:23 imap(wash@kictanet.or.ke): Info: Disconnected: IMAP session state is inconsistent, please relogin. bytes=13816863/907529
My doveconf output is here -> http://pastebin.com/6yNP5ygt
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223
I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email.
On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote:
I have a situation where I need to migrate e-mails from Outlook 2011 (Mac) to Apple Mail. Having looked at all options, I have resorted to the, perhaps, most difficult way: Create folders on the IMAP server, copy e-mails into them from Outlook, connect Apple Mail and do the reverse. However, I have hit a wall.
I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage.
In my first attempt, I have 1792 messages in the "Sent Items" folder for Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent Items folder, but the process dies at some point. When that happens, Outlook pops a screen saying "IMAP session state is inconsistent, please relogin". .. Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4 (around offset=894): msg header has bad magic value
Well, this isn't good. The mdbox was empty when you first started copying the mails? This is a normal local FreeBSD filesystem (not NFS or something else weird)?
What happens if you now run:
doveadm force-resync -u wash@kictanet.or.ke INBOX
Does it show any errors? If not, and if you try to copy the mails again, does it still fail?
It's interesting if you can reproduce this. I wonder if it's because of FreeBSD or if it's related to single instance storage.
On Fri, Mar 16, 2012 at 16:24, Timo Sirainen <tss@iki.fi> wrote:
I have a situation where I need to migrate e-mails from Outlook 2011 (Mac) to Apple Mail. Having looked at all options, I have resorted to the, perhaps, most difficult way: Create folders on the IMAP server, copy e-mails into them from Outlook, connect Apple Mail and do the reverse. However, I have hit a wall.
I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage.
In my first attempt, I have 1792 messages in the "Sent Items" folder for Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent Items folder, but the process dies at some point. When that happens, Outlook
On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: pops
a screen saying "IMAP session state is inconsistent, please relogin". .. Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around offset=894): msg header has bad magic value
Well, this isn't good. The mdbox was empty when you first started copying the mails? This is a normal local FreeBSD filesystem (not NFS or something else weird)?
Yes, it was pretty much empty..actually, I just configured the Dovecot instance yesterday evening and only tested for "correct operation" by sending a test mail to myself and login to POP3/IMAP. Pretty much pristine.
And yes, if is FreeBSD ufs. [wash@jaribu ~]$ mount /dev/label/rootfs0 on / (ufs, local, noatime, journaled soft-updates) devfs on /dev (devfs, local, multilabel) procfs on /proc (procfs, local) linprocfs on /compat/linux/proc (linprocfs, local) /dev/ada1s1a on /disk2 (ufs, local, noatime, soft-updates)
What happens if you now run:
doveadm force-resync -u wash@kictanet.or.ke INBOX
[root@jaribu] /usr/home/wash# /opt/dovecot2.1/bin/doveadm force-resync -u wash@kictanet.or.ke INBOX doveadm(wash@kictanet.or.ke): Warning: mdbox /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage: rebuilding indexes
Does it show any errors? If not, and if you try to copy the mails again, does it still fail?
Unfortunately, the user has taken the MacBook now so I am unable to test copying again.
It's interesting if you can reproduce this. I wonder if it's because of FreeBSD or if it's related to single instance storage.
I should be able to test this again at some point next week, but just to add, I changed the storage to Maildir and I was able to copy all the mails to the IMAP folder without any issue so I think it's something with SIS.
PS: I wish I could test this with Outlook running on Windows, but I guess that introduces a completely different environment than what I had on the MacBook, right?
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223
I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email.
On Fri, Mar 16, 2012 at 16:24, Timo Sirainen <tss@iki.fi> wrote:
I have a situation where I need to migrate e-mails from Outlook 2011 (Mac) to Apple Mail. Having looked at all options, I have resorted to the, perhaps, most difficult way: Create folders on the IMAP server, copy e-mails into them from Outlook, connect Apple Mail and do the reverse. However, I have hit a wall.
I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage.
In my first attempt, I have 1792 messages in the "Sent Items" folder for Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent Items folder, but the process dies at some point. When that happens, Outlook
On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: pops
a screen saying "IMAP session state is inconsistent, please relogin". .. Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around offset=894): msg header has bad magic value
Well, this isn't good. The mdbox was empty when you first started copying the mails? This is a normal local FreeBSD filesystem (not NFS or something else weird)?
What happens if you now run:
doveadm force-resync -u wash@kictanet.or.ke INBOX
Does it show any errors? If not, and if you try to copy the mails again, does it still fail?
It's interesting if you can reproduce this. I wonder if it's because of FreeBSD or if it's related to single instance storage.
Ok. I have been able to reproduce it anyway.
Environment: Windows 8 Consumer Preview, Outlook 2010. I had 415 e-mails in the Inbox. While copying, Outlook popped up an error:
IMAP session state is inconsistent, please relogin. Protocol: IMAP Server: 192.168.40.252 Port: 143 Error Code: 0x800CCCDD
...and dovecot.log details at that time can be found here - http://196.200.26.114/~wash/dovecot.log.txt
And the output of the force-resync command is:
[root@jaribu] /usr/home/wash# cp /var/log/dovecot.log ~wash/public_html/dovecot.log.txt [root@jaribu] /usr/home/wash# /opt/dovecot2.1/bin/doveadm force-resync -u wash@kictanet.or.ke INBOX doveadm(wash@kictanet.or.ke): Warning: mdbox /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage: rebuilding indexes doveadm(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage/m.5 (around offset=30): Invalid dbox version doveadm(wash@kictanet.or.ke): Error: mdbox rebuild: Failed to fix file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.5 doveadm(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage/m.10 (around offset=30): Invalid dbox version doveadm(wash@kictanet.or.ke): Error: mdbox rebuild: Failed to fix file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.10
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223
I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email.
On Fri, Mar 16, 2012 at 16:24, Timo Sirainen<tss@iki.fi> wrote:
On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote:
I have a situation where I need to migrate e-mails from Outlook 2011 (Mac)
Personally I would just use readpst to export the standard Outlook
On 16/03/2012 14:00, Odhiambo Washington wrote: personal storage folders to mbox format...
-- Best Regards,
Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey@netsecspec.co.uk Email/MSN/Live Messenger: giles@coochey.net Skype: gilescoochey
On Fri, Mar 16, 2012 at 17:11, Giles Coochey <giles@coochey.net> wrote:
On 16/03/2012 14:00, Odhiambo Washington wrote:
On Fri, Mar 16, 2012 at 16:24, Timo Sirainen<tss@iki.fi> wrote:
On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote:
I have a situation where I need to migrate e-mails from Outlook 2011
(Mac)
Personally I would just use readpst to export the standard Outlook
personal storage folders to mbox format...
Outlook 2011 (Mac OS X - Lion) can export everything into (!pst) .olm I haven't no clue whether .olm and .pst are one and the same, but I highly doubt. With Outlook 2011, the guys at Redmond intended to lock the user to Outlook! I have seen Outlook->Apple Mail migration procedures fraught with e-mail loses. Well, not quite a loss because you still have the e-mail inside Outlook, but that idea of running Outlook side-by-side with Apple Mail is not what I want to subject the user to. I also don't want to make them start searching for the e-mails that might not have been migrated and forward them to themselves. Migration should be complete & safe - no losses. There are commercial software out there that's said to be good at this process, but I wanted the "free"method. If there is *readpst *that can migrate from .olm to mbox, then I am willing to give it a shot, but I also want to see if Dovecot gets a fix for whatever problem I am facing.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223
I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email.
On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote:
Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around offset=894): msg header has bad magic value
Ok. I have been able to reproduce it anyway.
It looks like mdbox is completely broken in your setup. Don't try to use it until this is solved, or you'll probably end up losing mails.
Could you try if you can easily reproduce this using imaptest? http://imapwiki.org/ImapTest
Simply run it for an empty test account as:
imaptest host=localhost user=testuser pass=testpass
Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else can report that they can reproduce this problem that would be helpful..
On Fri, Mar 16, 2012 at 18:05, Timo Sirainen <tss@iki.fi> wrote:
On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote:
Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage/m.4(around<http://kictanet.or.ke/wash/mdbox/storage/m.4%28around> offset=894): msg header has bad magic value
Ok. I have been able to reproduce it anyway.
It looks like mdbox is completely broken in your setup. Don't try to use it until this is solved, or you'll probably end up losing mails.
Could you try if you can easily reproduce this using imaptest? http://imapwiki.org/ImapTest
Simply run it for an empty test account as:
imaptest host=localhost user=testuser pass=testpass
Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else can report that they can reproduce this problem that would be helpful..
Must I edit src/settings.h to reflect my #define MBOX_PATH value? I think I had to do that, but things don't appear good at all.
[root@jaribu] ~wash/Tools/Dovecot/2.1/imaptest-20120129# ./src/imaptest host=localhost user=wash@kictanet.or.ke pass=XXX Fatal: Empty mbox file: /var/spool/virtual/kictanet.or.ke/wash/mdbox
You are welcome to access my box if you like, but I must warn you, things will be slow. Internationa links are slow as a result of fiber cuts in MSA, KE!
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223
I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email.
On 16.3.2012, at 17.57, Odhiambo Washington wrote:
imaptest host=localhost user=testuser pass=testpass
Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else can report that they can reproduce this problem that would be helpful..
Must I edit src/settings.h to reflect my #define MBOX_PATH value? I think I had to do that, but things don't appear good at all.
No, you can also specify it as mbox=path parameter.
[root@jaribu] ~wash/Tools/Dovecot/2.1/imaptest-20120129# ./src/imaptest host=localhost user=wash@kictanet.or.ke pass=XXX Fatal: Empty mbox file: /var/spool/virtual/kictanet.or.ke/wash/mdbox
But don't point the mbox there! You'll need to download http://www.dovecot.org/tmp/dovecot-crlf file and point the mbox to that. It's used to upload mails.
On 16.03.2012 16:05, wrote Timo Sirainen:
On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote:
Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around offset=894): msg header has bad magic value
This is kind of the same error message I'm getting with doveadm backup. I can reproduce this at will. Not sure however if this is related.
http://www.dovecot.org/list/dovecot/2012-March/064462.html
Ok. I have been able to reproduce it anyway.
It looks like mdbox is completely broken in your setup. Don't try to use it until this is solved, or you'll probably end up losing mails.
Could you try if you can easily reproduce this using imaptest? http://imapwiki.org/ImapTest
Simply run it for an empty test account as:
imaptest host=localhost user=testuser pass=testpass
Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else can report that they can reproduce this problem that would be helpful..
On 16.3.2012, at 18.16, e-frog wrote:
On 16.03.2012 16:05, wrote Timo Sirainen:
On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote:
Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around offset=894): msg header has bad magic value
This is kind of the same error message I'm getting with doveadm backup. I can reproduce this at will. Not sure however if this is related.
Yeah, it's the same. Looks like I just hadn't stress tested mdbox myself last few days.
On 16.03.2012 18:00, wrote Timo Sirainen:
On 16.3.2012, at 18.16, e-frog wrote:
On 16.03.2012 16:05, wrote Timo Sirainen:
On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote:
Mar 16 13:33:23 jaribu dovecot: imap(wash@kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around offset=894): msg header has bad magic value
This is kind of the same error message I'm getting with doveadm backup. I can reproduce this at will. Not sure however if this is related.
Yeah, it's the same. Looks like I just hadn't stress tested mdbox myself last few days.
Ok, just tested with 2.1.3 and it works again.
Thanks Timo!
participants (4)
-
e-frog
-
Giles Coochey
-
Odhiambo Washington
-
Timo Sirainen