[Dovecot] IMAP to Maildir Migration preserving UIDs?
Hello all,
I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible.
The mailbox format I would want to use is Maildir++.
The storage format used by the current server is unknown, and I don't look forward to trying to reverse-engineer it. This leaves me with the option of reading the mailboxes using IMAP. There are tools like offlineimap or mbsync, and they do store the UID and UIDVALIDITY info. The last piece of the puzzle is a process to properly create the dovecot-uidlist and dovecot-uidvalidity files. So far I wasn't able to find anything on this. Are there any tips? Are there any tools available to do this job, or part of it?
In either case I need this done, and I'll have to create whatever I can't find available. If there isn't anything out there that I'm yet to become aware of, then I'm looking at creating something like an offlineimap post-processing routine?
Any help would be much appreciated.
Gedalya
On 2012-01-25 11:31 PM, Gedalya gedalya@gedalya.net wrote:
This leaves me with the option of reading the mailboxes using IMAP. There are tools like offlineimap or mbsync,
Not familiar with those, but I think imapsync will do what you want?
I do see that it references those two though...
--
Best regards,
Charles
On 01/26/2012 07:06 AM, Charles Marcus wrote:
On 2012-01-25 11:31 PM, Gedalya gedalya@gedalya.net wrote:
This leaves me with the option of reading the mailboxes using IMAP. There are tools like offlineimap or mbsync,
Not familiar with those, but I think imapsync will do what you want?
I do see that it references those two though...
As I understand, there is no way an IMAP-to-IAMP process can preserve UIDs, since new UIDs are assigned for every message by the target server. Also, imapsync found 0 messages in all mailboxes on my evil to-be-eliminated server, something I didn't bother troubleshooting much. Timo's idea sounds interesting, time to look into 2.1 !
On 2012-01-26 8:11 AM, Gedalya gedalya@gedalya.net wrote:
As I understand, there is no way an IMAP-to-IAMP process can preserve UIDs, since new UIDs are assigned for every message by the target server. Also, imapsync found 0 messages in all mailboxes on my evil to-be-eliminated server, something I didn't bother troubleshooting much. Timo's idea sounds interesting, time to look into 2.1 !
Yep, it definitely sounds like the way to go...
--
Best regards,
Charles
On 26.1.2012, at 6.31, Gedalya wrote:
I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible.
The mailbox format I would want to use is Maildir++.
The storage format used by the current server is unknown, and I don't look forward to trying to reverse-engineer it. This leaves me with the option of reading the mailboxes using IMAP. There are tools like offlineimap or mbsync, and they do store the UID and UIDVALIDITY info. The last piece of the puzzle is a process to properly create the dovecot-uidlist and dovecot-uidvalidity files. So far I wasn't able to find anything on this. Are there any tips? Are there any tools available to do this job, or part of it?
Get Dovecot v2.1 and configure it to work. Then for migration add to dovecot.conf:
imapc_host = imap.example.com imapc_port = 993 imapc_ssl = imaps imapc_ssl_ca_dir = /etc/ssl/certs mail_prefetch_count = 50
And do the migration one user at a time:
doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:
On 01/26/2012 07:27 AM, Timo Sirainen wrote:
On 26.1.2012, at 6.31, Gedalya wrote:
I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible.
The mailbox format I would want to use is Maildir++.
The storage format used by the current server is unknown, and I don't look forward to trying to reverse-engineer it. This leaves me with the option of reading the mailboxes using IMAP. There are tools like offlineimap or mbsync, and they do store the UID and UIDVALIDITY info. The last piece of the puzzle is a process to properly create the dovecot-uidlist and dovecot-uidvalidity files. So far I wasn't able to find anything on this. Are there any tips? Are there any tools available to do this job, or part of it? Get Dovecot v2.1 and configure it to work. Then for migration add to dovecot.conf:
imapc_host = imap.example.com imapc_port = 993 imapc_ssl = imaps imapc_ssl_ca_dir = /etc/ssl/certs mail_prefetch_count = 50
And do the migration one user at a time:
doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:
Still working on it on my side, but for now:
# doveadm -o imapc_user=gedalya@thisdomain.com -o imapc_password=***** backup -u gedalya@thisdomain.com -R imapc: Segmentation fault
syslog:
Jan 26 18:34:29 imap01 kernel: [ 9055.766548] doveadm[8015]: segfault at 4 ip b7765752 sp bff90600 error 4 in libdovecot-storage.so.0.0.0[b769a000+ff000] Jan 26 18:34:53 imap01 kernel: [ 9078.883024] doveadm[8046]: segfault at 4 ip b7828752 sp bf964450 error 4 in libdovecot-storage.so.0.0.0[b775d000+ff000]
(I tried twice)
Also, I happen to have no idea what I'm doing, but still, segfault..
This is a debian testing "wheezy" machine I put up to do the initial playing around, i386, using Dovecot prebuilt binary packages from http://xi.rename-it.nl/debian/pool/testing-auto/dovecot-2.1/
On 27.1.2012, at 1.42, Gedalya wrote:
doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:
Still working on it on my side, but for now:
# doveadm -o imapc_user=gedalya@thisdomain.com -o imapc_password=***** backup -u gedalya@thisdomain.com -R imapc: Segmentation fault
gdb backtrace would be helpful. You should be able to get that by running (as root):
gdb --args doveadm ... bt full
(assuming you haven't changed base_dir, otherwise it might fail)
On 01/26/2012 06:46 PM, Timo Sirainen wrote:
On 27.1.2012, at 1.42, Gedalya wrote:
doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:
Still working on it on my side, but for now:
# doveadm -o imapc_user=gedalya@thisdomain.com -o imapc_password=***** backup -u gedalya@thisdomain.com -R imapc: Segmentation fault gdb backtrace would be helpful. You should be able to get that by running (as root):
gdb --args doveadm ... bt full
(assuming you haven't changed base_dir, otherwise it might fail)
Does this help?
GNU gdb (GDB) 7.3-debian Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/bin/doveadm...Reading symbols from /usr/lib/debug/usr/bin/doveadm...done. done. (gdb) run Starting program: /usr/bin/doveadm -o imapc_user=jedi@example.com -o imapc_password=**** backup -u jedi@example.com -R imapc: [Thread debugging using libthread_db enabled]
Program received signal SIGSEGV, Segmentation fault.
mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213
213 mailbox-log.c: No such file or directory.
in mailbox-log.c
(gdb) bt full
#0 mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213
No locals.
#1 0xb7fa7dd4 in mailbox_log_iter_init (log=0x0) at mailbox-log.c:239
iter = 0x80cbd90
#2 0x0806ffd3 in dsync_worker_get_list_mailbox_log (list=0x80b6180,
worker=0x80c3138) at dsync-worker-local.c:316
log = <optimized out>
iter = 0x8
rec = <optimized out>
#3 dsync_worker_get_mailbox_log (worker=0x80c3138) at
dsync-worker-local.c:386
ns = 0x80a5f90
ret = <optimized out>
#4 0x0807032f in dsync_worker_get_mailbox_log (worker=0x80c3138) at
dsync-worker-local.c:372
No locals.
#5 local_worker_mailbox_iter_init (_worker=0x80c3138) at
dsync-worker-local.c:410
worker = 0x80c3138
iter = 0x80b6920
patterns = {0x8076124 "*", 0x0}
#6 0x08065a2f in dsync_brain_mailbox_list_init (brain=0x80b68e8,
worker=0x80c3138) at dsync-brain.c:141
list = 0x80c5940
pool = 0x80c5930
#7 0x0806680f in dsync_brain_sync (brain=0x80b68e8) at dsync-brain.c:827
No locals.
#8 dsync_brain_sync (brain=0x80b68e8) at dsync-brain.c:813
No locals.
#9 0x08067038 in dsync_brain_sync_all (brain=0x80b68e8) at
dsync-brain.c:895
old_state = DSYNC_STATE_GET_MAILBOXES
__FUNCTION__ = "dsync_brain_sync_all"
#10 0x08064cfd in cmd_dsync_run (_ctx=0x8098ec0, user=0x80a9e98) at
doveadm-dsync.c:237
ctx = 0x8098ec0
worker1 = 0x80c3138
worker2 = 0x80aedb8
workertmp = <optimized out>
brain = 0x80b68e8
#11 0x0805371e in doveadm_mail_next_user (error_r=0xbffffa1c,
ctx=0x8098ec0, input=<optimized out>) at doveadm-mail.c:221
ret = <optimized out>
#12 doveadm_mail_next_user (ctx=0x8098ec0, input=<optimized out>,
error_r=0xbffffa1c) at doveadm-mail.c:187
error = <optimized out>
ret = <optimized out>
#13 0x08053b2e in doveadm_mail_single_user (ctx=0x8098ec0,
input=0xbffffa6c) at doveadm-mail.c:242
---Type <return> to continue, or q <return> to quit---
error = 0x0
ret = <optimized out>
__FUNCTION__ = "doveadm_mail_single_user"
#14 0x08053f58 in doveadm_mail_cmd (cmd=0x8096f60, argc=<optimized out>,
argv=0x80901e4) at doveadm-mail.c:425
input = {module = 0x0, service = 0x8076b3a "doveadm", username
= 0x8090242 "jedi@example.com", local_ip = {family = 0, u = {
ip6 = {__in6_u = {__u6_addr8 = '\000'
On 27.1.2012, at 2.00, Gedalya wrote:
# doveadm -o imapc_user=gedalya@thisdomain.com -o imapc_password=***** backup -u gedalya@thisdomain.com -R imapc: Segmentation fault gdb backtrace would be helpful. You should be able to get that by running (as root):
213 mailbox-log.c: No such file or directory. in mailbox-log.c (gdb) bt full #0 mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 No locals. #1 0xb7fa7dd4 in mailbox_log_iter_init (log=0x0) at mailbox-log.c:239 iter = 0x80cbd90 #2 0x0806ffd3 in dsync_worker_get_list_mailbox_log (list=0x80b6180, worker=0x80c3138) at dsync-worker-local.c:316
Ah, right, dsync really wants index files. Of course it shouldn't crash, I'll fix that, but you should be able to work around it:
rm -rf /tmp/imapc doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:/tmp/imapc
On 01/26/2012 07:06 PM, Timo Sirainen wrote:
On 27.1.2012, at 2.00, Gedalya wrote:
# doveadm -o imapc_user=gedalya@thisdomain.com -o imapc_password=***** backup -u gedalya@thisdomain.com -R imapc: Segmentation fault gdb backtrace would be helpful. You should be able to get that by running (as root):
213 mailbox-log.c: No such file or directory. in mailbox-log.c (gdb) bt full #0 mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 No locals. #1 0xb7fa7dd4 in mailbox_log_iter_init (log=0x0) at mailbox-log.c:239 iter = 0x80cbd90 #2 0x0806ffd3 in dsync_worker_get_list_mailbox_log (list=0x80b6180, worker=0x80c3138) at dsync-worker-local.c:316 Ah, right, dsync really wants index files. Of course it shouldn't crash, I'll fix that, but you should be able to work around it:
rm -rf /tmp/imapc doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:/tmp/imapc
# doveadm -o imapc_user=jedi@example.com -o imapc_password=***** backup -u jedi@example.com -R imapc:/tmp/imapc dsync(jedi@example.com): Error: Failed to sync mailbox Drafts: STATUS cannot access mailbox Drafts dsync(jedi@example.com): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not.
To be clear, I am trying to pull all the mailboxes from the old server on to this dovecot server, which has no mailboxes populated yet. It looks like this command would be pushing the messages from here to the imapc_host rather than pulling?
On 01/26/2012 07:17 PM, Gedalya wrote:
On 01/26/2012 07:06 PM, Timo Sirainen wrote:
On 27.1.2012, at 2.00, Gedalya wrote:
# doveadm -o imapc_user=gedalya@thisdomain.com -o imapc_password=***** backup -u gedalya@thisdomain.com -R imapc: Segmentation fault gdb backtrace would be helpful. You should be able to get that by running (as root):
213 mailbox-log.c: No such file or directory. in mailbox-log.c (gdb) bt full #0 mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 No locals. #1 0xb7fa7dd4 in mailbox_log_iter_init (log=0x0) at mailbox-log.c:239 iter = 0x80cbd90 #2 0x0806ffd3 in dsync_worker_get_list_mailbox_log (list=0x80b6180, worker=0x80c3138) at dsync-worker-local.c:316 Ah, right, dsync really wants index files. Of course it shouldn't crash, I'll fix that, but you should be able to work around it:
rm -rf /tmp/imapc doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:/tmp/imapc
# doveadm -o imapc_user=jedi@example.com -o imapc_password=***** backup -u jedi@example.com -R imapc:/tmp/imapc dsync(jedi@example.com): Error: Failed to sync mailbox Drafts: STATUS cannot access mailbox Drafts dsync(jedi@example.com): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not.
To be clear, I am trying to pull all the mailboxes from the old server on to this dovecot server, which has no mailboxes populated yet. It looks like this command would be pushing the messages from here to the imapc_host rather than pulling?
This got me somewhere...
# doveadm -o imapc_user=jedi1@example.com -o imapc_password=***** import -u jedi1@example.com imapc: "" all doveadm(jedi1@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=2 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=3 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=4 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=5 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=6 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=7 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=8 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=9 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=10 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=11 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=12 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=13 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=14 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=15 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=16 failed: Message GUID not available in this server (guid) doveadm(jedi1@example.com): Error: Copying box=INBOX uid=17 failed: Message GUID not available in this server (guid)
Should I / how can I disable this message GUID thing?
On 27.1.2012, at 2.33, Gedalya wrote:
# doveadm -o imapc_user=jedi@example.com -o imapc_password=***** backup -u jedi@example.com -R imapc:/tmp/imapc dsync(jedi@example.com): Error: Failed to sync mailbox Drafts: STATUS cannot access mailbox Drafts
Apparently your server doesn't like sending STATUS command to Drafts mailbox and returns a failure. This isn't very nice from it.
dsync(jedi@example.com): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not.
The -R parameter reversed the direction. It possibly fails because of the STATUS error. Or maybe some other problem, I'd need to look into it. You could try giving "-m INBOX" parameter to see if it works for one mailbox.
This got me somewhere...
# doveadm -o imapc_user=jedi1@example.com -o imapc_password=***** import -u jedi1@example.com imapc: "" all doveadm(jedi1@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid)
Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61
But doveadm import doesn't preserve UIDs.
On 01/26/2012 07:45 PM, Timo Sirainen wrote:
On 27.1.2012, at 2.33, Gedalya wrote:
# doveadm -o imapc_user=jedi@example.com -o imapc_password=***** backup -u jedi@example.com -R imapc:/tmp/imapc dsync(jedi@example.com): Error: Failed to sync mailbox Drafts: STATUS cannot access mailbox Drafts Apparently your server doesn't like sending STATUS command to Drafts mailbox and returns a failure. This isn't very nice from it.
This particular is broken - I'm pretty sure it doesn't do this for other accounts.
dsync(jedi@example.com): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not. The -R parameter reversed the direction. It possibly fails because of the STATUS error. Or maybe some other problem, I'd need to look into it. You could try giving "-m INBOX" parameter to see if it works for one mailbox. Must be that broken account. This got me somewhere...
# doveadm -o imapc_user=jedi1@example.com -o imapc_password=***** import -u jedi1@example.com imapc: "" all doveadm(jedi1@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61
But doveadm import doesn't preserve UIDs. OK - I got a different error from running doveadm backup on a non-broken account - see my other email :)
On 27.1.2012, at 2.57, Gedalya wrote:
# doveadm -o imapc_user=jedi1@example.com -o imapc_password=***** import -u jedi1@example.com imapc: "" all doveadm(jedi1@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61
But doveadm import doesn't preserve UIDs. OK - I got a different error from running doveadm backup on a non-broken account - see my other email :)
The GUID error is the same. The crash is probably the result of it. Try if upgrading fixes it.
On 01/26/2012 08:00 PM, Timo Sirainen wrote:
On 27.1.2012, at 2.57, Gedalya wrote:
# doveadm -o imapc_user=jedi1@example.com -o imapc_password=***** import -u jedi1@example.com imapc: "" all doveadm(jedi1@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61
But doveadm import doesn't preserve UIDs. OK - I got a different error from running doveadm backup on a non-broken account - see my other email :) The GUID error is the same. The crash is probably the result of it. Try if upgrading fixes it.
OK. Thank you very very much for everything so far. I'm going to wait for the changes to pop up in the prebuilt binary repository - I assume it's a matter of hours? For now I need to go eat something :-) and get back to this later, I'll post the results at that time.
On 01/26/2012 08:00 PM, Timo Sirainen wrote:
On 27.1.2012, at 2.57, Gedalya wrote:
# doveadm -o imapc_user=jedi1@example.com -o imapc_password=***** import -u jedi1@example.com imapc: "" all doveadm(jedi1@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61
But doveadm import doesn't preserve UIDs. OK - I got a different error from running doveadm backup on a non-broken account - see my other email :) The GUID error is the same. The crash is probably the result of it. Try if upgrading fixes it.
Yeap. Worked impeccably (doveadm backup)!! Pretty fast, too. Very impressed! I'll have to do some very thorough testing with various clients etc, will post interesting findings if any come up.
On 01/26/2012 08:00 PM, Timo Sirainen wrote:
On 27.1.2012, at 2.57, Gedalya wrote:
# doveadm -o imapc_user=jedi1@example.com -o imapc_password=***** import -u jedi1@example.com imapc: "" all doveadm(jedi1@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61
But doveadm import doesn't preserve UIDs. OK - I got a different error from running doveadm backup on a non-broken account - see my other email :) The GUID error is the same. The crash is probably the result of it. Try if upgrading fixes it.
This is what I ended up doing. I have the production machine acting as a dovecot imap server, and as a proxy for accounts not yet migrated. Running dovecot 2.0.15, with directly attached 6 TB of storage.
Per Timo's instructions, I set up a quick VM running debian wheezy and the latest dovecot 2.1, and copied the config from the production server with tiny modifications, and connected it to the same mysql user database. I gave this machine the same hostname as the production machine, just so that the maildir filenames end up looking neat. I don't know if this has anything more than psychological value :-)
I mounted the storage from the production machine (sshfs surprisingly didn't seem slower than NFS) and set up dovecot 2.1 to find the mailboxes under there, then things like
doveadm -o imapc_user=jedi1@example.com -o imapc_password=****** backup -u jedi1@example.com -R imapc:/tmp/imapc
started doing the job. No output, no problems.
So far the only glitch I noticed is that I have dovecot autocreate a Spam folder and when starting a Windows Live Mail which was reading a proxied account, after it was migrated and served by dovecot, it doesn't find the Spam folder until I click "Download all folders". We have thousands of mailboxes being read from every conceivable client, so there will be more tiny issues like this. Can't wait to test a blackberry.
Other than that, things work as intended - UID and UIDVALIDITY seem to be preserved, the clients don't seem to notice the migration or react to it in any way.
What's left is to wrap around this a proper process to lock the mailbox, essentially put the right things in the database in the beginning and in the end of the process. Looks beautiful.
On 01/26/2012 07:17 PM, Gedalya wrote:
On 01/26/2012 07:06 PM, Timo Sirainen wrote:
On 27.1.2012, at 2.00, Gedalya wrote:
# doveadm -o imapc_user=gedalya@thisdomain.com -o imapc_password=***** backup -u gedalya@thisdomain.com -R imapc: Segmentation fault gdb backtrace would be helpful. You should be able to get that by running (as root):
213 mailbox-log.c: No such file or directory. in mailbox-log.c (gdb) bt full #0 mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 No locals. #1 0xb7fa7dd4 in mailbox_log_iter_init (log=0x0) at mailbox-log.c:239 iter = 0x80cbd90 #2 0x0806ffd3 in dsync_worker_get_list_mailbox_log (list=0x80b6180, worker=0x80c3138) at dsync-worker-local.c:316 Ah, right, dsync really wants index files. Of course it shouldn't crash, I'll fix that, but you should be able to work around it:
rm -rf /tmp/imapc doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:/tmp/imapc
# doveadm -o imapc_user=jedi@example.com -o imapc_password=***** backup -u jedi@example.com -R imapc:/tmp/imapc dsync(jedi@example.com): Error: Failed to sync mailbox Drafts: STATUS cannot access mailbox Drafts dsync(jedi@example.com): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not.
To be clear, I am trying to pull all the mailboxes from the old server on to this dovecot server, which has no mailboxes populated yet. It looks like this command would be pushing the messages from here to the imapc_host rather than pulling?
Sorry, my bad. That was a malfunction on the old IMAP server - that mailbox is inaccessible.
Tried with another account:
doveadm -o imapc_user=jedi1@example.com -o imapc_password=***** backup -u jedi1@example.com -R imapc:/tmp/imapc dsync(jedi1@example.com): Error: msg guid lookup failed: Message GUID not available in this server dsync(jedi1@example.com): Error: msg guid lookup failed: Message GUID not available in this server dsync(jedi1@example.com): Panic: file dsync-brain.c: line 901 (dsync_brain_sync_all): assertion failed: (brain->state != old_state) dsync(jedi1@example.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x3e98a) [0xb756a98a] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x41) [0xb756aa91] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb753f66b] -> doveadm() [0x8067095] -> doveadm() [0x8064cfd] -> doveadm() [0x805371e] -> doveadm(doveadm_mail_single_user+0x5e) [0x8053b2e] -> doveadm() [0x8053f58] -> doveadm(doveadm_mail_try_run+0x139) [0x80543d9] -> doveadm(main+0x3a7) [0x8053347] -> /lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0xb73e8e46] -> doveadm() [0x8053519] Aborted
So there :D
On 27.1.2012, at 2.00, Gedalya wrote:
Starting program: /usr/bin/doveadm -o imapc_user=jedi@example.com -o imapc_password=**** backup -u jedi@example.com -R imapc:
Program received signal SIGSEGV, Segmentation fault. mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 213 mailbox-log.c: No such file or directory. in mailbox-log.c
This crash is now fixed, so there's no need to give /tmp/imapc path anymore: http://hg.dovecot.org/dovecot-2.1/rev/7b94d1c8a6e7
On 01/28/2012 12:45 PM, Timo Sirainen wrote:
On 27.1.2012, at 2.00, Gedalya wrote:
Starting program: /usr/bin/doveadm -o imapc_user=jedi@example.com -o imapc_password=**** backup -u jedi@example.com -R imapc:
Program received signal SIGSEGV, Segmentation fault. mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 213 mailbox-log.c: No such file or directory. in mailbox-log.c This crash is now fixed, so there's no need to give /tmp/imapc path anymore: http://hg.dovecot.org/dovecot-2.1/rev/7b94d1c8a6e7
Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current versions are putting the body of the last message in "Sent Items" in place of every single email in INBOX. In other words, for every email that sits in INBOX in the source, I get a copy of the last email in "Sent Items" instead. This happens for every account I try to migrate. Very strange. I noticed this only now, and the last package I have left in the local apt cache which still works is 2.1.rc7-0~auto+0.
Am 16.3.2011 20:59, schrieb Gedalya:
Starting program: /usr/bin/doveadm -o imapc_user=jedi at example.com -o imapc_password=**** backup -u jedi at example.com -R imapc:
Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current versions are putting the body of the last message in "Sent Items" in place of every single email in INBOX. In other words, for every email that sits in INBOX in the source, I get a copy of the last email in "Sent Items" instead. This happens for every account I try to migrate. Very strange. I noticed this only now, and the last package I have left in the local apt cache which still works is 2.1.rc7-0~auto+0.
i see the same regression (2.1.3-0~auto+4) :(
doveadm sync/backup via impac puts the same message all over the place...
On 3/21/2012 10:17 PM, Martin Schitter wrote:
Am 16.3.2011 20:59, schrieb Gedalya:
Starting program: /usr/bin/doveadm -o imapc_user=jedi at example.com -o imapc_password=**** backup -u jedi at example.com -R imapc:
Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current versions are putting the body of the last message in "Sent Items" in place of every single email in INBOX. In other words, for every email that sits in INBOX in the source, I get a copy of the last email in "Sent Items" instead. This happens for every account I try to migrate. Very strange. I noticed this only now, and the last package I have left in the local apt cache which still works is 2.1.rc7-0~auto+0.
i see the same regression (2.1.3-0~auto+4) :(
doveadm sync/backup via impac puts the same message all over the place...
Thanks Martin, I've set up a test platform to investigate this further but I've been short on time...
Am 2012-03-22 03:46, schrieb Gedalya:
doveadm sync/backup via impac puts the same message all over the place...
Thanks Martin, I've set up a test platform to investigate this further but I've been short on time...
after some debugging a few more remarks about this problem:
the bug only appears on recursive folder hierarchies. if you specity option "-m INBOX" everything works fine.
for recursive hierarchies the rawlog (-o imapc_rawlog_dir=...) shows that "UID FETCH 1:* FLAGS" will be called for all folders but "UID FETCH NNN (INTERNALDATE)" and "UID FETCH NNN (BODY.PEEK[])" only happens for the messages in first found subfolder! the last message in this folder will substitute all other messages on the target side... :(
has anyone a clue how to fix this problem in the source code?
On 22.3.2012, at 19.09, Martin Schitter wrote:
Am 2012-03-22 03:46, schrieb Gedalya:
doveadm sync/backup via impac puts the same message all over the place...
Thanks Martin, I've set up a test platform to investigate this further but I've been short on time...
after some debugging a few more remarks about this problem:
the bug only appears on recursive folder hierarchies. if you specity option "-m INBOX" everything works fine.
for recursive hierarchies the rawlog (-o imapc_rawlog_dir=...) shows that "UID FETCH 1:* FLAGS" will be called for all folders but "UID FETCH NNN (INTERNALDATE)" and "UID FETCH NNN (BODY.PEEK[])" only happens for the messages in first found subfolder! the last message in this folder will substitute all other messages on the target side... :(
has anyone a clue how to fix this problem in the source code?
http://hg.dovecot.org/dovecot-2.1/rev/078697a32109 should fix it.
Am 2012-03-30 02:07, schrieb Timo Sirainen:
has anyone a clue how to fix this problem in the source code?
http://hg.dovecot.org/dovecot-2.1/rev/078697a32109 should fix it.
thanks! -- now it works! :)
On 01/26/2012 07:27 AM, Timo Sirainen wrote:
On 26.1.2012, at 6.31, Gedalya wrote:
I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible.
The mailbox format I would want to use is Maildir++.
The storage format used by the current server is unknown, and I don't look forward to trying to reverse-engineer it. This leaves me with the option of reading the mailboxes using IMAP. There are tools like offlineimap or mbsync, and they do store the UID and UIDVALIDITY info. The last piece of the puzzle is a process to properly create the dovecot-uidlist and dovecot-uidvalidity files. So far I wasn't able to find anything on this. Are there any tips? Are there any tools available to do this job, or part of it? Get Dovecot v2.1 and configure it to work. Then for migration add to dovecot.conf:
imapc_host = imap.example.com imapc_port = 993 imapc_ssl = imaps imapc_ssl_ca_dir = /etc/ssl/certs mail_prefetch_count = 50
And do the migration one user at a time:
doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:
Now, to the issue of POP3.
The old system uses the message filename for UIDL, but we need to migrate via IMAP in order to preserve IMAP info and UIDs (which have nothing to do with the POP3 UIDL in this case). So I've just finished writing a script to insert X-UIDL headers, and pop3_reuse_xuidl is doing the job.
Question: Since the system currently serves in excess of 10 pop3 connections per second, would there be any performance gain from using pop3_save_uidl? Would it be faster or slower to fetch the UIDL list from the uidlist rather than look up the X-UIDL in the index? Just wondering.
Also, what order does dovecot return the UIDLs in?
On 30.1.2012, at 5.29, Gedalya wrote:
The old system uses the message filename for UIDL, but we need to migrate via IMAP in order to preserve IMAP info and UIDs (which have nothing to do with the POP3 UIDL in this case). So I've just finished writing a script to insert X-UIDL headers, and pop3_reuse_xuidl is doing the job.
You could also store the UIDL to dovecot-uidlist file directly without keeping it in header.
Question: Since the system currently serves in excess of 10 pop3 connections per second, would there be any performance gain from using pop3_save_uidl? Would it be faster or slower to fetch the UIDL list from the uidlist rather than look up the X-UIDL in the index? Just wondering.
Currently with pop3_reuse_xuidl=yes setting the X-UIDL header is cached to dovecot.index.cache file. With pop3_save_uidl=yes I doubt you'd get much of a performance improvement, but if it enables you to later set pop3_reuse_xuidl=no you can delete all of the dovecot.index.cache files for pop3-only users, since they don't really need it.
Also, what order does dovecot return the UIDLs in?
The messages are in the same order as in dovecot-uidlist. That of course isn't necessarily the same ordering as in your POP3 server. If you directly add the UIDLs to dovecot-uidlist file in migration script you can also change the UIDL ordering in v2.0.13+:
+ pop3: Added support for showing messages in "POP3 order", which can
be different from IMAP message order. This can be useful for
migrations from other servers. Implemented it for Maildir as 'O'
field in dovecot-uidlist.On 02/01/2012 03:46 PM, Timo Sirainen wrote:
On 30.1.2012, at 5.29, Gedalya wrote:
The old system uses the message filename for UIDL, but we need to migrate via IMAP in order to preserve IMAP info and UIDs (which have nothing to do with the POP3 UIDL in this case). So I've just finished writing a script to insert X-UIDL headers, and pop3_reuse_xuidl is doing the job. You could also store the UIDL to dovecot-uidlist file directly without keeping it in header.
Question: Since the system currently serves in excess of 10 pop3 connections per second, would there be any performance gain from using pop3_save_uidl? Would it be faster or slower to fetch the UIDL list from the uidlist rather than look up the X-UIDL in the index? Just wondering. Currently with pop3_reuse_xuidl=yes setting the X-UIDL header is cached to dovecot.index.cache file. With pop3_save_uidl=yes I doubt you'd get much of a performance improvement, but if it enables you to later set pop3_reuse_xuidl=no you can delete all of the dovecot.index.cache files for pop3-only users, since they don't really need it.
Also, what order does dovecot return the UIDLs in? The messages are in the same order as in dovecot-uidlist. That of course isn't necessarily the same ordering as in your POP3 server. If you directly add the UIDLs to dovecot-uidlist file in migration script you can also change the UIDL ordering in v2.0.13+:
- pop3: Added support for showing messages in "POP3 order", which can be different from IMAP message order. This can be useful for migrations from other servers. Implemented it for Maildir as 'O' field in dovecot-uidlist.
Thank you so much! This clears up a lot of fine details.
On Thursday 26 January 2012 04:31:20 Gedalya wrote:
I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible.
Ignoring the migration of individual mailboxes addressed in other replies, I trust you've met Perdition - very useful for this sort of situation, http://horms.net/projects/perdition/
to provide an IMAP "Server" (actually proxy) that knows where the real mailboxes are located, and directs connections accordingly. That way you can migrate mailboxes one-by-one as you've migrated them, helpful to test a few mailboxes first without affecting the bulk of users' mailboxes atall.
cheers,
Andrew.
On 01/26/2012 07:38 AM, Andrew Richards wrote:
I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible. Ignoring the migration of individual mailboxes addressed in other replies, I
On Thursday 26 January 2012 04:31:20 Gedalya wrote: trust you've met Perdition - very useful for this sort of situation, http://horms.net/projects/perdition/
to provide an IMAP "Server" (actually proxy) that knows where the real mailboxes are located, and directs connections accordingly. That way you can migrate mailboxes one-by-one as you've migrated them, helpful to test a few mailboxes first without affecting the bulk of users' mailboxes atall.
cheers,
Andrew. Sounds very cool. I already have dovecot set up as a proxy, working, and it should allow me to forcefully disconnect users and lock them out while they are being migrated and then once they are done they'll be served locally rather than proxied. My main problem is that most connections are simply coming directly to the old server, using the deprecated hostname. I need all clients to use the right hostnames, or clog up this new server with redirectors and proxies for all the junk done on the old server.. bummer.
What I might want to look into is actually setting up a proxy like this but on the evil (windows) server - to get *him* to pass on those requests he shouldn't be handling.
Hi
Sounds very cool. I already have dovecot set up as a proxy, working, and it should allow me to forcefully disconnect users and lock them out while they are being migrated and then once they are done they'll be served locally rather than proxied. My main problem is that most connections are simply coming directly to the old server, using the deprecated hostname. I need all clients to use the right hostnames, or clog up this new server with redirectors and proxies for all the junk done on the old server.. bummer.
Why not put the old server IP to redirect to the new machine, then give the old machine some new temp IP in order to proxy back to it? That way you can do the proxying on the dovecot machine, which as you already established is working ok?
Good luck
Ed W
On 01/26/2012 11:02 AM, Ed W wrote:
Hi
Sounds very cool. I already have dovecot set up as a proxy, working, and it should allow me to forcefully disconnect users and lock them out while they are being migrated and then once they are done they'll be served locally rather than proxied. My main problem is that most connections are simply coming directly to the old server, using the deprecated hostname. I need all clients to use the right hostnames, or clog up this new server with redirectors and proxies for all the junk done on the old server.. bummer.
Why not put the old server IP to redirect to the new machine, then give the old machine some new temp IP in order to proxy back to it?
That way you can do the proxying on the dovecot machine, which as you already established is working ok?Good luck
Ed W
Yeap, taht's what I'm doing to do, except that I would have to proxy more than just IMAP and POP - it's a one-does-it-all kind of machine accepting mail delivered from the outside, relaying outgoing mail, does webmail, does all this things very poorly... I have the choice of forcing all users to change to the new, dedicated servers doing these things, or reimplementing / porxying all of this on my new dovecot server which I so desperately want to keep neat and tidy...
Hi
Yeap, taht's what I'm doing to do, except that I would have to proxy more than just IMAP and POP - it's a one-does-it-all kind of machine accepting mail delivered from the outside, relaying outgoing mail, does webmail, does all this things very poorly... I have the choice of forcing all users to change to the new, dedicated servers doing these things, or reimplementing / porxying all of this on my new dovecot server which I so desperately want to keep neat and tidy...
In that case I would suggest perhaps that the IP is taken over by a dedicated firewall box (running the OS of your choice). The firewall could then be used to port forward the services to the individual machines responsible for each service. This would give you the benefit that you could easily move other services off/around
We are clearly off topic to dovecot...
Plenty of good firewall options. If you want small, compact and low power, then you can pickup a bunch off intel compatible boards around the low couple hundred £s mark fairly easily. Run your favourite distro and firewall on them. If you hadn't seen them before, I quite like Lanner for appliances, eg: http://www.lannerinc.com/x86_Network_Appliances/x86_Desktop_Appliances
For example if you added a small appliance running linux which runs that IP, then you could add intrusion detection, bounce the web traffic to the windows box (or even just certain URLs, other URLs could go to some hypothetical linux box, etc), port forwarding the mail to the new dovecot box, etc, etc. Incremental price would be surprisingly low, but lots of extra flexibility?
Just a thought
Good luck
Ed W
On Wed, 25 Jan 2012 23:31:20 -0500, Gedalya wrote:
I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible.
setup dovecot and make it listen on 127.0.0.2 only, modify your current to only listen on 127.0.0.1 this so you now can have 2 imap servers running at the same time
next step is here
http://www.howtoforge.com/how-to-migrate-mailboxes-between-imap-servers-with...
when all accounts is transfered, stop the old server, make dovecot listen on any ip, done, it worked for me when i changed from courier-imap to dovecot
participants (7)
-
Andrew Richards
-
Benny Pedersen
-
Charles Marcus
-
Ed W
-
Gedalya
-
Martin Schitter
-
Timo Sirainen