Help with migration to Dovecot from raw files
Hello list,
I recently migrated to Dovecot from a very old version of IceWarp running on failing hardware running Windows XP. I had a lot of issues downloading mail through imap. What I have instead are the email files directly off the harddrive from IceWarp's mail store.
I rearranged the mail files to look like a Dovecot maildir and tried to use these emails directly. Dovecot did not complain, but I ran into my first problem. The emails were out of order on the client side. None of my imap clients store email. They all just fetch headers and download mail as it's required. So a resync is not a problem. I just can't figure out how and I don't know if the filenames are a problem.
The files I'm working with have this format:
2020032512105639B0-000001000000D2D45079615C.imap
The first part is the date and all the the filenames appear to be unique. Is it okay to leave them this way or should I rename them in the same fashion as Dovecot? If so, is there a tool I can use or do I need to write a script? If I have to write a script, could someone help me decipher this filename:
1585197800.M440284P10160.hostname,S=2812,W=2875:2,S
I understand the first bit is a unix timestamp. I couldn't find out what M440284P10160 means or how I might create that number myself to use in a renaming script. Additionally, I'd like to customise the hostname without changing the machine's hostname and I understand that the S and W can also be omitted. I'm not using quotas. I couldn't find anywhere in the documentation where it talked about any configuration options for the filenames. Are there any or do I have to patch the source?
I was also planning on directly backing up the email using rsync, but the changing filenames with flags is obviously an issue. I understand that dovecot can store flags in a file instead, if the mailbox is shared or public. Is there any way to have this behaviour with a private mailbox? Ideally the filenames are not touched after creation in which case rsync will work nicely to my NAS.
The last issue is that since Dovecot went live, I've started receiving mail. For the time being, I have left my "yet to be migrated" mail outside of maildir. So Dovecot doesn't know about it. I will need to merge this newly received mail with my "yet to migrated mail" once I sort out exactly how to migrate that mail. And I would really like all the mail to sort properly.
From what I can tell doveadm is the tool for the job, but I just can't make sense of the documentation.
Thanks, Stef
Dovecot version: 2.3.4.1 (f79e8e7e4)
You could try something simple like just
for n in *; do cat $n | doveadm save -u targetuser; done
or if the IceWarp has IMAP interface, you could try
doveadm -o imapc_host=hostname -o imapc_user=username -o imapc_password=password sync -1R targetuser imapc:
Aki
On 26/03/2020 12:11 Stefano Antonelli <dovecot@intheworks.ca> wrote:
Hello list,
I recently migrated to Dovecot from a very old version of IceWarp running on failing hardware running Windows XP. I had a lot of issues downloading mail through imap. What I have instead are the email files directly off the harddrive from IceWarp's mail store.
I rearranged the mail files to look like a Dovecot maildir and tried to use these emails directly. Dovecot did not complain, but I ran into my first problem. The emails were out of order on the client side. None of my imap clients store email. They all just fetch headers and download mail as it's required. So a resync is not a problem. I just can't figure out how and I don't know if the filenames are a problem.
The files I'm working with have this format:
2020032512105639B0-000001000000D2D45079615C.imap
The first part is the date and all the the filenames appear to be unique. Is it okay to leave them this way or should I rename them in the same fashion as Dovecot? If so, is there a tool I can use or do I need to write a script? If I have to write a script, could someone help me decipher this filename:
1585197800.M440284P10160.hostname,S=2812,W=2875:2,S
I understand the first bit is a unix timestamp. I couldn't find out what M440284P10160 means or how I might create that number myself to use in a renaming script. Additionally, I'd like to customise the hostname without changing the machine's hostname and I understand that the S and W can also be omitted. I'm not using quotas. I couldn't find anywhere in the documentation where it talked about any configuration options for the filenames. Are there any or do I have to patch the source?
I was also planning on directly backing up the email using rsync, but the changing filenames with flags is obviously an issue. I understand that dovecot can store flags in a file instead, if the mailbox is shared or public. Is there any way to have this behaviour with a private mailbox? Ideally the filenames are not touched after creation in which case rsync will work nicely to my NAS.
The last issue is that since Dovecot went live, I've started receiving mail. For the time being, I have left my "yet to be migrated" mail outside of maildir. So Dovecot doesn't know about it. I will need to merge this newly received mail with my "yet to migrated mail" once I sort out exactly how to migrate that mail. And I would really like all the mail to sort properly.
From what I can tell doveadm is the tool for the job, but I just can't make sense of the documentation.
Thanks, Stef
Dovecot version: 2.3.4.1 (f79e8e7e4)
On 2020-03-26 03:37, Aki Tuomi wrote:
You could try something simple like just
for n in *; do cat $n | doveadm save -u targetuser; done
Thank you. This "works" in that it renames the files for me. However,
- the filename timestamps are all similar (ie. time of import)
- the emails showed up in new rather than cur
- the emails are unseen.
The seen state can probably be fixed after the fact using doveadm flags, but there doesn't appear to be a way to do it with the doveadm save command.
I'd also like the filenames to reflect their original timestamps, not the time of import. Is that possible using doveadm?
or if the IceWarp has IMAP interface, you could try
doveadm -o imapc_host=hostname -o imapc_user=username -o imapc_password=password sync -1R targetuser imapc:
The server is down for good. All I have are files.
Thanks, Stef
On 2020-03-26 12:03, Stefano Antonelli wrote:
On 2020-03-26 03:37, Aki Tuomi wrote:
You could try something simple like just
for n in *; do cat $n | doveadm save -u targetuser; done
Thank you. This "works" in that it renames the files for me. However,
- the filename timestamps are all similar (ie. time of import)
- the emails showed up in new rather than cur
- the emails are unseen.
As mentioned previously, I don't have access to the mail server any
more. I did manage to bring up a virtual machine and installed the
server software on to it and restored the mail to the virtual machine.
However, no imap backup tools (not even doveadm) could complete
operation without failing. In all cases, none of the tools preserved
the timestamp of the email anyway (not even with mbsync using
CopyArrivalDate).
So I went back to working with the raw .imap files. I could not coerce doveadm into listing the uids in any sane order. I thought this might have something to do with the filename timestamp or mtime on the file itself. So I set out to decode the .imap filenames and convert them to proper maildir filenames. Of course, this was only possible with a server I could query. Then I wrote a script that renamed files as best as I could from the old mail server:
2012122513472903CA-000001020000000141996DEF.imap
to maildir files:
1356472049.M000000P12345.icewarp,S=10354,W=10354:2,S
I also set the m-time and a-time after copying into the maildir directory. Followed by:
chown -R mail:mail /path/to/maildir/user doveadm force-resync -u user@domain INBOX
At first this didn't work. dovecot-uidlist was not correct. So then I removed all the dovecot files and did a resync again:
rm /path/to/maildir/user/dovecot* doveadm force-resync -u user@domain INBOX
And this gave me the dovecot-uidlist I was looking for and mails displayed properly at the imap client.
I've attached my script in case it's helpful to anyone in a similar situation. It took quite a bit of effort to get this right, highly depends on the format of the original mailstore, and I've only tested this on one smaller mailbox (<400 messages). And I wasn't concerned with any client having to download everything again since this was a fresh import.
In any case, it worked for me.
Thanks, Stef
participants (2)
-
Aki Tuomi
-
Stefano Antonelli