[Dovecot] UW-IMAP to Dovecot conversion - How to migrate the folders?
Hi all,
I've read so much information on this that I'm a little confused as to what to follow. Here's what's going on:
We just migrated away from our AIX machines which were running Postfix and UW-IMAP. We're now running Postfix with Dovecot. I already converted everyone's mbox file to maildir, but now I'm having a little troubles with how to exactly populate their IMAP folders. People are saying they cannot see their folders. These 'folders' they are talking about can be found in their /home directory.
I have the following URLs I've been looking at: http://wiki.dovecot.org/Migration/MailFormat http://people.redhat.com/rkeech/maildir-migration.txt http://wiki.dovecot.org/Plugins/Convert http://www.dovecot.org/list/dovecot/2007-June/023757.html
What would be the suggested method to convert them? So far I understand that I need to somehow populate the /home/USERNAME/Maildir/subscriptions file with the items located in /home/USERNAME/.mailboxlist and then I don't know what to do from there. I don't know if I need to incorporate a Dovecot plugin, or if there is a script I can run to do this for me.
Since this is a live server, I would prefer to test on a single account before I run a plugin that would effect all users globally.
Any help would be very appreciated - thanks in advance!
Patrick
Patrick - South Valley Internet spake the following on 8/20/2007 3:21 PM:
Hi all,
I've read so much information on this that I'm a little confused as to what to follow. Here's what's going on:
We just migrated away from our AIX machines which were running Postfix and UW-IMAP. We're now running Postfix with Dovecot. I already converted everyone's mbox file to maildir, but now I'm having a little troubles with how to exactly populate their IMAP folders. People are saying they cannot see their folders. These 'folders' they are talking about can be found in their /home directory.
I have the following URLs I've been looking at: http://wiki.dovecot.org/Migration/MailFormat http://people.redhat.com/rkeech/maildir-migration.txt http://wiki.dovecot.org/Plugins/Convert http://www.dovecot.org/list/dovecot/2007-June/023757.html
What would be the suggested method to convert them? So far I understand that I need to somehow populate the /home/USERNAME/Maildir/subscriptions file with the items located in /home/USERNAME/.mailboxlist and then I don't know what to do from there. I don't know if I need to incorporate a Dovecot plugin, or if there is a script I can run to do this for me.
Since this is a live server, I would prefer to test on a single account before I run a plugin that would effect all users globally.
Any help would be very appreciated - thanks in advance!
Patrick
Have you tried to edit their subscriptions file? Or log in with a client to one of the accounts and use its subscribe feature? With the second method, you could get an idea of what the subscriptions layout is and use a copy / sed script to fix each one.
--
MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
I have, but it didn't do anything that I could tell.
I tried resyncing my IMAP but I didn't see the new folder.
Does it matter that the UW-IMAP folders are in mbox-like format? It appears each "folder" is a single file, similar to mbox format.
Thanks again.
Patrick
Scott Silva wrote:
Patrick - South Valley Internet spake the following on 8/20/2007 3:21 PM:
Hi all,
I've read so much information on this that I'm a little confused as to what to follow. Here's what's going on:
We just migrated away from our AIX machines which were running Postfix and UW-IMAP. We're now running Postfix with Dovecot. I already converted everyone's mbox file to maildir, but now I'm having a little troubles with how to exactly populate their IMAP folders. People are saying they cannot see their folders. These 'folders' they are talking about can be found in their /home directory.
I have the following URLs I've been looking at: http://wiki.dovecot.org/Migration/MailFormat http://people.redhat.com/rkeech/maildir-migration.txt http://wiki.dovecot.org/Plugins/Convert http://www.dovecot.org/list/dovecot/2007-June/023757.html
What would be the suggested method to convert them? So far I understand that I need to somehow populate the /home/USERNAME/Maildir/subscriptions file with the items located in /home/USERNAME/.mailboxlist and then I don't know what to do from there. I don't know if I need to incorporate a Dovecot plugin, or if there is a script I can run to do this for me.
Since this is a live server, I would prefer to test on a single account before I run a plugin that would effect all users globally.
Any help would be very appreciated - thanks in advance!
Patrick
Have you tried to edit their subscriptions file? Or log in with a client to one of the accounts and use its subscribe feature? With the second method, you could get an idea of what the subscriptions layout is and use a copy / sed script to fix each one.
On Monday, August 20 at 04:07 PM, quoth Patrick - South Valley Internet:
I have, but it didn't do anything that I could tell.
I tried resyncing my IMAP but I didn't see the new folder.
Does it matter that the UW-IMAP folders are in mbox-like format? It appears each "folder" is a single file, similar to mbox format.
Yeah, it does matter. Dovecot doesn't allow you to mix-and-match different mail storage formats very easily. You may be able to set something like that up with the Namespaces support in Dovecot, but the easiest thing to do is to convert your mbox files into Maildir.
~Kyle
Liberty means responsibility. That is why most men dread it. -- George Bernard Shaw
Kyle Wheeler wrote:
On Monday, August 20 at 04:07 PM, quoth Patrick - South Valley Internet:
I have, but it didn't do anything that I could tell.
I tried resyncing my IMAP but I didn't see the new folder.
Does it matter that the UW-IMAP folders are in mbox-like format? It appears each "folder" is a single file, similar to mbox format.
Yeah, it does matter. Dovecot doesn't allow you to mix-and-match different mail storage formats very easily. You may be able to set something like that up with the Namespaces support in Dovecot, but the easiest thing to do is to convert your mbox files into Maildir.
~Kyle
Thanks Kyle, but how do I convert the mbox-like IMAP folders into something Dovecot can read with the new config?
Patrick
On Tuesday, August 21 at 02:06 PM, quoth Patrick - South Valley Internet:
Thanks Kyle, but how do I convert the mbox-like IMAP folders into something Dovecot can read with the new config?
One way (the most straightforward) is to use any of the available mbox-to-maildir converter scripts. Search google for the text "mbox to maildir" and you should get references to a bunch of them.
You may be able to use Dovecot's "convert" plugin, but I've never played with it, so I don't know much about setting it up.
~Kyle
The only fool bigger than the person who knows it all is the person who argues with him. -- Stanislaw Jerszy Lec
On Wed, 22 Aug 2007 07:11:24 -0600 Kyle Wheeler wrote:
On Tuesday, August 21 at 02:06 PM, quoth Patrick - South Valley Internet:
Thanks Kyle, but how do I convert the mbox-like IMAP folders into something Dovecot can read with the new config?
One way (the most straightforward) is to use any of the available mbox-to-maildir converter scripts. Search google for the text "mbox to maildir" and you should get references to a bunch of them.
I had success using "perfect_maildir.pl", surf to http://perfectmaildir.home-dn.net/
--Frank Elsner
On Mon, Aug 20, 2007 at 03:21:20PM -0700, Patrick - South Valley Internet wrote:
We just migrated away from our AIX machines which were running Postfix and UW-IMAP. We're now running Postfix with Dovecot. I already converted everyone's mbox file to maildir, but now I'm having a little troubles with how to exactly populate their IMAP folders. People are saying they cannot see their folders. These 'folders' they are talking about can be found in their /home directory.
We just moved from HPUX to linux and there we had the problem, that with IMAP and UW-IMAPd you had to specify a Folder Path in the client. With Dovecot and Maildir you don't need this.
Check if your users have set this folder path. That solved a lot of problems here.
Philipp
-- If you have problems in Windows: REBOOT If you have problems in Linux: BE ROOT
Philipp Kolmann wrote:
On Mon, Aug 20, 2007 at 03:21:20PM -0700, Patrick - South Valley Internet wrote:
We just migrated away from our AIX machines which were running Postfix and UW-IMAP. We're now running Postfix with Dovecot. I already converted everyone's mbox file to maildir, but now I'm having a little troubles with how to exactly populate their IMAP folders. People are saying they cannot see their folders. These 'folders' they are talking about can be found in their /home directory.
We just moved from HPUX to linux and there we had the problem, that with IMAP and UW-IMAPd you had to specify a Folder Path in the client. With Dovecot and Maildir you don't need this.
Check if your users have set this folder path. That solved a lot of problems here.
Philipp
Hi Philipp,
Our users are using Outlook Express and Outlook. I just set up an IMAP account that is having the issue and I didn't specify any folders. I just used the defaults.
What's even more odd is that when I created a new folder within Outlook Express, I see it in /home/USERNAME/Maildir/subscriptions, but I don't see the folder anywhere...how does Dovecot see these IMAP folders? Is there some sort of tag that is contained within each Maildir email that tells what folder it goes in to?
Patrick
On Tuesday, August 21 at 02:15 PM, quoth Patrick - South Valley Internet:
What's even more odd is that when I created a new folder within Outlook Express, I see it in /home/USERNAME/Maildir/subscriptions, but I don't see the folder anywhere...how does Dovecot see these IMAP folders? Is there some sort of tag that is contained within each Maildir email that tells what folder it goes in to?
The new folders are created with names prefixed by a period.
For example, imagine you have the following IMAP folders (subfolders separated by a slash (/)):
INBOX
Sent
Trash
Drafts
INBOX/Business
INBOX/Family
These would typically be stored like so:
/home/USERNAME/Maildir/ <-- "INBOX"
/home/USERNAME/Maildir/.Sent/
/home/USERNAME/Maildir/.Trash/
/home/USERNAME/Maildir/.Drafts/
/home/USERNAME/Maildir/.INBOX.Business/
/home/USERNAME/Maildir/.INBOX.Family/
Doing a simple ls
in the Maildir won't show the other folders,
because they begin with a period (and are considered "hidden" by ls).
Do an ls -a
and they'll show up. That's standard Maildir++ layout,
which is what you get when your config says:
mail_location = maildir:%h/Maildir
~Kyle
It ain't the parts of the Bible that I can't understand that bother me, it is the parts that I do understand. -- Mark Twain
Kyle Wheeler wrote:
On Tuesday, August 21 at 02:15 PM, quoth Patrick - South Valley Internet:
What's even more odd is that when I created a new folder within Outlook Express, I see it in /home/USERNAME/Maildir/subscriptions, but I don't see the folder anywhere...how does Dovecot see these IMAP folders? Is there some sort of tag that is contained within each Maildir email that tells what folder it goes in to?
The new folders are created with names prefixed by a period.
For example, imagine you have the following IMAP folders (subfolders separated by a slash (/)):
INBOX Sent Trash Drafts INBOX/Business INBOX/Family
These would typically be stored like so:
/home/USERNAME/Maildir/ <-- "INBOX" /home/USERNAME/Maildir/.Sent/ /home/USERNAME/Maildir/.Trash/ /home/USERNAME/Maildir/.Drafts/ /home/USERNAME/Maildir/.INBOX.Business/ /home/USERNAME/Maildir/.INBOX.Family/
Doing a simple
ls
in the Maildir won't show the other folders, because they begin with a period (and are considered "hidden" by ls). Do anls -a
and they'll show up. That's standard Maildir++ layout, which is what you get when your config says:mail_location = maildir:%h/Maildir
~Kyle
Yes I finally figured that out yesterday. Once I figured that out, everything else fell into place.
Thanks again for all the help.
Patrick
Hi Patrick,
On 20 Aug 2007, at 23:21, Patrick - South Valley Internet wrote:
I've read so much information on this that I'm a little confused as
to what to follow. Here's what's going on:We just migrated away from our AIX machines which were running
Postfix and UW-IMAP. We're now running Postfix with Dovecot. I
already converted everyone's mbox file to maildir, but now I'm
having a little troubles with how to exactly populate their IMAP
folders. People are saying they cannot see their folders. These
'folders' they are talking about can be found in their /home
directory.
I'm not sure this is relevant but...
Here we found that the standard installation of Outlook will by
default only show folders that you have subscribed to. This means if
you simply convert the folders from Mbox to Maildir and put them into
place Outlook users won't be shown them unless either:
a) You also carefully convert the subscriptions data too (which
the UW
server keeps in a file called .mailboxlist), or
b) You have everyone change their Outlook setting to show ALL
folders instead
only the subscribed ones.
Aside: the latter may be problematic in the longer run: I encountered
Windows Mail (the Outlook Express replacement under Vista) for the
first time yesterday, and this doesn't seem to have a "Show all
folders" setting ... at least not one that I could find. :-(
We decided it was too risky to convert people's mailboxes and
subscriptions files: we were using UW's MBX format, which meant the
conversion process would have been MBX --> Mbox --> Maildir :-(
Instead we're using the very-wonderful "imapsync" utility. This is a
Perl script that does everything using IMAP. In particular it
obtains the folder listing over IMAP, reads the messages over IMAP,
writes folders/messages over IMAP and (most importantly) with the
appropriate command line option can also copy over your subscriptions
list.
Because all of this is done over IMAP you then don't have to worry
about finding your old server's subscriptions file and mailbox
structure/formats and converting them: it "just works" and everything
ends up correctly in Dovecot's folders.
The only downsides are that imapsync is CPU-intensive, and can take
some (wall-clock) time. But we thought it was worth doing that way
for safety. :-)
Cheers, Mike B-)
-- The Computing Service, University of York, Heslington, York Yo10 5DD, UK Tel:+44-1904-433811 FAX:+44-1904-433740
- Unsolicited commercial e-mail is NOT welcome at this e-mail address. *
Mike Brudenell wrote:
Hi Patrick,
On 20 Aug 2007, at 23:21, Patrick - South Valley Internet wrote:
I've read so much information on this that I'm a little confused as to what to follow. Here's what's going on:
We just migrated away from our AIX machines which were running Postfix and UW-IMAP. We're now running Postfix with Dovecot. I already converted everyone's mbox file to maildir, but now I'm having a little troubles with how to exactly populate their IMAP folders.
People are saying they cannot see their folders. These 'folders' they are talking about can be found in their /home directory.I'm not sure this is relevant but...
Here we found that the standard installation of Outlook will by default only show folders that you have subscribed to. This means if you simply convert the folders from Mbox to Maildir and put them into place Outlook users won't be shown them unless either:
a) You also carefully convert the subscriptions data too (which the UW server keeps in a file called .mailboxlist), or
b) You have everyone change their Outlook setting to show ALL folders instead only the subscribed ones.
Aside: the latter may be problematic in the longer run: I encountered Windows Mail (the Outlook Express replacement under Vista) for the first time yesterday, and this doesn't seem to have a "Show all folders" setting ... at least not one that I could find. :-(
We decided it was too risky to convert people's mailboxes and subscriptions files: we were using UW's MBX format, which meant the conversion process would have been MBX --> Mbox --> Maildir :-(
Instead we're using the very-wonderful "imapsync" utility. This is a Perl script that does everything using IMAP. In particular it obtains the folder listing over IMAP, reads the messages over IMAP, writes folders/messages over IMAP and (most importantly) with the appropriate command line option can also copy over your subscriptions list.
Because all of this is done over IMAP you then don't have to worry about finding your old server's subscriptions file and mailbox structure/formats and converting them: it "just works" and everything ends up correctly in Dovecot's folders.
The only downsides are that imapsync is CPU-intensive, and can take some (wall-clock) time. But we thought it was worth doing that way for safety. :-)
Cheers, Mike B-)
Just to give everyone an update as to what happened and how I fixed it:
I didn't understand that the mailboxes that UW-IMAP created for folders were actually mbox files. I then read the following URL:
http://people.redhat.com/rkeech/maildir-migration.txt
Which then led me to understand that I can use the same mb2md conversion script that I did with /var/spool/mail. I then ran into other issues, such as the .mailboxlist file. So I wrote a script that would read that file (if it existed) and created the necessary Maildirs into the appropriate directories.
Most customers who use Outlook are not having much issues, but we do
have some who have to reenable all the folders after we migrated them.
This seems to fix the issue.
Thanks again to everyone who helped me out!
Patrick
participants (6)
-
Frank Elsner
-
Kyle Wheeler
-
Mike Brudenell
-
Patrick - South Valley Internet
-
Philipp Kolmann
-
Scott Silva