[Dovecot] More Migration Questions
Hello,
I was on here a month or two ago asking lots of questions about migrating from uw-imapd/qpopper/mbox to dovecot/maildir. I am now at the stage where i'm trying to convert all my emails, but i have a few questions...
Where should i convert my mbox mailboxes into? "/home/username/Maildir/" or "/somewhere/maildirs/"?
Does it actually matter? Is there a standard location that they *should* go?
I've tried converting them into "/somewhere/maildirs/username/", but then i found a dovecot page that says they are usually stored in "/home/username/Maildir" so now i don't know where to put them!
Also, having converted the mbox files into maildir files, i do seem to have one email per file, as intended, but the files have names like "1250253965.000586.mbox,S=8167,W=8427:2,S". Should they have the word "mbox" in their filename if they are maildir files?
Finally, it appears that our existing IMAP users (all mbox) have mailboxes stored simply in their home directory, not necessarily inside the "~/mail/" directory - is this going to be a problem?
The command i'm using to convert is this one:
./migrateuser.sh username /home/username /vol2/maildirs/username
This script (and the latest mb2md.py that goes with it, to preserve msg IDs) were obtained from here: http://wiki.dovecot.org/Migration/MailFormat
The stage i'm at with the server overall is that uw-imapd and qpopper have been completely removed, exim is currently setup to deliver via it's own method into the mbox files in /var/spool/mail but the exim service is stopped because i'm migrating, and dovecot 1.1 is installed but stopped, and has had no configuration done to it yet whatsoever.
So... any advice?
Thanks in advance!
Richard.
-- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.hobbs@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999 Mobile: +44 7811 803377
On Aug 14, 2009, at 8:57 AM, Richard Hobbs wrote:
Where should i convert my mbox mailboxes into? "/home/username/ Maildir/" or "/somewhere/maildirs/"?
Does it actually matter? Is there a standard location that they
*should* go?
It doesn't matter. Maildir/ is a bit more standard, but Dovecot
doesn't care.
Also, having converted the mbox files into maildir files, i do seem to have one email per file, as intended, but the files have names like "1250253965.000586.mbox,S=8167,W=8427:2,S". Should they have the word "mbox" in their filename if they are maildir files?
The file names don't matter, as long as they're unique. The S= and W=
parts are about optimizing looking up file sizes.
Finally, it appears that our existing IMAP users (all mbox) have mailboxes stored simply in their home directory, not necessarily
inside the "~/mail/" directory - is this going to be a problem?
That's better actually, you don't then have to add those ~/mail/
compabilitity namespaces.
The command i'm using to convert is this one:
./migrateuser.sh username /home/username /vol2/maildirs/username
I'd have separate "home" and maildir directories. For example:
home = /vol2/maildirs/username mail_location = maildir: ~/Maildir
Hello,
Thanks for the reply... everything seem to be working with regard to my previous questions now, which is great.
Regarding the ~/mail/ issues though, you said:
That's better actually, you don't then have to add those ~/mail/ compabilitity namespaces.
Well, i haven't added any ~/mail/ compatibility namespaces knowingly... i guess they might be in the default config (my version has been built for debian), but how would i check?
Basically, the problem is that some users have their folders in "~/mail/" and others have their folders in "~/".
I want to convert both, and also and up with a standard location in the maildir setup. I guess the best way to do this is to just copy everyone's mailboxes from "~/" into "~/mail/" before i run the conversion, right? As long as people don't object to their folders potentially changing places within their IMAP client, everything will be OK, right?
Thanks again!
Richard.
Timo Sirainen wrote:
On Aug 14, 2009, at 8:57 AM, Richard Hobbs wrote:
Where should i convert my mbox mailboxes into? "/home/username/Maildir/" or "/somewhere/maildirs/"?
Does it actually matter? Is there a standard location that they *should* go?
It doesn't matter. Maildir/ is a bit more standard, but Dovecot doesn't care.
Also, having converted the mbox files into maildir files, i do seem to have one email per file, as intended, but the files have names like "1250253965.000586.mbox,S=8167,W=8427:2,S". Should they have the word "mbox" in their filename if they are maildir files?
The file names don't matter, as long as they're unique. The S= and W= parts are about optimizing looking up file sizes.
Finally, it appears that our existing IMAP users (all mbox) have mailboxes stored simply in their home directory, not necessarily inside the "~/mail/" directory - is this going to be a problem?
That's better actually, you don't then have to add those ~/mail/ compabilitity namespaces.
The command i'm using to convert is this one:
./migrateuser.sh username /home/username /vol2/maildirs/username
I'd have separate "home" and maildir directories. For example:
home = /vol2/maildirs/username mail_location = maildir: ~/Maildir
http://wiki.dovecot.org/VirtualUsers#Home_directories
This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email____________________________________________...
-- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.hobbs@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999 Mobile: +44 7811 803377
On Fri, 2009-08-14 at 15:39 +0000, Richard Hobbs wrote:
That's better actually, you don't then have to add those ~/mail/ compabilitity namespaces.
Well, i haven't added any ~/mail/ compatibility namespaces knowingly... i guess they might be in the default config (my version has been built for debian), but how would i check?
Basically, the problem is that some users have their folders in "~/mail/" and others have their folders in "~/".
This is kind of what I meant by the compatibility namespaces. If some users have ~/mail/ and whatelse, you'll probably want to add such namespaces to Dovecot. See http://wiki.dovecot.org/Namespaces#Backwards_Compatibility:_UW-IMAP
I want to convert both, and also and up with a standard location in the maildir setup. I guess the best way to do this is to just copy everyone's mailboxes from "~/" into "~/mail/" before i run the conversion, right?
That should work, as long as you can find all the mailboxes (or are all files in ~/ mailboxes?)
As long as people don't object to their folders potentially changing places within their IMAP client, everything will be OK, right?
With the compatibility namespace it should be pretty transparent to clients. Also you'll probably want to set separator=/ in all the namespaces.
Hello,
Thanks again - your reply will help greatly.
I have a rather more annoying problem at the moment, best described by my conversation in the dovecot IRC channel...
i've telnetted to 110 on old server
the "uidl" command shows 3 UIDLs
uidl command also shows 3 UIDLs from new server...
trouble is, despite seeing the same 3 UIDLs on the old mail server *and* the new mail server, when issuing the "uidl" command through port 110 after logging in, i just setup the account in my mail client against the old server, downloaded 3 msgs, changed the config for the new server, and it downloaded the same msgs again!
ok, on the old server, in the received email, towards the bottom of the headers, i see this:
X-UIDL: a=="!(3N"!&D'"!87D"!
on the new server, in the duplicate received email, right at the top, before any other headers, i see this:
X-UIDL: a=="!(3N"!&D'"!87D"!
whaton earth is going on here...
both now have the same X-UIDL, but are being downloaded separately!
So... does anyone have any ideas?
Thanks again!
Richard.
Timo Sirainen wrote:
On Fri, 2009-08-14 at 15:39 +0000, Richard Hobbs wrote:
That's better actually, you don't then have to add those ~/mail/ compabilitity namespaces. Well, i haven't added any ~/mail/ compatibility namespaces knowingly... i guess they might be in the default config (my version has been built for debian), but how would i check?
Basically, the problem is that some users have their folders in "~/mail/" and others have their folders in "~/".
This is kind of what I meant by the compatibility namespaces. If some users have ~/mail/ and whatelse, you'll probably want to add such namespaces to Dovecot. See http://wiki.dovecot.org/Namespaces#Backwards_Compatibility:_UW-IMAP
I want to convert both, and also and up with a standard location in the maildir setup. I guess the best way to do this is to just copy everyone's mailboxes from "~/" into "~/mail/" before i run the conversion, right?
That should work, as long as you can find all the mailboxes (or are all files in ~/ mailboxes?)
As long as people don't object to their folders potentially changing places within their IMAP client, everything will be OK, right?
With the compatibility namespace it should be pretty transparent to clients. Also you'll probably want to set separator=/ in all the namespaces.
-- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.hobbs@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999 Mobile: +44 7811 803377
Hello,
We've also tried setting the size of the msgs to be the same, just in case that made a difference. Here is some output from our old mail server during the test:
====================================================================== uidl +OK uidl command accepted. 1 a=="!(3N"!&D'"!87D"! 2 _fg!!)%>!!Ic>!!-@'"! 3 Yh]!!Zlk"!QfA"!Lj1!! . list +OK 3 visible messages (42852 octets) 1 28359 2 2918 3 11575 .
Here are the same commands against the new server during our test:
====================================================================== uidl +OK 1 a=="!(3N"!&D'"!87D"! 2 _fg!!)%>!!Ic>!!-@'"! 3 Yh]!!Zlk"!QfA"!Lj1!! . list +OK 3 messages: 1 28359 2 2918 3 11575 .
As far as i can tell, they are identical.
If i setup the old server in my mail client, it downloads all 3 msgs. If i then mark them all as "read" and change the hostname of the mail server to point to the new server, it then downloads the same 3 msgs so i end up with duplicates in my inbox, despite them having the same UIDLs!
The only difference i can see is that the old mail server has the X-UIDL header towards the end of the headers, and the new server puts the X-UIDL header right at the top above all other headers.
Does anyone know what on earth is going on here?
Thanks again!
Richard.
Richard Hobbs wrote:
Hello,
Thanks again - your reply will help greatly.
I have a rather more annoying problem at the moment, best described by my conversation in the dovecot IRC channel...
i've telnetted to 110 on old server
the "uidl" command shows 3 UIDLs
uidl command also shows 3 UIDLs from new server...
trouble is, despite seeing the same 3 UIDLs on the old mail server *and* the new mail server, when issuing the "uidl" command through port 110 after logging in, i just setup the account in my mail client against the old server, downloaded 3 msgs, changed the config for the new server, and it downloaded the same msgs again!
ok, on the old server, in the received email, towards the bottom of the headers, i see this:
X-UIDL: a=="!(3N"!&D'"!87D"!
on the new server, in the duplicate received email, right at the top, before any other headers, i see this:
X-UIDL: a=="!(3N"!&D'"!87D"!
whaton earth is going on here...
both now have the same X-UIDL, but are being downloaded separately!
So... does anyone have any ideas?
Thanks again!
Richard.
Timo Sirainen wrote:
On Fri, 2009-08-14 at 15:39 +0000, Richard Hobbs wrote:
That's better actually, you don't then have to add those ~/mail/ compabilitity namespaces. Well, i haven't added any ~/mail/ compatibility namespaces knowingly... i guess they might be in the default config (my version has been built for debian), but how would i check?
Basically, the problem is that some users have their folders in "~/mail/" and others have their folders in "~/". This is kind of what I meant by the compatibility namespaces. If some users have ~/mail/ and whatelse, you'll probably want to add such namespaces to Dovecot. See http://wiki.dovecot.org/Namespaces#Backwards_Compatibility:_UW-IMAP
I want to convert both, and also and up with a standard location in the maildir setup. I guess the best way to do this is to just copy everyone's mailboxes from "~/" into "~/mail/" before i run the conversion, right? That should work, as long as you can find all the mailboxes (or are all files in ~/ mailboxes?)
As long as people don't object to their folders potentially changing places within their IMAP client, everything will be OK, right? With the compatibility namespace it should be pretty transparent to clients. Also you'll probably want to set separator=/ in all the namespaces.
-- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.hobbs@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999 Mobile: +44 7811 803377
Hello,
OK, I've found the problem...
If i migrate all email from one server to another, preserving msg UIDs, then the msg UIDs are ignored by the POP3 client if it has it's hostname/IP changed in order to point to the new server.
So... mail client points to 192.168.1.1. mail is all downloaded but left on server. migration to new server occurs, preserving msg UIDs. mail client IP adjusted to point to 192.168.1.2. All mail downloaded again, duplicates now exist.
The way to get around this is to either set your mail client to hostname in the first place, and then make the change in DNS, or to remove all msgs from the server before migration, so the msg UIDs are irrelevant.
It's also annoying that you cannot fix the problem before migration because if i tell people to change from the old mail server's IP to the old mail server's hostname, then again - the mail client will download all their mail again and duplicate it!
Anyway... i'm glad i've found the problem, even if it does make the migration a little annoying! :-)
Regards, Richard.
Richard Hobbs wrote:
Hello,
We've also tried setting the size of the msgs to be the same, just in case that made a difference. Here is some output from our old mail server during the test:
====================================================================== uidl +OK uidl command accepted. 1 a=="!(3N"!&D'"!87D"! 2 _fg!!)%>!!Ic>!!-@'"! 3 Yh]!!Zlk"!QfA"!Lj1!! . list +OK 3 visible messages (42852 octets) 1 28359 2 2918 3 11575 .
Here are the same commands against the new server during our test:
====================================================================== uidl +OK 1 a=="!(3N"!&D'"!87D"! 2 _fg!!)%>!!Ic>!!-@'"! 3 Yh]!!Zlk"!QfA"!Lj1!! . list +OK 3 messages: 1 28359 2 2918 3 11575 .
As far as i can tell, they are identical.
If i setup the old server in my mail client, it downloads all 3 msgs. If i then mark them all as "read" and change the hostname of the mail server to point to the new server, it then downloads the same 3 msgs so i end up with duplicates in my inbox, despite them having the same UIDLs!
The only difference i can see is that the old mail server has the X-UIDL header towards the end of the headers, and the new server puts the X-UIDL header right at the top above all other headers.
Does anyone know what on earth is going on here?
Thanks again!
Richard.
Richard Hobbs wrote:
Hello,
Thanks again - your reply will help greatly.
I have a rather more annoying problem at the moment, best described by my conversation in the dovecot IRC channel...
i've telnetted to 110 on old server
the "uidl" command shows 3 UIDLs
uidl command also shows 3 UIDLs from new server...
trouble is, despite seeing the same 3 UIDLs on the old mail server *and* the new mail server, when issuing the "uidl" command through port 110 after logging in, i just setup the account in my mail client against the old server, downloaded 3 msgs, changed the config for the new server, and it downloaded the same msgs again!
ok, on the old server, in the received email, towards the bottom of the headers, i see this:
X-UIDL: a=="!(3N"!&D'"!87D"!
on the new server, in the duplicate received email, right at the top, before any other headers, i see this:
X-UIDL: a=="!(3N"!&D'"!87D"!
whaton earth is going on here...
both now have the same X-UIDL, but are being downloaded separately!
So... does anyone have any ideas?
Thanks again!
Richard.
Timo Sirainen wrote:
On Fri, 2009-08-14 at 15:39 +0000, Richard Hobbs wrote:
That's better actually, you don't then have to add those ~/mail/ compabilitity namespaces. Well, i haven't added any ~/mail/ compatibility namespaces knowingly... i guess they might be in the default config (my version has been built for debian), but how would i check?
Basically, the problem is that some users have their folders in "~/mail/" and others have their folders in "~/". This is kind of what I meant by the compatibility namespaces. If some users have ~/mail/ and whatelse, you'll probably want to add such namespaces to Dovecot. See http://wiki.dovecot.org/Namespaces#Backwards_Compatibility:_UW-IMAP
I want to convert both, and also and up with a standard location in the maildir setup. I guess the best way to do this is to just copy everyone's mailboxes from "~/" into "~/mail/" before i run the conversion, right? That should work, as long as you can find all the mailboxes (or are all files in ~/ mailboxes?)
As long as people don't object to their folders potentially changing places within their IMAP client, everything will be OK, right? With the compatibility namespace it should be pretty transparent to clients. Also you'll probably want to set separator=/ in all the namespaces.
-- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.hobbs@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999 Mobile: +44 7811 803377
participants (2)
-
Richard Hobbs
-
Timo Sirainen