[Dovecot] Suggested IMAP Directory Size..
Hi,
ive walked into a enviroment where the previous admin has being using pop and making the email clients leaving a copy of the mail on the workstation..
Now overtime these mailboxes (mbox format) have grown between 1,5 to 1,8 Gigs large
I want to convert all these mailbox to IMAP (only 15) and want to know whats the largested suggest folder size per user dovecot IMAP runs smoothly with good responce times..
I know the users also have large OUT LOOK pst files 4.5GIGs and wondering if I could also intergrate that into IMAP?
Please any feedback would be great on this topic..??
On Mon, 15 Oct 2007, Ilo Lorusso wrote:
[...] I know the users also have large OUT LOOK pst files 4.5GIGs and wondering if I could also intergrate that into IMAP?
It can be done, but it is a nightmare. For post-2003(?) Outlook .PST's, the only sensible, non-commercial path I could find was through Thunderbird's import. Uploading directly to the server (Even if you ran a local server!) was horrendously, painstakingly slow, and rendered the Outlook user's computer unusable for that time.
(If you're feeling lucky, Google libpst. Maybe your Outlook is old enough that it supports the format.)
Via Thunderbird:
Open all the .PST's you want to convert in Outlook, and, if possible, make sure those were the only .PST's open.
Be sure to 'compact'/'compress' each one, to get rid of deleted messages (excluding those in 'Deleted Items'. Uggh.).
Make sure Outlook is completely closed, and not accessing any .PST's.
Open Thunderbird.
Import mail from Outlook.
This gets you mbox files with the same hierarchy that you had in Outlook. I then wrote some Perl scripts to deal with these. In my case, I was combining several users' folders into a single shared hierarchy. Maybe you can run some mbox2maildir program and be done with it.
Caveats:
If possible, change the location of Thunderbird's profile directory to a short path name. (e.g. C:\convert) The default path to local folders: C:\Documents and Settings\%USER%\Application Data\Thunderbird\Profiles\(random string)\Mail\Local Folders means that approximately 100 of your 255-character limit for filenames are chewed up.
Thunderbird will mangle folder names that contain 'odd' characters. I never figured out what characters caused trouble, but the following were definitely OK: [A-Za-z0-9. ] (I found the odd foldernames running: find (dirname) -type d | perl -lnwe 'print if /[\da-f]{8}/' They always ended in a string of hexadecimal digits.)
Thunderbird doesn't seem to like non-Latin-1 headers. (I didn't find this out until someone noticed it a while after the conversion.) This means QP-encoded headers. (In my case, ISO-2022-JP.)
Best of luck. I don't envy your task. :-)
-- Ben
Not likely.. The PST file format that libpst can read is limited to 2GB.
-ellie
On Mon, 2007-10-15 at 12:17 -0400, Benjamin R. Haskell wrote:
On Mon, 15 Oct 2007, Ilo Lorusso wrote:
[...] I know the users also have large OUT LOOK pst files 4.5GIGs and wondering if I could also intergrate that into IMAP?
It can be done, but it is a nightmare. For post-2003(?) Outlook .PST's, the only sensible, non-commercial path I could find was through Thunderbird's import. Uploading directly to the server (Even if you ran a local server!) was horrendously, painstakingly slow, and rendered the Outlook user's computer unusable for that time.
(If you're feeling lucky, Google libpst. Maybe your Outlook is old enough that it supports the format.)
Via Thunderbird:
Open all the .PST's you want to convert in Outlook, and, if possible, make sure those were the only .PST's open.
Be sure to 'compact'/'compress' each one, to get rid of deleted messages (excluding those in 'Deleted Items'. Uggh.).
Make sure Outlook is completely closed, and not accessing any .PST's.
Open Thunderbird.
Import mail from Outlook.
This gets you mbox files with the same hierarchy that you had in Outlook. I then wrote some Perl scripts to deal with these. In my case, I was combining several users' folders into a single shared hierarchy. Maybe you can run some mbox2maildir program and be done with it.
Caveats:
If possible, change the location of Thunderbird's profile directory to a short path name. (e.g. C:\convert) The default path to local folders: C:\Documents and Settings\%USER%\Application Data\Thunderbird\Profiles\(random string)\Mail\Local Folders means that approximately 100 of your 255-character limit for filenames are chewed up.
Thunderbird will mangle folder names that contain 'odd' characters. I never figured out what characters caused trouble, but the following were definitely OK: [A-Za-z0-9. ] (I found the odd foldernames running: find (dirname) -type d | perl -lnwe 'print if /[\da-f]{8}/' They always ended in a string of hexadecimal digits.)
Thunderbird doesn't seem to like non-Latin-1 headers. (I didn't find this out until someone noticed it a while after the conversion.) This means QP-encoded headers. (In my case, ISO-2022-JP.)
Best of luck. I don't envy your task. :-)
-- Ben
Elizabeth Greene IT Manager, GlobalOptions FSIU Division 615-665-5555 (Office) 615-456-9813 (Mobile)
Hi Benjamin,
[...] I know the users also have large OUT LOOK pst files 4.5GIGs and wondering if I could also intergrate that into IMAP? It can be done, but it is a nightmare. For post-2003(?) Outlook .PST's, the only sensible, non-commercial path I could find was through Thunderbird's import. Uploading directly to the server (Even if you ran a local server!) was horrendously, painstakingly slow, and rendered the Outlook user's computer unusable for that time.
It is possible to connect outlook directly to an imap server (if it was previously connected to an exchange, you first have to switch it pop/imap mode. I don't remember where you select that). Then you can create a imap connection in addition to the local folders and drag'n drop the mail folders to the new imap connexion. The major issue with this process is when you have a very large amount of folder...
Cheers,
Denis
(If you're feeling lucky, Google libpst. Maybe your Outlook is old enough that it supports the format.)
Via Thunderbird:
Open all the .PST's you want to convert in Outlook, and, if possible, make sure those were the only .PST's open.
Be sure to 'compact'/'compress' each one, to get rid of deleted messages (excluding those in 'Deleted Items'. Uggh.).
Make sure Outlook is completely closed, and not accessing any .PST's.
Open Thunderbird.
Import mail from Outlook.
This gets you mbox files with the same hierarchy that you had in Outlook. I then wrote some Perl scripts to deal with these. In my case, I was combining several users' folders into a single shared hierarchy. Maybe you can run some mbox2maildir program and be done with it.
Caveats:
If possible, change the location of Thunderbird's profile directory to a short path name. (e.g. C:\convert) The default path to local folders: C:\Documents and Settings\%USER%\Application Data\Thunderbird\Profiles\(random string)\Mail\Local Folders means that approximately 100 of your 255-character limit for filenames are chewed up.
Thunderbird will mangle folder names that contain 'odd' characters. I never figured out what characters caused trouble, but the following were definitely OK: [A-Za-z0-9. ] (I found the odd foldernames running: find (dirname) -type d | perl -lnwe 'print if /[\da-f]{8}/' They always ended in a string of hexadecimal digits.)
Thunderbird doesn't seem to like non-Latin-1 headers. (I didn't find this out until someone noticed it a while after the conversion.) This means QP-encoded headers. (In my case, ISO-2022-JP.)
Best of luck. I don't envy your task. :-)
-- Ben
-- Denis Cardon Tranquil IT Systems 44 bvd des pas enchantés 44230 Saint Sébastien sur Loire tel : +33 (0) 2.40.97.62.67 http://www.tranquil-it-systems.fr
Denis Cardon wrote:
Hi Benjamin,
[...] I know the users also have large OUT LOOK pst files 4.5GIGs and wondering if I could also intergrate that into IMAP? It can be done, but it is a nightmare. For post-2003(?) Outlook .PST's, the only sensible, non-commercial path I could find was through Thunderbird's import. Uploading directly to the server (Even if you ran a local server!) was horrendously, painstakingly slow, and rendered the Outlook user's computer unusable for that time.
It is possible to connect outlook directly to an imap server (if it was previously connected to an exchange, you first have to switch it pop/imap mode. I don't remember where you select that).
there's nothing to "switch". just create a new account, and then you are given the option to select imap.
Then you can create a imap connection in addition to the local folders and drag'n drop the mail folders to the new imap connexion. The major issue with this process is when you have a very large amount of folder...
yes, as Benjamin said, this is horribly slow (even with a LAN imap server) exceptionally for 4.5 Go files ! (I had one machine do that for almost one day with a much smaller pst file.
On Mon, 15 Oct 2007, Denis Cardon wrote:
Hi Benjamin,
[...] I know the users also have large OUT LOOK pst files 4.5GIGs and wondering if I could also intergrate that into IMAP? It can be done, but it is a nightmare. For post-2003(?) Outlook .PST's, the only sensible, non-commercial path I could find was through Thunderbird's import. Uploading directly to the server (Even if you ran a local server!) was horrendously, painstakingly slow, and rendered the Outlook user's computer unusable for that time.
It is possible to connect outlook directly to an imap server (if it was previously connected to an exchange, you first have to switch it pop/imap mode. I don't remember where you select that). Then you can create a imap connection in addition to the local folders and drag'n drop the mail folders to the new imap connexion. The major issue with this process is when you have a very large amount of folder...
This was the "uploading directly to the server" option I mentioned.
The .PST's we "converted" (using the process I described) were each about 3 or 4 GB (containing 10-20K messages each). Plus there was an extensive folder hierarchy involved (3 top-level folders with about 100 subfolders each, and each of those having an average of 3 subfolders -- basically /TOP/client-name/job-number).
Our first approach had been to just drag and drop a folder or two (with their sub-folders) at a time (totalling ~100MB) at the end of the day, so that they'd finish by the next morning. But, there were lots of arbitrary errors, making it difficult to both complete the process and to resume from errors. (Since the folders were created on the server side, trying to resume resulted in "Folder already exists"-type errors.)
In testing, our external server, which we accessed via ADSL at a pretty consistent 300Kbps, had about the same transfer rate and characteristics as a server I set up on the local (wired, very fast) LAN. Even on the LAN, Outlook was taking horribly long amounts of time to transfer. At one point, a back-of-the-envelope calculation put the estimate at about two weeks (based on what had made it over to the server). And that was for a single .PST on a decently fast network (total transfer time for that .PST file itself would've been around 8 hours).
I suspect the culprit is Microsoft's less-than-stellar IMAP support, and recommend you avoid it. The Thunderbird route has just as many hurdles (It's done through SimpleMAPI, which is a subset of Exchange's MAPI that runs locally to a machine), but at least it's computationally tractable (in some sense).
Best, Ben
Hi all,
if you have a few bucks to spend, you may use the emailchemy software (http://www.weirdkid.com/products/emailchemy/). It is not free but it once saved my day on an Outlook Express 2GB+ mbx folder (yes it is possible...) : Outlook Express was itself unable to transfert to imap (completly screwed) and converting to thunderbird using MAPI also failed. The emailchemy software on the other hand did work flawlessly.
Cheers,
Denis
Disclaimer : I have no relation to weirdkid.com other than using their software once.
On Mon, 15 Oct 2007, Ilo Lorusso wrote:
[...] I know the users also have large OUT LOOK pst files 4.5GIGs and wondering if I could also intergrate that into IMAP?
It can be done, but it is a nightmare. For post-2003(?) Outlook .PST's, the only sensible, non-commercial path I could find was through Thunderbird's import. Uploading directly to the server (Even if you ran a local server!) was horrendously, painstakingly slow, and rendered the Outlook user's computer unusable for that time.
(If you're feeling lucky, Google libpst. Maybe your Outlook is old enough that it supports the format.)
Via Thunderbird:
Open all the .PST's you want to convert in Outlook, and, if possible, make sure those were the only .PST's open.
Be sure to 'compact'/'compress' each one, to get rid of deleted messages (excluding those in 'Deleted Items'. Uggh.).
Make sure Outlook is completely closed, and not accessing any .PST's.
Open Thunderbird.
Import mail from Outlook.
This gets you mbox files with the same hierarchy that you had in Outlook. I then wrote some Perl scripts to deal with these. In my case, I was combining several users' folders into a single shared hierarchy. Maybe you can run some mbox2maildir program and be done with it.
Caveats:
If possible, change the location of Thunderbird's profile directory to a short path name. (e.g. C:\convert) The default path to local folders: C:\Documents and Settings\%USER%\Application Data\Thunderbird\Profiles\(random string)\Mail\Local Folders means that approximately 100 of your 255-character limit for filenames are chewed up.
Thunderbird will mangle folder names that contain 'odd' characters. I never figured out what characters caused trouble, but the following were definitely OK: [A-Za-z0-9. ] (I found the odd foldernames running: find (dirname) -type d | perl -lnwe 'print if /[\da-f]{8}/' They always ended in a string of hexadecimal digits.)
Thunderbird doesn't seem to like non-Latin-1 headers. (I didn't find this out until someone noticed it a while after the conversion.) This means QP-encoded headers. (In my case, ISO-2022-JP.)
Best of luck. I don't envy your task. :-)
-- Ben
-- Denis Cardon Tranquil IT Systems 44 bvd des pas enchantés 44230 Saint Sébastien sur Loire tel : +33 (0) 2.40.97.62.67 http://www.tranquil-it-systems.fr
on 10/15/2007 8:52 AM Ilo Lorusso spake the following:
Hi,
ive walked into a enviroment where the previous admin has being using pop and making the email clients leaving a copy of the mail on the workstation..
Now overtime these mailboxes (mbox format) have grown between 1,5 to 1,8 Gigs large
I want to convert all these mailbox to IMAP (only 15) and want to know whats the largested suggest folder size per user dovecot IMAP runs smoothly with good responce times..
I know the users also have large OUT LOOK pst files 4.5GIGs and wondering if I could also intergrate that into IMAP?
Please any feedback would be great on this topic..??
I have some users that have let their inboxes get to a gig with no noticeable speed problems except when you delete stuff from the middle with clients like thunderbird that want to purge on every delete. If you purge as a seperate step, it isn't as noticeable.
I have some little used folders of things like logwatch e-mails that are several gigs large, and I can read and search them fairly quickly.
The indexes in dovecot help a lot. Wuimap would slow to a crawl on large boxes.
-- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
On Mon, 2007-10-15 at 17:52 +0200, Ilo Lorusso wrote:
Hi,
ive walked into a enviroment where the previous admin has being using pop and making the email clients leaving a copy of the mail on the workstation..
Now overtime these mailboxes (mbox format) have grown between 1,5 to 1,8 Gigs large
I want to convert all these mailbox to IMAP (only 15) and want to know whats the largested suggest folder size per user dovecot IMAP runs smoothly with good responce times..
If old messages are deleted, maildir would work a lot faster than mbox. If not, Dovecot shouldn't have problems even with huge mbox files as long as you set mbox_very_dirty_syncs=yes. >4GB files should work, but they haven't been tested much.
I know the users also have large OUT LOOK pst files 4.5GIGs and wondering if I could also intergrate that into IMAP?
Sure, if you can figure out how to convert them.
on 10/20/2007 1:56 PM Timo Sirainen spake the following:
On Mon, 2007-10-15 at 17:52 +0200, Ilo Lorusso wrote:
Hi,
ive walked into a enviroment where the previous admin has being using pop and making the email clients leaving a copy of the mail on the workstation..
Now overtime these mailboxes (mbox format) have grown between 1,5 to 1,8 Gigs large
I want to convert all these mailbox to IMAP (only 15) and want to know whats the largested suggest folder size per user dovecot IMAP runs smoothly with good responce times..
If old messages are deleted, maildir would work a lot faster than mbox. If not, Dovecot shouldn't have problems even with huge mbox files as long as you set mbox_very_dirty_syncs=yes. >4GB files should work, but they haven't been tested much.
I know the users also have large OUT LOOK pst files 4.5GIGs and wondering if I could also intergrate that into IMAP?
Sure, if you can figure out how to convert them.
One more note on this; Since Outlook PST files can have multiple folders, but mbox files can't, the process of creating the folders on the server and migrating the mail there might break up the size of the PST files.
-- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
participants (7)
-
Benjamin R. Haskell
-
Denis Cardon
-
Elizabeth Greene
-
Ilo Lorusso
-
mouss
-
Scott Silva
-
Timo Sirainen