[Dovecot] Problem with huge IMAP Archive after Courier migration
I have a mail server running Debian 6.0 with Courier IMAP to store project related mail. Currently the maildir of the archive (one user) contains about 37GB of data. Our staff is acessing the archive via Outlook 2007 where they drag their Exchange inbox or sent files to it. The problem with courier is that is sometimes mixes up headers with message bodies, so I wanted to migrate to dovecot.
I tried this on my proxy running Debian 7.0 with some test data and this worked fine (OK, spent some hours to get the config files done - Dovecot without authentication). Dovecot version here is 2.0.15.
Tried it with our productive system today, but got Dovecot 1.2.15 installed on Debian 6.0 Config files and parameters I took from my test system were not compatible and I didn't get it to work. So I forced to install the Debisn 7.0 packages with 2.0.15 and finally got the server running, I also restarted the whole machine to empty caches. But the problem I got was that in the huge folder hierarchy the downloaded headers in the individual folders disappeared, some folders showed a few very old messages, some none. Also some subfolders disappeared. I checked this with Outlook and Thunderbird. The difference was, that Thunderbird shows more messages (but not all) than Outlook in some folders, but also none in some others. Outlook brought up a message in some cases, that the connection timed out, although I set the timeout to 60s.
After being frustrated uninstalled dovecot, went back to Courier and folder contents are displayed correctly again.
Anyone a clue what's wrong here?
Finally some config information: proxy-server:~# dovecot -n # 2.0.15: /etc/dovecot/dovecot.conf # OS: Linux 3.1.0-1-686-pae i686 Debian wheezy/sid auth_debug_passwords = yes auth_mechanisms = plain login disable_plaintext_auth = no namespace { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { driver = pam } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = imap ssl = no ssl_cert =
On 1/2/2012 10:17 AM, Preacher wrote: ...
So I forced to install the Debisn 7.0 packages with 2.0.15 and finally got the server running, I also restarted the whole machine to empty caches. But the problem I got was that in the huge folder hierarchy the downloaded headers in the individual folders disappeared, some folders showed a few very old messages, some none. Also some subfolders disappeared. I checked this with Outlook and Thunderbird. The difference was, that Thunderbird shows more messages (but not all) than Outlook in some folders, but also none in some others. Outlook brought up a message in some cases, that the connection timed out, although I set the timeout to 60s. ... Anyone a clue what's wrong here?
Absolutely. What's wrong is a lack of planning, self education, and patience on the part of the admin.
Dovecot gets its speed from its indexes. How long do you think it takes Dovecot to index 37GB of maildir messages, many thousands per directory, hundreds of directories, millions of files total? Until those indexes are built you will not see a complete folder tree and all kinds of stuff will be missing.
For your education: Dovecot indexes every message and these indexes are the key to its speed. Normally indexing occurs during delivery when using deliver or lmtp, so the index updates are small and incremental, keeping performance high. You tried to do this and expected Dovecot to instantly process it all:
http://www.youtube.com/watch?v=THVz5aweqYU
If you don't know, that's a coal train car being dumped. 100 tons of coal in a few seconds. Visuals are always good teaching tools. I think this drives the point home rather well.
-- Stan
Actually I took a look inside the folders right after starting up and waited for two hours to let Dovecot work. Saving the whole Maildir into a tar on the same partition also took only 2 hours before. But nothing did change and when looking at activities with top, the server was idle, dovecot not indexing. Also I wasn't able to drag new messages to the folder hierachy.
With courier it takes not more than 5 seconds to download the headers in a folder containing more than 3.000 messages.
Stan Hoeppner schrieb:
So I forced to install the Debisn 7.0 packages with 2.0.15 and finally got the server running, I also restarted the whole machine to empty caches. But the problem I got was that in the huge folder hierarchy the downloaded headers in the individual folders disappeared, some folders showed a few very old messages, some none. Also some subfolders disappeared. I checked this with Outlook and Thunderbird. The difference was, that Thunderbird shows more messages (but not all) than Outlook in some folders, but also none in some others. Outlook brought up a message in some cases, that the connection timed out, although I set the timeout to 60s. ... Anyone a clue what's wrong here? Absolutely. What's wrong is a lack of planning, self education, and
On 1/2/2012 10:17 AM, Preacher wrote: ... patience on the part of the admin.
Dovecot gets its speed from its indexes. How long do you think it takes Dovecot to index 37GB of maildir messages, many thousands per directory, hundreds of directories, millions of files total? Until those indexes are built you will not see a complete folder tree and all kinds of stuff will be missing.
For your education: Dovecot indexes every message and these indexes are the key to its speed. Normally indexing occurs during delivery when using deliver or lmtp, so the index updates are small and incremental, keeping performance high. You tried to do this and expected Dovecot to instantly process it all:
http://www.youtube.com/watch?v=THVz5aweqYU
If you don't know, that's a coal train car being dumped. 100 tons of coal in a few seconds. Visuals are always good teaching tools. I think this drives the point home rather well.
On 1/3/2012 10:47 AM, Preacher wrote:
Actually I took a look inside the folders right after starting up and waited for two hours to let Dovecot work.
So two hours after clicking on an IMAP folder the contents of that folder were still not displayed correctly?
Saving the whole Maildir into a tar on the same partition also took only 2 hours before.
This doesn't have any relevance.
But nothing did change and when looking at activities with top, the server was idle, dovecot not indexing. Also I wasn't able to drag new messages to the folder hierachy.
Then something is seriously wrong. The fact that you "forced" the Wheezy Dovecot package into a Squeeze system may have something, if not everything, to do with your problem (somehow I missed this fact in your previous email).
Debian testing/sid packages are compiled against newer system libraries. If you check various logs you'll likely see problems related to this. This is also why the Backports project exists--TESTING packages are compiled against the STABLE libraries so newer application revs can be used on the STABLE distribution. Currently there is no Dovecot 2.x backport for Squeeze.
I would suggest you thoroughly remove the Wheezy 2.0.15 package and install the 1.2.15-7 STABLE package. Read the documentation for 1.2.x and configure it properly. Then things will likely work as they should.
-- Stan
On Tue, Jan 03, 2012 at 11:55:27AM -0600, Stan Hoeppner wrote:
[..]
I would suggest you thoroughly remove the Wheezy 2.0.15 package and
Not to use the Wheezy package might be wise.
install the 1.2.15-7 STABLE package. Read the documentation for 1.2.x
Alternatively you could use Stephan Bosch's repository:
deb http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.0 mainDespite the warning at http://wiki2.dovecot.org/PrebuiltBinaries#Automatically_Built_Packages they work very stable.
and configure it properly. Then things will likely work as they should.
Dennis
On 01/03/2012 05:48 PM, Dennis Guhl wrote:
On Tue, Jan 03, 2012 at 11:55:27AM -0600, Stan Hoeppner wrote:
[..]
I would suggest you thoroughly remove the Wheezy 2.0.15 package and Not to use the Wheezy package might be wise.
install the 1.2.15-7 STABLE package. Read the documentation for 1.2.x Alternatively you could use Stephan Bosch's repository:
deb http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.0 main
Despite the warning at http://wiki2.dovecot.org/PrebuiltBinaries#Automatically_Built_Packages they work very stable.
and configure it properly. Then things will likely work as they should. Dennis See http://www.prato.linux.it/~mnencia/debian/dovecot-squeeze/ and http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592959
I have the packages from this repository running in production on a squeeze system, working fine.
On Mon, 2012-01-02 at 17:17 +0100, Preacher wrote:
So I forced to install the Debisn 7.0 packages with 2.0.15 and finally got the server running, I also restarted the whole machine to empty caches. But the problem I got was that in the huge folder hierarchy the downloaded headers in the individual folders disappeared, some folders showed a few very old messages, some none. Also some subfolders disappeared. I checked this with Outlook and Thunderbird. The difference was, that Thunderbird shows more messages (but not all) than Outlook in some folders, but also none in some others. Outlook brought up a message in some cases, that the connection timed out, although I set the timeout to 60s.
Did you run the Courier migration script? http://wiki2.dovecot.org/Migration/Courier
Also explicitly setting mail_location would be a good idea.
Yes i did, followed this guide you mentioned, it said that it found the 3 mailboxes I have set-up in total, conversion took only a few moments. I guess the mail location was automaticall set correctly as the folder hierachy was displayed correctly
Timo Sirainen schrieb:
On Mon, 2012-01-02 at 17:17 +0100, Preacher wrote:
So I forced to install the Debisn 7.0 packages with 2.0.15 and finally got the server running, I also restarted the whole machine to empty caches. But the problem I got was that in the huge folder hierarchy the downloaded headers in the individual folders disappeared, some folders showed a few very old messages, some none. Also some subfolders disappeared. I checked this with Outlook and Thunderbird. The difference was, that Thunderbird shows more messages (but not all) than Outlook in some folders, but also none in some others. Outlook brought up a message in some cases, that the connection timed out, although I set the timeout to 60s. Did you run the Courier migration script? http://wiki2.dovecot.org/Migration/Courier
Also explicitly setting mail_location would be a good idea.
participants (5)
-
Dennis Guhl
-
Gedalya
-
Preacher
-
Stan Hoeppner
-
Timo Sirainen