[Dovecot] Problem with huge IMAP Archive after Courier migration

[Preacher]

Actually I took a look inside the folders right after starting up and 
waited for two hours to let Dovecot work.
Saving the whole Maildir into a tar on the same partition also took only 
2 hours before.
But nothing did change and when looking at activities with top, the 
server was idle, dovecot not indexing.
Also I wasn't able to drag new messages to the folder hierachy.

With courier it takes not more than 5 seconds to download the headers in 
a folder containing more than 3.000 messages.

Stan Hoeppner schrieb:
> On 1/2/2012 10:17 AM, Preacher wrote:
> ...
>> So I forced to install the Debisn 7.0 packages with 2.0.15 and finally
>> got the server running, I also restarted the whole machine to empty caches.
>> But the problem I got was that in the huge folder hierarchy the
>> downloaded headers in the individual folders disappeared, some folders
>> showed a few very old messages, some none. Also some subfolders
>> disappeared.
>> I checked this with Outlook and Thunderbird. The difference was, that
>> Thunderbird shows more messages (but not all) than Outlook in some
>> folders, but also none in some others. Outlook brought up a message in
>> some cases, that the connection timed out, although I set the timeout to
>> 60s.
> ...
>> Anyone a clue what's wrong here?
> Absolutely.  What's wrong is a lack of planning, self education, and
> patience on the part of the admin.
>
> Dovecot gets its speed from its indexes.  How long do you think it takes
> Dovecot to index 37GB of maildir messages, many thousands per directory,
> hundreds of directories, millions of files total?  Until those indexes
> are built you will not see a complete folder tree and all kinds of stuff
> will be missing.
>
> For your education:  Dovecot indexes every message and these indexes are
> the key to its speed.  Normally indexing occurs during delivery when
> using deliver or lmtp, so the index updates are small and incremental,
> keeping performance high.  You tried to do this and expected Dovecot to
> instantly process it all:
>
> http://www.youtube.com/watch?v=THVz5aweqYU
>
> If you don't know, that's a coal train car being dumped.  100 tons of
> coal in a few seconds.  Visuals are always good teaching tools.  I think
> this drives the point home rather well.
>