[Dovecot] Dovecot NFS Indexes and IMAP Migration

[Andy Dills]

Hi there,

After being extremely impressed from implementing it in some customer 
installations, I've decided to migrate our mail infrastructure to dovecot. 
Being able to have /bin/checkpassword support across the board for pop, 
imap, and smtp authentication, as well as being able to ditch stunnel for 
the SSL layer...where were you in 2001?!? :)

I have two concerns, the first is about the proper index configuration in 
our environment, and the second is about the best way to migrate 
mailboxes.

Some quick background: we've long been distributed, with an L4 switch up 
front and a Netapp in the back, Mysql for metadata. We have a cluster of 
several FreeBSD boxes doing filtering and maildir delivery 
(postfix/amavisd-new), and we have a server that users get sent to for 
pop/imap. If it's down, they get sent to one of the filter boxes instead.

So, regarding the indexes:

>From reading the docs, it appears I would not be able to use the dovecot 
LDA due to locking issues (bummer), and I should configure dovecot to 
store index files on a local disk of the primary pop/imap server rather 
than on the netapp.

My first question would be, how much of a performance hit should I expect 
from having to update the indexes when mail is checked rather then when 
mail is delivered since I can't use the LDA? Is this a minor issue like I 
expect, or can it be something the user actually notices when checking 
mail?

In a worst case scenario, if the pop/imap server is down, and another 
server is used, how long would it take to create an index for an account 
with a few hundered megs, several thousand messages? I'm wondering if, 
since I have plenty of overhead available on the netapp, I should just 
store the indexes there in case another server must be used. 

I'm basically looking for more information about the pros and cons of 
storing the indexes on the nfs volume in a situation where a single server 
would normally be providing pop/imap service.

Regarding migration:

My concerns regarding migration relate to how best to process existing 
maildirs by both converting the previous imap metadata and calculating an 
initial index for each account.

First off, I can't seem to find any info on a way to create an initial 
index file for all of the mailboxes. Can anybody suggest a command I can 
use for this? I'm sure I just overlooked it.

Regarding the migrations, we have both courier and bincimap to deal with. 
(We use qmail's pop, so no complications with migrating courier).

Fortunately, courier is used for normal imap on 143, bincimap is only used 
for webmail. So, it would seem that metadata from bincimap could be 
ignored; the only thing it would seem important to preserve is mailbox 
subscriptions. Please correct this assumption if I'm wrong.

bincimap uses a INBOX/folder format in the subscription file, however the 
actual folders are stored exactly the same as the INBOX.folder style of 
courier.

So, it seems like my best option is to modify the 
courier-dovecot-migrate.pl script to consider the mailboxes subscribed to 
via bincimap. Here's a modified convert_subscriptions that seems to work 
well in limited testing, feel free to point out any idiot mistakes, it's 
been a while since I've done anything substantial in perl. I would have 
sent a patch but I suspect I'm the only person who will ever use this 
change and critiquing it would be easier with a full context. 

sub convert_subscriptions {
  my ($dir, $owner_uid, $owner_gid) = @_;
      
  my $in_fname = "$dir/courierimapsubscribed";

  # ADDED: The bincimap file to parse

  my $in_fname2 = "$dir/.bincimap-subscribed";
  my $out_fname = "$dir/subscriptions";

  # MODIFIED: Return only if neither exist

  return if (!-f $in_fname && !-f $in_fname2);
      
  if (!$overwrite && -f $out_fname) {
    print "$out_fname already exists, not overwritten\n" if (!$quiet);
    return;
  }
      
  return if (!$do_conversion);

  my ($fin, $fin2, $fout);
  open ($fin, $in_fname) || die $!;
  open ($fin2, $in_fname2) || die $!;
  open ($fout, ">$out_fname") || die $!;

  # ADDED: An array to track mailboxes we've written in case both files exist

  my @subs;
  while (<$fin>) {
    chomp $_;
       
    if (/^INBOX$/i) {
      print $fout "INBOX\n";
      push @subs, "INBOX\n";
    } elsif (/^INBOX\.(.*)$/i) {
      print $fout "$1\n";
      push @subs, "$1\n";
    } else {
      # unknown. keep it as-is.
      print $fout "$_\n";
      push @subs, "$_\n";  
    }
  }
  close $fin;

  # ADDED: A second loop, same as the first, but also checking to make 
  # sure that the mailbox has not already been written to the new file to 
  # eliminate duplicates. 

  while (<$fin2>) {
    chomp $_;
    my $sub = $_;
    if (/^INBOX$/i && !grep (/^INBOX\n/i, @subs)) {
      print $fout "INBOX\n";
    } elsif (/^INBOX\/(.*)$/i && !grep (/^$1\n/i, @subs)) {
      print $fout "$1\n";
    } elsif (!grep(/^$sub\n/, @subs)) {
      # unknown. keep it as-is.
      print $fout "$sub\n";   
    }
  }
  close $fin;
  close $fin2;
  close $fout;
  chown $owner_uid, $owner_gid, $out_fname;
}


Thanks in advance for any help or suggestions!
Andy

---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---