[Dovecot] Providing shared folders with multiple backend servers

[Stan Hoeppner]

On 1/7/2012 4:20 PM, Sven Hartge wrote:
> Hi *,
> 
> I am currently in the planning stage for a "new and improved" mail
> system at my university.
> 
> Right now, everything is on one big backend server but this is causing
> me increasing amounts of pain, beginning with the time a full backup
> takes.

You failed to mention your analysis and diagnosis identifying the source
of the slow backup, and other issues your eluded to but didn't mention
specifically.  You also didn't mention how you're doing this full backup
(tar, IMAP; D2D or tape), where the backup bottleneck is, what mailbox
storage format you're using, total mailbox count and filesystem space
occupied.  What is your disk storage configuration?  Direct attach?
Hardware or software RAID?  What RAID level?  How many disks?  SAS or SATA?

It's highly likely your problems can be solved without the drastic
architecture change, and new problems it will introduce, that you
describe below.

> So naturally, I want to split this big server into smaller ones.

Naturally?  Many OPs spend significant x/y/z resources trying to avoid
the "shared nothing" storage backend setup below.

> To keep things simple, I want to pin a user to a server so I can avoid
> things like NFS or cluster aware filesystems. The mapping for each
> account is then inserted into the LDAP object for each user and the
> frontend proxy (perdition at the moment) then uses this information to
> route each access to the correct backend storage server running dovecot.

Splitting the IMAP workload like this isn't keeping things simple, but
increases complexity, on many levels.  And there's nothing wrong with
NFS and cluster filesystems if they are used correctly.

> So far this has been working nice with my test setup.
> 
> But: I also have to provide shared folders for users. Thankfully users
> don't have the right to share their own folders, which makes things
> easier (I hope).
> 
> Right now, the setup works like this, using Courier:
> 
>  - complete virtual mail setup
>  - global shared folders configured in /etc/courier/shared/index
>  - inside /home/shared-folder-name/Maildir/courierimapacl specific user
>    get access to a folder
>  - each folder a user has access is mapped to the namespace #shared
>    like #shared.shared-folder-name
> 
> Now, if I split my backend storage server into multiple ones and user-A
> is on server-1 and user-B is on server-2, but both need to access the
> same shared folder, I have a problem.

Yes, you do.

> I could of course move all users needing access to a shared folder to
> the same server, but in the end, this will be a nightmare for me,
> because I forsee having to move users around on a daily basis.

See my comments above.

> Right now, I am pondering with using an additional server with just the
> shared folders on it and using NFS (or a cluster FS) to mount the shared
> folder filesystem to each backend storage server, so each user has
> potential access to a shared folders data.

So you're going to implement a special case of what you're desperately
trying to avoid?  This makes no sense.

> Ideas? Suggestions? Nudges in the right direction?

Yes.  We need more real information.  Please provide:

1.  Mailbox count, total maildir file count and size
2.  Average/peak concurrent user connections
3.  CPU type/speed/total core count, total RAM, free RAM (incl buffers)
4.  Storage configuration--total spindles, RAID level, hard or soft RAID
5.  Filesystem type
6.  Backup software/method
7.  Operating system

Instead of telling us what you think the solution to your unidentified
bottleneck is and then asking "yeah or nay", tell us what the problem is
and allow us to recommend solutions.  This way you'll get some education
and multiple solutions that may very well be a better fit, will perform
better, and possibly cost less in capital outlay and administration
time/effort.

-- 
Stan