On 12/31/2011 12:56 AM, Jan-Frode Myklebust wrote:
On Fri, Dec 30, 2011 at 06:38:28PM -0600, Stan Hoeppner wrote:
Roger that. Good strategy. You using SAN storage or local RAID? What filesystem do you plan to use for the new mailbox location? What OS is the Dovecot host?
IBM DS4800 SAN-storage. Filesystem is IBM GPFS, which stripe all I/O over all the RAID5 LUNs it has assigned. Kind of like RAID5+0. To guard against disaster if one RAID5 array should fail, we plan on replicating the filesystem metadata on different sets for LUNs.
Nice setup. I've mentioned GPFS for cluster use on this list before, but I think you're the only operator to confirm using it. I'm sure others would be interested in hearing of your first hand experience: pros, cons, performance, etc. And a ball park figure on the licensing costs, whether one can only use GPFS on IBM storage or if storage from others vendors is allowed in the GPFS pool.
To this point IIRC everyone here doing clusters is using NFS, GFS, or OCFS. Each has its downsides, mostly because everyone is using maildir. NFS has locking issues with shared dovecot index files. GFS and OCFS have filesystem metadata performance issues. How does GPFS perform with your maildir workload?
OS is RHEL (currently RHEL4 and RHEL5, but new servers are implemented on RHEL6).
Lastly, how many users you have? Sorry for prying,
I'd rather not say.. but we're an ISP, with about 250.000 residential customers and multiple mailboxes per customer.
I'm always really curious about system details when someone states they have 12TB of mailbox data. ;)
$ df -h /usr/local/atmail/users Filesystem Size Used Avail Use% Mounted on /dev/atmailusers 14T 12T 2.1T 85% /usr/local/atmail/users $ df -hi /usr/local/atmail/users Filesystem Inodes IUsed IFree IUse% Mounted on /dev/atmailusers 145M 109M 37M 75% /usr/local/atmail/users
Looking forward to reducing the number of inodes when we finally move to mdbox.. Should do wonders to the backup process.
That will depend to a large degree on your mdbox_rotate_size value. The default is 2MB, which means you'll get multiple ~2MB mdbox files. If we assume the average email size including headers and attachments is 32KB, Dovecot will place ~64 such emails in a single mdbox file with the default 2MB setting. 32KB may be a high or low average depending on your particular users.
Considering there is no inherent performance downside to going larger than the default, and significant gains to be made, consider a setting of 8MB to 16MB. This will dramatically reduce both inode consumption and filesystem metadata IOPS vs maildir. Reducing IOPS on a shared SAN is always a plus, especially if you're going to be adding some extra GPFS replication traffic.
Timo, is there any technical or sanity based upper bound on mdbox size? Anything wrong with using 64MB, 128MB, or even larger for mdbox_rotate_size?
-- Stan