NFS Config vs Ceph / GlusterFS

[Marc]

> 
> I have about 100TB of mailboxes in Maildir format on NFS (NetApp FAS)
> and works very well, for performance but also stability.

Hmmm, I would like to read something else. Eg that the design/elementary properties of distributed storage result into that all such systems are performing about the same.
Maybe there should be more focus on ceph performance development instead of this cephadm?

> The main problem of using Ceph or GlusterFS to store Maildir is the high
> use of metadata that dovecot require for check new messages and others
> activity. On my storage/NFS the main part of the traffic and I/O is
> metadata traffic on small file (high file count workload).

That is why I am using mdbox files of 4MB. I hope that should give me hardly any write amplification. I am also seperating between ssd and hdd pools by auto archiving email to the hdd pools

> 
> And Ceph or GlusterFS are very inefficient with this kind of workload
> (many metadata GETATTR/ACCESS/LOOKUP and high numer of small files).

I am using rbd. After luminuous I had some issues with the cephfs and do not want to store operational stuff on it yet.