That is why I am using mdbox files of 4MB. I hope that should give me hardly any write amplification. I am also seperating between ssd and hdd pools by auto archiving email to the hdd pools I am using rbd. After luminuous I had some issues with the cephfs and do not want to store operational stuff on it yet.
I am very interested in that setup, since I am currently planning to reshape my cluster in a similar way (currently from sole distribution via director to distribution + HA). Could you post a short overview (scheme) and some important configurations of your setup? Did you do any performancetesting? Also, when you say rbd in a clustered context, is that one block device per node while the director still spreads the accounts over the nodes?
Thanks in advance,
M. Schmidt