"Miloslav" == Miloslav Hůla <miloslav.hula@gmail.com> writes:

Miloslav> Dne 09.09.2020 v 17:52 John Stoffel napsal(a):
Miloslav> There is a one PCIe RAID controller in a chasis. AVAGO
Miloslav> MegaRAID SAS 9361-8i. And 16x SAS 15k drives conneced to
Miloslav> it. Because the controller does not support pass-through for
Miloslav> the drives, we use 16x RAID-0 on controller. So, we get
Miloslav> /dev/sda ... /dev/sdp (roughly) in OS. And over that we have
Miloslav> single btrfs RAID-10, composed of 16 devices, mounted as
Miloslav> /data.

I will bet that this is one of your bottlenecks as well.  Get a secord
or third controller and split your disks across them evenly.

Miloslav> That's plan for a next step.

Miloslav> We run 'rsync' to remote NAS daily. It takes about 6.5 hours to finish,
Miloslav> 12'265'387 files last night.

That's.... sucky.  So basically you're hitting the drives hard with
random IOPs and you're probably running out of performance.  How much
space are you using on the filesystem?

Miloslav> It's not so sucky how it seems. rsync runs during the
Miloslav> night. And even reading is high, server load stays low. We
Miloslav> have problems with writes.

Ok.  So putting in an SSD pair to cache things should help.

And why not use brtfs send to ship off snapshots instead of using
rsync?  I'm sure that would be an improvement...

Miloslav> We run backup to external NAS (NetApp) for a disaster
Miloslav> recovery scenario.  Moreover NAS is spreaded across multiple
Miloslav> locations. Then we create NAS snapshot, tens days
Miloslav> backward. All snapshots easily available via NFS mount. And
Miloslav> NAS capacity is cheaper.

So why not run the backend storage on the Netapp, and just keep the
indexes and such local to the system?  I've run Netapps for many years
and they work really well.  And then you'd get automatic backups using
schedule snapshots.

Keep the index files local on disk/SSDs and put the maildirs out to
NFSv3 volume(s) on the Netapp(s).  Should do wonders.  And you'll stop
needing to do rsync at night.

Miloslav> It's the option we have in minds. As you wrote, NetApp is very solid. 
Miloslav> The main reason for local storage is, that IMAP server is completely 
Miloslav> isolated from network. But maybe one day will use it.

It's not completely isolated, it can rsync data to another host that
has access to the Netapp.  *grin*

Miloslav> Unfortunately, to quickly fix the problem and make server
Miloslav> usable again, we already added SSD and moved indexes on
Miloslav> it. So we have no measurements in old state.

That's ok, if it's better, then its better.  How is the load now?
Looking at the output of 'iostat -x 30' might be a good thing.

Miloslav> Situation is better, but I guess, problem still exists. I
Miloslav> takes some time to load be growing. We will see.

Hmm... how did you setup the new indexes volume?  Did you just use
btrfs again?  Did you mirror your SSDs as well?

Do the indexes fill the SSD, or is there 20-30% free space?  When an
SSD gets fragmented, it's performance can drop quite a bit.  Did you
put the SSDs onto a seperate controller?  Probably not.  So now you've
just increased the load on the single controller, when you really
should be spreading it out more to improve things.

Another possible hack would be to move some stuff to a RAM disk,
assuming your server is on a UPS/Generator incase of power loss.  But
that's an unsafe hack. 

Also, do you have quotas turned on?  That's a performance hit for
sure. 

Miloslav> Thank you for the fio tip. Definetly I'll try that.

It's a good way to test and measure how the system will react.
Unfortunately, you will need to do your testing outside of normal work
hours so as to not impact your users too much.

Good luck!   Please post some numbers if you get them.  If you see
only a few disks are 75% or more busy, then *maybe* you have a bad
disk in the system, and moving off that disk or replacing it might
help.  Again, hard to know.

Rebalancing btrfs might also help, especially now that you've moved
the indexes off that volume.

Robert: 

Fio is an acronym for Flexible IO Tester and describes a tool for measuring IO performance. With Fio, devices such as hard drives or SSDs can be tested for their speed by executing a user-defined workload and collecting performance data. Therefore it might be difficult to really simulate the load with that, because You need to define the workload Yourself.
But, at least, You might use it to get an idea of maximum transfer rates, random I/O etc.

iotop shows the current I/O transfer rates for the currently running processes / threads. It uses the I/O usage information of the Linux kernel, so it might be a good tool for You.
htop might also be Your friend.
and of course, as someone else pointed out : 

> 

Looking at the output of 'iostat -x 30' might be a good thing.



my default setup for such diagnosis is to run iotop, htop, ntop to get an idea whats going on and measure ... 
for the index-SSD (on a different Channel ? NvMe ?) I strongly suggest to compare EXT4 vs BTRFS - some Benchmarks suggest that on fast SSD´s EXT4 is much (factor 4-5) faster then btrfs.

You might also try ZSTD transparent compression on the remaining (non-SSD) btrfs disks - this can boost the transfer rate (since the data is compressed) if the Server CPU´s have enough juice.
But from my guts feeling, its not the transfer rate - its the random I/O (in that case the situation will get worse with compression), but You really need to measure that. 

In general I dont understand that (old fashioned) configuration - using a bunch of 2TB or even 4TB TLC SSD in Raid10 is nowadays really affordable. You can keep the old disks for (local) rsync before You rsync from there to the NAS ... 

Robert