Btrfs RAID-10 performance
Miloslav Hůla
miloslav.hula at gmail.com
Mon Sep 7 12:38:25 EEST 2020
Hello,
I sent this into the Linux Kernel Btrfs mailing list and I got reply:
"RAID-1 would be preferable"
(https://lore.kernel.org/linux-btrfs/7b364356-7041-7d18-bd77-f60e0e2e2112@lechevalier.se/T/).
May I ask you for the comments as from people around the Dovecot?
We are using btrfs RAID-10 (/data, 4.7TB) on a physical Supermicro
server with Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz and 125GB of RAM.
We run 'btrfs scrub start -B -d /data' every Sunday as a cron task. It
takes about 50 minutes to finish.
# uname -a
Linux imap 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64
GNU/Linux
RAID is a composition of 16 harddrives. Harddrives are connected via
AVAGO MegaRAID SAS 9361-8i as a RAID-0 devices. All harddrives are SAS
2.5" 15k drives.
Server serves as a IMAP with Dovecot 2.2.27-3+deb9u6, 4104 accounts,
Mailbox format, LMTP delivery.
We run 'rsync' to remote NAS daily. It takes about 6.5 hours to finish,
12'265'387 files last night.
Last half year, we encoutered into performace troubles. Server load
grows up to 30 in rush hours, due to IO waits. We tried to attach next
harddrives (the 838G ones in a list below) and increase a free space by
rebalace. I think, it helped a little bit, not not so rapidly.
Is this a reasonable setup and use case for btrfs RAID-10? If so, are
there some recommendations to achieve better performance?
Thank you. With kind regards
Milo
# megaclisas-status
-- Controller information --
-- ID | H/W Model | RAM | Temp | BBU | Firmware
c0 | AVAGO MegaRAID SAS 9361-8i | 1024MB | 72C | Good | FW:
24.16.0-0082
-- Array information --
-- ID | Type | Size | Strpsz | Flags | DskCache | Status | OS
Path | CacheCade |InProgress
c0u0 | RAID-0 | 838G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdq | None |None
c0u1 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sda | None |None
c0u2 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdb | None |None
c0u3 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdc | None |None
c0u4 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdd | None |None
c0u5 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sde | None |None
c0u6 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdf | None |None
c0u7 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdg | None |None
c0u8 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdh | None |None
c0u9 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdi | None |None
c0u10 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdj | None |None
c0u11 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdk | None |None
c0u12 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdl | None |None
c0u13 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdm | None |None
c0u14 | RAID-0 | 558G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdn | None |None
c0u15 | RAID-0 | 838G | 256 KB | RA,WB | Enabled | Optimal |
/dev/sdr | None |None
-- Disk information --
-- ID | Type | Drive Model | Size | Status
| Speed | Temp | Slot ID | LSI ID
c0u0p0 | HDD | SEAGATE ST900MP0006 N003WAG0Q3S3 | 837.8 Gb | Online,
Spun Up | 12.0Gb/s | 53C | [8:14] | 32
c0u1p0 | HDD | HGST HUC156060CSS200 A3800XV250TJ | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 38C | [8:0] | 12
c0u2p0 | HDD | HGST HUC156060CSS200 A3800XV3XT4J | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 43C | [8:1] | 11
c0u3p0 | HDD | HGST HUC156060CSS200 ADB05ZG4XLZU | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 46C | [8:2] | 25
c0u4p0 | HDD | HGST HUC156060CSS200 A3800XV3DWRL | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 48C | [8:3] | 14
c0u5p0 | HDD | HGST HUC156060CSS200 A3800XV3XZTL | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 52C | [8:4] | 18
c0u6p0 | HDD | HGST HUC156060CSS200 A3800XV3VSKJ | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 55C | [8:5] | 15
c0u7p0 | HDD | SEAGATE ST600MP0006 N003WAF1LWKE | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 56C | [8:6] | 28
c0u8p0 | HDD | HGST HUC156060CSS200 A3800XV3XTDJ | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 55C | [8:7] | 20
c0u9p0 | HDD | HGST HUC156060CSS200 A3800XV3T8XL | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 57C | [8:8] | 19
c0u10p0 | HDD | HGST HUC156060CSS200 A7030XHL0ZYP | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 61C | [8:9] | 23
c0u11p0 | HDD | HGST HUC156060CSS200 ADB05ZG4VR3P | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 60C | [8:10] | 24
c0u12p0 | HDD | SEAGATE ST600MP0006 N003WAF195KA | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 60C | [8:11] | 29
c0u13p0 | HDD | SEAGATE ST600MP0006 N003WAF1LTZW | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 56C | [8:12] | 26
c0u14p0 | HDD | SEAGATE ST600MP0006 N003WAF1LWH6 | 558.4 Gb | Online,
Spun Up | 12.0Gb/s | 55C | [8:13] | 27
c0u15p0 | HDD | SEAGATE ST900MP0006 N003WAG0Q414 | 837.8 Gb | Online,
Spun Up | 12.0Gb/s | 47C | [8:15] | 33
# btrfs --version
btrfs-progs v4.7.3
# btrfs fi show
Label: 'DATA' uuid: 5b285a46-e55d-4191-924f-0884fa06edd8
Total devices 16 FS bytes used 3.49TiB
devid 1 size 558.41GiB used 448.66GiB path /dev/sda
devid 2 size 558.41GiB used 448.66GiB path /dev/sdb
devid 4 size 558.41GiB used 448.66GiB path /dev/sdd
devid 5 size 558.41GiB used 448.66GiB path /dev/sde
devid 7 size 558.41GiB used 448.66GiB path /dev/sdg
devid 8 size 558.41GiB used 448.66GiB path /dev/sdh
devid 9 size 558.41GiB used 448.66GiB path /dev/sdf
devid 10 size 558.41GiB used 448.66GiB path /dev/sdi
devid 11 size 558.41GiB used 448.66GiB path /dev/sdj
devid 13 size 558.41GiB used 448.66GiB path /dev/sdk
devid 14 size 558.41GiB used 448.66GiB path /dev/sdc
devid 15 size 558.41GiB used 448.66GiB path /dev/sdl
devid 16 size 558.41GiB used 448.66GiB path /dev/sdm
devid 17 size 558.41GiB used 448.66GiB path /dev/sdn
devid 18 size 837.84GiB used 448.66GiB path /dev/sdr
devid 19 size 837.84GiB used 448.66GiB path /dev/sdq
# btrfs fi df /data/
Data, RAID10: total=3.48TiB, used=3.47TiB
System, RAID10: total=256.00MiB, used=320.00KiB
Metadata, RAID10: total=21.00GiB, used=18.17GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
I do not attach whole dmesg log. It is almost empty, without errors.
Only lines about BTRFS are about relocations, like:
BTRFS info (device sda): relocating block group 29435663220736 flags 65
BTRFS info (device sda): found 54460 extents
BTRFS info (device sda): found 54459 extents
More information about the dovecot
mailing list