Btrfs RAID-10 performance

Miloslav Hůla miloslav.hula at gmail.com
Mon Sep 7 12:38:25 EEST 2020


Hello,

I sent this into the Linux Kernel Btrfs mailing list and I got reply: 
"RAID-1 would be preferable" 
(https://lore.kernel.org/linux-btrfs/7b364356-7041-7d18-bd77-f60e0e2e2112@lechevalier.se/T/). 
May I ask you for the comments as from people around the Dovecot?


We are using btrfs RAID-10 (/data, 4.7TB) on a physical Supermicro 
server with Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz and 125GB of RAM. 
We run 'btrfs scrub start -B -d /data' every Sunday as a cron task. It 
takes about 50 minutes to finish.

# uname -a
Linux imap 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64 
GNU/Linux

RAID is a composition of 16 harddrives. Harddrives are connected via 
AVAGO MegaRAID SAS 9361-8i as a RAID-0 devices. All harddrives are SAS 
2.5" 15k drives.

Server serves as a IMAP with Dovecot 2.2.27-3+deb9u6, 4104 accounts, 
Mailbox format, LMTP delivery.

We run 'rsync' to remote NAS daily. It takes about 6.5 hours to finish, 
12'265'387 files last night.


Last half year, we encoutered into performace troubles. Server load 
grows up to 30 in rush hours, due to IO waits. We tried to attach next 
harddrives (the 838G ones in a list below) and increase a free space by 
rebalace. I think, it helped a little bit, not not so rapidly.

Is this a reasonable setup and use case for btrfs RAID-10? If so, are 
there some recommendations to achieve better performance?

Thank you. With kind regards
Milo



# megaclisas-status
-- Controller information --
-- ID | H/W Model                  | RAM    | Temp | BBU    | Firmware
c0    | AVAGO MegaRAID SAS 9361-8i | 1024MB | 72C  | Good   | FW: 
24.16.0-0082

-- Array information --
-- ID | Type   |    Size |  Strpsz | Flags | DskCache |   Status |  OS 
Path | CacheCade |InProgress
c0u0  | RAID-0 |    838G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdq | None      |None
c0u1  | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sda | None      |None
c0u2  | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdb | None      |None
c0u3  | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdc | None      |None
c0u4  | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdd | None      |None
c0u5  | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sde | None      |None
c0u6  | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdf | None      |None
c0u7  | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdg | None      |None
c0u8  | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdh | None      |None
c0u9  | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdi | None      |None
c0u10 | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdj | None      |None
c0u11 | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdk | None      |None
c0u12 | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdl | None      |None
c0u13 | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdm | None      |None
c0u14 | RAID-0 |    558G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdn | None      |None
c0u15 | RAID-0 |    838G |  256 KB | RA,WB |  Enabled |  Optimal | 
/dev/sdr | None      |None

-- Disk information --
-- ID   | Type | Drive Model                       | Size     | Status 
      | Speed    | Temp | Slot ID  | LSI ID
c0u0p0  | HDD  | SEAGATE ST900MP0006 N003WAG0Q3S3  | 837.8 Gb | Online, 
Spun Up | 12.0Gb/s | 53C  | [8:14]   | 32
c0u1p0  | HDD  | HGST HUC156060CSS200 A3800XV250TJ | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 38C  | [8:0]    | 12
c0u2p0  | HDD  | HGST HUC156060CSS200 A3800XV3XT4J | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 43C  | [8:1]    | 11
c0u3p0  | HDD  | HGST HUC156060CSS200 ADB05ZG4XLZU | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 46C  | [8:2]    | 25
c0u4p0  | HDD  | HGST HUC156060CSS200 A3800XV3DWRL | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 48C  | [8:3]    | 14
c0u5p0  | HDD  | HGST HUC156060CSS200 A3800XV3XZTL | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 52C  | [8:4]    | 18
c0u6p0  | HDD  | HGST HUC156060CSS200 A3800XV3VSKJ | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 55C  | [8:5]    | 15
c0u7p0  | HDD  | SEAGATE ST600MP0006 N003WAF1LWKE  | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 56C  | [8:6]    | 28
c0u8p0  | HDD  | HGST HUC156060CSS200 A3800XV3XTDJ | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 55C  | [8:7]    | 20
c0u9p0  | HDD  | HGST HUC156060CSS200 A3800XV3T8XL | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 57C  | [8:8]    | 19
c0u10p0 | HDD  | HGST HUC156060CSS200 A7030XHL0ZYP | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 61C  | [8:9]    | 23
c0u11p0 | HDD  | HGST HUC156060CSS200 ADB05ZG4VR3P | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 60C  | [8:10]   | 24
c0u12p0 | HDD  | SEAGATE ST600MP0006 N003WAF195KA  | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 60C  | [8:11]   | 29
c0u13p0 | HDD  | SEAGATE ST600MP0006 N003WAF1LTZW  | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 56C  | [8:12]   | 26
c0u14p0 | HDD  | SEAGATE ST600MP0006 N003WAF1LWH6  | 558.4 Gb | Online, 
Spun Up | 12.0Gb/s | 55C  | [8:13]   | 27
c0u15p0 | HDD  | SEAGATE ST900MP0006 N003WAG0Q414  | 837.8 Gb | Online, 
Spun Up | 12.0Gb/s | 47C  | [8:15]   | 33



# btrfs --version
btrfs-progs v4.7.3



# btrfs fi show
Label: 'DATA'  uuid: 5b285a46-e55d-4191-924f-0884fa06edd8
         Total devices 16 FS bytes used 3.49TiB
         devid    1 size 558.41GiB used 448.66GiB path /dev/sda
         devid    2 size 558.41GiB used 448.66GiB path /dev/sdb
         devid    4 size 558.41GiB used 448.66GiB path /dev/sdd
         devid    5 size 558.41GiB used 448.66GiB path /dev/sde
         devid    7 size 558.41GiB used 448.66GiB path /dev/sdg
         devid    8 size 558.41GiB used 448.66GiB path /dev/sdh
         devid    9 size 558.41GiB used 448.66GiB path /dev/sdf
         devid   10 size 558.41GiB used 448.66GiB path /dev/sdi
         devid   11 size 558.41GiB used 448.66GiB path /dev/sdj
         devid   13 size 558.41GiB used 448.66GiB path /dev/sdk
         devid   14 size 558.41GiB used 448.66GiB path /dev/sdc
         devid   15 size 558.41GiB used 448.66GiB path /dev/sdl
         devid   16 size 558.41GiB used 448.66GiB path /dev/sdm
         devid   17 size 558.41GiB used 448.66GiB path /dev/sdn
         devid   18 size 837.84GiB used 448.66GiB path /dev/sdr
         devid   19 size 837.84GiB used 448.66GiB path /dev/sdq



# btrfs fi df /data/
Data, RAID10: total=3.48TiB, used=3.47TiB
System, RAID10: total=256.00MiB, used=320.00KiB
Metadata, RAID10: total=21.00GiB, used=18.17GiB
GlobalReserve, single: total=512.00MiB, used=0.00B



I do not attach whole dmesg log. It is almost empty, without errors. 
Only lines about BTRFS are about relocations, like:

BTRFS info (device sda): relocating block group 29435663220736 flags 65
BTRFS info (device sda): found 54460 extents
BTRFS info (device sda): found 54459 extents


More information about the dovecot mailing list