Hi All,
We're on migration stage to a new 2-node active/active cluster with shared iSCSI storage, everything is ready, but we are in the process of I/O testing, before we going to in production.
Cluster info:
2 nodes with CentOS 7.4
balancer ( keepalived + ldirectord ), 2 dovecot directors, 2 dovecot IMAP backends, 2 Postfix SMTP, gfs2 ( corosync + dlm ), LMTP between Postfix and Dovecot directors ( proxy ) and Dovecot backends.
Everything seems normal, the users are mapped to dovecot backends, the balancers and the directors are working properly.
The measurements were created under the following conditions :
- operating system : CentOS Linux release 7.4.1708 (Core)- virtual server (KVM)
- (2 cores per virtual socket) and 4 GB RAM
- RAID 10 ( 6 x 2TB Seagate ST2000NX0273 SAS 7200 RPM ( iSCSI SAN Infrotrend Storage DS 4024 ), partition 5.45 TB formatted with gfs2.
- Dovecot 2.2.10
- imaptest v.1.0.0
We run a test with ./imaptest userfile=users.txt mbox=dovecot-crlf secs=300 clients=50 append=100,0 logout=0 msgs=10000000, part of the output is that, but we can't understand what mean these values, anyone who has experience with imaptest can explain what they mean?
100% 50% 50% 100% 100% 100% 50% 100% 100% 100%
30%
0 223 216 0 442 657 219 218 443 438 50/ 50
0 229 212 0 418 596 194 224 428 424 50/ 50
0 217 226 0 436 632 227 223 429 428 50/ 50
0 214 226 0 432 644 207 193 429 435 50/ 50
0 213 204 0 414 587 209 213 414 417 50/ 50
0 195 206 0 427 611 213 203 426 420 50/ 50
0 225 200 0 435 642 215 226 439 445 50/ 50
0 229 194 0 441 637 223 213 441 436 50/ 50
0 213 219 0 434 616 212 232 431 428 50/ 50
0 210 213 0 427 622 203 219 434 440 50/ 50
0 112 113 0 35 30 30 56 87 111 ms/cmd avg
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe
100% 50% 50% 100% 100% 100% 50% 100% 100% 100%
30%
0 231 246 0 443 622 204 211 444 440 50/ 50
0 215 211 0 431 601 194 219 428 430 50/ 50
0 230 228 0 445 626 216 235 445 440 50/ 50
0 218 234 0 429 624 182 225 420 428 50/ 50
0 223 222 0 427 640 214 215 430 431 50/ 50
0 210 219 0 424 582 201 213 433 428 50/ 50
0 198 208 0 418 579 212 210 416 412 50/ 50
0 221 211 0 419 617 209 215 418 420 50/ 50
0 208 196 0 405 566 223 189 413 413 50/ 50
0 215 244 0 441 639 198 212 431 435 50/ 50
0 115 114 0 35 30 30 57 87 112 ms/cmd avg
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe
100% 50% 50% 100% 100% 100% 50% 100% 100% 100%
30%
0 199 219 0 422 608 206 205 424 420 50/ 50
0 214 206 0 429 616 209 208 432 435 50/ 50
0 227 204 0 413 583 200 213 412 410 50/ 50
0 192 214 0 416 609 212 219 411 415 50/ 50
0 212 215 0 426 625 204 225 434 428 50/ 50
0 202 209 0 423 585 224 197 415 421 50/ 50
0 216 221 0 422 599 190 197 427 426 50/ 50
0 208 207 0 414 579 209 211 414 410 50/ 50
0 205 211 0 432 616 212 213 423 427 50/ 50
0 225 207 0 432 642 208 215 436 434 50/ 50
0 115 115 0 36 31 32 59 89 113 ms/cmd avg
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe
100% 50% 50% 100% 100% 100% 50% 100% 100% 100%
30%
0 211 216 0 431 614 212 221 437 439 50/ 50
0 165 159 0 324 487 160 148 322 315 50/ 50
0 225 199 0 406 568 215 209 403 410 50/ 50
0 213 217 0 441 617 202 220 448 442 50/ 50
0 221 230 0 439 609 229 216 428 437 50/ 50
0 227 211 0 429 595 210 206 432 430 50/ 50
0 196 218 0 423 619 209 207 425 419 50/ 50
0 218 194 0 422 596 199 212 417 422 50/ 50
0 223 213 0 427 599 221 220 434 432 50/ 50
0 185 203 0 405 558 226 211 399 403 50/ 50
0 118 118 0 37 32 31 58 91 116 ms/cmd avg
Totals:
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe
100% 50% 50% 100% 100% 100% 50% 100% 100% 100%
30%
50 61115 61109 50 122229 174619 59582 60969 122229 122229
Both nodеs pick up load average about 20-30 and io wait 14-20%.
top - 16:34:50 up 7 days, 1:35, 1 user, load average: 22.22, 9.18, 3.47
Tasks: 244 total, 4 running, 240 sleeping, 0 stopped, 0 zombie
%Cpu(s): 28.8 us, 29.8 sy, 0.0 ni, 16.4 id, 14.9 wa, 0.0 hi, 10.1 si, 0.0 st
KiB Mem : 4013876 total, 134212 free, 1083648 used, 2796016 buff/cache
KiB Swap: 6292476 total, 5974836 free, 317640 used. 2449504 avail Mem
Do you think that's normal?
Тhis test is only for 50 users, if run it for 100 have many stalled commands...
We have about 500-700 mailboxes and we want to testing a load close to a real environment.
Can anyone help with ideas?
Thank you very much!