Lustre FS BMT – 2016

116: 4194304 bytes 90 times –> 57624.10 Mbps in 555.32 usec
117: 4194307 bytes 90 times –> 57595.43 Mbps in 555.60 usec
118: 6291453 bytes 89 times –> 57859.07 Mbps in 829.60 usec
119: 6291456 bytes 80 times –> 57753.10 Mbps in 831.12 usec
120: 6291459 bytes 80 times –> 57741.74 Mbps in 831.29 usec
121: 8388605 bytes 40 times –> 57931.57 Mbps in 1104.75 usec
122: 8388608 bytes 45 times –> 57828.93 Mbps in 1106.71 usec
123: 8388611 bytes 45 times –> 57818.01 Mbps in 1106.92 usec

1. iozone

cmd : iozone -i 0 -i 1 -c -e -w -r 1024k -s 10g -t 1
infiniband qdr 40gb + MDS, OSS (1대) + OST 3개

local disk : 132486.09 KB/sec

writers : 413665.66 KB/sec (2.8.53)
readers : 5543444.00 KB/sec
writers : 425039.97 KB/sec (2.8.50)

TCP 1gb + ISCSI SVR (MDT 1개, OST 6개) + MDS (1대) + OSS (2대) + OST 6개

local disk : 100353.77 KB/sec

writers : 111260.23 KB/sec
readers : 157678.58 KB/sec

TCP 1gb + NFS

local disk : 83102.07 KB/sec

writers : 86601.88 KB/sec
2. obdfilter-survey
ib :

# objhi=2 thrhi=2 size=1024 targets=”gcfs-OST0001 gcfs-OST0002 gcfs-OST0003″ obdfilter-survey
Tue May 17 16:58:05 KST 2016 Obdfilter-survey for case=disk from master01
ost 3 sz 3145728K rsz 1024K obj 3 thr 3 write 485.59 [ 112.99, 242.97] rewrite 388.49 [ 110.99, 135.91] read 52388.72 SHORT
ost 3 sz 3145728K rsz 1024K obj 3 thr 6 write 399.23 [ 108.99, 134.99] rewrite 398.57 [ 108.99, 138.99] read 42676.97 SHORT
ost 3 sz 3145728K rsz 1024K obj 6 thr 6 write 393.78 [ 107.99, 136.99] rewrite 386.39 [ 87.98, 135.99] read 91887.34 SHORT

tcp :

# objhi=2 thrhi=2 size=1024 targets=”gcfs-OST0000 gcfs-OST0001 gcfs-OST0002″ obdfilter-survey
Tue May 17 16:58:26 KST 2016 Obdfilter-survey for case=disk from OSS001
ost 3 sz 3145728K rsz 1024K obj 3 thr 3 write 106.47 [ 34.00, 37.00] rewrite 106.50 [ 34.94, 37.00] read 43694.52 SHORT
ost 3 sz 3145728K rsz 1024K obj 3 thr 6 write 109.08 [ 32.00, 41.00] rewrite 109.43 [ 29.00, 41.00] read 48396.54 SHORT
ost 3 sz 3145728K rsz 1024K obj 6 thr 6 write 109.45 [ 35.00, 38.00] rewrite 109.50 [ 35.00, 38.00] read 66188.09 SHORT
done!
3. ost-survey

ib :

# ost-survey -s 1000 /home

Ost# Read(MB/s) Write(MB/s) Read-time Write-time
—————————————————-
0 1113.214 415.951 0.898 2.404
1 773.853 268.545 1.292 3.724
2 823.993 263.385 1.214 3.797

tcp :

Ost# Read(MB/s) Write(MB/s) Read-time Write-time
—————————————————-
0 330.635 102.068 3.024 9.797
1 323.059 105.008 3.095 9.523
2 312.162 102.320 3.203 9.773
3 297.242 100.523 3.364 9.948
4 299.927 104.588 3.334 9.561
5 301.015 103.178 3.322 9.692
4. ior

mpirun -np 8 /root/bmt/IOR/src/C/IOR -v -F -t 1m -b 1000m -o `date +%Y-%m-%d_%H-%M-%S`
ib :

Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize

——— ——— ——— ———- ——- ——— ——— ———- ——- ——–
write 408.04 408.04 408.04 0.00 408.04 408.04 408.04 0.00 19.60585 8 8 1 1 0 1 0 0 1 1048576000 1048576 8388608000 -1 POSIX EXCEL
read 17677.41 17677.41 17677.41 0.00 17677.41 17677.41 17677.41 0.00 0.45255 8 8 1 1 0 1 0 0 1 1048576000 1048576 8388608000 -1 POSIX EXCEL

Max Write: 408.04 MiB/sec (427.86 MB/sec)
Max Read: 17677.41 MiB/sec (18536.11 MB/sec)

tcp :

peration Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize

——— ——— ——— ———- ——- ——— ——— ———- ——- ——–
write 101.59 101.59 101.59 0.00 101.59 101.59 101.59 0.00 39.37228 8 4 1 1 0 1 0 0 1 524288000 1048576 4194304000 -1 POSIX EXCEL
read 13498.42 13498.42 13498.42 0.00 13498.42 13498.42 13498.42 0.00 0.29633 8 4 1 1 0 1 0 0 1 524288000 1048576 4194304000 -1 POSIX EXCEL

Max Write: 101.59 MiB/sec (106.53 MB/sec)
Max Read: 13498.42 MiB/sec (14154.12 MB/sec)
5. mdtest

mpirun -np 8 /root/bmt/mdtest/mdtest -I 10 -i 1 -z 3 -b 3 -d ./

ib :

mdtest-1.9.3 was launched with 8 total task(s) on 1 node(s)
Command line used: /root/bmt/mdtest/mdtest -I 10 -i 1 -z 3 -b 3 -d ./
Path: /home
FS: 5.4 TiB Used FS: 0.4% Inodes: 7.3 Mi Used Inodes: 0.0%

8 tasks, 3200 files/directories

SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
——— — — —- ——-
Directory creation: 29132.548 29132.548 29132.548 0.000
Directory stat : 43832.065 43832.065 43832.065 0.000
Directory removal : 33516.558 33516.558 33516.558 0.000
File creation : 18309.891 18309.891 18309.891 0.000
File stat : 27824.585 27824.585 27824.585 0.000
File read : 29267.095 29267.095 29267.095 0.000
File removal : 30069.524 30069.524 30069.524 0.000
Tree creation : 5204.981 5204.981 5204.981 0.000
Tree removal : 6958.902 6958.902 6958.902 0.000
tcp :

— started at 03/31/2016 20:47:51 —

mdtest-1.9.3 was launched with 8 total task(s) on 2 node(s)
Command line used: /APP/enhpc/mdtest/mdtest -I 10 -i 2 -z 5 -b 5 -d /home/mdtest
Path: /home
FS: 5.4 TiB Used FS: 0.1% Inodes: 41.2 Mi Used Inodes: 0.0%

8 tasks, 312480 files/directories

SUMMARY: (of 2 iterations)
Operation Max Min Mean Std Dev
——— — — —- ——-
Directory creation: 9914.734 7713.628 8814.181 1100.553
Directory stat : 16537.097 14552.808 15544.952 992.144
Directory removal : 11672.575 9810.903 10741.739 930.836
File creation : 6572.272 4553.086 5562.679 1009.593
File stat : 9076.555 8897.806 8987.180 89.374
File read : 7959.927 7811.317 7885.622 74.305
File removal : 6977.820 6678.429 6828.124 149.696
Tree creation : 3108.665 2539.653 2824.159 284.506
Tree removal : 1780.642 1667.998 1724.320 56.322

// zfs

local disk : 46958.58 KB/sec
writers : 102552.48 KB/sec

매우 성능이 안좋음…ㅡ.ㅡ
// Doc 에 나온 tuning 후 ..

– iozone

writers : 401670.97 KB/sec 으로 별반 차이 없음 .. 오히려 조금 떨어져 보임..

– obdfilter-survey

조금 좋아진듯..그래도 비슷

# objhi=2 thrhi=2 size=1024 targets=”gcfs-OST0001 gcfs-OST0002 gcfs-OST0003″ obdfilter-survey
Thu May 19 09:28:54 KST 2016 Obdfilter-survey for case=disk from master01
ost 3 sz 3145728K rsz 1024K obj 3 thr 3 write 482.52 [ 96.99, 275.96] rewrite 368.84 [ 39.00, 161.98] read 56096.92 SHORT
ost 3 sz 3145728K rsz 1024K obj 3 thr 6 write 415.41 [ 110.99, 137.99] rewrite 397.90 [ 106.00, 135.99] read 42537.34 SHORT
ost 3 sz 3145728K rsz 1024K obj 6 thr 6 write 403.54 [ 114.00, 134.99] rewrite 384.88 [ 67.99, 135.99] read 88516.48 SHORT
– ost-survey

비슷..

Ost# Read(MB/s) Write(MB/s) Read-time Write-time
—————————————————-
0 752.870 431.051 1.328 2.320
1 634.774 289.960 1.575 3.449
2 581.304 262.809 1.720 3.805
– IOR

비슷 ..

Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize

——— ——— ——— ———- ——- ——— ——— ———- ——- ——–
write 402.24 402.24 402.24 0.00 402.24 402.24 402.24 0.00 79.55380 8 8 1 1 0 1 0 0 1 4194304000 1048576 33554432000 -1 POSIX EXCEL
read 18188.68 18188.68 18188.68 0.00 18188.68 18188.68 18188.68 0.00 1.75934 8 8 1 1 0 1 0 0 1 4194304000 1048576 33554432000 -1 POSIX EXCEL

Max Write: 402.24 MiB/sec (421.78 MB/sec)
Max Read: 18188.68 MiB/sec (19072.21 MB/sec)

– mdtest

다소 떨어짐.

SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
——— — — —- ——-
Directory creation: 26058.203 26058.203 26058.203 0.000
Directory stat : 51273.151 51273.151 51273.151 0.000
Directory removal : 32509.102 32509.102 32509.102 0.000
File creation : 17190.805 17190.805 17190.805 0.000
File stat : 25999.347 25999.347 25999.347 0.000
File read : 28903.809 28903.809 28903.809 0.000
File removal : 29713.515 29713.515 29713.515 0.000
Tree creation : 4462.263 4462.263 4462.263 0.000
Tree removal : 9035.554 9035.554 9035.554 0.000

// ldisk + LVM

– iozone
iozone -i 0 -c -e -w -r 1024k -s 10g -t 1
writers = 416205.94 KB/sec

– obdfilter-survey
objhi=2 thrhi=2 size=1024 targets=”gcfs-OST0001 gcfs-OST0002 gcfs-OST0003″ obdfilter-survey

Thu May 19 16:07:44 KST 2016 Obdfilter-survey for case=disk from master01
ost 3 sz 3145728K rsz 1024K obj 3 thr 3 write 454.08 [ 107.99, 248.98] rewrite 364.25 [ 59.99, 134.99] read 58285.32 SHORT
ost 3 sz 3145728K rsz 1024K obj 3 thr 6 write 448.04 [ 130.99, 168.98] rewrite 375.58 [ 97.98, 134.99] read 48271.44 SHORT
ost 3 sz 3145728K rsz 1024K obj 6 thr 6 write 424.08 [ 111.99, 179.99] rewrite 375.87 [ 73.99, 153.99] read 87206.87 SHORT

– IOR
mpirun -np 8 /root/bmt/IOR/src/C/IOR -v -F -t 1m -b 1000m -o `date +%Y-%m-%d_%H-%M-%S`

Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize

——— ——— ——— ———- ——- ——— ——— ———- ——- ——–
write 423.36 423.36 423.36 0.00 423.36 423.36 423.36 0.00 18.89623 8 8 1 1 0 1 0 0 1 1048576000 1048576 8388608000 -1 POSIX EXCEL
read 17756.00 17756.00 17756.00 0.00 17756.00 17756.00 17756.00 0.00 0.45055 8 8 1 1 0 1 0 0 1 1048576000 1048576 8388608000 -1 POSIX EXCEL

Max Write: 423.36 MiB/sec (443.93 MB/sec)
Max Read: 17756.00 MiB/sec (18618.51 MB/sec)

– mdtest
mpirun -np 8 /root/bmt/mdtest/mdtest -I 10 -i 1 -z 3 -b 3 -d ./
SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
——— — — —- ——-
Directory creation: 43677.728 43677.728 43677.728 0.000
Directory stat : 46887.798 46887.798 46887.798 0.000
Directory removal : 33176.059 33176.059 33176.059 0.000
File creation : 15498.010 15498.010 15498.010 0.000
File stat : 22317.547 22317.547 22317.547 0.000
File read : 29228.599 29228.599 29228.599 0.000
File removal : 32864.041 32864.041 32864.041 0.000
Tree creation : 4869.171 4869.171 4869.171 0.000
Tree removal : 7996.766 7996.766 7996.766 0.000
// 정상유무 판단하기

1. 우선 OST 단위별 기본 성능 측정
ldiskfs 로 mount 하거나 lustre format 전에 ext4 로 format 하여
해당 OST 의 iozone 실행

2. obdfilter-survey 로 OST 별 Block 레벨 성능 측정

3. collectl 로 oss 에 들어오는 ib 성능 측정

4. 최종 client 에서의 iozone 성능이 위 3개 테스트의 성능과 일치하는지
확인

// DELL
MDS : 1대

Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz * 24core
64GB
MDT -> RAID-1 SAS 1.2TB

OSS : 1대
Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz * 24core
OST : 3개 -> RAID-6 1.2TB * 8 (6:data, 2:패럴티)

SAS HBA : 24Gb

LNET : QDR 40Gb
OSS> nobjhi=2 thrhi=2 size=1024 targets=”gcfs-OST0000 gcfs-OST0001 gcfs-OST0002″ obdfilter-survey
Sat Jun 4 14:12:33 KST 2016 Obdfilter-survey for case=disk from jo-oss01
ost 3 sz 3145728K rsz 1024K obj 3 thr 3 write 2474.18 SHORT rewrite 2290.67 SHORT read 64311.96 SHORT
ost 3 sz 3145728K rsz 1024K obj 3 thr 6 write 2396.60 SHORT rewrite 2534.07 SHORT read 40313.19 SHORT
ost 3 sz 3145728K rsz 1024K obj 6 thr 6 write 2712.60 SHORT rewrite 2316.08 SHORT read 100146.14 SHORT
OSS> nobjhi=2 thrhi=2 size=2048 targets=”gcfs-OST0000 gcfs-OST0001 gcfs-OST0002″ obdfilter-survey
Sat Jun 4 14:16:24 KST 2016 Obdfilter-survey for case=disk from jo-oss01
ost 3 sz 6291456K rsz 1024K obj 3 thr 3 write 2428.32 [ 732.96, 927.95] rewrite 2503.84 [ 721.96, 908.86] read 59632.03 SHORT
ost 3 sz 6291456K rsz 1024K obj 3 thr 6 write 2613.22 [ 795.97, 863.97] rewrite 2504.75 [ 803.95, 846.97] read 33903.62 SHORT
ost 3 sz 6291456K rsz 1024K obj 6 thr 6 write 2501.44 [ 811.94, 837.97] rewrite 2551.33 [ 824.94, 831.97] read 102343.16 SHORT
done!

client>

# iozone -i 0 -c -e -w -r 1024k -s 20g -t 1
Children see throughput for 1 initial writers = 1772764.00 KB/sec
Children see throughput for 1 rewriters = 2460944.25 KB/sec

# iozone -i 0 -c -e -w -r 1024k -s 5g -t 1
Children see throughput for 1 initial writers = 1821404.38 KB/sec
Children see throughput for 1 rewriters = 2540168.50 KB/sec
# iozone -i 0 -c -e -w -r 1024k -s 5g -t 4
Children see throughput for 4 initial writers = 2466344.50 KB/sec
Parent sees throughput for 4 initial writers = 2393456.64 KB/sec
Avg throughput per process = 616586.12 KB/sec

Children see throughput for 4 rewriters = 2455132.69 KB/sec
Parent sees throughput for 4 rewriters = 2453261.08 KB/sec
Avg throughput per process = 613783.17 KB/sec
# iozone -i 0 -i 1 -c -e -w -r 1024k -s 20g -t 1
Children see throughput for 1 initial writers = 2505180.00 KB/sec
Parent sees throughput for 1 initial writers = 2505101.80 KB/sec
Avg throughput per process = 2505180.00 KB/sec

Children see throughput for 1 rewriters = 2501997.00 KB/sec
Parent sees throughput for 1 rewriters = 2501914.19 KB/sec
Avg throughput per process = 2501997.00 KB/sec

Children see throughput for 1 readers = 6602296.00 KB/sec
Parent sees throughput for 1 readers = 6601737.13 KB/sec
Avg throughput per process = 6602296.00 KB/sec

Children see throughput for 1 re-readers = 5552512.50 KB/sec
Parent sees throughput for 1 re-readers = 5552086.64 KB/sec
Avg throughput per process = 5552512.50 KB/sec
# iozone -i 0 -c -e -w -r 1024k -s 50g -t 1
Children see throughput for 1 initial writers = 1488265.00 KB/sec
Children see throughput for 1 rewriters = 1544182.25 KB/sec

# iozone -i 0 -c -e -w -r 1024k -s 20g -t 2 -+n -+m ./client_write
Test running:
Children see throughput for 2 initial writers = 2536592.12 KB/sec
Min throughput per process = 1261948.88 KB/sec
Max throughput per process = 1274643.25 KB/sec
Avg throughput per process = 1268296.06 KB/sec
Min xfer = 20762624.00 KB
// stripe count 3

# mpirun -np 8 -machinefile ./nodelist ./mdtest -I 10 -i 1 -z 3 -b 3 -d /home/mdtest
— started at 06/04/2016 14:58:45 —

mdtest-1.9.3 was launched with 8 total task(s) on 2 node(s)
Command line used: ./mdtest -I 10 -i 1 -z 3 -b 3 -d /home/mdtest
Path: /home
FS: 29.2 TiB Used FS: 0.0% Inodes: 19.6 Mi Used Inodes: 0.0%

8 tasks, 3200 files/directories

SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
——— — — —- ——-
Directory creation: 31911.242 31911.242 31911.242 0.000
Directory stat : 70385.300 70385.300 70385.300 0.000
Directory removal : 42422.413 42422.413 42422.413 0.000
File creation : 22867.796 22867.796 22867.796 0.000
File stat : 36521.731 36521.731 36521.731 0.000
File read : 37598.425 37598.425 37598.425 0.000
File removal : 39367.195 39367.195 39367.195 0.000
Tree creation : 5258.986 5258.986 5258.986 0.000
Tree removal : 3798.328 3798.328 3798.328 0.000

— finished at 06/04/2016 14:58:46 —
# mpirun -np 8 -machinefile ./nodelist ./mdtest -I 10 -i 1 -z 5 -b 5 -d /home/mdtest
— started at 06/04/2016 15:00:07 —

mdtest-1.9.3 was launched with 8 total task(s) on 2 node(s)
Command line used: ./mdtest -I 10 -i 1 -z 5 -b 5 -d /home/mdtest
Path: /home
FS: 29.2 TiB Used FS: 0.0% Inodes: 19.6 Mi Used Inodes: 0.0%

8 tasks, 312480 files/directories

SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
——— — — —- ——-
Directory creation: 6064.029 6064.029 6064.029 0.000
Directory stat : 97942.174 97942.174 97942.174 0.000
Directory removal : 19754.684 19754.684 19754.684 0.000
File creation : 8186.819 8186.819 8186.819 0.000
File stat : 38601.782 38601.782 38601.782 0.000
File read : 46221.119 46221.119 46221.119 0.000
File removal : 26270.054 26270.054 26270.054 0.000
Tree creation : 9396.038 9396.038 9396.038 0.000
Tree removal : 4152.139 4152.139 4152.139 0.000

— finished at 06/04/2016 15:02:24 —

// stripe count 1

# mpirun -np 8 -machinefile ./nodelist ./mdtest -I 10 -i 1 -z 3 -b 3 -d /home/mdtest
— started at 06/04/2016 15:08:09 —

mdtest-1.9.3 was launched with 8 total task(s) on 2 node(s)
Command line used: ./mdtest -I 10 -i 1 -z 3 -b 3 -d /home/mdtest
Path: /home
FS: 29.2 TiB Used FS: 0.0% Inodes: 58.4 Mi Used Inodes: 0.0%

8 tasks, 3200 files/directories

SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
——— — — —- ——-
Directory creation: 42051.586 42051.586 42051.586 0.000
Directory stat : 62541.461 62541.461 62541.461 0.000
Directory removal : 38661.857 38661.857 38661.857 0.000
File creation : 25529.103 25529.103 25529.103 0.000
File stat : 44634.797 44634.797 44634.797 0.000
File read : 39853.473 39853.473 39853.473 0.000
File removal : 49130.350 49130.350 49130.350 0.000
Tree creation : 5723.278 5723.278 5723.278 0.000
Tree removal : 3782.486 3782.486 3782.486 0.000

— finished at 06/04/2016 15:08:09 —

# mpirun -np 8 -machinefile ./nodelist ./mdtest -I 10 -i 1 -z 5 -b 5 -d /home/mdtest
— started at 06/04/2016 15:08:30 —

mdtest-1.9.3 was launched with 8 total task(s) on 2 node(s)
Command line used: ./mdtest -I 10 -i 1 -z 5 -b 5 -d /home/mdtest
Path: /home
FS: 29.2 TiB Used FS: 0.0% Inodes: 58.4 Mi Used Inodes: 0.0%

8 tasks, 312480 files/directories

SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
——— — — —- ——-
Directory creation: 17579.561 17579.561 17579.561 0.000
Directory stat : 97257.354 97257.354 97257.354 0.000
Directory removal : 5179.125 5179.125 5179.125 0.000
File creation : 26014.655 26014.655 26014.655 0.000
File stat : 49831.208 49831.208 49831.208 0.000
File read : 43628.377 43628.377 43628.377 0.000
File removal : 51289.037 51289.037 51289.037 0.000
Tree creation : 9416.496 9416.496 9416.496 0.000
Tree removal : 3985.962 3985.962 3985.962 0.000

— finished at 06/04/2016 15:10:24 —

# ost-survey -s 1000 /home
/usr/bin/ost-survey: 06/04/16 OST speed survey on /home from 10.0.0.150@o2ib
Number of Active OST devices : 3
Worst Read OST indx: 0 speed: 490.472783
Best Read OST indx: 1 speed: 519.569594
Read Average: 507.891920 +/- 12.553337 MB/s
Worst Write OST indx: 1 speed: 1053.816365
Best Write OST indx: 2 speed: 1189.011315
Write Average: 1122.279527 +/- 55.206684 MB/s
Ost# Read(MB/s) Write(MB/s) Read-time Write-time
—————————————————-
0 490.473 1124.011 2.039 0.890
1 519.570 1053.816 1.925 0.949
2 513.633 1189.011 1.947 0.841