[클러스터][파일] PVFS2 병렬 파일 시스템 환경 구축
##### PVFS2 설치하기 #######
작성일 : 2007년 12월 12일
작성자 : 서진우
1. PVFS2 설치
– meta server, ioserver, pvfs_client 에 모두 설치
# cd /usr/local/src
# tar xzvf pvfs-2.6.3.tar.gz
# cd pvfs-2.6.3
# ./configure –prefix=/usr/local/pvfs2 –with-kernel=/usr/src/kernels/2.6.9-55.EL-smp-x86_64
# make
# make install
# make kmod
# make kmod_install
# depmod -a /lib/modules/2.6.9-55.ELsmp
# modprobe pvfs2
2. PVFS2 설정
– meta server 에 적용
# cd /usr/local/pvfs2/bin
# ./pvfs2-genconfig /etc/pvfs2-fs.conf /etc/pvfs2-server.conf
—————————————————————————————————
**********************************************************************
Welcome to the PVFS2 Configuration Generator:
This interactive script will generate configuration files suitable
for use with a new PVFS2 file system. Please see the PVFS2 quickstart
guide for details.
**********************************************************************
You must first select the network protocol that your file system will use.
The only currently supported options are “tcp”, “gm”, and “ib”.
(For multi-homed configurations, use e.g. “ib,tcp”.)
* Enter protocol type [Default is tcp]:
Choose a TCP/IP port for the servers to listen on. Note that this
script assumes that all servers will use the same port number.
* Enter port number [Default is 3334]:
Choose a directory for each server to store data in.
* Enter directory name: [Default is /pvfs2-storage-space]: /clusterfs
Choose a file for each server to write log messages to.
* Enter log file location [Default is /tmp/pvfs2-server.log]: /var/log/pvfs2-server.log
Next you must list the hostnames of the machines that will act as
I/O servers. Acceptable syntax is “node1, node2, …” or “node{#-#,#,#}”.
* Enter hostnames [Default is localhost]: dm002,dm003,dm004
Now list the hostnames of the machines that will act as Metadata
servers. This list may or may not overlap with the I/O server list.
* Enter hostnames [Default is localhost]: dm001
Configured a total of 4 servers:
3 of them are I/O servers.
1 of them are Metadata servers.
* Would you like to verify server list (y/n) [Default is n]?
Writing fs config file… Done.
Writing 4 server config file(s)… Done.
———————————————————————————————————
# /usr/local/src/pvfs-2.6.3/examples
# cp pvfs2-server.rc /etc/rc.d/init.d/pvfs2-server
# chmod 755 /etc/rc.d/init.d/pvfs2-server
# ensync /etc/pvfs*
# ensync /etc/rc.d/init.d/pvfs2-server
# ensh /sbin/chkconfig –add pvfs2-server
# ensh /sbin/chkconfig –level 345 pvfs2-server on
3. pvfs2 서비스 시작하기..
– 모든 시스템에 적용
저장공간이 만들어지지 않고 처음 구동하실때 사용하는 명령어입니다.
# /usr/local/pvfs2/sbin/pvfs2-server /etc/pvfs2-fs.conf /etc/pvfs2-server.conf-`hostname` -f
해당 저장공간이 이미 만들어져 있다면 아래의 명령어만 실행하시면됩니다.
# /usr/local/pvfs/sbin/PVFS2-server /etc/PVFS2-fs.conf /etc/PVFS2-server.conf-`hostname`
# ls /clusterfs
5d8a54bb collections.db storage_attributes.db
# /etc/rc.d/init.d/pvfs2-server start
# vi /etc/pvfs2tab
——————————————————————————————————-
tcp://dm001:3334/pvfs2-fs /pvfs2-data pvfs2 default,noauto 0 0
4. 테스트
# /usr/local/pvfs2/bin/pvfs2-ping -m /pvfs2-data
——————————————————————————————————-
[root@dm001 ~]# /usr/local/pvfs2/bin/pvfs2-ping -m /pvfs2-data
(1) Parsing tab file…
(2) Initializing system interface…
(3) Initializing each file system found in tab file: /etc/pvfs2tab…
PVFS2 servers: tcp://dm001:3334
Storage name: pvfs2-fs
Local mount point: /pvfs2-data
/pvfs2-data: Ok
(4) Searching for /pvfs2-data in pvfstab…
PVFS2 servers: tcp://dm001:3334
Storage name: pvfs2-fs
Local mount point: /pvfs2-data
meta servers:
tcp://dm001:3334
data servers:
tcp://dm002:3334
tcp://dm003:3334
tcp://dm004:3334
(5) Verifying that all servers are responding…
meta servers:
tcp://dm001:3334 Ok
data servers:
tcp://dm002:3334 Ok
tcp://dm003:3334 Ok
tcp://dm004:3334 Ok
(6) Verifying that fsid 1292344335 is acceptable to all servers…
Ok; all servers understand fs_id 1292344335
(7) Verifying that root handle is owned by one server…
Root handle: 1048576
Ok; root handle is owned by exactly one server.
=============================================================
The PVFS2 filesystem at /pvfs2-data appears to be correctly configured.
——————————————————————————————————
# /usr/local/pvfs2/bin/pvfs2-ls /pvfs2-data
lost+found
# /usr/local/pvfs2/bin/pvfs2-cp -t pvfs_test_data.tar /pvfs2-data
– 상태 모니터링
/usr/local/pvfs2/bin/karma ( XGUI)
5. Kernel Interface (mount)를 이용한 사용 방법
– 모든 서버에 ..
# /usr/local/pvfs2/sbin/pvfs2-client -f -p /usr/local/pvfs2/sbin/pvfs2-client-core &
# mount -t pvfs2 tcp://dm001:3334/pvfs2-fs /pvfs2-data
# vi /etc/rc.d/rc.local
—————————————————————————————————–
/usr/local/pvfs2/sbin/pvfs2-client -f -p /usr/local/pvfs2/sbin/pvfs2-client-core &
/bin/mount -t pvfs2 tcp://dm001:3334/pvfs2-fs /pvfs2-data
6. BMT 결과 ..
6GB 파일 copy 작업 시 성능
– local-pvfs ( local_load 0.4 pvfs_io_load 0.1 )
/usr/local/pvfs2/bin/pvfs2-cp -t pvfs_test_data.tar /pvfs2-data
Wrote 7203962880 bytes in 162.155675 seconds. 42.368140 MB/seconds
real 2m37.612s
user 0m0.009s
sys 0m10.518s
– pvfs-local
– local-same partition ( load 2 )
real 6m26.416s
user 0m0.264s
sys 0m17.787s
18MB/seconds
– local-other partition ( load 1.5 )
real 6m17.658s
user 0m0.231s
sys 0m17.661s
17MB/seconds
– nfs-local partition ( nfs_load : 8 ,local_load 6 )
real 3m12.046s
user 0m0.256s
sys 0m17.945s
– local-iscsi partition
real 7m16.244s
user 0m0.244s
sys 0m19.668s
[root@alang00 tmpdoc]# cat local_bench.txt
Version 1.03 ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
dm002 2000M 69055 86 150770 26 37456 3 80171 99 3417821 99 +++++ +++
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
dm002 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
[root@alang00 tmpdoc]# cat nfs_bench.txt
Version 1.03 ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
dm004 2000M 33579 39 54127 5 8374 69 39581 59 3203894 99 +++++ +++
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
dm004 16 1037 3 805 86 2581 7 998 3 8001 7 1901 4
[root@alang00 tmpdoc]# cat iscsi_bench.txt
Version 1.03 ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
dm002 2000M 66477 85 396865 70 12579 1 88356 99 3170283 99 62.7 0
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
dm002 16 5367 99 +++++ +++ +++++ +++ 5410 99 +++++ +++ 17105 100
[root@alang00 clunix_doc]# cat /home/alang/tmpdoc/pvfs_bench.txt
Version 1.03 ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
dm002 2000M 61735 69 23101 1 11576 1 58388 73 24594 1 4897.2 3
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
dm002 16 276 0 1138 1 230 0 169 0 1066 1 62 0
relaxing bossa nova