[클러스터][파일] PVFS2 병렬 파일 시스템 환경 구축

##### PVFS2 설치하기 #######

작성일 : 2007년 12월 12일

작성자 : 서진우

1. PVFS2 설치

– meta server, ioserver, pvfs_client 에 모두 설치

# cd /usr/local/src

# tar xzvf pvfs-2.6.3.tar.gz

# cd pvfs-2.6.3

# ./configure –prefix=/usr/local/pvfs2 –with-kernel=/usr/src/kernels/2.6.9-55.EL-smp-x86_64

# make

# make install

# make kmod

# make kmod_install

# depmod -a /lib/modules/2.6.9-55.ELsmp

# modprobe pvfs2

2. PVFS2 설정

– meta server 에 적용

# cd /usr/local/pvfs2/bin

# ./pvfs2-genconfig /etc/pvfs2-fs.conf /etc/pvfs2-server.conf

—————————————————————————————————

**********************************************************************

    Welcome to the PVFS2 Configuration Generator:

This interactive script will generate configuration files suitable

for use with a new PVFS2 file system.  Please see the PVFS2 quickstart

guide for details.

**********************************************************************

You must first select the network protocol that your file system will use.

The only currently supported options are “tcp”, “gm”, and “ib”.

(For multi-homed configurations, use e.g. “ib,tcp”.)

* Enter protocol type [Default is tcp]:

Choose a TCP/IP port for the servers to listen on.  Note that this

script assumes that all servers will use the same port number.

* Enter port number [Default is 3334]:

Choose a directory for each server to store data in.

* Enter directory name: [Default is /pvfs2-storage-space]: /clusterfs

Choose a file for each server to write log messages to.

* Enter log file location [Default is /tmp/pvfs2-server.log]: /var/log/pvfs2-server.log

Next you must list the hostnames of the machines that will act as

I/O servers.  Acceptable syntax is “node1, node2, …” or “node{#-#,#,#}”.

* Enter hostnames [Default is localhost]: dm002,dm003,dm004

Now list the hostnames of the machines that will act as Metadata

servers.  This list may or may not overlap with the I/O server list.

* Enter hostnames [Default is localhost]: dm001

Configured a total of 4 servers:

3 of them are I/O servers.

1 of them are Metadata servers.

* Would you like to verify server list (y/n) [Default is n]?

Writing fs config file… Done.

Writing 4 server config file(s)… Done.

———————————————————————————————————

# /usr/local/src/pvfs-2.6.3/examples

# cp pvfs2-server.rc /etc/rc.d/init.d/pvfs2-server

# chmod 755 /etc/rc.d/init.d/pvfs2-server

# ensync /etc/pvfs*

# ensync /etc/rc.d/init.d/pvfs2-server

# ensh /sbin/chkconfig –add pvfs2-server

# ensh /sbin/chkconfig –level 345 pvfs2-server on

3. pvfs2 서비스 시작하기..

– 모든 시스템에 적용

저장공간이 만들어지지 않고 처음 구동하실때 사용하는 명령어입니다.

# /usr/local/pvfs2/sbin/pvfs2-server /etc/pvfs2-fs.conf /etc/pvfs2-server.conf-`hostname` -f

해당 저장공간이 이미 만들어져 있다면 아래의 명령어만 실행하시면됩니다.

# /usr/local/pvfs/sbin/PVFS2-server /etc/PVFS2-fs.conf /etc/PVFS2-server.conf-`hostname`

# ls /clusterfs

5d8a54bb collections.db storage_attributes.db

# /etc/rc.d/init.d/pvfs2-server start

# vi /etc/pvfs2tab

——————————————————————————————————-

tcp://dm001:3334/pvfs2-fs /pvfs2-data pvfs2 default,noauto 0 0

4. 테스트

# /usr/local/pvfs2/bin/pvfs2-ping -m /pvfs2-data

——————————————————————————————————-

[root@dm001 ~]# /usr/local/pvfs2/bin/pvfs2-ping -m /pvfs2-data

(1) Parsing tab file…

(2) Initializing system interface…

(3) Initializing each file system found in tab file: /etc/pvfs2tab…

   PVFS2 servers: tcp://dm001:3334

   Storage name: pvfs2-fs

   Local mount point: /pvfs2-data

   /pvfs2-data: Ok

(4) Searching for /pvfs2-data in pvfstab…

   PVFS2 servers: tcp://dm001:3334

   Storage name: pvfs2-fs

   Local mount point: /pvfs2-data

   meta servers:

   tcp://dm001:3334

   data servers:

   tcp://dm002:3334

   tcp://dm003:3334

   tcp://dm004:3334

(5) Verifying that all servers are responding…

   meta servers:

   tcp://dm001:3334 Ok

   data servers:

   tcp://dm002:3334 Ok

   tcp://dm003:3334 Ok

   tcp://dm004:3334 Ok

(6) Verifying that fsid 1292344335 is acceptable to all servers…

   Ok; all servers understand fs_id 1292344335

(7) Verifying that root handle is owned by one server…

   Root handle: 1048576

     Ok; root handle is owned by exactly one server.

=============================================================

The PVFS2 filesystem at /pvfs2-data appears to be correctly configured.

——————————————————————————————————

# /usr/local/pvfs2/bin/pvfs2-ls /pvfs2-data

lost+found

# /usr/local/pvfs2/bin/pvfs2-cp -t pvfs_test_data.tar /pvfs2-data

– 상태 모니터링

/usr/local/pvfs2/bin/karma ( XGUI)

5. Kernel Interface (mount)를 이용한 사용 방법

– 모든 서버에 ..

# /usr/local/pvfs2/sbin/pvfs2-client -f -p /usr/local/pvfs2/sbin/pvfs2-client-core &

# mount -t pvfs2 tcp://dm001:3334/pvfs2-fs /pvfs2-data

# vi /etc/rc.d/rc.local

—————————————————————————————————–

/usr/local/pvfs2/sbin/pvfs2-client -f -p /usr/local/pvfs2/sbin/pvfs2-client-core &

/bin/mount -t pvfs2 tcp://dm001:3334/pvfs2-fs /pvfs2-data

6. BMT 결과 ..

6GB 파일 copy 작업 시 성능

– local-pvfs ( local_load 0.4 pvfs_io_load 0.1 )

/usr/local/pvfs2/bin/pvfs2-cp -t pvfs_test_data.tar /pvfs2-data

Wrote 7203962880 bytes in 162.155675 seconds. 42.368140 MB/seconds

real    2m37.612s

user    0m0.009s

sys     0m10.518s

– pvfs-local

– local-same partition ( load 2 )

real    6m26.416s

user    0m0.264s

sys     0m17.787s

18MB/seconds

– local-other partition ( load 1.5 )

real    6m17.658s

user    0m0.231s

sys     0m17.661s

17MB/seconds

– nfs-local partition ( nfs_load : 8 ,local_load 6 )

real    3m12.046s

user    0m0.256s

sys     0m17.945s

– local-iscsi partition

real    7m16.244s

user    0m0.244s

sys     0m19.668s

[root@alang00 tmpdoc]# cat local_bench.txt

Version  1.03      ——Sequential Output—— –Sequential Input- –Random-

                    -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

dm002         2000M 69055  86 150770  26 37456   3 80171  99 3417821  99 +++++ +++

                    ——Sequential Create—— ——–Random Create——–

                    -Create– –Read— -Delete– -Create– –Read— -Delete–

files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

dm002            16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

[root@alang00 tmpdoc]# cat nfs_bench.txt

Version  1.03      ——Sequential Output—— –Sequential Input- –Random-

                    -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

dm004         2000M 33579  39 54127   5  8374  69 39581  59 3203894  99 +++++ +++

                    ——Sequential Create—— ——–Random Create——–

                    -Create– –Read— -Delete– -Create– –Read— -Delete–

files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

dm004            16  1037   3   805  86  2581   7   998   3  8001   7  1901   4

[root@alang00 tmpdoc]# cat iscsi_bench.txt

Version  1.03      ——Sequential Output—— –Sequential Input- –Random-

                    -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

dm002         2000M 66477  85 396865  70 12579   1 88356  99 3170283  99  62.7   0

                    ——Sequential Create—— ——–Random Create——–

                    -Create– –Read— -Delete– -Create– –Read— -Delete–

files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

dm002            16  5367  99 +++++ +++ +++++ +++  5410  99 +++++ +++ 17105 100

[root@alang00 clunix_doc]# cat /home/alang/tmpdoc/pvfs_bench.txt

Version  1.03      ——Sequential Output—— –Sequential Input- –Random-

                    -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

dm002         2000M 61735  69 23101   1 11576   1 58388  73 24594   1 4897.2   3

                    ——Sequential Create—— ——–Random Create——–

                    -Create– –Read— -Delete– -Create– –Read— -Delete–

files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

dm002            16   276   0  1138   1   230   0   169   0  1066   1    62   0