Openfiler(share disk) + Heartbeat version2

OpenFiler + Heartbeat Version2 + Cluster(3 Node)

OpenFiler Download : http://sourceforge.net/project/showfiles.php?group_id=90725

  

Heartbeat Version2 Download :

          = http://mirror.secuidc.com/centos/4.5/extras/i386/RPMS/heartbeat-2.1.2-3.el4.centos.i386.rpm

            – yum -y install lm_sensors

          = http://mirror.secuidc.com/centos/4.5/extras/i386/RPMS/heartbeat-pils-2.1.2-3.el4.centos.i386.rpm

          = http://mirror.secuidc.com/centos/4.5/extras/i386/RPMS/heartbeat-stonith-2.1.2-3.el4.centos.i386.rpm

            – yum -y install OpenIPMI

            – yum -y install OpenIPMI-devel

          = http://mirror.secuidc.com/centos/4.5/extras/i386/RPMS/heartbeat-gui-2.1.2-3.el4.centos.i386.rpm

– IPaddress 설정

——————————

   Openfiler : 153 – 4

   Node1 : 151

   Node2 : 152

   Node3 : 155

– Volume 설정(Openfiler)

——————————

   lv01(to Node1)

   lv02(to Node2)

   lv03(to Node3)

  

– Heartbeat 설정

——————————

   Node1 mcast 255.1.1.1

   Node2 mcast 255.1.1.1

   Node3 mcast 255.1.1.1

  

1. Openfiler에 iSCS타입의 볼륨 설정.

https://210.220.224.153:446

  – username : openfiler

  – password : password

  

1.1 Service – Enable/Disable

    iSCSItarget = Enabled

1.2 General – Local Networks

    Local networks configuration

      – Node1 210.220.224.151 255.255.255.255 Share

      – Node2 210.220.224.152 255.255.255.255 Share

      – Node3 210.220.224.155 255.255.255.255 Share

      

1.3 Volumes – Physical storage Mgmt

    

    Physical storage Management

      – /dev/sda  SCSI  8.00GB  msdos   3

      – /dev/sdb  SCSI  8.00GB  gpt     1

      

1.4 Volumes – Volume Group Mgmt

    

    Volume Group Management

      – vg1 7.97GB  0bytes  7.97GB

1.5 Volume – Create New Volume

    Select volume group

      – vg1

    

    Create a volume in “vg1”

      – lv01  2048MB

      – lv02  2048MB

      – lv03  2048MB

1.6 Volume – List of Existing Volumes

    Volumes in volume group “vg1″(8160MB)

      lv01 – Edit

          Node1 210.220.224.151 255.255.255.255   allow

          Node2 210.220.224.152 255.255.255.255   allow

          Node3 210.220.224.155 255.255.255.255   allow

      lv02 – Edit

          Node1 210.220.224.151 255.255.255.255   allow

          Node2 210.220.224.152 255.255.255.255   allow

          Node3 210.220.224.155 255.255.255.255   allow

      lv03 – Edit

          Node1 210.220.224.151 255.255.255.255   allow

          Node2 210.220.224.152 255.255.255.255   allow

          Node3 210.220.224.155 255.255.255.255   allow

2. 각 노드에 iSCSI Initiator 설정.

  – 각 노드에서 iSCSI 관련된 서비스를 동작할 경우 출력되는 메시지가 깨질경우 정상적으로 동작하지 않는다.

    따라서, 해당 노드의 언어셋을 ‘en_US.utf8’으로 설정해 주는 것이 좋다.(/etc/sysconfig/i18n)

>Node1 / Node2 / Node3

  # yum -y install iscsi-initiator-utils

  

  # vi /etc/iscsi.conf

    DiscoveryAddress=210.220.224.153   //openfiler IP address

  

  # vi /etc/scsi_id.conf

    #options = -b

    options = -g

  

>Openfiler – each Nodes

  Openfiler# service iscsi-target restart

      Stopping iSCSI target service:                             [  OK  ]

      Starting iSCSI target service:                             [  OK  ]

  

  Node1# service iscsi restart

  Node2# service iscsi restart

  Node3# service iscsi restart

      Searching for iscsi-based multipath maps

      Found 0 maps

      Stopping iscsid:                                           [  OK  ]

      Removing iscsi driver:                                     [  OK  ]

      Checking iscsi config:                                     [  OK  ]

      Loading iscsi driver:                                      [  OK  ]

      mknod: `/dev/iscsictl’: File exists

      Starting iscsid:                                           [  OK  ]

  Node1# iscsi-ls

  Node2# iscsi-ls

  Node3# iscsi-ls

*******************************************************************************

SFNet iSCSI Driver Version …4:0.1.11-4(15-Jan-2007)

*******************************************************************************

TARGET NAME             : iqn.2006-01.com.openfiler:vg1.lv03

TARGET ALIAS            :

HOST ID                 : 4

BUS ID                  : 0

TARGET ID               : 0

TARGET ADDRESS          : 210.220.224.153:3260,1

SESSION STATUS          : ESTABLISHED AT Wed Aug 29 08:35:42 KST 2007

SESSION ID              : ISID 00023d000001 TSIH 400

*******************************************************************************

TARGET NAME             : iqn.2006-01.com.openfiler:vg1.lv01

TARGET ALIAS            :

HOST ID                 : 5

BUS ID                  : 0

TARGET ID               : 0

TARGET ADDRESS          : 210.220.224.153:3260,1

SESSION STATUS          : ESTABLISHED AT Wed Aug 29 08:35:42 KST 2007

SESSION ID              : ISID 00023d000001 TSIH 500

*******************************************************************************

TARGET NAME             : iqn.2006-01.com.openfiler:vg1.lv02

TARGET ALIAS            :

HOST ID                 : 6

BUS ID                  : 0

TARGET ID               : 0

TARGET ADDRESS          : 210.220.224.153:3260,1

SESSION STATUS          : ESTABLISHED AT Wed Aug 29 08:35:41 KST 2007

SESSION ID              : ISID 00023d000001 TSIH 600

*******************************************************************************

3. 각 노드에 파일시스템 및 마운트 포인트 설정.

  – 공유된 볼륨이 정상적으로 보인다면 각 노드에서 공유된 볼륨을 자신의 디스크로 인식한다.

  – 공유된 볼륨이 꼬이지 않도록 가상 디바이스를 생성하여 매핑한다.

  Node1# fdisk -l

  Node2# fdisk -l

  Node3# fdisk -l

Disk /dev/sda: 8589 MB, 8589934592 bytes

255 heads, 63 sectors/track, 1044 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *           1          65      522081   83  Linux

/dev/sda2              66         460     3172837+  83  Linux

/dev/sda3             461         855     3172837+  83  Linux

/dev/sda4             856        1044     1518142+   5  Extended

/dev/sda5             856         986     1052226   82  Linux swap

Disk /dev/sdb: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Disk /dev/sdb doesn’t contain a valid partition table

Disk /dev/sdc: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Disk /dev/sdc doesn’t contain a valid partition table

Disk /dev/sdd: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Disk /dev/sdd doesn’t contain a valid partition table  

3.1 공유된 자원을 가상디바이스로 매핑

    Node1# scsi_id -g -s /block/sdb

          14f70656e66696c0000000000010000005dfd07000e000000

    Node1# scsi_id -g -s /block/sdc

          14f70656e66696c0000000000020000005dfd07000e000000

    Node1# scsi_id -g -s /block/sdd

          14f70656e66696c0000000000030000005dfd07000e000000

    Node1# vi /etc/udev/rules.d/20-mapping.rules

KERNEL=”sd*”,BUS=”scsi”,PROGRAM=”/sbin/scsi_id”,RESULT=”14f70656e66696c0000000000010000003bf807000e000000″,NAME=”NodeA%n”

KERNEL=”sd*”,BUS=”scsi”,PROGRAM=”/sbin/scsi_id”,RESULT=”14f70656e66696c000000000002000000abfa07000e000000″,NAME=”NodeB%n”

KERNEL=”sd*”,BUS=”scsi”,PROGRAM=”/sbin/scsi_id”,RESULT=”14f70656e66696c0000000000030000005dfd07000e000000″,NAME=”NodeC%n”

    Node1# /sbin/start_udev

    //’Node2′,’Node3’에 ’20-mapping.rules’를 복사한다.

3.2 파티션 및 파일시스템 생성

    Node1# fdisk /dev/NodeA

         # mkfs -j /dev/NodeA1

         # tun2fs -c -1 -i 0 /dev/NodeA1

    

    Node2# fdisk /dev/NodeB

         # mkfs -j /dev/NodeB1

         # tun2fs -c -1 -i 0 /dev/NodeB1

    

    Node3# fdisk /dev/NodeC

         # mkfs -j /dev/NodeC1

         # tun2fs -c -1 -i 0 /dev/NodeC1

3.3 마운트 포인터 생성

    Node1# mount /dev/NodeA1 /Node1

    Node2# mount /dev/NodeB1 /Node2

    Node3# mount /dev/NodeC1 /Node3

4. 각 노드에 Heartbeat Version 2 설정.

>Node1 / Node2 / Node2

# cp /usr/share/doc/heartbeat-2.1.2/authkeys /etc/ha.d/

# cp /usr/share/doc/heartbeat-2.1.2/ha.cf /etc/ha.d/

# vi /etc/ha.d/ha.cf

debugfile /var/log/ha-debug

logfile /var/log/ha-log

logfacility     local0

keepalive 2

deadtime 30

udpport 694

mcast eth0 255.1.1.1 694 1 0

auto_failback off

node    Node1

node    Node2

node    Node3

apiauth stonithd uid=root

respawn root    /usr/lib/heartbeat/stonithd

apiauth cibmon uid=hacluster

respawn hacluster       /usr/lib/heartbeat/cibmon -d

crm on

# vi /etc/ha.d/authkeys

auth 1

1    crc

# chmod 600 /etc/ha.d/authkeys

5. HA GUI를 통해 각 자원을 설정.

>Node1

  # hb_gui &

  

5.1 아파치 설정

//각각의 노드마다 설치한다.

//단, ‘failover’상황을 대비하여 서로다른 노드의 아파치를 구동할 수 있는 스크립트를 포함

  총 3개의 Init 스크립트를 갖고 있어야 한다.

>Node1 / Node2 / Node3

  # wget http://archive.apache.org/dist/httpd/httpd-2.0.59.tar.gz

  # tar xvzf httpd-2.0.59.tar.gz

  # cd httpd-2.0.59

  # vi conf

   ./configure –prefix=/Node1/apache \\

    –enable-modules=so \\

    –enable-so \\

    –enable-rule=SHARED_CODE \\

    –enable-file-cache \\

    –enable-cache \\

    –enable-disk-cache \\

    –enable-headers \\

    –enable-mods-shared=most  

  

  # make && make install

  

  # vi httpd.conf           //추가

  …

  SetHandler server-status

  Order allow,deny

  Allow from all

  …  

  

  # cp httpd.init /etc/init.d/httpd

   //’httpd.init’파일은 RedHat에서 제공하는 아파치 RPM에서 추출한 스크립트이다.

  

  # service httpd start

    stopped : echo $? == ‘0’

    started : echo $? == ‘0’                //수정

  

  # service httpd status

    stopped : echo $? == ‘3’

    started : echo $? == ‘0’

  

  # service httpd stop

    stopped : echo $? == ‘0’                //수정

    started : echo $? == ‘0’

5.2 Group_resource 자원에 대한 ‘failover’ 우선순위 지정하기..

    Nodes

      – Node1

      – Node2

      – Node3

    

    Groups

      – group_Node1    resource_stickiness(0)

      – group_Node2    resource_stickiness(0)

      – group_Node3    resource_stickiness(0)

    

    Places

      – place_GN1_Node1     score(300)    #uname eq Node1

      – place_GN1_Node2     score(200)    #uname eq Node2

      – place_GN1_Node3     score(100)    #uname eq Node3

      – place_GN2_Node1     score(200)    #uname eq Node1

      – place_GN2_Node2     score(300)    #uname eq Node2

      – place_GN2_Node3     score(100)    #uname eq Node3

      – place_GN3_Node1     score(100)    #uname eq Node1

      – place_GN3_Node2     score(200)    #uname eq Node2

      – place_GN3_Node3     score(300)    #uname eq Node3

      

    

    – ‘Place’를 지정할 때 ‘score’값에 따라 해당 노드가 ‘shutdown or standby’될 때(failover) 이동하고자 하는 노드의 우선순위가 달라진다.

    – ‘score’값이 클수록 지정된 노드에 대한 우선순위가 높아지므로, 최초 설정시 해당 resource가 수행될 노드를 지정하는 것이 좋다.