ISCSI 기반 Multipath 구성

발췌 : http://mydoc.digimoon.net/board/skin/ggambo7002_board/print.php?id=board&no=706

기가비트 스위치를 사이에 두고 2대의 노드(iSCSI initiator), 1대의 스토리지(iSCSI target) 준비

노드, 스토리지 각각 2개의 라인씩 뽑아내어 기가비트 스위치에 연결(기가비트 스위치 총 6개 포트 소모)

노드1
172.16.0.101
172.16.0.102

노드2
172.16.0.103
172.16.0.104

스토리지
172.16.0.105
172.16.0.106

iSCSI target 스토리지에 2개의 LUN 설정
[root@localhost ~]# /usr/sbin/tgtadm –lld iscsi –op show –mode target
Target 1: iqn.storage:iscsi-storage
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1045 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/vda1
Backing store flags:
LUN: 2
Type: disk
SCSI ID: IET 00010002
SCSI SN: beaf12
Size: 1045 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/vdb1
Backing store flags:
Account information:
ACL information:
ALL
[root@localhost ~]#

각 노드 device-mapper-multipath 패키지 설치 유무 확인
[root@node1 ~]# rpm -qa | grep device-mapper-multipath
device-mapper-multipath-0.4.7-42.el5
[root@node1 ~]#

노드1, 노드2에서 아래 커맨드 실행(스토리지에 2개의 IP가 할당되어 있으므로)
[root@node1 ~]# iscsiadm -m discovery -t st -p 172.16.0.105 -P 1
[root@node1 ~]# iscsiadm -m discovery -t st -p 172.16.0.105 -l
[root@node1 ~]# iscsiadm -m discovery -t st -p 172.16.0.106 -P 1
[root@node1 ~]# iscsiadm -m discovery -t st -p 172.16.0.106 -l

스토리지에선 iSCSI target 정보가 아래와 같이 확인된다.
[root@storage ~]# /usr/sbin/tgtadm –lld iscsi –op show –mode target
Target 1: iqn.storage:iscsi-storage
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 5
Initiator: iqn.1994-05.com.redhat:3dd8dc7ed525
Connection: 0
IP Address: 172.16.0.103
I_T nexus: 6
Initiator: iqn.1994-05.com.redhat:3dd8dc7ed525
Connection: 0
IP Address: 172.16.0.101
I_T nexus: 7
Initiator: iqn.1994-05.com.redhat:3dd8dc7ed525
Connection: 0
IP Address: 172.16.0.103
I_T nexus: 8
Initiator: iqn.1994-05.com.redhat:3dd8dc7ed525
Connection: 0
IP Address: 172.16.0.101
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1045 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/vda1
Backing store flags:
LUN: 2
Type: disk
SCSI ID: IET 00010002
SCSI SN: beaf12
Size: 1045 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/vdb1
Backing store flags:
Account information:
ACL information:
ALL
[root@storage ~]#

이제 각 노드에서 iSCSI target의 공유 블록 디바이스를 확인하면 4개로 보인다(LUN 갯수 x 2).
아래 결과물을 보면 /dev/sda~/dev/sdd까지 확인되는 것이 스토리지의 공유 블록이다(1G 2ea).
[root@node1 ~]# fdisk -l

Disk /dev/vda: 21.4 GB, 21495808000 bytes
255 heads, 63 sectors/track, 2613 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/vda1 * 1 13 104391 83 Linux
/dev/vda2 14 26 104422+ 82 Linux swap / Solaris
/dev/vda3 27 2613 20780077+ 83 Linux

Disk /dev/vdb: 5242 MB, 5242880000 bytes
16 heads, 63 sectors/track, 10158 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes

Device Boot Start End Blocks Id System
/dev/vdb1 1 195 98248+ 83 Linux
/dev/vdb2 196 390 98280 83 Linux
/dev/vdb3 391 779 196056 83 Linux
/dev/vdb4 780 10158 4727016 5 Extended
/dev/vdb5 780 1749 488848+ 83 Linux
/dev/vdb6 1750 1944 98248+ 83 Linux
/dev/vdb7 1945 3929 1000408+ 83 Linux
/dev/vdb8 3930 4318 196024+ 83 Linux
/dev/vdb9 5095 5676 293296+ 83 Linux
/dev/vdb10 4319 5094 391072+ 83 Linux

Partition table entries are not in disk order

Disk /dev/sda: 1044 MB, 1044578304 bytes
33 heads, 61 sectors/track, 1013 cylinders
Units = cylinders of 2013 * 512 = 1030656 bytes

Disk /dev/sda doesn’t contain a valid partition table

Disk /dev/sdb: 1044 MB, 1044578304 bytes
33 heads, 61 sectors/track, 1013 cylinders
Units = cylinders of 2013 * 512 = 1030656 bytes

Disk /dev/sdb doesn’t contain a valid partition table

Disk /dev/sdc: 1044 MB, 1044578304 bytes
33 heads, 61 sectors/track, 1013 cylinders
Units = cylinders of 2013 * 512 = 1030656 bytes

Disk /dev/sdc doesn’t contain a valid partition table

Disk /dev/sdd: 1044 MB, 1044578304 bytes
33 heads, 61 sectors/track, 1013 cylinders
Units = cylinders of 2013 * 512 = 1030656 bytes

Disk /dev/sdd doesn’t contain a valid partition table

[root@node1 ~]#

scsi_id 확인

[root@node1 ~]# scsi_id -gus /block/sda
1IET_00010001
[root@node1 ~]# scsi_id -gus /block/sdb
1IET_00010002
[root@node1 ~]# scsi_id -gus /block/sdc
1IET_00010001
[root@node1 ~]# scsi_id -gus /block/sdd
1IET_00010002
[root@node1 ~]#

위 출력값을 참고로 /etc/multipath.conf 작성
[root@node1 ~]# cat /etc/multipath.conf
[root@node1 ~]# cat /etc/multipath.conf
multipaths {
multipath {
wwid 1IET_00010001
alias red
}
multipath {
wwid 1IET_00010002
alias blue
}
}
[root@node1 ~]#

multipathd 서비스 시작
[root@node1 ~]# service multipathd start

fdisk로 다시 확인하면 /dev/dm-0, /dev/dm-1 장치가 확인됨
[root@node1 ~]# fdisk -l

Disk /dev/vda: 21.4 GB, 21495808000 bytes
255 heads, 63 sectors/track, 2613 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/vda1 * 1 13 104391 83 Linux
/dev/vda2 14 26 104422+ 82 Linux swap / Solaris
/dev/vda3 27 2613 20780077+ 83 Linux

Disk /dev/vdb: 5242 MB, 5242880000 bytes
16 heads, 63 sectors/track, 10158 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes

Device Boot Start End Blocks Id System
/dev/vdb1 1 195 98248+ 83 Linux
/dev/vdb2 196 390 98280 83 Linux
/dev/vdb3 391 779 196056 83 Linux
/dev/vdb4 780 10158 4727016 5 Extended
/dev/vdb5 780 1749 488848+ 83 Linux
/dev/vdb6 1750 1944 98248+ 83 Linux
/dev/vdb7 1945 3929 1000408+ 83 Linux
/dev/vdb8 3930 4318 196024+ 83 Linux
/dev/vdb9 5095 5676 293296+ 83 Linux
/dev/vdb10 4319 5094 391072+ 83 Linux

Partition table entries are not in disk order

Disk /dev/sda: 1044 MB, 1044578304 bytes
33 heads, 61 sectors/track, 1013 cylinders
Units = cylinders of 2013 * 512 = 1030656 bytes

Disk /dev/sda doesn’t contain a valid partition table

Disk /dev/sdb: 1044 MB, 1044578304 bytes
33 heads, 61 sectors/track, 1013 cylinders
Units = cylinders of 2013 * 512 = 1030656 bytes

Disk /dev/sdb doesn’t contain a valid partition table

Disk /dev/sdc: 1044 MB, 1044578304 bytes
33 heads, 61 sectors/track, 1013 cylinders
Units = cylinders of 2013 * 512 = 1030656 bytes

Disk /dev/sdc doesn’t contain a valid partition table

Disk /dev/sdd: 1044 MB, 1044578304 bytes
33 heads, 61 sectors/track, 1013 cylinders
Units = cylinders of 2013 * 512 = 1030656 bytes

Disk /dev/sdd doesn’t contain a valid partition table

Disk /dev/dm-0: 1044 MB, 1044578304 bytes
255 heads, 63 sectors/track, 126 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-0 doesn’t contain a valid partition table

Disk /dev/dm-1: 1044 MB, 1044578304 bytes
255 heads, 63 sectors/track, 126 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-1 doesn’t contain a valid partition table
[root@node1 ~]#

[root@node1 ~]# multipath -ll
mulipath.conf line 1, invalid keyword: multipath
mulipath.conf line 2, invalid keyword: wwid
mulipath.conf line 3, invalid keyword: alias
mulipath.conf line 4, invalid keyword: }
mulipath.conf line 6, invalid keyword: multipath
mulipath.conf line 7, invalid keyword: wwid
mulipath.conf line 8, invalid keyword: alias
mulipath.conf line 9, invalid keyword: }
1IET_00010001 dm-0 IET,VIRTUAL-DISK
[size=996M][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 2:0:0:1 sde 8:64 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 3:0:0:1 sdf 8:80 [active][ready]
1IET_00010002 dm-1 IET,VIRTUAL-DISK
[size=996M][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 2:0:0:2 sdg 8:96 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 3:0:0:2 sdh 8:112 [active][ready]
[root@node1 ~]#

mulipath.conf line 1, invalid keyword: multipath
위와 같은 에러가 나는 건 원본 multipath.conf 파일의 주석을 풀고 고쳐 쓰는 것으로 해결 봄
아마도 공백을 띄울 때 스페이스바로 띄우는 거랑 탭키로 띄우는 걸 가리는 모양….

[root@node1 ~]# cat /etc/multipath.conf
multipaths {
multipath {
wwid 1IET_00010001
alias red
}
multipath {
wwid 1IET_00010002
alias blue
}
}
[root@node1 ~]#

이제 이상없이 잘 뜸
[root@node1 ~]# multipath -ll
blue (1IET_00010002) dm-1 IET,VIRTUAL-DISK
[size=996M][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 1:0:0:2 sdb 8:16 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 0:0:0:2 sdd 8:48 [active][ready]
red (1IET_00010001) dm-0 IET,VIRTUAL-DISK
[size=996M][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 1:0:0:1 sda 8:0 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 0:0:0:1 sdc 8:32 [active][ready]
[root@node1 ~]#