[클러스터] GFS+GNBD 설치 관련 문서
DataCore:
RedHat Cluster FAQ
http://sourceware.org/cluster/gnbd/gnbd_usage.txt
http://sources.redhat.com/cluster/doc/usage.txt
https://open.datacore.ch/DCwiki.open/Wiki.jsp?page=GFS
GFS.GNBD.Usage
GFS.usage
RedHat
1. Installation of Required Software Packages
This does the trick on Scientific Linux v4.3 — in no particular order:
apt-get install cman cman-kernel-smp
apt-get install gulm ccs
apt-get install gnbd gnbd-kernel-smp
apt-get install dlm dlm-kernel-smp
apt-get install magma-plugins
apt-get install fence
apt-get install lvm2-cluster
apt-get install GFS-kernel-smp GFS
Versions of cluster software must be the same on all nodes in cluster (some versions are not compatible resulting in communication problems).
2. Post Installation
The packages install init s-links which cause Big Trouble sometimes, so remove:
chkconfig –level 35 ccsd off
chkconfig –level 35 cman off
chkconfig –level 35 lock_gulmd off
chkconfig –level 35 fenced off
chkconfig –level 35 clvmd off
chkconfig –level 35 gfs off
3. Initialising the Cluster
Load kernel modules:
modprobe cman
modprobe dlm
modprobe lock_dlm
modprobe gnbd
modprobe gfs
Create /etc/cluster/cluster.conf on one node and copy to all nodes. Use an appropriate cluster.conf — manual fencing is a bad idea except with only two nodes.
As nearly simultanously as possible, enter the following commands on all nodes:
ccsd
Test if ccsd is working properly:
# second may be required
#
# — if the first fails, try the second — <clustername> is given
# in cluster.conf :
#
ccs_test connect
ccs_test connect force block <clustername>
#
# …the second is required if the cluster is not yet quorate (see
# man page for ccs_test)…
Also:
ccs_test get node ‘//cluster/@name’
On all nodes, simultaneously:
/sbin/cman_tool join
Ensure all nodes have joined the cluster before the next step (fencing):
cat /proc/cluster/nodes
Node Votes Exp Sts Name # …be patient, this might take a while
1 1 3 M R5-14 # to appear…
2 1 3 M R5-13
3 1 3 M R5-15
and
cat /proc/cluster/status
Protocol version: 5.0.1
Config version: 1
Cluster name: clusterR5-13
Cluster ID: 36497
Cluster Member: Yes
Membership state: Cluster-Member
Nodes: 3
Expected_votes: 3
Total_votes: 3
Quorum: 3
Active subsystems: 9
Node name: R5-13
Node ID: 2
Node addresses: 10.10.1.13
Ensure all nodes have joined (see above) before this step — fencing. Simultaneously on all nodes:
/sbin/fence_tool join
4. Starting Cluster Services
/sbin/clvmd
# on both
vgchange -aly
# on both
cat /proc/cluster/services
5. cluster.conf Examples
Contents
Two-node, manual fencing;
three-node, GNBD-fencing.
5.1. Two-Node Cluster with Manual Fencing
/etc/cluster/cluster.conf:
<?xml version=”1.0″?>
<cluster name=”cluster1″ config_version=”1″>
<cman two_node=”1″ expected_votes=”1″>
</cman>
<clusternodes>
<clusternode name=”login” votes=”1″>
<fence>
<method name=”single”>
<device name=”human” ipaddr=”10.10.1.250″/>
</method>
</fence>
</clusternode>
<clusternode name=”R5-13″ votes=”1″>
<fence>
<method name=”single”>
<device name=”human” ipaddr=”10.10.1.13″/>
</method>
</fence>
</clusternode>
</clusternodes>
<fence_devices>
<device name=”human” agent=”fence_manual”/>
</fence_devices>
</cluster>
5.2. Three-Node Cluster with fence_gnbd
<?xml version=”1.0″?>
<cluster name=”clusterR5-13″ config_version=”1″>
<cman></cman>
<clusternodes>
<clusternode name=”R5-13″>
<fence>
<method name=”single”>
<device name=”gnbd” nodename=”R5-13″/>
</method>
</fence>
</clusternode>
<clusternode name=”R5-14″>
<fence>
<method name=”single”>
<device name=”gnbd” nodename=”R5-14″/>
</method>
</fence>
</clusternode>
<clusternode name=”R5-15″>
<fence>
<method name=”single”>
<device name=”gnbd” nodename=”R5-15″/>
</method>
</fence>
</clusternode>
</clusternodes>
<fence_devices>
<device name=”gnbd” agent=”fence_gnbd” servers=”R5-14 R5-15″/>
</fence_devices>
</cluster>
6. Adding a Node to the Cluster
As described in the RedHat Cluster FAQ:
a third node cannot be added to a two-node cluster as the latter is a special case (see the cluster.conf examples, above);
adding a subsequent node to a cluster of three-or-more is easy.
To add a node to a cluster of three-or-more:
Start the cluster software on the addition node (load required kernel modules;
7. GNBD and GFS
Here:
exporting nodes: R5-14, R5-15
importing nodes: R5-13
7.1. GNBD
On each exporting node:
R5-14> /sbin/gnbd_serv -v
R5-14> /sbin/gnbd_export -v -e R5-14-sdb1 -d /dev/sdb1
#
# …for example: the “-e” label
R5-15> /sbin/gnbd_serv -v
R5-15> /sbin/gnbd_export -v -e R5-15-sdb1 -d /dev/sdb1
R5-14> /sbin/gnbd_export -v -l
R5-15> /sbin/gnbd_export -v -l
On importing node:
R5-13> gnbd_import -v -i R5-14
R5-13> gnbd_import -v -i R5-15
R5-13> gnbd -v -l
Device name : R5-14-sdb1
———————-
Minor # : 0
sysfs name : /block/gnbd0
Server : R5-14
.
Device name : R5-15-sdb1
———————-
Minor # : 1
sysfs name : /block/gnbd1
Server : R5-15
.
ls -l /dev/gnbd
total 0
brw-r–r– 1 root root 251, 0 Feb 16 14:43 R5-14-sdb1
brw-r–r– 1 root root 251, 1 Feb 16 14:44 R5-15-sdb1
7.2. GNBD and EXT2/3
Use remote device as local on importing node:
R5-13> mke2fs /dev/gnbd/R5-14-sdb1
R5-13> mke2fs /dev/gnbd/R5-15-sdb1
R5-13> mount /dev/gnbd/R5-14-sdb1 /mnt/gfs-R5-14-sdb1
R5-13> mount /dev/gnbd/R5-15-sdb1 /mnt/gfs-R5-15-sdb1
#
# …N.B. One can then, on the exporting node, mount /dev/sdb1
# as ext2 — but the two nodes won’t sync, so this
# spells disaster…
#
7.3. GNFS and GFS
#
# — Either “mkfs” locally on the exporters…
#
R5-14> gfs_mkfs -p lock_dlm -t clusterR5-13:R5-14 -j 3 /dev/sdb1
R5-15> gfs_mkfs -p lock_dlm -t clusterR5-13:R5-15 -j 3 /dev/sdb1
#
# …or “mkfs” on the importer…
#
R5-13> gfs_mkfs -p lock_dlm -t clusterR5-13:R5-14-sdb1 -j 3 /dev/gnbd/R5-14-sdb1
R5-13> gfs_mkfs -p lock_dlm -t clusterR5-13:R5-15-sdb1 -j 3 /dev/gnbd/R5-15-sdb1
#
# …then mount each file-system on both exporter and importer…
#
R5-14> mount -t gfs /dev/sdb1 /mnt/sdb1
R5-15> mount -t gfs /dev/sdb1 /mnt/sdb1
#
R5-13> mount -t gfs /dev/gnbd/R5-14-sdb1 /mnt/gfs-R5-14-sdb1
R5-13> mount -t gfs /dev/gnbd/R5-15-sdb1 /mnt/gfs-R5-15-sdb1
#
# …creating and deleting files, see that local and remote
# mounted filesystems are synced, hooray…
#
8. Trouble and Troubleshooting
This is a very good place to look:
sourceware.org/cluster/faq.
Look in the system-logs for messages re. what the cluster-daemons are up to:
tail -f /var/log/messages # …or kernel or daemon, or whatever,
# looking for “kernel: CMAN:”- and
# ccsd-related messages…
#
8.1. Trouble 1
A cluster of R5-13, R5-14 and R5-15 worked fine; R5-16 would not add to this cluster. On R5-13:
Feb 16 11:45:46 R5-13 kernel: CMAN: removing node R5-16 from the cluster : No response to messages
and on R5-16
Feb 16 11:46:14 R5-16 kernel: CMAN: No functional network interfaces, leaving cluster
The FAQ (above) suggested “…inconsistent versions of cman software…” and sure enough the version number was not the same. Patching the four nodes to the same level solved the problem.
