GlusterFS 설치하기 (v3.1.2)
출처 : http://redmine.nehome.net/redmine/activity?from=2011-05-04
Install – GlusterFS v3.1.2-1 (by RPMs)
User documentation
2011-04-26
테스트 환경
CentOS 5.4(x86_64) 3대 + CentOS 5.4(i386) 3대 = 총 6-VM으로 테스트 진행
Hostname = IP
GFS1 = 192.168.100.31 (x86_64) GFS2 = 192.168.100.32 (x86_64) GFS3 = 192.168.100.33 (x86_64) GFS4 = 192.168.100.34 (i386) GFS5 = 192.168.100.35 (i386) GFS6 = 192.168.100.36 (i386)
6대 모두 하나의 스위치에 연결
GlusterFS로 사용될 Physicial HDD
200GB
/dev/sdb -> "fdisk" -> /dev/sdb1 -> "mount" -> /data
본 문서의 기본 작업 디렉토리는 /usr/local/src 로 함.
처음 테스트에는 4개 Node만 이용하고, 2개 노드는 운영중 확장/추가 테스트에 이용.
6-VM 공통 OS환경
편의상 /etc/hosts에 모든 GlusterFS Node들의 host정보 등록
192.168.100.31 GFS1 192.168.100.32 GFS2 192.168.100.33 GFS3 192.168.100.34 GFS4 192.168.100.35 GFS5 192.168.100.36 GFS6
iptables off # chkconfig iptables off # service iptables stop portmap on # chkconfig portmap on # service portmap start nfs-server off # chkconfig nfs off # chkconfig nfslock off # service nfs stop # service nfslock stop mount point 생성 # mkdir /data /mnt/nfs /mnt/glusterfs fuse 모듈 로딩 처리 # modprobe fuse # echo "modprobe fuse" >> /etc/rc.local Install GlusterFS (on all GlusterFS Nodes) Download SourceRPM http://download.gluster.com/pub/gluster/glusterfs/3.1/LATEST/ # wget http://download.gluster.com/pub/gluster/glusterfs/3.1/3.1.2/CentOS/glusterfs-core-3.1.2-1.x86_64.rpm # wget http://download.gluster.com/pub/gluster/glusterfs/3.1/3.1.2/CentOS/glusterfs-debuginfo-3.1.2-1.x86_64.rpm # wget http://download.gluster.com/pub/gluster/glusterfs/3.1/3.1.2/CentOS/glusterfs-fuse-3.1.2-1.x86_64.rpm # wget http://download.gluster.com/pub/gluster/glusterfs/3.1/3.1.2/CentOS/glusterfs-rdma-3.1.2-1.x86_64.rpm
Install Dependant RPM # yum -y install libibverbs-devel Install GlusterFS RPMs on x86_64 : # rpm -Uvh glusterfs-* Run GlusterFS Daemon # service glusterd start Probe Peer Node (On GFS1 Node) # gluster peer status No peers present # gluster peer probe gfs1 Probe on localhost not needed # gluster peer probe gfs2 Probe successful # gluster peer probe gfs3 Probe successful # gluster peer probe gfs4 Probe successful # gluster peer status Number of Peers: 3 Hostname: gfs2 Uuid: 8c5f1b10-4f15-4305-86b9-260e0dc2b8dc State: Peer in Cluster (Connected) Hostname: gfs3 Uuid: ba864454-e73e-4317-b90c-7f7c6142eef3 State: Peer in Cluster (Connected) Hostname: gfs4 Uuid: 9c01a6b7-c5c0-4bcd-af75-1b7fd1f9e6c3 State: Peer in Cluster (Connected) Create GlusterFS Volume and Start Volume Create a 4-Node(GFS1, GFS2, GFS3, GFS4) distributed (replicated) volume with a two-way mirror (Run On Any Nodes(4-Nodes)) # gluster volume info No volume present # gluster volume create test-vol replica 2 gfs1:/data/ gfs2:/data/ gfs3:/data/ gfs4:/data/ Creation of volume test-vol has been successful. Please start the volume to access data. # gluster volume info Volume Name: test-vol Type: Distributed-Replicate Status: Created Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gfs1:/data Brick2: gfs2:/data Brick3: gfs3:/data Brick4: gfs4:/data # gluster volume start test-vol Starting volume test-vol has been successful * test-vol : Volume Name Mount by NFS/GlusterFS Befor Mount # df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/sda3 ext3 61G 4.8G 53G 9% / /dev/sda1 ext3 996M 46M 899M 5% /boot tmpfs tmpfs 187M 0 187M 0% /dev/shm /dev/sdb1 ext3 197G 188M 187G 1% /data Mount by NFS (run on any...) # mount -t nfs gfs1:/test-vol /mnt/nfs (gfs1 = gfs2 or gfs3 gfs4...) df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/sda3 ext3 61G 4.8G 53G 9% / /dev/sda1 ext3 996M 46M 899M 5% /boot tmpfs tmpfs 187M 0 187M 0% /dev/shm /dev/sdb1 ext3 197G 188M 187G 1% /data gfs1:/test-vol nfs 394G 375M 374G 1% /mnt/nfs Mount by GlusterFS (run on any...) # lsmod | grep fuse fuse 83057 0 # mount -t glusterfs gfs1:/test-vol /mnt/glusterfs (gfs1 = gfs2 or gfs3 gfs4...) # df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/sda3 ext3 61G 4.8G 53G 9% / /dev/sda1 ext3 996M 46M 899M 5% /boot tmpfs tmpfs 187M 0 187M 0% /dev/shm /dev/sdb1 ext3 197G 188M 187G 1% /data gfs1:/test-vol nfs 394G 375M 374G 1% /mnt/nfs glusterfs#gfs1:/test-vol fuse 394G 375M 374G 1% /mnt/glusterfs Add-Node (or Brick) GFS5, GFS6에 각각 존재하는 200GB 용량을 Online상테에서 확장 # gluster peer probe gfs5 Probe successful # gluster peer probe gfs6 Probe successful # gluster peer status Number of Peers: 5 Hostname: gfs2 Uuid: 8c5f1b10-4f15-4305-86b9-260e0dc2b8dc State: Peer in Cluster (Connected) Hostname: gfs3 Uuid: ba864454-e73e-4317-b90c-7f7c6142eef3 State: Peer in Cluster (Connected) Hostname: gfs4 Uuid: 9c01a6b7-c5c0-4bcd-af75-1b7fd1f9e6c3 State: Peer in Cluster (Connected) Hostname: gfs5 Uuid: cd9a8813-f9e4-4613-a24f-43ac907f2908 State: Peer in Cluster (Connected) Hostname: gfs6 Uuid: fc3b6b09-2143-459a-9886-92b271a779b5 State: Peer in Cluster (Connected) # df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/sda3 ext3 61G 4.8G 53G 9% / /dev/sda1 ext3 996M 46M 899M 5% /boot tmpfs tmpfs 187M 0 187M 0% /dev/shm /dev/sdb1 ext3 197G 188M 187G 1% /data gfs1:/test-vol nfs 394G 375M 374G 1% /mnt/nfs glusterfs#gfs1:/test-vol fuse 394G 375M 374G 1% /mnt/glusterfs # gluster volume add-brick test-vol gfs5:/data/ gfs6:/data/ Add Brick successful # df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/sda3 ext3 61G 4.8G 53G 9% / /dev/sda1 ext3 996M 46M 899M 5% /boot tmpfs tmpfs 187M 0 187M 0% /dev/shm /dev/sdb1 ext3 197G 188M 187G 1% /data gfs1:/test-vol nfs - - - - /mnt/nfs glusterfs#gfs1:/test-vol fuse 394G 375M 374G 1% /mnt/glusterfs # df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/sda3 ext3 61G 4.8G 53G 9% / /dev/sda1 ext3 996M 46M 899M 5% /boot tmpfs tmpfs 187M 0 187M 0% /dev/shm /dev/sdb1 ext3 197G 188M 187G 1% /data gfs1:/test-vol nfs 591G 563M 561G 1% /mnt/nfs glusterfs#gfs1:/test-vol fuse 394G 375M 374G 1% /mnt/glusterfs Man Page of gluster # man gluster 참조 권장
Reading your article helped me a lot and I agree with you. But I still have some doubts, can you clarify for me? I’ll keep an eye out for your answers.