csync2 – 내용정리

1. 아래내용을 참조하여 각 서버들에게 csync2 를 설치한다

2. hosts 파일에 각서버정보를 입력하고 다른서버들도 같이 편집한다

3. 설치후 /etc/csync2.cfg 파일을 적절히 편집한다..

– host 부분은 server1, (server1)csync2 -k ( ) 부분에 따라서 각 서버들이

   각각 서버  역할을 하는지 아님 master slave 역할을 하는지 구분된다

– key 부분은 csync2 -k /etc/해당이름.key 명령으로 생성한후 각 서버들에게 복사해 넣는다

– master / slave 모드로 동작할때는 마스터 쪽에서만 변경된 부분을 적용시킬수있고  

각각이 서버모드일때는 각 서버에서 변경되거나 추가된 부분을 적용할수있다

– backup-directory 부분은 초기에 디렉토리가 생성되어있지 않으므로 생성시켜줘야 한다..

4. 셋팅완료후 csync2 -x 명령으로 실행시 수많은 에러가 발생할수있다

그럴경우 아래내용처럼 csync2 -f /”해당파일”  로 실행시킨후 csync2 -xv 로 다시 실행해본다..

5. 기본적으로 파일변경이나 추가된 서버에서 csync2 -xv 명령을 실행시켜야한다..

—————— [ 설정관련 내용 1 ]————————

When you search csync2 on google, you’ll get a hint:

Did you mean: sync2

That is so unfair. csync2 is one of the coolest tool to do Server Farm Synchronization among what I have tried. It solved so many problems I had before. Working as a sys admin, administrating a big server farm with hundres of linux machines, how to synchronize the data is always a big headache. You can use your own script to scp, you can use rsync, you can use subversion. Yeah, there are so many tools out there you can utilize. But, none of them are built for server synchronization.

You can also use SAN, yeah, that would be nice. But SAN solution is not cheap. You know that, right?

Ok, let’s have a look at csync2. the full name is “cluster synchronization tool, 2nd generation”. You can visit their website: http://oss.linbit.com/csync2/. Basicly everything you need is over there. But you’ll noticed there’s only one doc available: http://oss.linbit.com/csync2/paper.pdf.

The author Clifford Wolf <CLIFFORD@CLIFFORD.AT>put almost everything there in order for you to use this fantastic tool. But if you are a RedHat family user, you’ll find it not that straightforward to build and finally use it.

Here is what I have done, I include a step by step buiding and configuration mini how to here. Note that I am buiding from source, since the author’s rpm spec file has some problem. Maybe I can add a howto for buiding the rpm later.

For everybody who build linux from source before, the dependency hell is something always nasty. Here is a list of dependency I have encountered:

libgcrypt-1.2.2.tar.gz

libtasn1-0.3.2.tar.gz

libgpg-error-1.3.tar.gz

sqlite-2.8.17.tar.gz

gnutls-1.2.10.tar.bz2

librsync-0.9.7.tar.gz

Note that you have to use sqlite-2.x version, not the newest 3.x one.

And here is my script to buid the csync2:

cd /root/csync2

tar xvfz libgpg-error-1.3.tar.gz -C /usr/local/src/

cd /usr/local/src/libgpg-error-1.3/  <– (RPM 기본설치 여부확인)

./configure

make

make install

cd /root/csync2

tar xvfz libgcrypt-1.2.2.tar.gz -C /usr/local/src

cd /usr/local/src/libgcrypt-1.2.2/ <– (RPM 기본설치 여부확인)

./configure

make

make install

cd /root/csync2

tar xvfz libtasn1-0.3.2.tar.gz -C /usr/local/src/

cd /usr/local/src/ libtasn1-0.3.2/

./configure

make

make install

cd /root/csync2

tar xvfz sqlite-2.8.17.tar.gz -C /usr/local/src/

cd /usr/local/src/ sqlite-2.8.17/  <–반드시 2점대 사용할것

./configure

make

make install

cd /root/csync2

tar xvfz librsync-0.9.7.tar.gz -C /usr/local/src/

cd /usr/local/src/ librsync-0.9.7/

./configure

make

make install

cd /root/csync2

tar xvfj gnutls-1.2.10.tar.bz2 -C /usr/local/src/

cd /usr/local/src/ gnutls-1.2.10/  <– 1점대 버젼을 사용할것

./configure

make

make install

echo “/usr/local/lib” >> /etc/ld.so.conf

ldconfig

cd /root/csync2

tar xvfz csync2-1.31.tar.gz -C /usr/local/src/

cd /usr/local/src/ csync2-1.31/

./configure

make

make install

make cert

echo “csync2 30865/tcp” >> /etc/services< /FONT >

Notice this:

echo “/usr/local/lib” >> /etc/ld.so.conf

ldconfig

This is to tell the system to add /usr/local/lib to the ld directory, which is not included by default for redhat distros. Without this, the build will keep complaining missing some dependencies but you DID install.CLIFFORD@CLIFFORD.AT>

One advantage building from source here is the part to build a certificate, notice the line make cert;, just that, your certificate is made.

Now you have the csync2.

you can run csync2-ii on each machine to do some test. but you’d better make it an inetd service. Everyone who uses redhat knows: RedHat use xinetd to replace inetd.

so you’ll need this file named csyc2 to add to your /etc/xinetd.d/

service csync2

{

disable = no

protocol = tcp

socket_type = stream

wait = no

user = root

server = /usr/local/sbin/csync2

server_args = -i

}

chkconfig xinetd on

service xinetd start

CLIFFORD@CLIFFORD.AT>

<CLIFFORD@CLIFFORD.AT>Of course you have to tell the new csync2 service by adding the port to /etc/services. That I have included in my script, it’s the last line:

CLIFFORD@CLIFFORD.AT>

echo “csync2 30865/tcp” >> /etc/services

Be careful about turning on the xinetd service though. You may want to double check your /etc/xinetd.d/, make sure no other unwanted services would be turned on by this.

<CLIFFORD@CLIFFORD.AT>Now you are ready to go. Next thing you need is a good config file.

FYI: the default config file csync2 uses is /etc/csync2.cfg. Believe it or not, I didn’t find it on the paper.pdf. it took me quite a while to figure out that one is the default one.

Here is a sample /etc/csync2.cfg

group serverfarm

{

host server34.domain.com;

host (server35.domain.com);

host (server36.domain.com);

host (server37.domain.com);

key /etc/serverfarm.key;

include /etc/hosts;

include /etc/csync2.cfg;

include /usr/local/apache2/conf;

include /var/www;

action

{

pattern /usr/local/apache2/conf/httpd.conf;

exec “/usr/local/apache2/bin/apachectl graceful”;

logfile “/var/log/csync2_action.log”;

do-local;

}

backup-directory /var/backups/csync2;

backup-generations 3;

auto none;

}

I am only synchronizing the hosts file, the csync2.cfg file, the httpd.conf file, and the server document root. Notice the backup-directory, that directory, the csync2 won’t create for you, and it will complain some weird error if you don’t create them by yourself. So do that now.

This line: key /etc/serverfarm.key;

that’s their preshared key. generate is in any machine with:

csync2 -k /etc/serverfarm.key, and synchronize it to all your other machines. hopefully this is the last time you do a sync use your stone age solution, maybe rsync?

The coolest part IMO is the action part.

pattern /usr/local/apache2/conf/httpd.conf;

exec “/usr/local/apache2/bin/apachectl graceful”;

This means whenever I change the httpd.conf file, it’ll restart the apache for me. How nice!

Notice also I put () for most servers. That means, those servers are just slaves. well, it’s a server farm, who’s gonna change things in slaves if not for updating code? this also solves the conflict problem. Who cares who changed sth in slave machines, server34 is the one I need to put code on and propagate!

Now everything is set, ready… Go, let’s make some test.

First time sync, according to the author, need to run something like -I. But in my case, I just ran a csync2 -x on server34. There are lots of conflicts, sure. but my setting is only server34 should win, so did that machine win. The first time sync make take a while, not much.

Then I added a virtual host on server34, modifed httpd.conf, added the documentroot at /var/www/dummy-host, put some files in there, then I issued again the csync2 -x, I counted, 1,2,3,4,5, when I counted to 5, the execution finished, without any message.

Then try the new virtual host name in your browser, wow, it’s there, refresh, refresh, no missing server, they are all synced. only 5 seconds, 60 servers, that’s easy!

————— [ 설정파일 example 2 ]—————————–

Configure  

Even if csync2 is already configurated from the script we want remind the main steps.

We configure csync2 to mantain synchronized the configuration directories for bind9 (/etc/bind) and for apache (/etc/apache):

# /etc/csync2.cfg

group netkit {

  host www ns;                           # hosts list

  key /etc/csync2.key_netkit;            # pre-shared key

  include /etc/apache;

  include /etc/bind;

  action {                               # action section for apache

    pattern /etc/apache/httpd.conf;

    exec “apachectl graceful”;

    logfile “/var/log/csync2_action.log”;

  }

  action {                               # action section bind

    pattern /etc/bind/named.conf*;

    exec “/etc/init.d/bind9 restart”;

    logfile “/var/log/csync2_action.log”;

  }

  backup-directory /var/backups/csync2;

  backup-generations 3;                  # backup old files

}

The servers talk with each other using TCP port 30865. The servers use xinetd (or inetd) to startup:

# /etc/xinetd.d/csync2

service csync2

{

  flags       = REUSE

  socket_type = stream

  wait        = no

  user        = root

  group       = root

  server      = /usr/sbin/csync2

  server_args = -i

  disable     = no

  # only_from = 192.168.100.1 192.168.100.2

}

To complete the csync2 configuration we must generate an SSL certificate for each node. First we setup main openssl configuration file:

# /etc/ssl/openssl.cnf

[…]

countryName_default             = IT

stateOrProvinceName_default     = Italy

localityName_default            = Mestre-Venezia

0.organizationName_default      = MIUR

organizationalUnitName_default  = R&D

[…]

To generate the pre shared key:

# csync2 -k /etc/csync2.key_netkit

and we copy it in both nodes.

Now we generate the certificate for ns node:

ns# openssl genrsa \\

      -out      /etc/csync2_ssl_key.pem 1024 && \\

    openssl req -new \\

      -key      /etc/csync2_ssl_key.pem \\

      -out      /etc/csync2_ssl_cert.csr && \\

    openssl x509 -req -days 600 \\

      -in       /etc/csync2_ssl_cert.csr \\

      -signkey  /etc/csync2_ssl_key.pem \\

      -out      /etc/csync2_ssl_cert.pem && \\

    chmod go= /etc/csync2_ssl_key.pem

last we generate the certificate for www node:

www# openssl genrsa \\

       -out      /etc/csync2_ssl_key.pem 1024 && \\

     openssl req -new \\

       -key      /etc/csync2_ssl_key.pem \\

       -out      /etc/csync2_ssl_cert.csr && \\

     openssl x509 -req -days 600 \\

       -in       /etc/csync2_ssl_cert.csr \\

       -signkey  /etc/csync2_ssl_key.pem \\

       -out      /etc/csync2_ssl_cert.pem && \\

     chmod go= /etc/csync2_ssl_key.pem

Now test it  

Download the tarball, untar it in a HOME subdirectory and execute the following command:

realHost$ ./lab start

Now we are ready to test the csync2 capability. The first step is synchronizing the files:

www# csync2 -x

ns# csync2 -x

the first time csync2 load its database and after it try to synchronize. Now we simulate a conflict:

www# echo “#a” >> /etc/apache/httpd.conf  ns#  echo

“#b” >>   /etc/apache/httpd.conf

csync2 can not know which version is the right one:

www# csync2 -x

While syncing file /etc/apache/httpd.conf:

ERROR from peer ns: File is also marked dirty here!

Finished with 1 errors.

ns# csync2 -x

While syncing file /etc/apache/httpd.conf:

ERROR from peer www: File is also marked dirty here!

Finished with 1 errors.

We decide that ns node contains the right file (screenshot):

ns# csync2 -f /etc/apache/httpd.conf

ns# csync2 -xv

Connecting to host www (SSL) …

Updating /etc/apache/httpd.conf on www …

Finished with 0 errors.

www# csync2 -xv

As side effect it is performed the action to restart apache. Also if we change (from www node) the bind9 configuration files it is performed a bind9 restart.