Nutch & Hadoop Inst. Guide
Nutch(crawler) using the Hadoop (formerly NDFS) distributed file system (HDFS) 과 MapReduce
Nutch Hadoop의 세팅 작업하면서 정리한번 해놓은 좋지않겠니~
Nutch는 Nightly버젼으로 하면되고 Hadoop은 0.8을 기준으로 하였다.
Nutch 받기
Hadoop 받기
hadoop 설치
당근 JDK와 Ant는 설치하고환경설정(참조)한다.
1. 압축풀린 hadoop 디렉으로
2. #ant package
(멈춰진곳은 주석처리?)
nutch 설치
#cd nutch압축풀린디렉
#vi build.properties
dist.dir=/nutch/search
#ant package
(멈춰진곳은 주석처리?)
nutch hadoop global filesystem 구성 ()
mkdir /nutch
mkdir /nutch/search
mkdir /nutch/filesystem
mkdir /nutch/local
mkdir /nutch/home
! /nutch 상위디렉토리는 어디든 상관없다.(멀티서버는 각서버별로 다 맞추는것이..) 여기선 루트 밑으로
groupadd users
useradd -d /nutch/home -g users nutch
chown -R nutch:users /nutch
passwd nutch nutchuserpassword
! 그룹 만들기 귀찮으면 그냥 staff 적용한다. (유저퍼미션확인)
Hadoop환경 설정
#cd /nutch/search/conf
#vi hadoop-env.sh
export HADOOP_HOME=/nutch/search
export JAVA_HOME=/usr/java/jdk1.5.0_06
export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
ssh Key 생성
#cd $HOME
#ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/onnet/.ssh/id_rsa): [enter]
Enter passphrase (empty for no passphrase): [enter]
Enter same passphrase again: [enter]
#cd .ssh
#cp id_rsa.pub authorized_keys
hadoop-site.xml 편집
#/nutch/search/conf/hadoop-site.xml
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!– Put site-specific property overrides in this file. –>
<configuration>
<property>
<name>fs.default.name</name>
<value>호스트명:9000</value>
<description>
The name of the default file system. Either the literal string
“local” or a host:port for NDFS.
</description>
</property>
<property>
<name>mapred.job.tracker</name>
<value>호스트명:9001</value>
<description>
The host and port that the MapReduce job tracker runs at. If
“local”, then jobs are run in-process as a single map and
reduce task.
</description>
</property>
<property>
<name>mapred.map.tasks</name>
<value>2</value>
<description>
define mapred.map tasks to be number of slave hosts
</description>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>2</value>
<description>
define mapred.reduce tasks to be number of slave hosts
</description>
</property>
<property>
<name>dfs.name.dir</name>
<value>/nutch/filesystem/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/nutch/filesystem/data</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/nutch/filesystem/mapreduce/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/nutch/filesystem/mapreduce/local</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
! 호스트명 확인, 디렉 파일uri 확인
hadoop네임노드 파일 시스템 포맷
#bin/hadoop namenode -format
hadoop Start
#cd /nutch/search
#bin/start-all.sh
사용자 암호 입력
! localhost: ssh_exchange_identification: Connection closed by remote host 출력시
#vi conf/slaves
호스트명 변경
hadoop Stop
#bin/stop-all.sh
구동후 hadoop 로그 확인
#ls /nutch/search/logs/
Nutch crawl을 위한 url 리스트 작성
#cd /nutch/search
#mkdir urls
#vi urls/urllist.txt
http://lucene.apache.org
http://kangho.egloos.com
You should now have a urls/urllist.txt file with the one line pointing to the apache lucene site. Now we are going to add that directory to the filesystem. Later the nutch crawl will use this file as a list of urls to crawl. To add the urls directory to the filesystem run the following command:
name,data 노드 실행상태 확인후(logs)
#cd /nutch/search#bin/hadoop dfs -put urls urls
Nutch Crawl url Filter설정(정규표현)
#cd /nutch/search
#vi conf/crawl-urlfilter.txt
# +^http://([a-z0-9]*\\.)*MY.DOMAIN.NAME/
+^http://([a-z0-9]*\\.)*apache.org/
Nutch Crawl 실행
#cd /nutch/search
#bin/nutch crawl urls -dir crawled -depth 3
Crawl 상태 확인
#bin/nutch readdb crawld/crawldb -stats
#bin/hadoop dfs -du
