[ hadoop install & setup guide ]
- reference : http://blog.acronym.co.kr/329
- download http://ftp.daum.net/apache/hadoop/common/hadoop-1.0.3/
http://ftp.daum.net/apache/hadoop/common/hadoop-1.0.3/hadoop-1.0.3.tar.gz
- extract : hadoop-1.0.3.tar.gz to /home/mulder/apps/hadoop-1.0.3
- shell environment vars setup
mulder@vlinux:~$ tail .bashrc
export JAVA_HOME=/home/mulder/apps/jdk
export HADOOP_INSTALL=/home/mulder/apps/hadoop-1.0.3
export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_INSTALL/bin
export HADOOP_HOME_WARN_SUPPRESS=TRUE
- test hadoop
mulder@vlinux:~$ hadoop version
- hdfs namenode format
mulder@vlinux:~$ hadoop namenode -format
12/10/12 00:06:25 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = vlinux/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.3
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
************************************************************/
Re-format filesystem in /home/mulder/apps/hadoop-1.0.3/dfs/name ? (Y or N) Y
12/10/12 00:06:47 INFO util.GSet: VM type = 64-bit
12/10/12 00:06:47 INFO util.GSet: 2% max memory = 19.33375 MB
12/10/12 00:06:47 INFO util.GSet: capacity = 2^21 = 2097152 entries
12/10/12 00:06:47 INFO util.GSet: recommended=2097152, actual=2097152
12/10/12 00:06:47 INFO namenode.FSNamesystem: fsOwner=mulder
12/10/12 00:06:48 INFO namenode.FSNamesystem: supergroup=supergroup
12/10/12 00:06:48 INFO namenode.FSNamesystem: isPermissionEnabled=true
12/10/12 00:06:48 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
12/10/12 00:06:48 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
12/10/12 00:06:48 INFO namenode.NameNode: Caching file names occuring more than 10 times
12/10/12 00:06:48 INFO common.Storage: Image file of size 112 saved in 0 seconds.
12/10/12 00:06:48 INFO common.Storage: Storage directory /home/mulder/apps/hadoop-1.0.3/dfs/name has been successfully formatted.
12/10/12 00:06:48 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at vlinux/127.0.0.1
************************************************************/
mulder@vlinux:~$
- hadoop startup
mulder@vlinux:~/apps/hadoop-1.0.3/bin$ sh start-all.sh
starting namenode, logging to /home/mulder/apps/hadoop-1.0.3/logs/hadoop-mulder-namenode-vlinux.out
Enter passphrase for key '/home/mulder/.ssh/id_rsa':
localhost: Warning: $HADOOP_HOME is deprecated.
localhost:
localhost: starting datanode, logging to /home/mulder/apps/hadoop-1.0.3/logs/hadoop-mulder-datanode-vlinux.out
Enter passphrase for key '/home/mulder/.ssh/id_rsa':
localhost: Warning: $HADOOP_HOME is deprecated.
localhost:
localhost: starting secondarynamenode, logging to /home/mulder/apps/hadoop-1.0.3/logs/hadoop-mulder-secondarynamenode-vlinux.out
starting jobtracker, logging to /home/mulder/apps/hadoop-1.0.3/logs/hadoop-mulder-jobtracker-vlinux.out
Enter passphrase for key '/home/mulder/.ssh/id_rsa':
localhost: Warning: $HADOOP_HOME is deprecated.
localhost:
localhost: starting tasktracker, logging to /home/mulder/apps/hadoop-1.0.3/logs/hadoop-mulder-tasktracker-vlinux.out
mulder@vlinux:~/apps/hadoop-1.0.3/bin$
goto MapReduce
http://apmvlinux:50030/
goto HDFS
http://apmvlinux:50070/
mulder@vlinux:~/apps/hadoop-1.0.3/bin$
mulder@vlinux:~/apps/hadoop-1.0.3/bin$ cd ..
mulder@vlinux:~/apps/hadoop-1.0.3$ hadoop dfs -mkdir input
mulder@vlinux:~/apps/hadoop-1.0.3$ hadoop dfs -put NOTICE.txt input/
