hadoop伪分布式环境搭建记录

Hadoop快3个月没接触了，这阶段全是阿里云的东西，乘着领导让我给同事讲讲Hadoop的机会，复习下Hadoop的知识。

hadoop伪分布式环境搭建记录20160511
1.解压hadoop、jdk压缩包

tar -xvzf hadoop-2.6.0.tar.gz
tar -xvzf jdk-7u79-linux-x64.tar.gz

2.环境变量添加

vi ~/.bash_profile
export JAVA_HOME=/home/hadoop/jdk1.7.0_79
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/home/hadoop/hadoop-2.6.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

修改完成，生效配置文件
source ~/.bash_profile

检查是否成功
java -version
hadoop version

3.修改hadoop配置文件

<?xml version="1.0"?>
<!-- core-site.xml -->
<configuration>
 <property>
  <name>fs.default.name</name>
  <value>hdfs://127.0.0.1/</value>
 </property>
</configuration>
<?xml version="1.0"?>
<!-- hdfs-site.xml -->
<configuration>
 <property>
  <name>dfs.replication</name>
  <value>1</value>
 </property>
</configuration>
<?xml version="1.0"?>
<!-- mapred-site.xml -->
<configuration>
 <property>
  <name>mapred.job.tracker</name>
  <value>127.0.0.1:8021</value>
 </property>
<property>
  <name>mapreduce.jobhistory.address</name> 
  <value>127.0.0.1:10020</value> 
</property> 
<property> 
  <name>mapreduce.jobhistory.webapp.address</name> 
  <value>127.0.0.1:19888</value> 
</property>
</configuration>
<?xml version="1.0"?>
<!-- yarn-site.xml -->
<configuration>
 <property>
  <name>yarn.resourcemanager.address</name>
  <value>127.0.0.1:8032</value>
 </property>
 <property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce.shuffle</value>
 </property>
</configuration>

4.安装ssh并配置免密码登录

yum install ssh
#或者
apt-get install ssh
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh localhost

5.格式化hdfs

hadoop namenode -format
#此处遇到报错：
java.net.UnknownHostException
#修改
vi /etc/sysconfig/network
#HOSTNAME修改为localhost(随自己喜好，我这里设置的是Hadoop,确认/etc/hosts 里有对应的host配置就行)
#重启
reboot

6.启动hadoop

#start-all.sh 不推荐使用，用这个nodemanager总是起不起来
start-dfs.sh
start-yarn.sh

此处遇到报错：
localhost: Error: JAVA_HOME is not set and could not be found.

#修改/etc/hadoop/hadoop-env.sh中设JAVA_HOME
source  hadoop-env.sh

重新执行，此处依旧报错：
Cannot find configuration directory: /etc/hadoop
在hadoop-env.sh 配置一条hadoop配置文件所在目录：

export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.6.0/etc/hadoop
source  hadoop-env.sh

7.访问web
http://hadoop:8088/
http://hadoop:50070/
如果无法访问，关闭防火墙

service iptables stop

8.执行hdfs语句

hdfs dfs -ls

至此hadoop环境搭建完毕!