基于zookeeper的Hadoop集群搭建详细步骤
目录
一、一些基本概念
二、集群配置图
三、Hadoop高可用集群配置步骤
1.在第一台虚拟机解压hadoop-3.1.3.tar.gz到/opt/soft/目录
2.修改文件名、属主和属组
3.配置windows四台虚拟机的ip映射
4.修改hadoop配置文件
(1)hadoop-env.sh
(2)workers
(3)crore-site.xml
(4)hdfs-site.xml
(5)mapred-site.xml
(6)yarn-site.xml
5.拷贝hadoop到其他三台虚拟机
6.分配环境变量
7.重启环境变量，检验四台虚拟机安装是否成功
四、首次启动hadoop集群
1.高可用启动之前要启动zookeeper
2.三台机器启动JournalNode
3.第一台机器格式化
4.第一台机器启动namenode
5.在第二台机器同步namenode信息
6.第二台机器启动namenode
7.每台机器查看namenode的状态都是standby
8.关闭所有的与dfs有关的服务
9.格式化zookeeper
10.zkCli.sh
11.启动dfs
12.查看namenode节点状态
13.打开网页登录查看
14.每台虚拟机下载主备切换工具
15.启动yarn
16.有resourcemanager的主机名登录8088端口
17.查看resourcemanager节点状态
18.关闭集群
zookeeper集群的安装步骤参考博文《搭建zookeeper高可用集群详细步骤》，注意主机名，这里换了主机名ant161=ant165;ant162=ant166;ant163=ant167;ant164=ant168
一、一些基本概念
JournalNode的作用
Hadoop集群中的DFSZKFailoverController进程的作用
二、集群配置图
| ant161 | ant162 | ant163 | ant164 |
| NameNode | NameNode | ||
| DataNode | DataNode | DataNode | DataNode |
| NodeManager | NodeManager | NodeManager | NodeManager |
| ResourceManager | ResourceManager | ||
| JournalNode监控NameNode是否同步 | JournalNode | JournalNode | |
| DFSZKFailoverController监控NameNode是否存活 | DFSZKFailoverController | ||
| zookeeper0 | zookeeper1 | zookeeper2 | |
| JobHistory |
三、Hadoop高可用集群配置步骤
1.在第一台虚拟机解压hadoop-3.1.3.tar.gz到/opt/soft/目录
[root@ant161 install]# tar -zxf ./hadoop-3.1.3.tar.gz -C /opt/soft/
2.修改文件名、属主和属组
[root@ant161 soft]# mv ./hadoop-3.1.3/ hadoop313[root@ant161 soft]# chown -R root:root ./hadoop313/
3.配置windows四台虚拟机的ip映射
C:\Windows\System32\drivers\etc目录下的host文件，添加以下的主机ip配置
4.修改hadoop配置文件
(1)hadoop-env.sh
# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
export JAVA_HOME=/opt/soft/jdk180
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_JOURNALNODE_USER=root
export HDFS_ZKFC_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root(2)workers
输入四台虚拟机主机名
ant161
ant162
ant163
ant164
(3)crore-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://gky</value><description>逻辑名称，必须与hdfs-site.cml中的dfs.nameservices值保持一致</description></property><property><name>hadoop.tmp.dir</name><value>/opt/soft/hadoop313/tmpdata</value><description>namenode上本地的hadoop临时文件夹</description></property><property><name>hadoop.http.staticuser.user</name><value>root</value><description>默认用户</description></property><property><name>hadoop.proxyuser.root.hosts</name><value>*</value><description></description></property><property><name>hadoop.proxyuser.root.groups</name><value>*</value><description></description></property><property><name>io.file.buffer.size</name><value>131072</value><description>读写文件的buffer大小为：128K</description></property><property><name>ha.zookeeper.quorum</name><value>ant161:2181,ant162:2181,ant163:2181</value><description></description></property><property><name>ha.zookeeper.session-timeout.ms</name><value>10000</value><description>hadoop链接zookeeper的超时时长设置为10s</description></property>
</configuration>(4)hdfs-site.xml
<configuration><property><name>dfs.replication</name><value>3</value><description>Hadoop中每一个block的备份数</description></property><property><name>dfs.namenode.name.dir</name><value>/opt/soft/hadoop313/data/dfs/name</value><description>namenode上存储hdfs名字空间元数据目录</description></property><property><name>dfs.datanode.data.dir</name><value>/opt/soft/hadoop313/data/dfs/data</value><description>datanode上数据块的物理存储位置</description></property><property><name>dfs.namenode.secondary.http-address</name><value>ant161:9869</value><description></description></property><property><name>dfs.nameservices</name><value>gky</value><description>指定hdfs的nameservice,需要和core-site.xml中保持一致</description></property><property><name>dfs.ha.namenodes.gky</name><value>nn1,nn2</value><description>gky为集群的逻辑名称，映射两个namenode逻辑名</description></property><property><name>dfs.namenode.rpc-address.gky.nn1</name><value>ant161:9000</value><description>namenode1的RPC通信地址</description></property><property><name>dfs.namenode.http-address.gky.nn1</name><value>ant161:9870</value><description>namenode1的http通信地址</description></property><property><name>dfs.namenode.rpc-address.gky.nn2</name><value>ant162:9000</value><description>namenode2的RPC通信地址</description></property><property><name>dfs.namenode.http-address.gky.nn2</name><value>ant162:9870</value><description>namenode2的http通信地址</description></property><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://ant161:8485;ant162:8485;ant163:8485/gky</value><description>指定NameNode的edits元数据的共享存储位置(JournalNode列表)</description></property><property><name>dfs.journalnode.edits.dir</name><value>/opt/soft/hadoop313/data/journaldata</value><description>指定JournalNode在本地磁盘存放数据的位置</description></property><!-- 容错 --><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value><description>启用NameNode故障自动切换</description></property><property><name>dfs.client.failover.proxy.provider.gky</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value><description>失败后自动切换的实现方式</description></property><property><name>dfs.ha.fencing.methods</name><value>sshfence</value><description>防止脑裂的处理</description></property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_rsa</value><description>使用sshfence隔离机制时需要ssh免登陆</description></property><property><name>dfs.permissions.enabled</name><value>false</value><description>关闭HDFS操作权限验证</description></property><property><name>dfs.image.transfer.bandwidthPerSec</name><value>1048576</value><description>1M</description></property><property><name>dfs.block.scanner.volume.bytes.per.second</name><value>1048576</value><description>如果该值为0，则DataNode的块扫描程序将被禁用。如果这是正数，则这是DataNode的块扫描程序将尝试从每个卷扫描的每秒字节数。</description></property>
</configuration>(5)mapred-site.xml
<property><name>mapreduce.framework.name</name><value>yarn</value><description>job执行框架： local, classic or yarn</description>
</property>
<property><name>mapreduce.application.classpath</name><value>/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*</value><description></description>
</property>
<property><name>mapreduce.jobhistory.address</name><value>ant161:10020</value></property><property><name>mapreduce.jobhistory.webapp.address</name><value>ant161:19888</value></property>
<property><name>mapreduce.map.memory.mb</name><value>1024</value><description>设置map阶段的task工作内存</description>
</property>
<property><name>mapreduce.reduce.memory.mb</name><value>2048</value><description>设置reduce阶段的task工作内存</description>
</property>(6)yarn-site.xml
<configuration><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value><description>开启resourcemanager高可用</description></property><property><name>yarn.resourcemanager.cluster-id</name><value>yrcabc</value><description>指定yarn集群中的id</description></property><property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value><description>指定resourcemanager的名字</description></property><property><name>yarn.resourcemanager.hostname.rm1</name><value>ant163</value><description>设置rm1的名字</description></property><property><name>yarn.resourcemanager.hostname.rm2</name><value>ant164</value><description>设置rm2的名字</description></property><property><name>yarn.resourcemanager.webapp.address.rm1</name><value>ant163:8088</value><description></description></property><property><name>yarn.resourcemanager.webapp.address.rm2</name><value>ant164:8088</value><description></description></property><property><name>yarn.resourcemanager.zk-address</name><value>ant161:2181,ant162:2181,ant163:2181</value><description>指定zookeeper集群地址</description></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>运行mapreduce程序必须配置的附属服务</description></property><property><name>yarn.nodemanager.local-dirs</name><value>/opt/soft/hadoop313/tmpdata/yarn/local</value><description>nodemanager本地存储目录</description></property><property><name>yarn.nodemanager.log-dirs</name><value>/opt/soft/hadoop313/tmpdata/yarn/log</value><description>nodemanager本地日志目录</description></property><property><name>yarn.nodemanager.resource.memory-mb</name><value>2048</value><description>resource进程的工作内存</description></property><property><name>yarn.nodemanager.resource.cpu-vcores</name><value>2</value><description>resource工作中所能使用机器的内核数</description></property><!--下面三个配置在公司要删除--><property><name>yarn.scheduler.minimum-allocation-mb</name><value>256</value><description></description></property><property><name>yarn.log-aggregation-enable</name><value>true</value><description></description></property><property><name>yarn.log-aggregation.retain-seconds</name><value>86400</value><description>日志保留多少秒</description></property><property><name>yarn.nodemanager.vmem-check-enabled</name><value>false</value><description></description></property><property><name>yarn.application.classpath</name><value>/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*</value></property><property><name>yarn.nodemanager.env-whitelist</name><value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value></property>
</configuration>5.拷贝hadoop到其他三台虚拟机
[root@ant161 soft]# scp -r ./hadoop313/ root@ant162:/opt/soft/
[root@ant161 soft]# scp -r ./hadoop313/ root@ant163:/opt/soft/
[root@ant161 soft]# scp -r ./hadoop313/ root@ant164:/opt/soft/
6.分配环境变量
[root@ant161 soft]# scp /etc/profile root@ant162:/etc/
profile 100% 2202 1.4MB/s 00:00
[root@ant161 soft]# scp /etc/profile root@ant163:/etc/
profile 100% 2202 1.4MB/s 00:00
[root@ant161 soft]# scp /etc/profile root@ant164:/etc/
profile 7.重启环境变量，检验四台虚拟机安装是否成功
source /etc/profilehadoop hadoop version四、首次启动hadoop集群
1.高可用启动之前要启动zookeeper
[root@ant161 soft]# /opt/shell/zkop.sh start
------------ ant161 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
------------ ant162 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
------------ ant163 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED[root@ant161 soft]# /opt/shell/showjps.sh
---------------- ant161 服务启动状态 -----------------
2532 QuorumPeerMain
2582 Jps
---------------- ant162 服务启动状态 -----------------
2283 QuorumPeerMain
2335 Jps
---------------- ant163 服务启动状态 -----------------
2305 Jps
2259 QuorumPeerMain
---------------- ant164 服务启动状态 -----------------
2233 Jps[root@ant161 soft]# /opt/shell/zkop.sh status
------------ ant161 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Mode: follower
------------ ant162 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Mode: leader
------------ ant163 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Mode: follower
2.三台机器启动JournalNode
[root@ant161 soft]# hdfs --daemon start journalnode
WARNING: /opt/soft/hadoop313/logs does not exist. Creating.[root@ant162 soft]# hdfs --daemon start journalnode
WARNING: /opt/soft/hadoop313/logs does not exist. Creating.[root@ant163 soft]# hdfs --daemon start journalnode
WARNING: /opt/soft/hadoop313/logs does not exist. Creating.
3.第一台机器格式化
[root@ant161 soft]# hdfs namenode -format
4.第一台机器启动namenode
[root@ant161 hadoop]# hdfs --daemon start namenode
5.在第二台机器同步namenode信息
[root @ant162 soft]# hdfs namenode -bootstrapStandby
6.第二台机器启动namenode
[root @ant162 soft]# hdfs --daemon start namenode
7.每台机器查看namenode的状态都是standby
[root @ant161 soft]# hdfs haadmin -getServiceState nn1
standby
[root @ant161 soft]# hdfs haadmin -getServiceState nn2
standby8.关闭所有的与dfs有关的服务
[root @ant161 soft]# stop-dfs.sh
9.格式化zookeeper
[root @ant161 soft]# hdfs zkfc -formatZK
10.zkCli.sh
11.启动dfs
12.查看namenode节点状态
[root@ant161 soft]# hdfs haadmin -getServiceState nn1
standby
[root@ant161 soft]# hdfs haadmin -getServiceState nn2
active
13.打开网页登录查看
14.每台虚拟机下载主备切换工具
[root@ant161 soft]# yum install psmisc -y
此时如果停止active那一台的namenode，7218是active那一台机器namenode的进程号
[root@ant162 soft]# kill -9 7218此时active那一台网页无法连接，另一台没有关闭namenode的机器的网页变为active
再重新启动关闭的namenode，网页端的两个网址刷新，就会发现刚刚变为active的那一台机器，还是active状态，而另一台重启的机器就编程了standby等待状态
15.启动yarn
16.有resourcemanager的主机名登录8088端口
哪一台机器是active的状态，就会自动跳转到那一台机器的主机名
17.查看resourcemanager节点状态
[root@ant161 soft]# yarn rmadmin -getServiceState rm1
active
[root@ant161 soft]# yarn rmadmin -getServiceState rm2
standby
18.关闭集群
(1)关闭dfs
[root@ant161 soft]# stop-dfs.sh
(2)关闭yarn
[root@ant161 soft]# stop-yarn.sh
(3)关闭journalnode
[root@ant161 soft]# hdfs --daemon stop journalnode
(4)关闭zookeeper
[root@ant161 soft]# /opt/shell/zkop.sh stop