银河麒麟服务器操作系统部署Hadoop集群
银河麒麟服务器操作系统部署Hadoop集群
一、 Hadoop集群环境规则
1.Hadoop介绍
Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不了解分布式底层细节的情况下，开发分布式程序。
充分利用集群的威力进行高速运算和存储。
2.Hadoop集群简介
HADOOP集群具体来说包含两个集群：HDFS集群和YARN集群，两者逻辑上分离，但物理上常在一起。
HDFS集群负责海量数据的存储，集群中的角色主要有：NameNode、DataNode、SecondaryNameNode
YARN集群负责海量数据运算时的资源调度，集群中的角色主要有：ResourceManager、NodeManager
那mapreduce是什么呢？它其实是一个分布式运算编程框架，是应用程序开发包，由用户按照编程规范进行程序开发，后打包运行在
HDFS集群上，并且受到YARN集群的资源调度管理。
3.Hadoop核心组件有：
HDFS（分布式文件系统）：解决海量数据存储
YARN（作业调度和集群资源管理的框架）：解决资源任务调度
MAPREDUCE（分布式运算编程框架）：解决海量数据计算
4.节点规划说明：
zookeeper集群需要至少3个节点，并且节点数为奇数个，可以部署在任意独立节点上，NameNode及ResourceManager依赖zookeeper
进行主备选举和切换
NameNode至少需要2个节点，一主多备，可以部署在任意独立节点上，用于管理HDFS的名称空间和数据块映射，
依赖zookeeper和zkfc实现高可用和自动故障转移，并且依赖journalnode实现状态同步
ZKFailoverController即zkfc，在所有NameNode节点上启动，用于监视和管理NameNode状态，参与故障转移
DataNode至少需要3个节点，因为hdfs默认副本数为3，可以部署在任意独立节点上，用于实际数据存储
ResourceManager：至少需要2个节点，一主多备，可以部署在任意独立节点上，依赖zookeeper实现高可用和自动故障转移，
用于资源分配和调度
NodeManager部署在所有DataNode节点上，用于节点资源管理和监控
journalnode至少需要3个节点，并且节点数为奇数个，可以部署在任意独立节点上，用于主备NameNode状态信息同步
5.服务器角色
| 主机名 | IP地址 | 组件 | | | | | | |
|----------------|-------------|----------|----------|----------|------------|-----------------|---------------------|-------------|
| idc-bigdata-01 | 10.62.1.181 | zookeeper| NodeNode | DataNode | NodeManage | ResourceManager | ZKFailoverControlle | Journalnode |
| idc-bigdata-02 | 10.62.1.182 | zookeeper| NodeNode | DataNode | NodeManage | ResourceManager | ZKFailoverControlle | Journalnode |
| idc-bigdata-03 | 10.62.1.183 | zookeeper| NodeNode | DataNode | NodeManage | ResourceManager | ZKFailoverControlle | Journalnode |
| idc-bigdata-04 | 10.62.1.184 | zookeeper| NodeNode | DataNode | NodeManage | ResourceManager | ZKFailoverControlle | Journalnode |
| idc-bigdata-05 | 10.62.1.185 | zookeeper| NodeNode | DataNode | NodeManage | ResourceManager | ZKFailoverControlle | Journalnode |
二、系统初始化
所有环境使用hadoop用户来安装
2. 环境初始化
2.1 查看内核
# uname -r
4.19.90-24.4.v2101.ky10.x86_64
默认内核为4.19
2.2 创建用户名
创建 用户
useradd hadoop
echo WLwl@2023|passwd --stdin hadoop
visudo # 可选
添加
hadoop ALL=(ALL) NOPASSWD: ALL
切换方法 su - hadoop
2.3配置主机名
hostnamectl set-hostname idc-bigdata-01 && hostname idc-bigdata-01 && hostname -f
hostnamectl set-hostname idc-bigdata-02 && hostname idc-bigdata-02 && hostname -f
hostnamectl set-hostname idc-bigdata-03 && hostname idc-bigdata-03 && hostname -f
hostnamectl set-hostname idc-bigdata-04 && hostname idc-bigdata-04 && hostname -f
hostnamectl set-hostname idc-bigdata-05 && hostname idc-bigdata-05 && hostname -f
2.4 配置主机名显示
为每个机器单独配置主机名，历史记录显示
cat > /etc/profile.d/sys.sh<2.5 配置主机名解析
# 集群所有主机
cat >/etc/hosts <2.6 关闭防火墙和selinux
# 集群所有主机
systemctl disable --now firewalld && systemctl stop firewalld
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config && setenforce 0
2.7 配置时间同步
添加定时任务
如果没有时间服务器需要自己安装，并指向时间服务器，集群时间不同步会导致集群节点宕机
yum install -y ntp 所有节点
ntp服务器
cat > /etc/ntp.conf << EOF
# 日志配置
logfile /var/log/ntpd.log
# 配置时间服务器（阿里云）
driftfile /var/lib/ntp/drift
restrict default nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
server ntp1.aliyun.com
server ntp2.aliyun.com
#外部时间服务器不可用时，以本地时间作为时间服务
server 127.127.1.0
fudge 127.127.1.0 stratum 10
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
EOF
客户端
cat > /etc/ntp.conf << EOF
driftfile /var/lib/ntp/drift
restrict default nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
server 10.61.1.189
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
EOF
systemctl restart ntpd && systemctl enable ntpd
可选 默认时区Asia/Shanghai
timedatectl
timedatectl set-timezone "Asia/Shanghai"
timedatectl set-ntp true
ntpq -p
客户端配置
echo "*/30 * * * * /usr/sbin/ntpdate -u 10.62.1.191 >/dev/null 2>&1" > /var/spool/cron/root
crontab -l
2.8 关闭swap
每台机器设置交换空间
建议将交换空间设置为0，过多的交换空间会引起GC耗时的激增
echo "vm.swappiness = 0" >> /etc/sysctl.conf && sysctl -p
sed -i '/swap/s/^(.*)$/#/g' /etc/fstab
2.9 关闭大页面压缩
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.d/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local
2.10 关闭sshd DNS配置
可选
sed -i 's/#UseDNS yes/UseDNS no/g' /etc/ssh/sshd_config
systemctl restart sshd
设置 StrictHostKeyChecking 为no
sed -i 's/^# StrictHostKeyChecking ask/StrictHostKeyChecking no/g' /etc/ssh/ssh_config
grep -w StrictHostKeyChecking /etc/ssh/ssh_config
2.11 优化句柄 打开文件数
sed -i '/^*/d' /etc/security/limits.conf
cat >>/etc/security/limits.conf <2.12 优化内核参数
cat >/etc/sysctl.d/kernel.conf <2.13 配置ssh免密
所有NameNode节点需要对DataNode节点免密，用于NameNode管理DataNode进程，
所有NameNode节点之间也需要相互免密，用于故障切换。
3个节点生成公钥和私钥，3个节点上分别执行
切换用户，生成密钥对
su - hadoop
ssh-keygen -t rsa -m PEM
一直按回车，都设置为默认值，然后再当前用户的Home目录下的.ssh目录中会生成公钥文件（id_rsa.pub）和私钥文件（id_rsa）
注意：
ssh-keygen -t rsa 直接生产出来的格式是OPENSSH，后面HDFS无法实现高可用自动切换。，所以，需要加上 -m PEM
错误记录，详见 Hadoop 之 高可用不自动切换(ssh密钥无效 Caused by: com.jcraft.jsch.JSchException: invalid privatekey )
分发脚本 每台主机上都执行
for i in idc-bigdata-{01..05};do ssh-copy-id $i -o StrictHostKeyChecking=no ;done
另外两台机器执行同样操作，生成公钥和私钥后，分发给是三台机器
验证
for host in idc-bigdata-{01..05}
do
echo ======print hostname=======
ssh $host hostname
echo ======print ip addr =========
ssh $host ip a|awk -F "[ /]+" '/inet/ && /brd/{print $3}'
echo
done
2.14 配置ssh免密其它方法
su - hadoop
ssh-keygen -t rsa -m PEM 生成RSA 类型
5个节点对idc-bigdata-01节点免密，5个节点上分别执行
ssh-copy-id idc-bigdata-01
idc-bigdata-01节点执行，分发authorized_keys文件到其他节点
scp /root/.ssh/authorized_keys idc-bigdata-02:/root/.ssh/authorized_keys
scp /root/.ssh/authorized_keys idc-bigdata-02:/root/.ssh/authorized_keys
idc-bigdata-01节点执行，所有节点公钥写入known_hosts
ssh-keyscan -t ecdsa idc-bigdata-01 idc-bigdata-02 idc-bigdata-03 > /root/.ssh/known_hosts
idc-bigdata-01节点执行，分发known_hosts文件到其他节点
scp /root/.ssh/known_hosts idc-bigdata-02:/root/.ssh/
scp /root/.ssh/known_hosts idc-bigdata-03:/root/.ssh/
验证免密成功，任意namenode节点ssh到其他节点能够免密登录
ssh idc-bigdata-01
ssh idc-bigdata-02
ssh idc-bigdata-03
2.15 安装java环境
所有节点安装java，因为zookeeper及hadoop依赖java环境。
如果使用open-jdk 这里可以不需要安装
手动安装步骤
卸载open-jdk
如果已安装open-jdk ，需要卸载open-jdk
rpm -qa|grep java
[root@idc-bigdata-03 ~]# rpm -qa|grep java
javapackages-tools-5.3.0-2.ky10.noarch
javapackages-filesystem-5.3.0-2.ky10.noarch
java-11-openjdk-headless-11.0.9.11-4.ky10.x86_64
java-1.8.0-openjdk-headless-1.8.0.272.b10-7.ky10.x86_64
java-1.8.0-openjdk-1.8.0.272.b10-7.ky10.x86_64
tzdata-java-2020a-8.p01.ky10.noarch
java-11-openjdk-11.0.9.11-4.ky10.x86_64
yum remove java-openjdk -y
[root@idc-bigdata-01 conf]# rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.342.b07-0.p01.ky10.x86_64 java-11-openjdk-headless-11.0.9.11-4.ky10.x86_64 tzdata-java-2020a-8.p01.ky10.noarch
[root@idc-bigdata-01 conf]# rpm -qa|grep java
javapackages-tools-5.3.0-2.ky10.noarch
javapackages-filesystem-5.3.0-2.ky10.noarch
或
rpm -qa|grep java|xargs -i rpm -e --nodeps {}
安装jdk
下载jdk 忽略
上传jdk-8u181-linux-x64.tar.gz 到/opt
mkdir -p /usr/java
tar xf /opt/jdk-8u181-linux-x64.tar.gz -C /usr/java
cat > /etc/profile.d/jdk8.sh << EOF
export JAVA_HOME=/usr/java/jdk1.8.0_181
export PATH=$PATH:$JAVA_HOME/bin:/usr/local/bin
export CLASSPATH=$JAVA_HOME/lib
EOF
source /etc/profile.d/jdk8.sh
java -version
其它主机安装
for host in idc-bigdata-{02..03}
do
ssh $host mkdir -p /usr/java
scp -r /usr/java/jdk1.8.0_181 $host:/usr/java
done
cat > /etc/profile.d/jdk8.sh << EOF
export JAVA_HOME=/usr/java/jdk1.8.0_181
export PATH=$PATH:$JAVA_HOME/bin:/usr/local/bin
export CLASSPATH=$JAVA_HOME/lib
EOF
source /etc/profile.d/jdk8.sh
java -version
报错
ln -sf /usr/lib64/libLLVM-7.0.0.so /usr/lib64/libLLVM-7.so
2.16 基本工具安装
yum istall -y psmisc vim net-tools wget iotop iftop
NameNode主备切换依赖fuser，所有nameNode节点安装psmisc
三、 zookeeper 集群安装
生产环境需要3到5个节点做集群
Zookeeper 版本3.8
以下操作在3个节点上执行。
所有节点安装zookeeper
3.1 解压安装
所有节点
tar xf apache-zookeeper-3.8.0-bin.tar.gz -C /usr/local
ln -sf /usr/local/apache-zookeeper-3.8.0-bin /usr/local/zookeeper
ls -lh /usr/local/
scp -r /usr/local/apache-zookeeper-3.8.0-bin idc-bigdata-02:/usr/local
scp -r /usr/local/apache-zookeeper-3.8.0-bin idc-bigdata-03:/usr/local
软连接
ln -sf /usr/local/apache-zookeeper-3.8.0-bin /usr/local/zookeeper
for host in idc-bigdata-{02..03}
do
scp -r apache-zookeeper-3.8.0-bin.tar.gz $host:/usr/local
done
3.2 配置ZK环境变量
所有节点执行
cat > /etc/profile.d/zookeeper.sh <3.3 修改zookeeper配置文件
所有节点执行
cat >/usr/local/zookeeper/conf/zoo.cfg<3.4 配置zookeeper myid
创建myid文件，id在整体中必须是唯一的，并且应该具有1到255之间的值，
主配置文件中的server id 要和其当前主节点中的myid保持一致，分别在3个节点上执行
创建data和log目录：
mkdir -p /data/zookeeper/{data,logs}
#idc-bigdata-01
mkdir -p /data/zookeeper/data
echo "1" > /data/zookeeper/data/myid
#idc-bigdata-02
mkdir -p /data/zookeeper/data
echo "2" > /data/zookeeper/data/myid
#idc-bigdata-03
mkdir -p /data/zookeeper/data
echo "3" > /data/zookeeper/data/myid
#idc-bigdata-04
mkdir -p /data/zookeeper/data
echo "4" > /data/zookeeper/data/myid
#idc-bigdata-05
mkdir -p /data/zookeeper/data
echo "5" > /data/zookeeper/data/myid
3.5 配置systemd 服务
创建zookeeper用户 可选 这里使用root启动服务
useradd -r -s /bin/false zookeeper
chown -R zookeeper:zookeeper /usr/local/zookeeper
使用systemd管理zookeeper服务
cat > /usr/lib/systemd/system/zookeeper.service << EOF
[Unit]
Description=Zookeeper Service
[Service]
Type=simple
Environment="JAVA_HOME=/usr/java/jdk1.8.0_181"
WorkingDirectory=/usr/local/zookeeper/
PIDFile=/usr/local/zookeeper/data/zookeeper_server.pid
SyslogIdentifier=zookeeper
User=root
Group=root
ExecStart=/usr/local/zookeeper/bin/zkServer.sh start
ExecStop=/usr/local/zookeeper/bin/zkServer.sh stop
Restart=always
TimeoutSec=20
SuccessExitStatus=130 143
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
3.6 启动ZK服务
启动zookeeper服务，并设为开机启动
systemctl daemon-reload && systemctl enable zookeeper && systemctl restart zookeeper && systemctl status zookeeper
#systemctl enable --now zookeeper
3.7 验证状态
查看zookeeper主备状态
[hadoop@idc-bigdata-01 ~]$ xshell zkServer.sh status
==================xshell zkServer.sh status @idc-bigdata-01======================
Authorized users only. All activities may be monitored and reported.
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
==================xshell zkServer.sh status @idc-bigdata-02======================
Authorized users only. All activities may be monitored and reported.
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
==================xshell zkServer.sh status @idc-bigdata-03======================
Authorized users only. All activities may be monitored and reported.
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
四、Hadoop集群部署
Hadoop安装配置
官网下载地址：https://hadoop.apache.org/releases.html
4.1 下载hadoop
将下载好的hadoop-3.3.5.tar.gz包解压到 /usr/module下
mkdir -p /opt/software
chown -R hadoop:hadoop /opt/software
wget https://mirrors.aliyun.com/apache/hadoop/common/hadoop-3.3.5/hadoop-3.3.5.tar.gz -P /opt/software
4.2 解压hadoop
mkdir -p /usr/module
tar xf /opt/software/hadoop-3.3.5.tar.gz -C /usr/module
ln -sf /opt/module/hadoop-3.3.5 /opt/module/hadoop
chown -R hadoop:hadoop /opt/module/hadoop*
4.3 hadoop 环境变量配置
添加以下参数
vim /etc/profile.d/myenv.sh
# HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-3.3.5
export HADOOP_CONF_DIR=/opt/module/hadoop-3.3.5/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
# ZOO_HOME
#export ZOO_HOME=/opt/module/apache-zookeeper-3.8.0-bin
#export PATH=$PATH:$ZOO_HOME/bin
# HIVE_HOME
export HIVE_HOME=/opt/module/apache-hive-3.1.2-bin
export PATH=$PATH:$HIVE_HOME/bin
# SPARK_HOME
#export SPARK_HOME=/opt/module/spark-3.1.3-bin-hadoop3.2
export SPARK_HOME=/opt/module/spark-3.3.2-bin-hadoop3
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
# KAFKA_HOME
#export KAFKA_HOME=/opt/module/kafka_2.12-3.2.0
#export PATH=$PATH:$KAFKA_HOME/bin
# HBASE_HOME
#export HBASE_HOME=/opt/module/hbase-2.4.13
export HBASE_HOME=/opt/module/hbase-2.5.3
#export HBASE_HOME=/opt/module/hbase-2.5.3-hadoop3
export PATH=$PATH:$HBASE_HOME/bin
# PHOENIX_HOME
export PHOENIX_HOME=/opt/module/phoenix-hbase-2.4-5.1.2-bin
export PHOENIX_CLASSPATH=$PHOENIX_HOME
export PATH=$PATH:$PHOENIX_HOME/bin
# JAVA_HOME
#export JAVA_HOME=/opt/module/jdk1.8.0_181
#export PATH=$PATH:$JAVA_HOME/bin
# M2_HOME
#export M2_HOME=/opt/module/maven-3.8.6
#export PATH=$PATH:$M2_HOME/bin
# EFAK HOME
#export KE_HOME=/opt/module/efak-web-3.0.1
#export PATH=$PATH:$KE_HOME/bin
# FLINK_HOME
#export FLINK_HOME=/opt/module/flink-1.13.6
export FLINK_HOME=/opt/module/flink-1.13.1
export PATH=$PATH:$FLINK_HOME/bin
#KYUUBI_HOME
export KYUUBI_HOME=/opt/module/apache-kyuubi-1.7.1-bin
export PATH=$PATH:$KYUUBI_HOME/bin
# HADOOP_CLASSPATH
export HADOOP_CLASSPATH=$(hadoop classpath)
加载变量
source /etc/profile.d/myenv.sh
查看 hadoop 版本
[root@idc-bigdata-01 ~]# hdfs version
环境变量根据需要修改
4.4 hadoop安装目录介绍
bin：Hadoop最基本的管理脚本和使用脚本的目录，这些脚本是sbin目录下管理脚本的基础实现，用户可以直接使用这些脚本管理和使用Hadoop。
etc：Hadoop配置文件所在的目录，包括core-site.xml、hdfs-site.xml、mapred-site.xml等从Hadoop1.0继承而来的配置文件和yarn-site.xml等Hadoop2.0新增的配置文件。
include：对外提供的编程库头文件（具体动态库和静态库在lib目录中），这些头文件均是用C++定义的，通常用于C++程序访问HDFS或者编写MapReduce程序。
lib：该目录包含了Hadoop对外提供的编程动态库和静态库，与include目录中的头文件结合使用。
libexec：各个服务对用的shell配置文件所在的目录，可用于配置日志输出、启动参数（比如JVM参数）等基本信息。
sbin：Hadoop管理脚本所在的目录，主要包含HDFS和YARN中各类服务的启动/关闭脚本。
share：Hadoop各个模块编译后的jar包所在的目录，官方自带示例。
4.5 配置hadoop
Hadoop配置文件修改（注意：以下所有操作都在idc-bigdata-01主机进行）
Hadoop安装主要就是配置文件的修改，一般在主节点进行修改，完毕后scp下发给其他各个从节点机器
Hadoop安装目录下的etc/hadoop目录中，需修改core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml、workers文件，根据实际情况修改配置信息。
4.5.1 修改hadoop-env.sh
该文件中设置的是Hadoop运行时需要的环境变量。JAVA_HOME是必须设置的，即使我们当前的系统中设置了JAVA_HOME
，它也是不认识的，因为Hadoop即使是在本机上执行，它也是把当前的执行环境当成远程服务器。
vi $HADOOP_HOME/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_181
export HDFS_NAMENODE_USER=hadoop
export HDFS_DATANODE_USER=hadoop
export HADOOP_SHELL_EXECNAME=hadoop
export HDFS_SECONDARYNAMENODE_USER=hadoop
export YARN_RESOURCEMANAGER_USER=hadoop
export YARN_NODEMANAGER_USER=hadoop
export HDFS_JOURNALNODE_USER=hadoop
export HDFS_ZKFC_USER=hadoop
4.5.2 修改yarn-env.sh
这一步可选
vi $HADOOP_HOME/etc/hadoop/yarn-env.sh
export YARN_REGISTRYDNS_SECURE_USER=hadoop
export YARN_RESOURCEMANAGER_USER=hadoop
export YARN_NODEMANAGER_USER=hadoop
4.5.3修改core-site.xml
vi $HADOOP_HOME/etc/hadoop/core-site.xml
fs.defaultFS
hdfs://idc-bigdata-cluster
hadoop.tmp.dir
/datas/dfs/tmp
hadoop.http.staticuser.user
hadoop
hadoop.proxyuser.hadoop.hosts
*
hadoop.proxyuser.hadoop.groups
*
hadoop.proxyuser.hadoop.users
*
io.file.buffer.size
131072
fs.trash.interval
4320
fs.trash.checkpoint.interval
0
ha.zookeeper.quorum
idc-bigdata-01:2181,idc-bigdata-02:2181,idc-bigdata-03:2181
配置说明：
fs.defaultFS 指定HDFS中NameNode的地址
hadoop.tmp.dir 指定hadoop运行时产生文件的存储目录，是其他临时目录的父目录 如果不指定dfs.namenode.name.dir dfs.datanode.data.dir 默认数据会存hadoop.tmp.dir所指路径下的dfs路径中
ha.zookeeper.quorum ZooKeeper地址列表，ZKFailoverController将在自动故障转移中使用这些地址。
io.file.buffer.size 在序列文件中使用的缓冲区大小，流文件的缓冲区为4K
fs.defaultFS：指定namenode的hdfs协议的文件系统通信地址，可以指定一个主机+端口
hadoop.tmp.dir：hadoop集群在工作时存储的一些临时文件存放的目录
4.5.4 修改hdfs-site.xml
vi $HADOOP_HOME/etc/hadoop/hdfs-site.xml
dfs.nameservices
idc-bigdata-cluster
dfs.ha.namenodes.idc-bigdata-cluster
nn1,nn2,nn3
dfs.namenode.rpc-address.idc-bigdata-cluster.nn1
idc-bigdata-01:8020
dfs.namenode.rpc-address.idc-bigdata-cluster.nn2
idc-bigdata-02:8020
dfs.namenode.rpc-address.idc-bigdata-cluster.nn3
idc-bigdata-03:8020
dfs.namenode.http-address.idc-bigdata-cluster.nn1
idc-bigdata-01:9870
dfs.namenode.http-address.idc-bigdata-cluster.nn2
idc-bigdata-02:9870
dfs.namenode.http-address.idc-bigdata-cluster.nn3
idc-bigdata-03:9870
dfs.replication
3
dfs.blocksize
134217728
dfs.namenode.name.dir
file:///datas/dfs/nn
dfs.datanode.data.dir
/datas/dfs/dn1,/datas/dfs/dn2
dfs.journalnode.edits.dir
/datas/dfs/jn
dfs.namenode.shared.edits.dir
qjournal://idc-bigdata-01:8485;idc-bigdata-02:8485;idc-bigdata-03:8485/idc-bigdata-cluster
dfs.client.failover.proxy.provider.idc-bigdata-cluster
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.ha.automatic-failover.enabled
true
dfs.ha.fencing.methods
sshfence
shell(/bin/true)
dfs.ha.fencing.ssh.private-key-files
/home/hadoop/.ssh/id_rsa
dfs.ha.fencing.ssh.connect-timeout
30000
>
dfs.client.use.datanode.hostname
true
dfs.permissions
false
配置说明:
dfs.nameservices 配置命名空间，所有namenode节点配置在命名空间mycluster下
dfs.replication 指定dataNode存储block的副本数量，默认值是3个
dfs.blocksize 大型文件系统HDFS块大小为256MB，默认是128MB
dfs.namenode.rpc-address 各个namenode的 rpc通讯地址
dfs.namenode.http-address 各个namenode的http状态页面地址
dfs.namenode.name.dir 存放namenode名称表（fsimage）的目录
dfs.datanode.data.dir 存放datanode块的目录
dfs.namenode.shared.edits.dir HA集群中多个NameNode之间的共享存储上的目录。此目录将由活动服务器写入，由备用服务器读取，以保持名称空间的同步。
dfs.journalnode.edits.dir 存储journal edit files的目录
dfs.ha.automatic-failover.enabled 是否启用故障自动处理
dfs.ha.fencing.methods 处于故障状态的时候hadoop要防止脑裂问题，所以在standby机器切换到active后，hadoop还会试图通过内部网络的ssh连过去，并把namenode的相关进程给kill掉，一般是sshfence 就是ssh方式
dfs.ha.fencing.ssh.private-key-files 配置了 ssh用的 key 的位置。
dfs.secondary.http.address：secondarynamenode运行节点的信息，应该和namenode存放在不同节点
dfs.repliction：hdfs的副本数设置，默认为3
dfs.namenode.name.dir：namenode数据的存放位置，元数据存放位置
dfs.datanode.data.dir：datanode数据的存放位置，block块存放的位置
4.5.5修改mapred-site.xml
vi $HADOOP_HOME/etc/hadoop/mapred-site.xml
mapreduce.framework.name
yarn
mapreduce.job.ubertask.enable
true
mapreduce.jobhistory.address
idc-bigdata-03:10020
mapreduce.jobhistory.webapp.address
idc-bigdata-03:19888
mapreduce.map.memory.mb
1536
mapreduce.map.java.opts
-Xmx1024M
mapreduce.reduce.memory.mb
3072
mapreduce.reduce.java.opts
-Xmx2560M
配置说明
mapreduce.framework.name 设置MapReduce运行平台为yarn
mapreduce.jobhistory.address 历史服务器的地址
mapreduce.jobhistory.webapp.address 历史服务器页面的地址
mapreduce.framework.name：指定mapreduce框架为yarn方式
mapreduce.jobhistory.address：指定历史服务器的地址和端口
mapreduce.jobhistory.webapp.address：查看历史服务器已经运行完的Mapreduce作业记录的web地址，需要启动该服务才行
默认 MapReduce Job 是存在内存中的，重启之后就会丢失，所以需要配置历史服务器，将信息存储在 HDFS 中
4.5.6 修改yarn-site.xml
vi $HADOOP_HOME/etc/hadoop/yarn-site.xml
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.resourcemanager.connect.retry-interval.ms
2000
yarn.resourcemanager.ha.enabled
true
启动Yran HA模式
yarn.resourcemanager.cluster-id
idc-bigdata-yarn-cluster
yarn.resourcemanager.ha.rm-ids
rm1,rm2
yarn.resourcemanager.recovery.enabled
true
yarn.resourcemanager.ha.automatic-failover.enabled
true
开启 ResourceManager 故障自动切换
yarn.resourcemanager.ha.automatic-failover.embedded
true
yarn.resourcemanager.hostname.rm1
idc-bigdata-01
yarn.resourcemanager.webapp.address.rm1
idc-bigdata-01:8088
yarn.resourcemanager.address.rm1
idc-bigdata-01:8032
yarn.resourcemanager.scheduler.address.rm1
idc-bigdata-01:8030
yarn.resourcemanager.resource-tracker.address.rm1
idc-bigdata-01:8031
yarn.resourcemanager.hostname.rm2
idc-bigdata-02
yarn.resourcemanager.webapp.address.rm2
idc-bigdata-02:8088
yarn.resourcemanager.address.rm2
idc-bigdata-02:8032
yarn.resourcemanager.scheduler.address.rm2
idc-bigdata-02:8030
yarn.resourcemanager.resource-tracker.address.rm2
idc-bigdata-02:8031
yarn.resourcemanager.zk-address
idc-bigdata-01:2181,idc-bigdata-02:2181,idc-bigdata-03:2181
yarn.resourcemanager.store.class
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
yarn.nodemanager.env-whitelist
JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
Number of threads to handle scheduler interface.
yarn.resourcemanager.scheduler.client.thread-count
8
yarn.nodemanager.resource.count-logical-processors-as-cores
false
yarn.nodemanager.resource.detect-hardware-capabilities
false
yarn.nodemanager.resource.pcores-vcores-multiplier
1.0
yarn.scheduler.minimum-allocation-mb
1024
yarn.scheduler.maximum-allocation-mb
10240
yarn.nodemanager.resource.memory-mb
56320
yarn.scheduler.minimum-allocation-vcores
1
yarn.nodemanager.resource.cpu-vcores
4
yarn.scheduler.maximum-allocation-vcores
4
yarn.nodemanager.pmem-check-enabled
false
yarn.nodemanager.vmem-check-enabled
false
yarn.nodemanager.vmem-pmem-ratio
2.1
yarn.log-aggregation-enable
true
yarn.log.server.url
http://idc-bigdata-03:19888/jobhistory/logs
yarn.log-aggregation.retain-seconds
604800
yarn.acl.enable
false
4.5.7 yarn 队列配置
创建两个队列分别为online和offline，将这两个队列的资源分别分配为40%、40%，且允许在资源不足时借用其他队列的资源，
online队列里面运行实时任务
offline队列里面运行离线任务
vi $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml
yarn.scheduler.capacity.maximum-applications
10000
yarn.scheduler.capacity.maximum-am-resource-percent
0.2
yarn.scheduler.capacity.resource-calculator
org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
yarn.scheduler.capacity.root.queues
default,online,offline
yarn.scheduler.capacity.root.default.capacity
20
yarn.scheduler.capacity.root.default.maximum-capacity
80
yarn.scheduler.capacity.root.default.user-limit-factor
1
yarn.scheduler.capacity.root.default.state
RUNNING
yarn.scheduler.capacity.root.default.acl_submit_applications
*
yarn.scheduler.capacity.root.default.acl_administer_queue
*
yarn.scheduler.capacity.root.default.acl_application_max_priority
*
yarn.scheduler.capacity.root.default.maximum-application-lifetime
-1
yarn.scheduler.capacity.root.default.default-application-lifetime
-1
yarn.scheduler.capacity.root.online.capacity
40
yarn.scheduler.capacity.root.online.maximum-capacity
80
yarn.scheduler.capacity.root.online.user-limit-factor
1
yarn.scheduler.capacity.root.online.state
RUNNING
yarn.scheduler.capacity.root.online.acl_submit_applications
*
yarn.scheduler.capacity.root.online.acl_administer_queue
*
yarn.scheduler.capacity.root.online.acl_application_max_priority
*
yarn.scheduler.capacity.root.online.maximum-application-lifetime
-1
yarn.scheduler.capacity.root.online.default-application-lifetime
-1
yarn.scheduler.capacity.root.offline.capacity
40
yarn.scheduler.capacity.root.offline.maximum-capacity
80
yarn.scheduler.capacity.root.offline.user-limit-factor
1
yarn.scheduler.capacity.root.offline.state
RUNNING
yarn.scheduler.capacity.root.offline.acl_submit_applications
*
yarn.scheduler.capacity.root.offline.acl_administer_queue
*
yarn.scheduler.capacity.root.offline.acl_application_max_priority
*
yarn.scheduler.capacity.root.offline.maximum-application-lifetime
-1
yarn.scheduler.capacity.root.offline.default-application-lifetime
-1
yarn.scheduler.capacity.node-locality-delay
40
yarn.scheduler.capacity.rack-locality-additional-delay
-1
Number of additional missed scheduling opportunities over the node-locality-delay
ones, after which the CapacityScheduler attempts to schedule off-switch containers,
instead of rack-local ones.
Example: with node-locality-delay=40 and rack-locality-delay=20, the scheduler will
attempt rack-local assignments after 40 missed opportunities, and off-switch assignments
after 40+20=60 missed opportunities.
When setting this parameter, the size of the cluster should be taken into account.
We use -1 as the default value, which disables this feature. In this case, the number
of missed opportunities for assigning off-switch containers is calculated based on
the number of containers and unique locations specified in the resource request,
as well as the size of the cluster.
yarn.scheduler.capacity.queue-mappings
A list of mappings that will be used to assign jobs to queues
The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]*
Typically this list will be used to map users to queues,
for example, u:%user:%user maps all users to queues with the same name
as the user.
yarn.scheduler.capacity.queue-mappings-override.enable
false
If a queue mapping is present, will it override the value specified
by the user? This can be used by administrators to place jobs in queues
that are different than the one specified by the user.
The default is false.
yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments
1
Controls the number of OFF_SWITCH assignments allowed
during a node's heartbeat. Increasing this value can improve
scheduling rate for OFF_SWITCH containers. Lower values reduce
"clumping" of applications on particular nodes. The default is 1.
Legal values are 1-MAX_INT. This config is refreshable.
yarn.scheduler.capacity.application.fail-fast
false
Whether RM should fail during recovery if previous applications'
queue is no longer valid.
yarn.scheduler.capacity.workflow-priority-mappings
A list of mappings that will be used to override application priority.
The syntax for this list is
[workflowId]:[full_queue_name]:[priority][,next mapping]*
where an application submitted (or mapped to) queue "full_queue_name"
and workflowId "workflowId" (as specified in application submission
context) will be given priority "priority".
yarn.scheduler.capacity.workflow-priority-mappings-override.enable
false
If a priority mapping is present, will it override the value specified
by the user? This can be used by administrators to give applications a
priority that is different than the one specified by the user.
The default is false.
4.5.8 修改workers
cat > $HADOOP_HOME/etc/hadoop/workers <配置说明
workers 配置datanode工作的机器，而datanode主要是用来存放数据文件的
4.5.9分发配置文件
for host in idc-bigdata-{02..03}
do
echo =========$host hadoop install=======================
scp -r /opt/module/hadoop-3.3.5 $host:/opt/module
ssh $host ln -sf /opt/module/hadoop-3.3.5 /opt/module/hadoop
ssh $host chown -R hadoop:hadoop /opt/module/hadoop*
scp -r /etc/profile.d/myenv.sh $host:/etc/profile.d/myenv.sh
ssh $host source /etc/profile.d/myenv.sh
echo
done
4.6 磁盘分区挂载
parted /dev/sdb
mklabel gpt
mkpart primary xfs 0% 100%
mkfs.xfs /dev/sdb1
parted /dev/sdc
mklabel gpt
mkpart primary xfs 0% 100%
mkfs.xfs /dev/sdc1
mkdir -p /datas/dfs/{dn1,dn2}
chown -R hadoop:hadoop /datas/dfs
添加开机挂载
vim /etc/fstab
/dev/sdb1 /datas/dfs/dn1 xfs defaults 0 0
/dev/sdc1 /datas/dfs/dn2 xfs defaults 0 0
加载
mount -a
查看
[root@idc-bigdata-01 ~]# df -hT|grep datas
/dev/sdb1 xfs 100G 747M 100G 1% /datas/dfs/dn1
/dev/sdc1 xfs 100G 747M 100G 1% /datas/dfs/dn2
4.7 启动Hadoop
按以下顺序启动hadoop相关服务：
4.7.1 第一次启动集群（必须按顺序执行否则启动不成功）
1.启动zookeeper集群
zkServer.sh start
检查启动情况： jps ，每个节点包含QuorumPeerMain进程代表zk集群启动成功或者每个节点执行：
zkServer.sh status，查看是否有leader节点。
2.每个namenode节点上启动journalnode进程
hdfs --daemon start journalnode
3.在第一台namenode节点上格式化namenode
hdfs namenode -format
启动这台节点的namenode
hdfs --daemon start namenode
4.其他Namenode节点上首次要手动同步一次数据，在另外俩个节点执行：
hdfs namenode -bootstrapStandby
同步完成后启动namenode
hdfs --daemon start namenode
5.初始化ZKFC 注这里的初始化也可以放在第一步操作
ZKFC用于监控active namenode节点是否挂掉，通知其它节点上的ZKFC强行杀死自己ZKFC节点上的namenode
（防止其假死状态产生集群namenode脑裂的发生），然后选举出其他namenode为active节点
集群首次搭建需要在zookeeper中初始化namenode信息，在namenode1节点执行命令：
hdfs zkfc -formatZK
主节点启动zkfc
hdfs --daemon start zkfc
#### 启动HDFS相关进程
在namenode1节点上，启动HDFS整个集群（包括其他节点）的其他进程（Datanode/DFSZKFailoverController），执行命令：
start-dfs.sh
start-yarn.sh
或者执行start-all.sh
6.查看namenode active节点
hdfs haadmin -getAllServiceState
[hadoop@idc-bigdata-01 ~]$ hdfs haadmin -getAllServiceState
idc-bigdata-01:8020 active
idc-bigdata-02:8020 standby
idc-bigdata-03:8020 standby
7. 启动日志服务
idc-bigdata-03
mapred --daemon start historyserver
8. 验证高可用
kill -9 active namenode进程，查看页面状态，可发现另外某个namenode自动切换成active状态。
记住kill掉的，实验结束后再启动起来。
kill 或 hdfs --daemon stop namenode
集群一共一有namenode 最多可以容忍两个宕机
9. 验证集群是否可用
hdfs dfs -mkdir /data
hdfs dfs -mkdir hdfs://idc-bigdata-cluster/wordcount
hdfs dfs -put wordcount.txt hdfs://idc-bigdata-cluster/wordcount
hdfs dfs -ls hdfs://idc-bigdata-cluster/wordcount
4.8 具体操作步骤
4.8.1 初始化zkfc
格式化ZooKeeper集群，目的是在ZooKeeper集群上建立HA的相应节点，任意节点执行
这里使用第一个节点 idc-bigdata-01
su - hadoop
hdfs zkfc -formatZK
执行输出
2023-06-02 04:45:17,095 INFO ha.ActiveStandbyElector: Session connected.
2023-06-02 04:45:17,139 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/idc-bigdata-cluster in ZK.
2023-06-02 04:45:17,251 INFO zookeeper.ZooKeeper: Session: 0x2000134fa350000 closed
2023-06-02 04:45:17,252 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2000134fa350000
2023-06-02 04:45:17,252 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x2000134fa350000
2023-06-02 04:45:17,257 INFO tools.DFSZKFailoverController: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at idc-bigdata-01/10.62.1.192
************************************************************/
验证zkfc是否格式化成功
# zkCli.sh
[zk: localhost:2181(CONNECTED) 1] ls /
[hadoop-ha, zookeeper]
[zk: localhost:2181(CONNECTED) 2] ls /hadoop-ha
[idc-bigdata-cluster]
4.8.2 启动journalnode
idc-bigdata-01、idc-bigdata-02及idc-bigdata-03节点启动journalnode
su - hadoop
hdfs --daemon start journalnode 每个节点都需要启动
jps 查看java进程
[hadoop@idc-bigdata-01 ~]$ jps -l
46520 org.apache.hadoop.hdfs.qjournal.server.JournalNode
每个节点上都有这个进程
监听端口为：8485
4.8.3 namenode节点格式化
在其中一个namenode节点执行格式化，以在idc-bigdata-01节点为例
su - hadoop
hdfs namenode -format
执行输出
2023-06-02 04:59:50,441 INFO util.GSet: VM type = 64-bit
2023-06-02 04:59:50,442 INFO util.GSet: 0.029999999329447746% max memory 3.2 GB = 1007.6 KB
2023-06-02 04:59:50,442 INFO util.GSet: capacity = 2^17 = 131072 entries
2023-06-02 04:59:51,244 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1983029622-10.62.1.192-1685653191244
2023-06-02 04:59:51,314 INFO common.Storage: Storage directory /datas/dfs/name has been successfully formatted.
2023-06-02 04:59:51,597 INFO namenode.FSImageFormatProtobuf: Saving image file /datas/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2023-06-02 04:59:51,716 INFO namenode.FSImageFormatProtobuf: Image file /datas/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 398 bytes saved in 0 seconds .
2023-06-02 04:59:51,745 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2023-06-02 04:59:51,835 INFO namenode.FSNamesystem: Stopping services started for active state
2023-06-02 04:59:51,835 INFO namenode.FSNamesystem: Stopping services started for standby state
2023-06-02 04:59:51,841 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2023-06-02 04:59:51,842 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at idc-bigdata-01/10.62.1.192
************************************************************/
4.8.4 启动idc-bigdata-01节点nameNode
启动idc-bigdata-01节点nameNode
su - hadoop
hdfs --daemon start namenode
查看进程
[hadoop@idc-bigdata-01 ~]$ jps
47863 NameNode
46520 JournalNode
47918 Jps
[hadoop@idc-bigdata-01 ~]$ ls -lh /datas/dfs/name/
total 4.0K
drwx------ 2 hadoop hadoop 112 Jun 2 04:59 current
-rw-r--r-- 1 hadoop hadoop 20 Jun 2 05:06 in_use.lock
4.8.5 同步其它节点nameNode数据
将idc-bigdata-01节点上namenode的数据同步到其他nameNode节点，在idc-bigdata-02、idc-bigdata-03节点执行：
hdfs namenode -bootstrapStandby
执行过程
idc-bigdata-02
=====================================================
2023-06-01 21:13:27,641 INFO common.Storage: Storage directory /datas/dfs/name has been successfully formatted.
2023-06-01 21:13:27,694 INFO namenode.FSEditLog: Edit logging is async:true
2023-06-01 21:13:27,776 INFO namenode.TransferFsImage: Opening connection to http://idc-bigdata-01:9870/imagetransfer?getimage=1&txid=0&storageInfo=-66:53130945:1685653191244:CID-c35ed3c1-4131-4a48-bf1b-7d8ffe34a40d&bootstrapstandby=true
2023-06-01 21:13:27,877 INFO common.Util: Combined time for file download and fsync to all disks took 0.00s. The file download took 0.00s at 0.00 KB/s. Synchronous (fsync) write to disk of /datas/dfs/name/current/fsimage.ckpt_0000000000000000000 took 0.00s.
2023-06-01 21:13:27,878 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 398 bytes.
2023-06-01 21:13:27,885 INFO ha.BootstrapStandby: Skipping InMemoryAliasMap bootstrap as it was not configured
2023-06-01 21:13:27,888 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at idc-bigdata-02/10.62.1.193
idc-bigdata-03
=====================================================
2023-06-02 05:00:58,927 INFO common.Storage: Storage directory /datas/dfs/name has been successfully formatted.
2023-06-02 05:00:58,971 INFO namenode.FSEditLog: Edit logging is async:true
2023-06-02 05:00:59,045 INFO namenode.TransferFsImage: Opening connection to http://idc-bigdata-01:9870/imagetransfer?getimage=1&txid=0&storageInfo=-66:53130945:1685653191244:CID-c35ed3c1-4131-4a48-bf1b-7d8ffe34a40d&bootstrapstandby=true
2023-06-02 05:00:59,059 INFO common.Util: Combined time for file download and fsync to all disks took 0.00s. The file download took 0.00s at 0.00 KB/s. Synchronous (fsync) write to disk of /datas/dfs/name/current/fsimage.ckpt_0000000000000000000 took 0.00s.
2023-06-02 05:00:59,059 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 398 bytes.
2023-06-02 05:00:59,077 INFO ha.BootstrapStandby: Skipping InMemoryAliasMap bootstrap as it was not configured
2023-06-02 05:00:59,081 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at idc-bigdata-03/10.62.1.194
************************************************************/
4.8.6 启动其它节点nameNode
启动idc-bigdata-02及idc-bigdata-03节点nameNode
hdfs --daemon start namenode
查看进程
======================idc-bigdata-01======================
47863 NameNode
46520 JournalNode
======================idc-bigdata-02======================
21208 NameNode
19946 JournalNode
======================idc-bigdata-03======================
19938 JournalNode
21182 NameNode
监控端口 8020 9870
浏览器访问NameNode,当前所有NameNode都是standby状态：
http://idc-bigdata-01:9870/
http://idc-bigdata-02:9870/
http://idc-bigdata-03:9870/
4.9 启动所有其他服务，包括zkfc
start-all.sh
此时再次查看nameNode界面，发现已经选举出一个active节点:
http://idc-bigdata-01:9870/
======================idc-bigdata-01======================
52096 NameNode
53428 ResourceManager
52949 DFSZKFailoverController
52310 DataNode
53646 NodeManager
52638 JournalNode
======================idc-bigdata-02======================
23856 NameNode
24962 NodeManager
24514 DFSZKFailoverController
24051 DataNode
24775 ResourceManager
24269 JournalNode
======================idc-bigdata-03======================
24178 DFSZKFailoverController
24469 NodeManager
23931 JournalNode
23723 DataNode
23532 NameNode
4.10 验证hadoop功能
4.10.1 验证HDFS上传等功能
[hadoop@idc-bigdata-01 ~]$ hdfs dfs -put .bash_history /
[hadoop@idc-bigdata-01 ~]$ hdfs dfs -ls /
Found 12 items
-rw-r--r-- 3 hadoop supergroup 7818 2023-06-02 10:23 /.bash_history
4.10.2 验证YARN功能
计算圆周率
[hadoop@idc-bigdata-01 ~]$ hadoop jar /opt/module/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar pi 10 100
......
Job Finished in 3.059 seconds
Estimated value of Pi is 3.14800000000000000000
4.10.3验证mapreduce功能
echo "hello hadoop" > wordtest
hadoop fs -put wordtest /wordtest
执行 mapreduce 后生成结果文件
hadoop jar /opt/module/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar wordcount /wordtest /result
查看统计结果
hadoop fs -cat /result/part-r-00000
[hadoop@idc-bigdata-01 ~]$ hadoop fs -cat /result/part-r-00000
hadoop 1
hello 1
4.10.4验证HA高可用性
测试是否能够完成自动故障转移。
之后刷新页面我们发现idc-bigdata-02节点（原standy）自动变成了 active namenode。
查看nodename状态
[hadoop@idc-bigdata-01 ~]$ hdfs haadmin -getAllServiceState
idc-bigdata-01:8020 standby
idc-bigdata-02:8020 active
idc-bigdata-03:8020 standby
停止idc-bigdata-02 上namenode进程
在idc-bigdata-02节点active namenode上执行 jps ，确定namenode进程，kill 将其杀掉
kill -9 2424 或 hdfs --daemon stop namenode
[hadoop@idc-bigdata-02 ~]$ hdfs --daemon stop namenode
[hadoop@idc-bigdata-02 ~]$ jps
60886 Jps
53494 DataNode
54248 ResourceManager
53706 JournalNode
54443 NodeManager
53918 DFSZKFailoverController
查看namenode状态
[hadoop@idc-bigdata-02 ~]$ hdfs haadmin -getAllServiceState
idc-bigdata-01:8020 standby
2023-06-02 11:37:47,527 INFO ipc.Client: Retrying connect to server: idc-bigdata-02/10.62.1.193:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
idc-bigdata-02:8020 Failed to connect: Call From idc-bigdata-02/10.62.1.193 to idc-bigdata-02:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
idc-bigdata-03:8020 active
[hadoop@idc-bigdata-02 ~]$ hdfs haadmin -getServiceState nn1
现在namenode 已切换到idc-bigdata-03
再次验证上传下载等功能是否正常
HA模式RM的active节点
yarn rmadmin -getAllServiceState
yarn rmadmin -getServiceState rm1
切换
yarn rmadmin -transitionToStandby rm1
重置hadoop集群
删除旧数据
检查hadoop集群状态，如果集群中存在hadoop的服务进程，将hadoop的所有服务组件都关闭
关闭hadoop集群服务组件
清理hadoop集群安装目录下的data和logs目录
停上zookeeper 集群，删除 zookeeper 相关删除
rm -rf /data/zookeeper/data/version-2
rm -rf /datas/dfs/dn1/
rm -rf /datas/dfs/dn2/*
rm -rf /datas/dfs/jn
rm -rf /datas/dfs/nn
rm -rf /datas/dfs/tmp
按顺序重启初始化
1.启动zookeeper集群
zkServer.sh start
检查启动情况： jps ，每个节点包含QuorumPeerMain进程代表zk集群启动成功或者每个节点执行：
zkServer.sh status，查看是否有leader节点。
2.每个namenode节点上启动journalnode进程
hdfs --daemon start journalnode
3.在第一台namenode节点上格式化namenode
hdfs namenode -format
启动这台节点的namenode
hdfs --daemon start namenode
4.其他Namenode节点上首次要手动同步一次数据，在另外俩个节点执行：
hdfs namenode -bootstrapStandby
同步完成后启动namenode
hdfs --daemon start namenode
5.初始化ZKFC 注这里的初始化也可以放在第一步操作
ZKFC用于监控active namenode节点是否挂掉，通知其它节点上的ZKFC强行杀死自己ZKFC节点上的namenode
（防止其假死状态产生集群namenode脑裂的发生），然后选举出其他namenode为active节点
集群首次搭建需要在zookeeper中初始化namenode信息，在namenode1节点执行命令：
hdfs zkfc -formatZK
主节点启动zkfc
hdfs --daemon start zkfc
#### 启动HDFS相关进程
在namenode1节点上，启动HDFS整个集群（包括其他节点）的其他进程（Datanode/DFSZKFailoverController），执行命令：
start-dfs.sh
start-yarn.sh
或者执行start-all.sh
6.查看namenode active节点
hdfs haadmin -getAllServiceState
[hadoop@idc-bigdata-01 ~]$ hdfs haadmin -getAllServiceState
idc-bigdata-01:8020 active
idc-bigdata-02:8020 standby
idc-bigdata-03:8020 standby
7. 启动日志服务
idc-bigdata-03
mapred --daemon start historyserver
8. 验证高可用
kill -9 active namenode进程，查看页面状态，可发现另外某个namenode自动切换成active状态。
记住kill掉的，实验结束后再启动起来。
kill 或 hdfs --daemon stop namenode
集群一共一有namenode 最多可以容忍两个宕机
9. 验证集群是否可用
hdfs dfs -mkdir /data
hdfs dfs -mkdir hdfs://idc-bigdata-cluster/wordcount
hdfs dfs -put wordcount.txt hdfs://idc-bigdata-cluster/wordcount
hdfs dfs -ls hdfs://idc-bigdata-cluster/wordcount
hadoop 文件分析
查看namenode 目录结构
ls -lh /datas/dfs/nn/current/
-rw-rw-r-- 1 hadoop hadoop 42 Jun 4 11:51 edits_0000000000000002362-0000000000000002363
-rw-rw-r-- 1 hadoop hadoop 1.0M Jun 4 11:51 edits_inprogress_0000000000000002364
-rw-rw-r-- 1 hadoop hadoop 9.0K Jun 4 10:28 fsimage_0000000000000002018
-rw-rw-r-- 1 hadoop hadoop 62 Jun 4 10:28 fsimage_0000000000000002018.md5
-rw-rw-r-- 1 hadoop hadoop 12K Jun 4 11:28 fsimage_0000000000000002263
-rw-rw-r-- 1 hadoop hadoop 62 Jun 4 11:28 fsimage_0000000000000002263.md5
-rw-rw-r-- 1 hadoop hadoop 5 Jun 4 11:51 seen_txid
-rw-r--r-- 1 hadoop hadoop 215 Jun 3 21:14 VERSION
fsimage是NameNode元数据在内存满了后，持久化保存到的文件。
fsimage*.md5 是校验文件，用于校验fsimage的完整性。
seen_txid 是hadoop的版本
vession文件里保存
namespaceID：NameNode的唯一ID
clusterID:集群ID NameNode和DataNode的集群ID应该一致，表明是一个集群
[root@idc-bigdata-01 ~]# more /datas/dfs/nn/current/VERSION
#Sat Jun 03 21:14:52 CST 2023
namespaceID=1498354829
clusterID=CID-24440b17-58ee-4d55-a589-ea0a5c5c0be6
cTime=1685798092376
storageType=NAME_NODE
blockpoolID=BP-899285325-10.62.1.192-1685798092376
layoutVersion=-66
注意：
如果需要重新格式化NameNode,需要先将原来NameNode和DataNode下的文件全部删除，不然会报错，NameNode和DataNode所在
目录是在core-site.xml中hadoop.tmp.dir、dfs.namenode.name.dir、dfs.datanode.data.dir属性配置的。
因为每次格式化，默认是创建一个集群ID，并写入NameNode和DataNode的VERSION文件中（VERSION文件所在目录为
dfs/name/current 和 dfs/data/current），重新格式化时，默认会生成一个新的集群ID,如果不删除原来的目录，
会导致namenode中的VERSION文件中是新的集群ID,而DataNode中是旧的集群ID，不一致时会报错。
