Hadoop HA集群配置

Hadoop HA集群配置

高可用ZooKeeper集群部署

实验任务一：ZooKeeper安装部署

步骤一：解压安装jdk

 [root@master ~]# tar -zxvf  /opt/software/jdk-8u152-linux-x64.tar.gz -C /usr/local/src

更改jdk的名称

[root@master ~]# mv /usr/local/src/jdk1.8.0_152/ /usr/local/src/java

步骤二：安装ZooKeeper

解压并安装zookeeper到apps下

[root@master ~]# tar -zxf /opt/software/zookeeper-3.4.8.tar.gz -C /usr/local/src/

[root@master ~]# cd /usr/local/src/

[root@master src]# mv zookeeper-3.4.8 zookeeper

步骤三：创建ZooKeeper数据目录

data是用来传输数据的，logs是用来记录日志的

[root@master src]# mkdir /usr/local/src/zookeeper/data

[root@master src]# mkdir /usr/local/src/zookeeper/logs

实验任务二：ZooKeeper文件参数配置

步骤一：配置ZooKeeper环境变量

[root@master src]# cd ./zookeeper

[root@master zookeeper]# vi /etc/profile

添加如下配置：

#java environment

export JAVA_HOME=/usr/local/src/java #JAVA_HOME指向JAVA安装目录

export PATH=$PATH:$JAVA_HOME/bin #将JAVA安装目录加入PATH路径

#zookeeper environment

export ZK_HOME=/usr/local/src/zookeeper

export PATH=$PATH:$ZK_HOME/bin

保存并退出

步骤二：修改zoo.cfg配置文件

首先先进入到conf目录下将zoo.cfg文件拷贝过来 

[root@master zookeeper]# cd conf/

[root@master conf]# ls

configuration.xsl  log4j.properties  zoo_sample.cfg

[root@master conf]# cp zoo_sample.cfg zoo.cfg

[root@master conf]# vi zoo.cfg

添加并更改如下配置：

#修改

dataDir=/usr/local/src/zookeeper/data

#增加

dataLogDir=/usr/local/src/zookeeper/logs

server.1=master:2888:3888

server.2=slave1:2888:3888

server.3=slave2:2888:3888

//上面的IP可以换成自己的主机地址，或者换成主机名，一般我们换成主机名

保存并退出

步骤三：创建myid配置文件

[root@master conf]# cd ..

[root@master zookeeper]# cd data/

[root@master data]# echo "1" > myid

实验任务三：ZooKeeper集群启动

步骤一：分发ZooKeeper集群

[root@master data]# scp -r /usr/local/src/zookeeper/ root@slave1:/usr/local/src/

[root@master data]# scp -r /usr/local/src/zookeeper/ root@slave2:/usr/local/src/

分发环境变量并使其生效

[root@master data]# scp /etc/profile root@slave1:/etc/

[root@master data]# scp /etc/profile root@slave2:/etc/

[root@master data]# source /etc/profile

[root@slave1 ~]# source /etc/profile

[root@slave2 ~]# source /etc/profile

步骤二：修改myid配置

master对应1,slave1对应2，slave2对应3

[root@master data]# cat myid

1

[root@slave1 ~]# cd /usr/local/src/zookeeper/data/

[root@slave1 data]# echo "2">myid

[root@slave1 data]# cat myid

2

[root@slave2 ~]# cd /usr/local/src/zookeeper/data/

[root@slave2 data]# echo "3">myid

[root@slave2 data]# cat myid

3

步骤三：修改ZooKeeper安装目录的归属用户为hadoop用户。

[root@master data]# chown -R hadoop:hadoop /usr/local/src/zookeeper

[root@slave1 data] # chown -R hadoop:hadoop /usr/local/src/zookeeper

[root@slave2 data] # chown -R hadoop:hadoop /usr/local/src/zookeeper

步骤四：启动ZooKeeper集群

关闭防火墙

[root@master data]# systemctl stop firewalld.service

[root@slave1 data]# systemctl stop firewalld.service

[root@slave2 data]# systemctl stop firewalld.service

关闭防火墙自启

[root@master data]# systemctl disable firewalld.service

Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

[root@slave1 data]# systemctl disable firewalld.service

Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

[root@slave2 data]# systemctl disable firewalld.service

Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

同时启动三个节点的zookeeper

[root@master data]# su – hadoop

[hadoop@master root]$ cd

[hadoop@master ~]$ source /etc/profile

[hadoop@master ~]$ zkServer.sh start   # ZooKeeper启动

JMX enabled by default

Jsing. config:' /usr/local/src/zookeeper/bin/ ../conf/zoo.cfg

starting zookeeper . .STARTED

[root@slave1 data]# su hadoop

[hadoop@slave1 root]$ cd

[hadoop@slave1 ~]$ source /etc/profile

[hadoop@slave1 ~]$ zkServer.sh start   # ZooKeeper启动

JMX enabled by default

Jsing. config:' /usr/local/src/zookeeper/bin/ ../conf/zoo.cfg

starting zookeeper . .STARTED

[root@slave2 data]# su hadoop

[root@slave2 data]# cd

[hadoop@slave2 ~]$ source /etc/profile

[hadoop@slave2 ~]$ zkServer.sh start   # ZooKeeper启动

JMX enabled by default

Jsing. config:' /usr/local/src/zookeeper/bin/ ../conf/zoo.cfg

starting zookeeper . .STARTED

查看状态

[hadoop@master ~]$ zkServer.sh status

JMX enabled by default

Using config: /usr/local/src/zookeeper/bin/ ../conf/zoo.cfg

Mode: fo11ower  # follower状态

#slave1节点状态

[hadoop@slave1 ~]$ zkServer.sh status

JMX enab1ed by default

Using config:' /usr/local/src/zookeeper/bin/ . ./conf/zoo.cfg

Mode:leader   # leader状态

#slave2节点状态

[hadoop@slave2 ~]$ zkServer.sh status

JMX enabled by default

Using config:' /usr/local/src/zookeeper/bin/ . ./conf/zoo.cfg

Mode: fo11ower  # follower状态

实验二：Hadoop HA集群部署

实验任务一：ssh免密配置(第四章已配置）

步骤一：创建免密（三个主机同时进行）

[root@master ~]# su - hadoop

[hadoop@master ~]$ ssh-keygen -t rsa -P  ""

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Created directory '/root/.ssh'.

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:12nHZiHa8tK+hgmqid/Vvq5lAxpMZFFoY4lmnkoU7s4 root@master

The key's randomart image is:

+---[RSA 2048]----+

|  .. .+=.        |

| .. +o*          |

| ..+ +..    . .  |

| .. oo     + + . |

| ...  o S + = =  |

| o.    o.+ = +   |

|  E   ....*oo    |

|   . o.. +o+.    |

|  ..+.. .o+oo.   |

+----[SHA256]-----+

[root@master ~]# su - hadoop

[hadoop@slave1 ~]$ ssh-keygen -t rsa -P  ""

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Created directory '/root/.ssh'.

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:iOZ1zxts951cUelPDLPXGY0OPrFi+HDbcOLchhKf2v0 root@slave1

The key's randomart image is:

+---[RSA 2048]----+

|                 |

|               .o|

|            o +oo|

|     . . . . =.==|

|    o o S * = o+=|

|   o . . & @ . oo|

|    .   . # =   o|

|         = * ...o|

|        . o ..Eo.|

+----[SHA256]-----+

[root@master ~]# su - hadoop

[hadoop@slave2 ~]$ ssh-keygen -t rsa -P  ""

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Created directory '/root/.ssh'.

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:LxeFYtxLWfV9ZKhh9M3vM6vwuLJl6sVoJbezjeHHf1w root@slave2

The key's randomart image is:

+---[RSA 2048]----+

|           .o...o|

|       . . +o..*.|

|        + =..o. *|

|       . o o.   o|

|        S.oo    .|

|         .*..  .E|

|        .ooX.  +o|

|        .+* Xo  *|

|        .+o*o+oo.|

+----[SHA256]-----+

步骤二：创建公钥

[hadoop@master ~]$ cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys

[hadoop@master ~]$ chmod 700 ~/.ssh/authorized_keys

步骤三：将masterr创建的公钥发给slave1

[hadoop@master ~]# scp ~/.ssh/authorized_keys root@slave1:~/.ssh/

The authenticity of host 'slave1 (192.168.1.7)' can't be established.

ECDSA key fingerprint is SHA256:Nnk2MJS3KmUzmXXzgE0DTgnq990XctFMFUV82UdgFnQ.

ECDSA key fingerprint is MD5:f3:fa:be:c7:52:1e:96:ee:1b:7d:1a:26:23:a9:66:ec.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave1,192.168.1.7' (ECDSA) to the list of known hosts.

root@slave1's password:

authorized_keys                                                                                        100%  393   319.0KB/s   00:00

步骤四：将slave1的私钥加到公钥里

[hadoop@slave1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

步骤五：将公钥发给slave2

[hadoop@slave1 ~]$ scp ~/.ssh/authorized_keys root@slave2:~/.ssh/

The authenticity of host 'slave2 (192.168.1.8)' can't be established.

ECDSA key fingerprint is SHA256:Nnk2MJS3KmUzmXXzgE0DTgnq990XctFMFUV82UdgFnQ.

ECDSA key fingerprint is MD5:f3:fa:be:c7:52:1e:96:ee:1b:7d:1a:26:23:a9:66:ec.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave2,192.168.1.8' (ECDSA) to the list of known hosts.

root@slave2's password:

authorized_keys                                                                                        100%  786   525.0KB/s   00:00  

步骤六：登陆测试

[hadoop@master ~]$ ssh slave1

Last login: Wed Jun 24 16:41:46 2020 from 192.168.1.7

[hadoop@slave1 ~]$ ssh slave2

Last login: Wed Jun 24 16:35:46 2020 from 192.168.1.1

[hadoop@slave2 ~]$ exit

登出

Connection to slave2 closed.

[hadoop@slave1 ~]$ exit

登出

Connection to slave1 closed.

[hadoop@master ~]$ 

实验任务二：Hadoop HA文件参数配置

步骤一：解压安装Hadoop

[hadoop@master ~]$ stop-all.sh

[hadoop@master ~]$ su root

删除第4章安装的hadoop

[root@master ~]# rm -r -f /usr/local/src/hadoop

[root@slave1 ~]# rm -r -f /usr/local/src/hadoop

[root@slave2 ~]# rm -r -f /usr/local/src/hadoop

[root@master ~]# tar -zxvf /opt/software/hadoop-2.7.1.tar.gz -C /usr/local/src/

步骤二：更改hadoop文件名

[root@master ~]# mv /usr/local/src/hadoop-2.7.1 /usr/local/src/hadoop

步骤三：配置hadoop环境变量

[root@master ~]# vi /etc/profile

进行如下配置(此处需先删除第四章配置的环境变量)

#hadoop enviroment

export HADOOP_HOME=/usr/local/src/hadoop  #HADOOP_HOME指向JAVA安装目录

export HADOOP_PREFIX=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_INSTALL=$HADOOP_HOME

export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin   

#java environment

export JAVA_HOME=/usr/local/src/java #JAVA_HOME指向JAVA安装目录

export PATH=$PATH:$JAVA_HOME/bin #将JAVA安装目录加入PATH路径

#zookeeper environment

export ZK_HOME=/usr/local/src/zookeeper

export PATH=$PATH:$ZK_HOME/bin  

保存并退出

步骤四：配置hadoop-env.sh配置文件

进入到hadoop/etc/hadoop下

[root@master ~]# cd /usr/local/src/hadoop/etc/hadoop

[root@master hadoop]# vi hadoop-env.sh

在最下面添加如下配置：    

export JAVA_HOME=/usr/local/src/java

步骤五：配置core-site.xml配置文件

[root@master hadoop]# vi core-site.xml

添加如下配置：

    

        fs.defaultFS

        hdfs://mycluster

    

    

        hadoop.tmp.dir

        file:/usr/local/src/hadoop/tmp

    

    

    

        ha.zookeeper.quorum

        master:2181,slave1:2181,slave2:2181

    

    

    

        ha.zookeeper.session-timeout.ms

        30000

        ms

    

    

        fs.trash.interval

        1440

    

保存并退出

步骤六：配置hdfs-site.xml配置文件

[root@master hadoop]# vi hdfs-site.xml

进行如下配置：



    

        dfs.qjournal.start-segment.timeout.ms

        60000

    

    

    

        dfs.nameservices

        mycluster

    

        

    

        dfs.ha.namenodes.mycluster

        master,slave1

    

    

    

        dfs.namenode.rpc-address.mycluster.master

        master:8020

    

    

    

        dfs.namenode.rpc-address.mycluster.slave1

        slave1:8020

    

     

    

        dfs.namenode.http-address.mycluster.master

        master:50070

    

    

    

        dfs.namenode.http-address.mycluster.slave1

        slave1:50070

    

    

    

        dfs.namenode.shared.edits.dir

        qjournal://master:8485;slave1:8485;slave2:8485/mycluster

    

    

    

        dfs.client.failover.proxy.provider.mycluster

        org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

    

    

    

        dfs.ha.fencing.methods

        

            sshfence

        shell(/bin/true)

        

    

    

       dfs.permissions.enabled

       false

    

    

        dfs.support.append

        true

    

    

    

        dfs.ha.fencing.ssh.private-key-files

        /home/hadoop/.ssh/id_rsa

    

    

    

        dfs.replication

        2

    

    

        dfs.namenode.name.dir

        /usr/local/src/hadoop/tmp/hdfs/nn

    

    

        dfs.datanode.data.dir

        /usr/local/src/hadoop/tmp/hdfs/dn

    

    

    

        dfs.journalnode.edits.dir

        /usr/local/src/hadoop/tmp/hdfs/jn

    

    

    

        dfs.ha.automatic-failover.enabled

        true

    

    

    

        dfs.webhdfs.enabled

        true

    

    

    

        dfs.ha.fencing.ssh.connect-timeout

        30000

    

    

        ha.failover-controller.cli-check.rpc-timeout.ms

        60000

    

保存并退出

步骤七：配置mapred-site.xml配置文件

[root@master hadoop]# cp mapred-site.xml.template mapred-site.xml

[root@master hadoop]# vi mapred-site.xml

进行如下配置：

    

        mapreduce.framework.name

        yarn



    

    

        mapreduce.jobhistory.address

        master:10020

    

    

    

        mapreduce.jobhistory.webapp.address

        master:19888

  

保存并退出

步骤八：配置yarn-site.xml配置文件

[root@master hadoop]# vi yarn-site.xml

进行如下配置：

 

    

    

        yarn.resourcemanager.ha.enabled

        true

    

    

    

        yarn.resourcemanager.cluster-id

        yrc

    

    

    

        yarn.resourcemanager.ha.rm-ids

        rm1,rm2

    

    

    

        yarn.resourcemanager.hostname.rm1

        master

    

    

        yarn.resourcemanager.hostname.rm2

        slave1

    

    

    

        yarn.resourcemanager.zk-address

        master:2181,slave1:2181,slave2:2181

    

    

        yarn.nodemanager.aux-services

        mapreduce_shuffle

    

    

        yarn.log-aggregation-enable

        true

    

    

        yarn.log-aggregation.retain-seconds

        86400

    

    

    

        yarn.resourcemanager.recovery.enabled

        true

    

    

    

        yarn.resourcemanager.store.class

   org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore

    

保存并退出

步骤九：配置slaves配置文件

[root@master hadoop]# vi slaves

进行如下配置：

master

slave1

slave2

步骤十：解压包到指定目录

namenode、datanode、journalnode等存放数据的公共目录为/usr/local/src/hadoop/tmp；

 在master上执行如下：

[root@master hadoop]#  mkdir -p /usr/local/src/hadoop/tmp/hdfs/nn

[root@master hadoop]#  mkdir -p /usr/local/src/hadoop/tmp/hdfs/dn

[root@master hadoop]#  mkdir -p /usr/local/src/hadoop/tmp/hdfs/jn

[root@master hadoop]#  mkdir -p /usr/local/src/hadoop/tmp/logs

步骤十一：分发文件

[root@master hadoop]# scp -r /etc/profile root@slave1:/etc/

[root@master hadoop]# scp -r /etc/profile root@slave2:/etc/

[root@master hadoop]# scp -r /usr/local/src/hadoop root@slave1:/usr/local/src/

[root@master hadoop]# scp -r /usr/local/src/hadoop root@slave2:/usr/local/src/

步骤十二：修改目录所有者和所有者组

上述安装完成的Hadoop软件只能让root用户使用，要让hadoop用户能够运行Hadoop软件，需要将目录/usr/local/src的所有者改为hadoop用户。

[root@master ~]# chown -R hadoop:hadoop /usr/local/src/hadoop

[root@master ~]# ll /usr/local/src/

[root@slave1~]# chown -R hadoop:hadoop /usr/local/src/hadoop/ 

[root@slave2~]# chown -R hadoop:hadoop /usr/local/src/hadoop/  

步骤十三：生效环境变量

[root@master hadoop]# su hadoop

[hadoop@master hadoop]$ source /etc/profile

[root@slave1 hadoop]# su hadoop

[hadoop@slave1 hadoop]$ source /etc/profile

[root@slave2 hadoop]# su hadoop

[hadoop@slave2 hadoop]$ source /etc/profile