hadoop hive 搭建的其他方法

hadoop hive 搭建的其他方法

1. Hive搭建

1）将apache-hive-3.1.2-bin.tar.gz压缩包放在/opt/software路径下

 

2）、[root@hadoop100 ~]# cd /opt/software

3）、解压apache-hive-3.1.2-bin.tar.gz到/opt/module/路径下

[root@hadoop100 software]# tar -zxvf apache-hive-3.1.2-bin.tar.gz -C /opt/module/

4）、进入解压之后的apache-hive-3.1.2-bin路径下的conf/文件夹下

[root@hadoop100 module]# cd /opt/module/apache-hive-3.1.2-bin/conf/

5）、创建并编辑hive-site.xml文件，输入以下内容：

注意：下列标红部分为按自己VMnet8的IP及数据库账号密码进行配置

[root@hadoop100 conf]# vi hive-site.xml

                           

                                                       

                                                                                   

                                                          

                                                                                      

    javax.jdo.option.ConnectionURL                                                        

                                                                                        jdbc:mysql://192.168.1.5:3306/hive?useSSL=false&characterEncoding=utf8&a                            mp;serverTimezone=UTC                                                                                    

                                                                                      

                                                          

                                                                                                          

    javax.jdo.option.ConnectionDriverName                                                    

    com.mysql.cj.jdbc.Driver                                                    

                                                                                                          

                                                      

                                                                                                          

    javax.jdo.option.ConnectionUserName                                                    

    root                                                                                                        

                                                                                                          

                                                                                                          

                                                                                                          

    javax.jdo.option.ConnectionPassword                                                                                                        

    123456                                                                                                        

                                                                                                          

                                                                                                          

                                                                                                          

    hive.metastore.schema.verification                                                    

    false                                                                                                        

                                                                                                          

                                                                                                          

                                                                                                          

    hive.metastore.event.db.notification.api.auth                                                    

    false                                                                                                        

                                                                                                          

                                                                                                          

                                                                                                          

    hive.metastore.uris                                                                                                        

    thrift://localhost:9083                                                                                                        

                                                                                                          

                                                                                                          

                                                                                                          

    hive.server2.thrift.bind.host                                                                                                        

    localhost                                                                                                        

                                                                                                          

                                                                                                          

                                                                                                          

    hive.server2.thrift.port                                                    

    10000                                                                                                        

                                                                                                          

                                                                                                          

    hive.cli.print.header                                                                                                        

    true                                                                                                        

                                                                                                                                                                                                                  

                                                                                                          

    hive.cli.print.current.db                                                                                                        

    true                                                                                                 

                                                                                               

                                                                                  

6）、将mysql的jar包放在/opt/module/apache-hive-3.1.2-bin/lib路径下

 

7）、查看lib目录下mysql文件是否齐全

[root@hadoop100 conf]# ls /opt/module/apache-hive-3.1.2-bin/lib/ | grep mysql

 

8）、将/opt/module/hadoop-3.1.3/share/hadoop/common/lib/guava-27.0-jre.jar剪切到/opt/module/apache-hive-3.1.2-bin/lib路径下

[root@hadoop100 apache-hive-3.1.2-bin]# cd /opt/module/apache-hive-3.1.2-bin/

[root@hadoop100 apache-hive-3.1.2-bin]# mv /opt/module/hadoop-3.1.3/share/hadoop/common/lib/guava-27.0-jre.jar ./lib/

出现如下提示直接按回车即可

 

9）、初始化mysql

[root@hadoop100 apache-hive-3.1.2-bin]# bin/schematool -dbType mysql -initSchema -verbose

10）、后台启动metastore及hiveserver2服务（若想脚本启动，可看按第三节进行配置）

[root@hadoop100 apache-hive-3.1.2-bin]# nohup bin/hive --service metastore &

[root@hadoop100 apache-hive-3.1.2-bin]# nohup bin/hive --service hiveserver2 &

11）、启动hive

[root@hadoop100 apache-hive-3.1.2-bin]# hive

看到如下界面,hive正常启动

(which: no hbase只是hbase没有装而已。对hive不影响，不是错误)

which: no hbase in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/module/jdk1.8.0_212/bin:/opt/module/hadoop-3.1.3/bin:/opt/module/hadoop-3.1.3/sbin:/opt/module/hive/bin:/home/soft863/.local/bin:/home/soft863/bin)

Hive Session ID = 36f90830-2d91-469d-8823-9ee62b6d0c26

Logging initialized using configuration in jar:file:/opt/module/hive/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true

Hive Session ID = 14f96e4e-7009-4926-bb62-035be9178b02

hive>

2、Hive环境变量配置如下：

1）修改profile文件

[root@hadoop100 apache-hive-3.1.2-bin]# vi /etc/profile

添加内容如下：

HIVE_HOME=/opt/module/apache-hive-3.1.2-bin

修改内容如下：

PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin

export PATH JAVA_HOME HADOOP_HOME HIVE_HOME

 

1. 刷新/etc/profile

3）[root@hadoop100 apache-hive-3.1.2-bin]# source /etc/profile

3、编写脚本后台启动metastore和hiveserver2

1. 为了方便使用，可以直接编写脚本来管理服务的启动和关闭

[soft863@hadoop102 hive]$ vim $HIVE_HOME/bin/hiveservices.sh

内容如下：此脚本的编写不要求掌握。直接拿来使用即可。

#!/bin/bash

HIVE_LOG_DIR=$HIVE_HOME/logs

if [ ! -d $HIVE_LOG_DIR ]

then

mkdir -p $HIVE_LOG_DIR

fi

#检查进程是否运行正常，参数1为进程名，参数2为进程端口

function check_process()

{

    pid=$(ps -ef 2>/dev/null | grep -v grep | grep -i $1 | awk '{print $2}')

    ppid=$(netstat -nltp 2>/dev/null | grep $2 | awk '{print $7}' | cut -d '/' -f 1)

    echo $pid

    [[ "$pid" =~ "$ppid" ]] && [ "$ppid" ] && return 0 || return 1

}

function hive_start()

{

    metapid=$(check_process Hivemetastore 9083)

    cmd="nohup hive --service metastore >$HIVE_LOG_DIR/metastore.log 2>&1 &"

    cmd=$cmd" sleep 4; hdfs dfsadmin -safemode wait >/dev/null 2>&1"

    [ -z "$metapid" ] && eval $cmd || echo "metastroe服务已启动"

    server2pid=$(check_process HiveServer2 10000)

    cmd="nohup hive --service hiveserver2 >$HIVE_LOG_DIR/hiveServer2.log 2>&1 &"

    [ -z "$server2pid" ] && eval $cmd || echo "HiveServer2服务已启动"

}

function hive_stop()

{

    metapid=$(check_process Hivemetastore 9083)

    [ "$metapid" ] && kill $metapid || echo "metastore服务未启动"

    server2pid=$(check_process HiveServer2 10000)

    [ "$server2pid" ] && kill $server2pid || echo "HiveServer2服务未启动"

}

case $1 in

"start")

    hive_start

    ;;

"stop")

    hive_stop

    ;;

"restart")

    hive_stop

    sleep 2

    hive_start

    ;;

"status")

    check_process Hivemetastore 9083 >/dev/null && echo "metastore服务运行正常" || echo "metastore服务运行异常"

    check_process HiveServer2 10000 >/dev/null && echo "HiveServer2服务运行正常" || echo "HiveServer2服务运行异常"

    ;;

*)

    echo Invalid Args!

    echo 'Usage: '$(basename $0)' start|stop|restart|status'

    ;;

esac

2）添加执行权限

[soft863@hadoop102 hive]$ chmod +x $HIVE_HOME/bin/hiveservices.sh

3）启动Hive后台服务

[soft863@hadoop102 hive]$ hiveservices.sh start

 

• Hive搭建方法二

此方法将数据库安装在虚拟机上

1、MySql安装

1）检查当前系统是否安装过Mysql

[soft863@hadoop102 ~]$ rpm -qa|grep mariadb

mariadb-libs-5.5.56-2.el7.x86_64 //如果存在通过如下命令卸载

[soft863 @hadoop102 ~]$  rpm -e --nodeps  mariadb-libs   //用此命令卸载mariadb

2）将MySQL安装包拷贝到/opt/software目录下

[soft863 @hadoop102 software]# ll

总用量 528384

-rw-r--r--. 1 root root 609556480 3月  21 15:41 mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar

3）解压MySQL安装包

[soft863 @hadoop102 software]# tar -xf mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar

1. 在安装目录下执行rpm安装

[soft863 @hadoop102 software]$ rpm -ivh mysql-community-common-5.7.28-1.el7.x86_64.rpm

[soft863 @hadoop102 software]$ rpm -ivh mysql-community-libs-5.7.28-1.el7.x86_64.rpm

[soft863 @hadoop102 software]$ rpm -ivh mysql-community-libs-compat-5.7.28-1.el7.x86_64.rpm

[soft863 @hadoop102 software]$ rpm -ivh mysql-community-client-5.7.28-1.el7.x86_64.rpm

[soft863 @hadoop102 software]$ rpm -ivh mysql-community-server-5.7.28-1.el7.x86_64.rpm

注意:按照顺序依次执行

如果Linux是最小化安装下错误

[soft863@hadoop102 software]$ sudo rpm -ivh mysql-community-server-5.7.28-1.el7.x86_64.rpm

警告：mysql-community-server-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY

错误：依赖检测失败：

        libaio.so.1()(64bit) 被 mysql-community-server-5.7.28-1.el7.x86_64 需要

        libaio.so.1(LIBAIO_0.1)(64bit) 被 mysql-community-server-5.7.28-1.el7.x86_64 需要

        libaio.so.1(LIBAIO_0.4)(64bit) 被 mysql-community-server-5.7.28-1.el7.x86_64 需要

通过yum安装缺少的依赖,然后重新安装my的，在安装mysql-community-server-5.7.28-1.el7.x86_64.rpm时可能会出 现如sql-community-server-5.7.28-1.el7.x86_64 即可

[soft863@hadoop102 software] yum install -y libaio

5）删除/etc/my.cnf文件中datadir指向的目录下的所有内容,如果有内容的情况下:

   查看datadir的值：

[mysqld]

datadir=/var/lib/mysql

   删除/var/lib/mysql目录下的所有内容:

cd /var/lib/mysql

[soft863 @hadoop102 mysql]# rm -rf *    //注意执行命令的位置

6）初始化数据库

[soft863 @hadoop102 opt]$ mysqld --initialize --user=mysql

7）查看临时生成的root用户的密码 

[soft863 @hadoop102 opt]$ cat /var/log/mysqld.log

 

8）启动MySQL服务

[soft863 @hadoop102 opt]$ systemctl start mysqld

9）登录MySQL数据库

[soft863 @hadoop102 opt]$ mysql -uroot -p

Enter password:   输入临时生成的密码

   登录成功.

10）必须先修改root用户的密码,否则执行其他的 *** 作会报错

mysql> set password = password("新密码");

11）修改mysql库下的user表中的root用户允许任意ip连接

mysql> update mysql.user set host='%' where user='root';

mysql> flush privileges;

2、Hive安装部署

1）把apache-hive-3.1.2-bin.tar.gz上传到linux的/opt/software目录下

2）解压apache-hive-3.1.2-bin.tar.gz到/opt/module/目录下面

[soft863@hadoop102 software]$ tar -zxvf /opt/software/apache-hive-3.1.2-bin.tar.gz -C /opt/module/

3）修改apache-hive-3.1.2-bin.tar.gz的名称为hive

[soft863@hadoop102 software]$ mv /opt/module/apache-hive-3.1.2-bin/ /opt/module/hive

4）修改/etc/profile，添加环境变量

[soft863@hadoop102 software]$ vim /etc/profile

5）添加内容，添加完后别忘记执行 source /etc/profile

#HIVE_HOME

export HIVE_HOME=/opt/module/hive

export PATH=$PATH:$HIVE_HOME/bin

6）解决日志Jar包冲突

[soft863@hadoop102 software]$ mv $HIVE_HOME/lib/log4j-slf4j-impl-2.10.0.jar $HIVE_HOME/lib/log4j-slf4j-impl-2.10.0.bak

3、Hive元数据配置到MySql 3.1、 修改core-site.xml

修改hadoop 配置文件 etc/hadoop/core-site.xml,加入如下配置项:

为了防止root在hive中 *** 作的权限问题；

    hadoop.proxyuser.root.hosts

    *

    hadoop.proxyuser.root.groups

    *

保存退出，并重启hdfs服务

3.2、 拷贝驱动

将MySQL的JDBC驱动拷贝到Hive的lib目录下

先把jar包放到 /opt/software下，然后再执行下面命令。

[soft863@hadoop102 software]$ cp /opt/software/mysql-connector-java-5.1.48.jar $HIVE_HOME/lib

3.3、 配置metastore到MySql

在$HIVE_HOME/conf目录下新建hive-site.xml文件

[soft863@hadoop102 software]$ vim $HIVE_HOME/conf/hive-site.xml

添加如下内容

    

    

        javax.jdo.option.ConnectionURL

        jdbc:mysql://hadoop100:3306/metastore?useSSL=false

    

    

        javax.jdo.option.ConnectionDriverName

        com.mysql.jdbc.Driver

    

        javax.jdo.option.ConnectionUserName

        root

    

    

    

        javax.jdo.option.ConnectionPassword

        123456

    

    

    

        hive.metastore.warehouse.dir

        /user/hive/warehouse

    

    

   

    

        hive.metastore.schema.verification

        false

    

    

    

        hive.metastore.uris

        thrift://hadoop100:9083

    

    

    

    hive.server2.thrift.port

    10000

    

   

    

        hive.server2.thrift.bind.host

        hadoop100

    

    

    

        hive.metastore.event.db.notification.api.auth

        false

    

3.4、 启动Hive

当在hive中执行增删改查 *** 作时，会在Yarn上运行任务，所以需要启动Yarn服务。在启动yarn之前，最好保持容量调度器中的队列只有一个，并增加AppMaster的运行内存，修改如下：

vi /opt/module/hadoop-3.1.3/etc/hadoop/capacity-scheduler.xml

    yarn.scheduler.capacity.maximum-am-resource-percent

    0.5

    

      Maximum percent of resources in the cluster which can be used to run

      application masters i.e. controls number of concurrent running

      applications.

    

  

此配置表示集群上AM最多可使用的资源比例，目的为限制过多的app数量，默认比例为10%，可适当调大比例。

为了防止执行hive出错，可调大虚拟机内存，比如6G。

3.5、 初始化元数据库

1）登陆MySQL

[soft863@hadoop102 software]$ mysql -uroot -p123456

2）新建Hive元数据库

mysql> create database metastore;

mysql> quit;

3）初始化Hive元数据库(会执行一会)

[soft863@hadoop102 software]$ schematool -initSchema -dbType mysql -verbose

 

3.6、 启动metastore和hiveserver2

1）Hive 2.x以上版本，要先启动这两个服务，否则会报错：

FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHivemetaStoreClient

1. 启动metastore

[soft863@hadoop202 hive]$ hive --service metastore

2020-04-24 16:58:08: Starting Hive metastore Server  

注意: 启动后窗口不能再 *** 作，需打开一个新的shell窗口做别的 *** 作

1. 启动 hiveserver2

[soft863@hadoop202 hive]$ hive --service hiveserver2

which: no hbase in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/module/jdk1.8.0_212/bin:/opt/module/hadoop-3.1.3/bin:/opt/module/hadoop-3.1.3/sbin:/opt/module/hive/bin:/home/soft863/.local/bin:/home/soft863/bin)

2020-04-24 17:00:19: Starting HiveServer2  

注意: 启动后窗口不能再 *** 作，需打开一个新的shell窗口做别的 *** 作

  

2）编写hive服务启动脚本

1. 前台启动的方式导致需要打开多个shell窗口，可以使用如下方式后台方式启动

nohup: 放在命令开头，表示不挂起,也就是关闭终端进程也继续保持运行状态

2>&1 : 表示将错误重定向到标准输出上

&: 放在命令结尾,表示后台运行

一般会组合使用: nohup  [xxx命令 *** 作]> file  2>&1 &  ， 表示将xxx命令运行的

结果输出到file中，并保持命令启动的进程在后台运行。

如上命令不要求掌握。

[soft863@hadoop202 hive]$ nohup hive --service metastore 2>&1 &

[soft863@hadoop202 hive]$ nohup hive --service hiveserver2 2>&1 &

1. 为了方便使用，可以直接编写脚本来管理服务的启动和关闭

[soft863@hadoop102 hive]$ vim $HIVE_HOME/bin/hiveservices.sh

内容如下：此脚本的编写不要求掌握。直接拿来使用即可。

#!/bin/bash

HIVE_LOG_DIR=$HIVE_HOME/logs

if [ ! -d $HIVE_LOG_DIR ]

then

mkdir -p $HIVE_LOG_DIR

fi

#检查进程是否运行正常，参数1为进程名，参数2为进程端口

function check_process()

{

    pid=$(ps -ef 2>/dev/null | grep -v grep | grep -i $1 | awk '{print $2}')

    ppid=$(netstat -nltp 2>/dev/null | grep $2 | awk '{print $7}' | cut -d '/' -f 1)

    echo $pid

    [[ "$pid" =~ "$ppid" ]] && [ "$ppid" ] && return 0 || return 1

}

function hive_start()

{

    metapid=$(check_process Hivemetastore 9083)

    cmd="nohup hive --service metastore >$HIVE_LOG_DIR/metastore.log 2>&1 &"

    cmd=$cmd" sleep 4; hdfs dfsadmin -safemode wait >/dev/null 2>&1"

    [ -z "$metapid" ] && eval $cmd || echo "metastroe服务已启动"

    server2pid=$(check_process HiveServer2 10000)

    cmd="nohup hive --service hiveserver2 >$HIVE_LOG_DIR/hiveServer2.log 2>&1 &"

    [ -z "$server2pid" ] && eval $cmd || echo "HiveServer2服务已启动"

}

function hive_stop()

{

    metapid=$(check_process Hivemetastore 9083)

    [ "$metapid" ] && kill $metapid || echo "metastore服务未启动"

    server2pid=$(check_process HiveServer2 10000)

    [ "$server2pid" ] && kill $server2pid || echo "HiveServer2服务未启动"

}

case $1 in

"start")

    hive_start

    ;;

"stop")

    hive_stop

    ;;

"restart")

    hive_stop

    sleep 2

    hive_start

    ;;

"status")

    check_process Hivemetastore 9083 >/dev/null && echo "metastore服务运行正常" || echo "metastore服务运行异常"

    check_process HiveServer2 10000 >/dev/null && echo "HiveServer2服务运行正常" || echo "HiveServer2服务运行异常"

    ;;

*)

    echo Invalid Args!

    echo 'Usage: '$(basename $0)' start|stop|restart|status'

    ;;

esac

3）添加执行权限

[soft863@hadoop102 hive]$ chmod +x $HIVE_HOME/bin/hiveservices.sh

4）启动Hive后台服务

[soft863@hadoop102 hive]$ hiveservices.sh start