Hadoop集群搭建之Hive on Spark

Hadoop集群搭建之Hive on Spark Hadoop集群搭建之Hive on Spark

用到的各个组件的版本

组件版本hadoop3.3.1hive3.1.2spark2.4.6scala2.12.10 1. 准备工作

# 下载spark的源码
wget https://downloads.apache.org/spark/spark-2.4.6/spark-2.4.6.tgz

2. 编译spark源码

# 解压源码到目录下
tar -zvxf spark-2.4.6.tgz && cd spark-2.4.6

修改spark源码中的make-distribution.sh文件：

# 编译源码
./dev/make-distribution.sh --name "hadoop3-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-3.3,parquet-provided"

3. Spark配置

将编译好的包解压到/opt/modules目录，并建立软链接为spark。

tar -zvxf spark-2.4.6-bin-hadoop3-without-hive.tgz
mv spark-2.4.6-bin-hadoop3-without-hive spark-2.4.6
ln -s spark-2.4.6 spark

• spark-env.sh

  export JAVA_HOME=/opt/modules/jdk
  export SCALA_HOME=/opt/modules/scala
  export HADOOP_HOME=/opt/modules/hadoop
  export HADOOP_CONF_DIR=/opt/modules/hadoop/etc/hadoop
  export HADOOP_YARN_CONF_DIR=/opt/modules/hadoop/etc/hadoop
  export SPARK_HOME=/opt/modules/spark
  export SPARK_WORKER_MEMORY=512m
  export SPARK_EXECUTOR_MEMORY=512m
  export SPARK_DRIVER_MEMORY=512m
  export SPARK_DIST_CLASSPATH=$(/opt/modules/hadoop/bin/hadoop classpath)
  

4. Hive配置

    
        
        hive.cli.print.header
        true
    
    
        
        hive.cli.print.current.db
        true
    
    
        
        hive.metastore.warehouse.dir
        /user/hive/warehouse
    
    
    	javax.jdo.option.ConnectionURL
      jdbc:mysql://hadoop01:3306/hive?createDatabaseIfNotExist=true
    
    
    	javax.jdo.option.ConnectionDriverName
        com.mysql.jdbc.Driver
    
    
    	javax.jdo.option.ConnectionUserName
        hive
    
    
    	javax.jdo.option.ConnectionPassword
        xiaoer
    
    
        hive.server2.transport.mode
        binary
    
    
        hive.server2.thrift.port
        10000
    
    
        hive.server2.webui.host
        hadoop01
    
    
        hive.server2.webui.port
        10002
    
    
        
        hive.execution.engine
        spark
    
    
        hive.enable.spark.execution.engine
        true
    
    
        spark.home
        /opt/modules/spark
    
    
        spark.master
        yarn
    
    
        spark.eventLog.enabled
        true
    
    
        
        spark.eventLog.dir
        hdfs://supercluster/spark-hive-jobhistory
    
    
        spark.executor.memory
        512m
    
    
        spark.driver.memory
        512m
    
    
        spark.serializer
        org.apache.spark.serializer.KryoSerializer
    
    
        
        spark.yarn.jars
        hdfs://supercluster/spark-jars/*
    
    
        hive.spark.client.server.connect.timeout
        300000
    

拷贝mysql的jar包到hive/lib目录下，并初始化hive元数据库：

cp mysql-connector-jdbc.jar /opt/modules/hive/lib
/opt/modules/hive/bin/schematool --initSchema -dbType

5. 整合jar包

将spark和hive的jar包整合：

cp /opt/modules/spark/jars/*.jar /opt/modules/hive/lib/
rm /opt/modules/spark/jars/*.jar
cp /opt/modules/hive/lib/*.jar /opt/modules/spark/jars/

将配置文件拷贝到saprk/conf目录下：

cp /opt/modules/hadoop/etc/hadoop/core-site.xml /opt/modules/hadoop/etc/hadoop/hdfs-site.xml /opt/modules/hadoop/etc/hadoop/yarn-site.xml /opt/modules/hive/conf/hive-site.xml /opt/modules/spark/conf

将整合后的jar包上传到hdfs目录spark-jars：

hdfs dfs -mkdir /spark-jars
hdfs dfs -put /opt/modules/spark/jars/*.jar /spark-jars

6. 启动服务

# 启动hadoop集群
start-dfs.sh
start-yarn.sh

# 启动spark集群
cd /opt/modules/spark/sbin && ./start-all.sh