hadoop，java教程视

hadoop，java教程视

4. 刷新设置，运行hadoop version 查看是否配置成功

source /etc/profile

5. 修改hadoop的核心配置文件

5.1 修改 hadoop-env.sh   mapred-env.sh   yarn-env.sh

vim hadoop-env.sh

export JAVA_HOME=/usr/local/jdk

5.2 修改 core-site.xml

fs.defaultFS

hdfs://master:9000

hadoop.tmp.dir

/usr/local/src/hadoop/data_tmp

5.3 修改 hdfs-site.xml

dfs.namenode.secondary.http-address

master:50090

5.4 修改 mapred-site.xml

mapreduce.framework.name

yarn

5.5 修改 yarn-site.xml

yarn.nodemanager.aux-services

mapreduce_shuffle

5.6 修改 slave

slave1

slave2

6、远程分发

7、格式化

hdfs namenode -format

8、启动hdfs和yarn

start-dfs.sh

start-yarn.sh

六、安装Hive

#1、安装Hive的元数据库 MySQL

1、下载mysql源安装包

sudo wget http://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm

2、安装mysql源

sudo yum localinstall mysql57-community-release-el7-8.noarch.rpm

3、安装mysql

sudo yum install mysql-community-server

4、启动MySQL服务

sudo systemctl start mysqld

5、查看mysql初始密码

[ec2-user@master ~]$ sudo grep “password” /var/log/mysqld.log

2021-03-19T07:56:41.030922Z 1 [Note] A temporary password is generated for root@localhost: v=OKXu0laSo;

v=OKXu0laSo;是密码

6、修改mysql登陆密码

把初始密码复制下来，在进入mysql需要输入密码时粘贴下来，回车，就可以进入MySQL命令行。

进入命令行

sudo mysql -uroot -p

修改密码，设置MySQL登陆密码为1234

新密码设置的时候如果设置的过于简单会报错

修改密码规则：

在mysql命令行中输入

mysql> set global validate_password_policy=0;

Query OK, 0 rows affected (0.00 sec)

mysql> set global validate_password_length=1;

Query OK, 0 rows affected (0.00 sec)

重新设置密码

mysql> set password for ‘root’@‘localhost’=password(‘1234’);

Query OK, 0 rows affected, 1 warning (0.00 sec)

7、设置远程登陆

先退出MySQL

以新密码登陆MySQL

[ec2-user@master ~]$ mysql -uroot -p1234

mysql: [Warning] Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor. Commands end with ; or g.

Your MySQL connection id is 10

Server version: 5.7.33 MySQL Community Server (GPL)

Copyright © 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type ‘help;’ or ‘h’ for help. Type ‘c’ to clear the current input statement.

mysql>

创建用户

mysql> create user ‘root’@‘172.%.%.%’ identified by ‘1234’;

Query OK, 0 rows affected (0.00 sec)

允许远程连接：

mysql> grant all privileges on . to ‘root’@‘172.%.%.%’ with grant option;

Query OK, 0 rows affected (0.00 sec)

刷新权限：

mysql> flush privileges;

Query OK, 0 rows affected (0.00 sec)

到此mysql安装完成

#2、安装Hive（依赖Hadoop）

1、把Hive解压到指定位置

tar -zxvf hadoop/apache-hive-1.1.0-bin.tar.gz -C /usr/local/src/

2、重命名

mv apache-hive-1.1.0-bin/ hive

3、配置全局环境变量

sudo vim /etc/profile

export HIVE_HOME=/usr/local/src/hive

export PATH= P A T H : PATH: PATH:HIVE_HOME/bin

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/src/hive/lib/*

刷新环境变量

source /etc/profile

4、在src/hive/conf下创建hive-site.xml文件

touch hive-site.xml

在hive-site.xml文件中添加如下内容:

hive.metastore.warehouse.dir

【一线大厂Java面试题解析+后端开发学习笔记+最新架构讲解视频+实战项目源码讲义】

浏览器打开：qq.cn.hn/FTf 免费领取

name>

/user/hive/warehouse

javax.jdo.option.ConnectionURL

jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=false

javax.jdo.option.ConnectionDriverName

com.mysql.jdbc.Driver

javax.jdo.option.ConnectionUserName

root

javax.jdo.option.ConnectionPassword

1234

注意：MySQL密码要改成自己设置的密码

5、添加驱动包

把MySQL驱动放到hive的lib目录下

cp /home/ec2-user/hadoop/mysql-connector-java-5.1.44-bin.jar $HIVE_HOME/lib

6、修改hive-env.sh配置文件

[ec2-user@master conf]$ vi hive-env.sh

#在里面添加如下配置

export HADOOP_HOME=/usr/local/src/hadoop

export HIVE_CONF_DIR=/usr/local/src/hive/conf

7、启动Hive

需要确保hdfs、mapreduce、mysql启动

若没启动

start-all.sh

初始化MySQL中的hive的数据库

schematool -dbType mysql -initSchema

启动hive

hive

#安装sqoop