HDP 01.Ambari
安装和
配置
1、
节点说明
1.2
ambari.tianlingqun.com
主要用于 Ambari管理( ambari-server )
用于存放元数据的MySQL数据库也安装到该节点（元数据主要包括：Ambari, Hive, Ranger, Hue, Azkaban, Superset等)
1.3 client-v[01,02].
tianlingqun.com
两台客户端机器，均安装 Azkaban-exec; Azkaban-web 独立安装(Azkaban-web不支持HA)到
client-v02.tianlingqun.com
1.4 两台 Master 机器 master-v[01,02].
tianlingqun.com
基中主要包括：hdfs, yarn, hbase 的高可用相关进程; Hive metadata, Hive server进程 及 Zookeeper进程等
1.5 5台 worker 节点 datanode-v[01~05].
tianlingqun.com
主要是 Datanode, NodeManager, HRegion Server等 进程
1.6 ipa-v[01,02].
tianlingqun.com
主要用于 FreeIPA“互为主从的两个节点”，FreeIPA是一个集 kerberos, ldap, dns于一体的管理工具，
详细请参考：
FreeIPA
2、对机器进行配置：
1）修改/etc/hosts，并分发到每个节点
2）免密配置：ssh-keygen -t rsa
分发到其它机器：ssh-copy-id -i
，ipa除外
3）命令脚本：ssh_to_all_node.sh
4）分发脚本：sync_to_all_node.sh
4）关闭防火墙：
systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld
3、ambari安装：
1）yum -y install wget
# ambari
wget
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari-2.7.4.0-centos7.tar.gz
wget
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari-2.7.4.0-centos7.tar.gz.md5
# hdp
wget
http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-centos7-rpm.tar.gz
wget
http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-centos7-rpm.tar.gz.md5
# hdp gpl
wget
http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/HDP-GPL-3.1.4.0-centos7-gpl.tar.gz
wget
http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/HDP-GPL-3.1.4.0-centos7-gpl.tar.gz.md5
# hdp utils
wget
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz.md5
以上文件可以在linux直接下载或者下载以后在进行上传到指定目录即可，为了避免下载过程中出现问题，要进行MD5校验
2）关闭SELINUX=disabled：
分发配置文件或者命令修改
yum -y install perl
ls /etc/selinux/config |xargs perl -pi -e 's|SELINUX=enforcing|SELINUX=disabled|g'
cat /etc/selinux/config|grep "SELINUX"|grep -v "grep"
3）安装MySQL：
yum -y localinstall
https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
yum -y install mysql-community-server
systemctl enable mysqld
systemctl start mysqld
systemctl status mysqld
grep 'temporary password' /var/log/mysqld.log
mysql_secure_installation
也可以登录MySQL以后在设置密码和降低密码复杂度：mysql -u root -p 密码
set password = password('123456');
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456';
SHOW VARIABLES LIKE 'validate_password%';
set global validate_password_policy=0;
set global validate_password_mixed_case_count=1;
set global validate_password_number_count=1;
set global validate_password_special_char_count=1;
set global validate_password_length=4;
SHOW VARIABLES LIKE 'validate_password%';
FLUSH PRIVILEGES;
4）创建ambari数据库，用于存放ambari元数据：
表示服务端版本号大于4.1.00时会被执行，使用utf8编码
drop database if exists ambari;
CREATE DATAbase `ambari` ;
drop user 'ambari'@'
ambari.tianlingqun.com';
GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'
ambari.tianlingqun.com' IDENTIFIED BY'Ambari_123' WITH
GRANT OPTION;
flush privileges;
show create database ambari;
5）创建hive数据库，存放元数据：
CREATE DATAbase `hive` ;
drop user 'hive'@'
master-v01.tianlingqun.com';
drop user 'hive'@'
master-v02.tianlingqun.com';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'
master-v01.tianlingqun.com' IDENTIFIED BY 'Wfhive_4321' WITH
GRANT OPTION;
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'
master-v02.tianlingqun.com' IDENTIFIED BY 'Wfhive_4321' WITH
GRANT OPTION;
flush privileges;
show create database hive;
6）创建ranger数据库，存放元数据：
drop database if exists `ranger`;
use mysql;
create database `ranger`;
delete from mysql.user where User='rangeradmin';
grant all privileges on ranger.* to 'rangeradmin'@'
master-v01.tianlingqun.com' identified by 'Wfrangeradmin_8899'
WITH GRANT OPTION;
grant all privileges on ranger.* to 'rangeradmin'@'
master-v02.tianlingqun.com' identified by 'Wfrangeradmin_8899'
WITH GRANT OPTION;
grant all privileges on ranger.* to 'rangeradmin'@'
client-v01.tianlingqun.com' identified by 'Wfrangeradmin_8899'
WITH GRANT OPTION;
grant all privileges on ranger.* to 'rangeradmin'@'
client-v02.tianlingqun.com' identified by 'Wfrangeradmin_8899'
WITH GRANT OPTION;
grant all privileges on ranger.* to 'rangeradmin'@'
ambari.tianlingqun.com' identified by 'Wfrangeradmin_8899'
WITH GRANT OPTION;
flush privileges;
7）创建rangerkms数据库，存放元数据：
drop database if exists `rangerkms`;
use mysql;
create database `rangerkms`;
delete from user where User='rangerkms';
grant all privileges on rangerkms.* to 'rangerkms'@'
master-v01.tianlingqun.com' identified by 'Rangerkms_123';
grant all privileges on rangerkms.* to 'rangerkms'@'
master-v02.tianlingqun.com' identified by 'Rangerkms_123';
grant all privileges on rangerkms.* to 'rangerkms'@'
ambari.tianlingqun.com' identified by 'Rangerkms_123';
flush privileges;
8）创建superset数据库，存放元数据：
drop database if exists `superset`;
use mysql;
CREATE DATAbase `superset` ;
grant all privileges on superset.* to 'superset'@'
client-v01.tianlingqun.com' identified by 'wfsuperset456';
grant all privileges on superset.* to 'superset'@'
client-v02.tianlingqun.com' identified by 'wfsuperset456';
flush privileges;
4、配置同步时间：
yum -y install chrony
systemctl enable chronyd
systemctl start chronyd
5、配置Maximum Open Files Requirements
服务器 vi /etc/security/limits.conf添加文件
* - nofile 409600
* - nproc 63535
root - nofile 409600
root - nproc 63535
执行 sysctl -p 命令，使上面修改生效 (注意：需要新开登陆会话，才能看到效果)
sh ./sync_to_all_node.sh /etc/security/limits.conf /etc/security/
sh ./ssh_to_all_node.sh "sysctl -p"
sh ./ssh_to_all_node.sh "ulimit -n"
6、安装相关文件并分发其它机器：
yum -y install yum-utils createrepo yum-plugin-priorities
cd ~/scripts/;
sh ./ssh_to_all_node.sh "yum -y install yum-utils createrepo yum-plugin-priorities;"
7、安装httpd
yum -y install yum-utils createrepo
yum -y install httpd
mkdir -p /var/www/html/
systemctl enable httpd
systemctl start httpd
systemctl status httpd
8、配置yum相关的 repo 文件
vi /etc/yum.repos.d/ambari.repo
#VERSION_NUMBER=2.7.4.0-118
[ambari-2.7.4.0]
json.url =
http://ambari.tianlingqun.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.4.0
baseurl=
http://ambari.tianlingqun.com/ambari/centos7/2.7.4.0-118
gpgcheck=1
gpgkey=
http://ambari.tianlingqun.com/ambari/centos7/2.7.4.0-118/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
vi /etc/yum.repos.d/hdp.repo
#VERSION_NUMBER=3.1.4.0-315
[HDP-3.1.4.0]
name=HDP Version - HDP-3.1.4.0
baseurl=
http://ambari.tianlingqun.com/HDP/centos7/3.1.4.0-315
gpgcheck=1
gpgkey=
http://ambari.tianlingqun.com/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=
http://ambari.tianlingqun.com/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=
http://ambari.tianlingqun.com/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
vi /etc/yum.repos.d/hdp.gpl.repo
#VERSION_NUMBER=3.1.4.0-315
[HDP-GPL-3.1.4.0]
name=HDP-GPL Version - HDP-GPL-3.1.4.0
baseurl=
http://ambari.tianlingqun.com/HDP-GPL/centos7/3.1.4.0-315
gpgcheck=1
gpgkey=
http://ambari.tianlingqun.com/HDP-GPL/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
9、校验
yum clean all
yum makecache
yum repolist
10、software解压缩jar包
tar -zxvf ambari-2.7.4.0-centos7.tar.gz
tar -zxvf HDP-3.1.4.0-centos7-rpm.tar.gz
tar -zxvf HDP-GPL-3.1.4.0-centos7-gpl.tar.gz
tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz
tar -zxvf jdk-8u231-linux-x64.tar.gz
11、创建软连接
cd /var/www/html/
ln -s /software/ambari ./ambari
ln -s /software/HDP ./HDP
ln -s /software/HDP-GPL ./HDP-GPL
ln -s /software/HDP-UTILS ./HDP-UTILS
ls -l ./
12、启动时激活互联网
ETWORKING=yes
HOSTNAME=
cd ~/scripts/;
sh ./ssh_to_all_node.sh 'echo "NETWORKING=yes" >> /etc/sysconfig/network;hostname=$(hostname -f);
echo "HOSTNAME=${hostname}" >> /etc/sysconfig/network;'
sh ./ssh_to_all_node.sh "cat /etc/sysconfig/network;"
13、Install MySQL JDBC Driver (只需在 ambari 服务器上执行)
yum -y install mysql-connector-java-5*
12、安装 JDK再同步到其它节点
cd /software/
tar -xvf jdk-8u201-linux-x64.tar.gz
mkdir -p /usr/java/
mv jdk1.8.0_231/ /usr/java/
cd /usr/java/
ln -s ./jdk1.8.0_231 ./current
chown -R root:root ./jdk1.8.0_231
ls -l ./
cd ~/scripts/
sh ./sync_to_all_node.sh /usr/java/jdk1.8.0_231 /usr/java/
sh ./ssh_to_all_node.sh "mkdir -p /usr/java/;"
sh ./sync_to_all_node.sh /usr/java/jdk1.8.0_231 /usr/java/;
sh ./ssh_to_all_node.sh "cd /usr/java/;ln -s ./jdk1.8.0_231 ./current;ls -l ./;"
echo "" >> /etc/profile;
echo "#set Java environment" >> /etc/profile;
echo "export JAVA_HOME=/usr/java/current" >> /etc/profile;
echo "export JRE_HOME=$JAVA_HOME/jre" >> /etc/profile;
echo "export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH" >> /etc/profile;
echo "export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH" >> /etc/profile;
java -version
sh ./sync_to_all_node.sh /etc/profile /etc/;
13、安装 ambari-server RPM包
yum -y install ambari-server
14、配置 ambari-server
ambari-server setup
15、生成 ambari 相关元数据
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
16、添加 jdbc 驱动到 ambari-server，执行如下命令
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
#ambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/mysql/mysql-connector-java.jar
17、启动 ambari-server，执行如下命令
ambari-server start
18、查看 ambari-server 当前状态的命令如下：
ambari-server status
19、启动成功后，可以执行类似如下命令，去查看日志，有没有报错信息
tail -n 400 /var/log/ambari-server/ambari-server.log
20、启动成功后，就可以浏览 Ambari 的 web 页面了，链接类似如下(端口默认 8080，也可以通过编辑
/etc/ambari-server/conf/ambari.properties 文件去个性化各参数)
http://xxx:8080
默认的用户名密码均为 admin
21、ambari安装配置：
1）开始配置：
22、HA高可用：
1、HDFS高可用
2、YARN高可用
3、hive高可用
微信扫一扫
支付宝扫一扫
评论列表（0条）