搭建ambri集群基础配置

搭建ambri集群基础配置 搭建ambri集群基础配置 adduser

如有需要添加用户

#!/bin/bash
userName=$1
userPassword=$2
userType=$3
ifneed=$4
groupadd  $userName
useradd -g $userName -d /home/$userName $userName
if [[ $ifneed = "yes" ]];then
echo "$userPassword" | passwd --stdin $userName
fi
if [[ $userType = "sudo" ]];
then
cp /etc/sudoers /etc/sudoers_bak
echo "$userName ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
fi

调整系统语言&时区（所有节点）

#修改系统语言为英文
echo 'export LANG=en_US.UTF-8' >> ~/.bashrc
#修改时区
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

验证

echo $LANG

firewalld

systemctl stop firewalld
systemctl disable firewalld
service iptables stop
chkconfig iptables off
systemctl status firewalld
service iptables status

jdk

#第一步检测mariadb安装目录是否存在，不存在就创建
basedir="/usr/local/java/";
jdkfile="jdk-8u301-linux-aarch64.tar.gz"
if [ ! -d "$basedir" ];
   then
   echo "$basedir目录不存在，开始创建该目录"
   mkdir -p $basedir
fi

#jdk安装文件是否存在
if [ ! -f "$jdkfile" ];
then
echo "$jdkfile文件不存在，请检测"
exit
fi

#第二步，解压jdk文件
if [ ! -f "/usr/local/java/jdk1.8.0_301/bin/java" ];
then
tar xvf $jdkfile -C $basedir
chown -R root:root $basedir
chmod -R 755 $basedir
else
echo "$jdkfile文件已解压"
exit
fi

echo -E 'export JAVA_HOME=/usr/local/java/jdk1.8.0_301' >> /etc/profile
echo -E 'export JRE_HOME=${JAVA_HOME}/jre' >> /etc/profile
echo -E 'export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib' >> /etc/profile
echo -E 'export PATH=${JAVA_HOME}/bin:$PATH' >> /etc/profile

source /etc/profile
java -version

hostname

脚本执行

sudo sed -i '2,100d'   /etc/sysconfig/network
echo 'NETWORKING=yes' >> /etc/sysconfig/network
echo "HOSTNAME=${hostname}" >> /etc/sysconfig/network
hostnamectl set-hostname $hostname
sudo sed -i '3,100d'  /etc/hosts
sudo sed -i '2r /home/hadoop/hostnamecontent' /etc/hosts

SELinux、THP、umask、ulimit、nproc 1、SELinux

#暂时禁用SELinux
setenforce 0
#禁用selinux(永久)：
vi /etc/selinux/config
SELINUX=disabled

或者使用脚本

sed -i '/SELINUX=/d' /etc/selinux/config
echo 'SELINUX=disabled' >> /etc/selinux/config

2、THP

关闭THP
关闭THP并给予文件权限， 禁用后对cpu的性能提升有帮助

echo -E 'if test -f /sys/kernel/mm/transparent_hugepage/enabled; then' >> /etc/rc.d/rc.local
echo -E 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.d/rc.local
echo -E 'fi' >> /etc/rc.d/rc.local
echo -E 'if test -f /sys/kernel/mm/transparent_hugepage/defrag; then' >> /etc/rc.d/rc.local
echo -E 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.d/rc.local
echo -E 'fi' >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local

3、umask

集群所有主机设置umask，设置用户创建目录默认权限

echo umask 0022 >> /etc/profile

4、设置ulimit和nproc

echo 'session required  /lib64/security/pam_limits.so' >> /etc/pam.d/login
echo "unicom  soft nofile  65535" >> /etc/security/limits.conf
echo "unicom  hard nofile  65535" >> /etc/security/limits.conf
echo "unicom  soft nproc  65535" >> /etc/security/limits.conf
echo "unicom  hard nproc  65535" >> /etc/security/limits.conf

检查设置：

[unicom@nn71 root]$ ulimit -Hn
65535
[unicom@nn71 root]$ ulimit -Sn
65535

ssh 1.集群每台主机产生ssh登录认证密钥对id_rsa ， id_rsa.pub秘钥文件

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

2：配置ssh无密登录： 2.1：Ambari server主机执行如下指令：

cd ~/.sshcat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

2.2：将Ambari Server生成的公钥文件id_rsa.pub拷贝至集群其他Ambari Agent主机(target_hosts)：

scp ~/.ssh/id_rsa.pub unicom@{target_hosts}:~

2.3：Agent主机分别执行下面命令：

cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

3：所有主机设置文件目录权限：

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Ambari Server主机分别执行ssh unicom@{target.host}, 进行免密码登录检查。

httpd 安装httpd

yum install httpd -y

修改httpd默认配置

vi /etc/httpd/conf/httpd.conf
1）修改 documentRoot为"/var/www/html"
2）修改 节：添加Options Indexes FollowSymlinks
3）修改“#ServerName www.example.com:80”去掉“#，改为ServerName localhost
4）删除默认页面：
rm -f /etc/httpd/conf.d/welcome.conf

设置httpd开启自启

systemctl enable httpd
systemctl restart httpd

验证

http://ip/

解压介质到/var/www/html

sudo tar -zxvf HDP-3.1.5.0-centos7-rpm.tar.gz -C /var/www/html
sudo tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html
sudo tar -zxvf ambari-2.7.5.0-centos7.tar.gz -C /var/www/html

删除默认页面

rm index.html index.html.bak

配置repo文件

[hadoop@sjsy-hh202-zbxh130w yum.repos.d]$ cat ambari.repo 
#VERSION_NUMBER=2.7.5.0-72
[ambari-2.7.5.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.5.0
baseurl=http://10.177.86.41/ambari/centos7/2.7.5.0-72/
gpgcheck=1
gpgkey=http://10.177.86.41/ambari/centos7/2.7.5.0-72/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[hadoop@sjsy-hh202-zbxh130w yum.repos.d]$ cat hdp.repo 
[hdp]
name=hdp repo
baseurl=http://10.177.86.41/HDP/centos7/3.1.5.0-152/
enabled=1
gpgcheck=0

[hdp-utils]
name=hdp-utils repo
baseurl=http://10.177.86.41/HDP-UTILS/centos7/1.1.0.22/
enabled=1
gpgcheck=0

删除snappy包（server节点）

yum list installed | grep snappy

yum -y remove snappy

mysql 安装mariadb服务

sudo yum install -y mariadb-server

更改配置

sudo sh -c "echo default-character-set=utf8 >> /etc/my.cnf.d/client.cnf"

启动mysql

systemctl start mariadb

设置服务开启自启动

systemctl enable mariadb

查看服务状态

systemctl status mariadb

使用命令行客户端尝试连接

mysql -uroot

初次使用无密码

查看mariadb版本号

select version();

查看字符集配置

show variables like '%char%';

创建新的用户、数据库

create database ambari character set utf8;
CREATE USER 'ambari'@'%'IDENTIFIED BY 'mKUgVPdaFU';
grant all privileges on ambari.* TO 'ambari'@'%';
flush privileges;

查看用户

SELECT DISTINCT CONCAt('User: ''',user,'''@''',host,''';') AS query FROM mysql.user;

mysql安全配置

mysql_secure_installation

可设置root的新密码

ambari 安装ambari

sudo yum install ambari-server

配置mysql连接器jar

cp /home/unicom/mysql-connector-java-5.1.46.jar /usr/share/java
ln -s /usr/share/java/mysql-connector-java-5.1.46.jar /usr/share/java/mysql-connector-java.jar

初始化ambari-server数据库连接配置

sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

配置ambari

sudo ambari-server setup

执行ambari初始化sql

在数据库中

use ambari；
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

启动ambari

sudo ambari-server start

注意：混部arm和x86机器时，在启动ambari之前，需要注释配置文件：
vi /var/lib/ambari-server/resources/stacks/HDP/3.0/services/SMARTSENSE/metainfo.xml
注释掉MANDATORY

web页面

http://yourip:8080/