Linux运维笔记|自动化运维攻城狮

Centos 7.5 离线安装CDH5.15

CDH 5.15.1   大数据  

一、 服务器初始化

  • 配置hosts和设置hostname(也可以直接通过DNS给每台主机解析一个域名,内网IP)
hostname  001.cdh.imdst.com  
echo  001.cdh.imdst.com >  /etc/hostname   ###每台主机依次执行
  • /etc/hosts统一配置
001.cdh.imdst.com   10.0.1.1  
002.cdh.imdst.com   10.0.1.2  
003.cdh.imdst.com   10.0.1.2
  • 安装需要的软件
yum -y install   vim telnet lrzsz rsync httpd ntp  
chkconfig ntpd on
  • 启用rc.local
chmod +x /etc/rc.d/rc.local  
systemctl enable rc-local.service
  • 挂载分区 
mkfs.ext4  /dev/vdb  
mount /dev/vdb /data && df -h  

data_blkid=`blkid |grep vdb |awk '{print $2}'|sed s/\"//g`  
echo "$data_blkid /data              ext4    defaults        0 0" >> /etc/fstab
  • 配置三台主机之间无密码SSH登陆
ssh-keygen  
cd .ssh/  
cat id_rsa.pub  >> /root/.ssh/authorized_keys  ###将authorized_keys和id_rsa拷贝至其他两台主机  
chmod 600 /root/.ssh/id_rsa
  • 其他优化
echo never > /sys/kernel/mm/transparent_hugepage/defrag  
echo never > /sys/kernel/mm/transparent_hugepage/enabled  
echo 10 >/proc/sys/vm/swappiness  
echo "vm.swappiness = 10" >> /etc/sysctl.conf  
echo "vm.max_map_count=655360" >> /etc/sysctl.conf  
sysctl -p

二、安装JDK和Mysql

  • JDK下载地址
    https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
  • 安装JDK,三台主机分别安装 
yum install jdk-8u181-linux-x64.rpm

三、下载CDH离线安装

mkdir /opt/cloudera-manager  
wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz  
tar xvzf cloudera-manager*.tar.gz -C /opt/cloudera-manager  

mkdir -p /opt/cloudera/parcel-repo && cd /opt/cloudera/parcel-repo  
wget http://archive.cloudera.com/cdh5/parcels/5.15.1/CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel  
wget http://archive.cloudera.com/cdh5/parcels/5.15.1/CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1 && mv CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha  
wget http://archive.cloudera.com/cdh5/parcels/5.15.1/manifest.json

四、开始部署SCM

  • 创建用户 cloudera-scm（所有节点）
useradd --system --home-dir /opt/cloudera-manager/cm-5.15.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

　 * 配置 CM Agent ,server_host为主节点主机名,将config.ini分发至其他节点

vim /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini  
server_host=001.cdh.imdst.com

scp /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini  002.cdh.imdst.com@:/opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/  
scp /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini  003.cdh.imdst.com@:/opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/
  • 配置 CM Server 的数据库
/opt/cloudera-manager/cm-5.15.1/share/cmf/schema/scm_prepare_database.sh mysql cm -h127.0.0.1 -uroot -p --scm-host 127.0.0.1 scm root ""
  • 参数说明
mysql             数据库用的是mysql，如果安装过程中用的oracle，那么该参数就应该改为oracle  
-h127.0.0.1       数据库建立在本机上面，也就是127.0.0.1上面.
-uroot            以root用户运行mysql
-p                mysql用户密码为空则不填
--scm-host        CMS的主机，一般是和mysql安装的主机是在同一个主机上
最后三个参数       分别为数据库名称、数据库用户名、数据库密码

五、启动 CM Manager&Agent 服务

  • 主节点启动
/opt/cloudera-manager/cm-5.15.1/etc/init.d/cloudera-scm-server start
/opt/cloudera-manager/cm-5.15.1/etc/init.d/cloudera-scm-agent start
  • 所有从节点
/opt/cloudera-manager/cm-5.15.1/etc/init.d/cloudera-scm-agent start

六、开始安装CDH

  • 登陆后根据提示进行安装即可，请先往后面看
  • 部分组件需要我们先创建数据库
CREATE DATABASE `hive` CHARACTER SET 'utf8mb4';  
CREATE DATABASE `hue` CHARACTER SET 'utf8mb4';  
CREATE DATABASE `am` CHARACTER SET 'utf8mb4';  
CREATE DATABASE `oozie` CHARACTER SET 'utf8mb4';
  • JDBC driver cannot be found.需要下载JDBC库 (所有节点)
cd /opt/cloudera-manager/cm-5.15.1/share/cmf/lib  
wget http://archiva-maven-storage-prod.oss-cn-beijing.aliyuncs.com/repository/central/mysql/mysql-connector-java/5.1.9/mysql-connector-java-5.1.9.jar
  • Hive服务也需要添加jdbc库 (所有节点)
cd /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hive/lib  
cp /opt/cloudera-manager/cm-5.15.1/share/cmf/lib/mysql-connector-java-5.1.9.jar .
  • oozie也没有
cp /opt/cloudera-manager/cm-5.15.1/share/cmf/lib/mysql-connector-java-5.1.9.jar /usr/share/java/
  • 注自己添加的parcel要重启Server和agent才可以看到