cdh阿里云安装

于 2020-06-04 00:32:53 发布
阅读量945 收藏
点赞数
文章标签： 数据库 linux hadoop centos 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/github_28583061/article/details/106500322
版权
这篇博客详细介绍了如何在阿里云上安装CDH 6.2.1集群，包括CDH和CM的离线包准备、主机配置、免密登录、JDK安装、MySQL JDBC驱动配置、数据库创建、CDH parcel安装以及集群的启动和测试。主要涉及Linux CentOS 7.3系统，涉及技术包括Hadoop、大数据、数据库和Linux系统管理。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

集群安装(cdh 6.2.1)

cdh 安装

cdh 版本与hadoop 版本对应关系

6.1文档

centos 源

服务器

  • viya02 master
  • viya01 master2
  • dm-inc-fin-1 data1
  • dm-inc-fin-2 data2
  • forecastedatamind data3

具体步骤

CDH离线包准备

配置host (所有节点)

  • vim /etc/hosts

ssh 免密登录（两个name节点）

ssh-keygen -trsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

scp ~/.ssh/authorized_keys root@dm-inc-fin-1:~/.ssh/
scp ~/.ssh/authorized_keys root@dm-inc-fin-2:~/.ssh/
scp ~/.ssh/authorized_keys root@forecastdatamind:~/.ssh/

centos 7.3 准备及基础配置 jdk （所有节点）

安装 MySQL JDBC 驱动(所有节点)

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar xf mysql-connector-java-5.1.46.tar.gz

mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

添加mysql 配置(namenode节点）

https://www.cloudera.com/documentation/enterprise/latest/topics/install_cm_mariadb.html

innodb_buffer_pool_size = 4G  ==>128m
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
#innodb_log_file_size = 512M    ==32m
  • 启动数据库
    • systemctl enable mariadb
    • systemctl start mariadb
    • vim /etc/rc.local 添加启动命令
  • 初始化数据库

    • /usr/bin/mysql_secure_installation * 问题：ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

      • /etc/my.cnf 配置 [mysqld]下 添加 skip-grant-tables * 出问题看日志： tail -n500 /var/log/mariadb/mariadb.log * 问题：Error: log file ./ib_logfile0 is of different size 0 5242880 bytes

      • innodb_log_file_size = 256M 注释

    • 账号密码：root/root123456

使用root登陆数据库，创建以下数据库和账号。

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '!dtmind&123';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '!dtmind&123';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '!dtmind&123';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '!dtmind&123';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '!dtmind&123';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '!dtmind&123';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '!dtmind&123';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '!dtmind&123';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '!dtmind&123';

flush privileges;

cdh manager 安装

文件拷贝

  • namenode : /home/datamind/cdh
    • 全部文件
  • datanode: /home/datamind/cdh
    • cloudera-manager-agent,cloudera-manager-daemons,cloudera-manager.repo (其他节点)
    • scp cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm root@viya01:/home/datamind/cdh
    • scp cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm root@viya01:/home/datamind/cdh
    • scp cloudera-manager.repo root@viya01:/home/datamind/cdh
安装 CM Server 和 Agent
namenode
  • mkdir /opt/cloudera-manager/
  • cp cloudera-manager.repo /etc/yum.repos.d/
  • yum search cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
  • rpm --import https://archive.cloudera.com/cm6/6.2.1/redhat7/yum/RPM-GPG-KEY-cloudera
  • //yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
  • yum -y install cyrus-sasl-gssapi bind-utils psmisc libxslt cyrus-sasl-plain openssl cyrus-sasl-gssap fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libpq.so.5
  • rpm -ivh cloudera-manager-daemons
  • rpm -ivh cloudera-manager-agent
  • rpm -ivh cloudera-manager-server
设置 Cloudera Manager 数据库
  • /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm * !dtmind&123

配置CDH的软件包 parcels(namenode01)

  • 拷贝 CDH-6.2.1-el7.parcel ==》 /opt/cloudera/parcel-repo
  • //拷贝 CDH-6.2.1-el7.parcel.sha256 ==》 /opt/cloudera/parcel-repo
  • 拷贝 manifest.json ==》 /opt/cloudera/parcel-repo
  • mv CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel manifest.json /opt/cloudera/parcel-repo/
  • 重要，不然选择不到版本，接下来打开manifest.json文件，里面是json格式的配置，我们需要的就是与我们系统版本相对应的 hash码， 
    •  "hash": "e3b31081a200ee9ee10aa64e48a0a3d62b25a7b1",
  "parcelName": "CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel",
  "replaces": "IMPALA, SOLR, SPARK, KAFKA, IMPALA_KUDU, KUDU, SPARK2"
    • echo "e3b31081a200ee9ee10aa64e48a0a3d62b25a7b1" > CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha
    • chown cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo/*

启动 cm (namenode - viya02)

  • vim /etc/cloudera-scm-agent/config.ini
    • server_host = viya02
  • systemctl start cloudera-scm-server
  • systemctl status cloudera-scm-server
  • tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
  • 问题： Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
    • journalctl -n 20 查看service启动日志 Cloudera Manager requires Oracle JDK 1.8 or later.
    • journalctl -f
  • scp oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm root@172.17.0.2:/home
  • rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

slave1-3

  • cdh源配置 * mkdir /opt/cloudera-manager/ * cp cloudera-manager.repo /etc/yum.repos.d/ * yum -y install cyrus-sasl-gssapi bind-utils psmisc libxslt cyrus-sasl-plain openssl cyrus-sasl-gssap fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libpq.so.5 * rpm -ivh cloudera-manager-daemons-6.2.1 * rpm -ivh cloudera-manager-agent-6.2.1 * vim /etc/cloudera-scm-agent/config.ini 修改server_host * jdk安装 * docker-machine ssh default * scp oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm root@172.17.0.2:/home * rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

通过网页安装

  • 选择CDH版本这里会显示你放在/opt/cloudera/parcel-repo/下的parcel包，若未显示
  • org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp. Name node is in safe mode.*
HDFS 根目录 /data/hbase

DataNode 数据目录 /data/dfs/dn

NameNode 数据目录 /data/dfs/nn

HDFS 检查点目录 /data/dfs/snn

Hive 仓库目录 /data/user/hive/warehouse

ShareLib 根目录 /data/user/oozie

NodeManager 本地目录 /data/yarn/nm

端口页面

运行测试

  • spark-shell:
    • Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x * usermod -s /bin/bash hdfs
    • adduser hdfs 用hdfs账号登录

hive 测试

  • Given NMToken for application : appattempt_1556161534641_0003_000002 is not valid for current node manager.expected : slave2:8041 found : slave1:8041
    • ip改变了，对不上
CREATE  TABLE test (   id int) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','  ;

insert into test values(1);

web页面

单独升级spark

评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值