星辰大数据的博客

一、安装准备1、下载地址https://www.apache.org/dyn/closer.cgi/hive2、参考文档3、ssh免密配置https://blog.csdn.net/qq262593421/article/details/1053255934、zookeeper安装https://blog.csdn.net/qq262593421/article/details/1069554855、hadoop安装...

win10安装Hadoop3.0.0：https://blog.csdn.net/qq262593421/article/details/105927625win10搭建hive3.0.0：https://blog.csdn.net/qq262593421/article/details/104961689

一、hadoop集群重置1、关闭hbase和hdfsstop-habse.shstop-all.sh2、关闭zookeeperzkServer.sh stop3、删除nn、dn、jn配置目录数据rm -rf /home/cluster/hadoop/data/nn/*rm -rf /home/cluster/hadoop/data/dn/*rm -rf /home/cluster/hadoop/data/jn/*4、删除hdfs和hbase的日志文件数据rm -

一、问题描述重新清理hdfs的时候，发现有一个文件删除不了，出现以下报错rm: 无法删除"xxx": 结构需要清理二、问题解决1、查看问题文件的挂载磁盘df -h2、卸载磁盘挂载umount /dev/mapper/centos-home3、磁盘挂载修复xfs_repair /dev/mapper/centos-home4、重启系统reboot5、重新清理文件rm -rf /home/cluster/hadoop/data/.

目录一、CDH自身组件1、cloudera-scm-server2、cloudera-scm-agent二、Hadoop生态组件1、hadoop2、hive3、hbase4、zookeeper5、spark三、CDH加载的环境变量一、CDH自身组件1、cloudera-scm-server/etc/cloudera-scm-server/...

一、依赖环境安装1、jdk和scala依赖下载jdk1.8：https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.htmlscala12.11：https://www.scala-lang.org/download/2、flink1.9.0下载flink官方中文文档：https://flink.apache.org/zh/downloads.htmlflink1.9.0安装包：https://...

一、代码实现package com.xtd.hdfsimport java.io.Fileimport org.apache.hadoop.conf.Configurationimport org.apache.hadoop.fs.{FileSystem, FileUtil, Path}import scala.collection.mutable.{ArrayBuffer, ListBuffer}object HDFSUtils { def main(args: Array[

1、查看hdfs /home/data/test 目录下的所有文件2、使用org.apache.hadoop.fs.FileSystem 类遍历hdfs文件package com.xtd.hdfsimport org.apache.hadoop.conf.Configurationimport org.apache.hadoop.fs.{FileSystem, FileUtil, Path}import scala.collection.mutable.{ArrayBuf...

一、安装准备下载地址：https://archive.apache.org/dist/spark/官方文档：http://spark.apache.org/docs/latest/二、解压安装解压缩文件cd /usr/local/hadooptar zxpf spark-2.4.0-bin-hadoop2.7.tgz2、创建软链接ln -s spark-2.4.0-bin-hadoop2.7.tgz spark三、修改配置文件slaveshadoop00.

一、安装准备下载地址：http://archive.apache.org/dist/hbase/参考文档：http://hbase.apache.org/book.htmlssh免密登录：https://blog.csdn.net/qq262593421/article/details/105325593zookeeper安装：https://blog.csdn.net/qq262593421/article/details/106955485hadoop集群安装：https://blo

一、安装准备下载地址：https://www.apache.org/dyn/closer.cgi/hadoop/common官方文档：https://hadoop.apache.org/docs/r3.0.0/zookeeper高可用配置：https://blog.csdn.net/qq262593421/article/details/106955485hadoop集群角色 集群节点 NameNode hadoop001、hadoop002 DataNode ha

一、安装准备下载地址：https://www.apache.org/dyn/closer.cgi/zookeeper/官方文档：https://zookeeper.apache.org/doc/r3.4.5/二、解压安装解压文件cd /usr/local/hadooptar zxpf zookeeper-3.4.5.tar.gz创建软连接ln -s zookeeper-3.4.5 zookeeper三、配置zoo.cfg文件cd /usr/local/hadoop

#!/bin/bashecho '.......................................'QuorumPeerMain=$(jps | grep ' QuorumPeerMain')ZKFC=$(jps | grep ' DFSZKFailoverController')NameNode=$(jps | grep ' NameNode')DataNode=$(jps | grep ' DataNode')HMaster=$(jps | grep ' HMaster...

ReadShipMMSITwopackage com.xtd.fileimport java.io.Fileimport org.apache.spark.rdd.RDDimport org.apache.spark.sql.{SaveMode, SparkSession}object ReadShipMMSITwo { def main(args: Array[String]): Unit = { // sparkSession val spark = S...

目录remoterepositorycomponent versiondependencytemplateartifact versionremoterepository<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM...

一、下载hadoop3.0.0下载地址：http://archive.apache.org/dist/hadoop/core/hadoop-3.0.0/下载文件：hadoop-3.0.0.tar.gz二、下载winutils工具下载地址：https://github.com/steveloughran/winutils下载文件：hadoop-3.0.0/bin三、解...

1、配置Oozie Load Balancer（1）进入Oozie配置页面，搜索关键字 port（2）在Oozie Load Balancer栏输入需要开启oozie的host，以逗号隔开cdh01,cdh02,cdh03,cdh04,cdh05,cdh062、配置 Oozie Load Balancer HTTP、HTTPSPort注意：定义的 htt...

CDH 6.3.2The following table lists the project name, groupId, artifactId, and version required to access each CDH artifact.Project groupId artifactId version Apache Avro org.apach...

hive官方文档：https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-RunningHiveServer2andBeeline一、下载 hive 二进制文件http://archive.apache.org/dist/hive/hive-2.1.1/二、配置环境变量三...

一、Java环境变量配置二、hadoop环境变量配置三、spark环境变量配置四、hdfs配置1、hadoop-2.7.7\etc\hadoophadoop-env.cmd

hadoop：https://hadoop.apache.org/docs/stable/hive：https://cwiki.apache.org/confluence/display/Hive/GettingStartedhbase：https://hbase.apache.org/book.htmlyarn：https://hadoop.apache.org/docs/cu...

目录1、core-size.xml2、hdfs-site.xml3、mapred-site.xml4、yarn-site.xmlhadoop的常用配置文件，不需要记，知道哪个配置有什么作用就行了，下次配置的时候可以直接拿来用。1、core-size.xml<?xml version="1.0" encoding="UTF-8"?><?xml-sty...

目录1、Maven导入hadoop-client包2、core-site.xml文件配置3、log4j.properties 文件配置4、Top5.java（主要代码）5、测试数据6、运行结果1、Maven导入hadoop-client包 <dependency> <groupId>org.apache.hadoop</gro...

Index of /docs Name Last modified Size Description Parent Directory - current/ 2019-09-24 14:45 - current1/ 2019-08-22 16:09 -...

Apache Hadoop 2.7.3 Shell 命令操作指南大全目录OverviewappendToFilecatchecksumchgrpchmodchowncopyFromLocalcopyToLocalcountcpcreateSnapshotdeleteSnapshotdfdudus...