hadoop-3.0.0分布式部署与配置

最新推荐文章于 2025-05-12 22:58:50 发布

wuxidemo

最新推荐文章于 2025-05-12 22:58:50 发布

阅读量2k

点赞数 1

CC 4.0 BY-SA版权

分类专栏： hadoop

本文链接：https://blog.csdn.net/wuxidemo/article/details/77115931

hadoop 专栏收录该内容

4 篇文章

订阅专栏

本文详细介绍如何在三台CentOS7机器上部署Hadoop3集群，包括SSH免密登录配置、Hadoop软件包下载与解压、核心配置文件(core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml)的设置、Java环境变量添加及Hadoop集群的启动步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

3台机器centos 7

hadoop.master 192.168.0.116

hadoop.slave1 192.168.0.117

hadoop.slave2 192.168.0.118

配置好ssh免密登陆

下载hadoop-3.0.0-alpha4.tar.gz到本机

解压tar xzvf hadoop-3.0.0-alpha4.tar.gz

修改配置文件(目录/home/glsc/hadoop-3.0.0/etc/hadoop/)：

1. core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop.master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:///home/glsc/hadoop-3.0.0/tmp</value>
</property>
</configuration>

2. hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/glsc/hadoop-3.0.0/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/glsc/hadoop-3.0.0/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop.slave1:9001</value>
</property>
</configuration>

3.yarn-site.xml

<configuration>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandle</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop.master:8025</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop.master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop.master:8040</value>
</property>
</configuration>

4.mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapreduce.application.classpath</name>
<value>
/home/glsc/hadoop-3.0.0/etc/hadoop,
/home/glsc/hadoop-3.0.0/share/hadoop/common/*,
/home/glsc/hadoop-3.0.0/share/hadoop/common/lib/*,
/home/glsc/hadoop-3.0.0/share/hadoop/hdfs/*,
/home/glsc/hadoop-3.0.0/share/hadoop/hdfs/lib/*,
/home/glsc/hadoop-3.0.0/share/hadoop/mapreduce/*,
/home/glsc/hadoop-3.0.0/share/hadoop/mapreduce/lib/*,
/home/glsc/hadoop-3.0.0/share/hadoop/yarn/*,
/home/glsc/hadoop-3.0.0/share/hadoop/yarn/lib/*
</value>
</property>
</configuration>