[root@hadoop hadoop]# start-all.sh Starting namenodes on [hadoop] hadoop: namenode is running as process 7644. Stop it first and ensure /tmp/hadoop-root-namenode.pid file is empty before retry. Starting datanodes hadoop02: mv: cannot stat ‘/export/servers/hadoop-3.4.1/logs/hadoop-root-datanode-hadoop.out.4’: No such file or directory hadoop02: mv: cannot stat ‘/export/servers/hadoop-3.4.1/logs/hadoop-root-datanode-hadoop.out.3’: No such file or directory hadoop02: mv: cannot stat ‘/export/servers/hadoop-3.4.1/logs/hadoop-root-datanode-hadoop.out.2’: No such file or directory hadoop02: mv: cannot stat ‘/export/servers/hadoop-3.4.1/logs/hadoop-root-datanode-hadoop.out.1’: No such file or directory hadoop02: mv: cannot stat ‘/export/servers/hadoop-3.4.1/logs/hadoop-root-datanode-hadoop.out’: No such file or directory Starting secondary namenodes [hadoop02] hadoop02: secondarynamenode is running as process 5763. Stop it first and ensure /tmp/hadoop-root-secondarynamenode.pid file is empty before retry. Starting resourcemanager resourcemanager is running as process 15332. Stop it first and ensure /tmp/hadoop-root-resourcemanager.pid file is empty before retry. Starting nodemanagers hadoop01: mv: cannot stat ‘/export/servers/hadoop-3.4.1/logs/hadoop-root-nodemanager-hadoop.out.4’: No such file or directory hadoop01: mv: cannot stat ‘/export/servers/hadoop-3.4.1/logs/hadoop-root-nodemanager-hadoop.out.3’: No such file or directory hadoop01: mv: cannot stat ‘/export/servers/hadoop-3.4.1/logs/hadoop-root-nodemanager-hadoop.out.2’: No such file or directory hadoop01: mv: cannot stat ‘/export/servers/hadoop-3.4.1/logs/hadoop-root-nodemanager-hadoop.out.1’: No such file or directory hadoop01: mv: cannot stat ‘/export/servers/hadoop-3.4.1/logs/hadoop-root-nodemanager-hadoop.out’: No such file or directory

### Hadoop 3.4.1 集群启动失败的解决方法 Hadoop 集群启动失败可能由多种原因引起，包括配置文件错误、权限问题、进程未正确停止或日志文件缺失等。以下是针对 `start-all.sh` 启动失败时 NameNode 和 DataNode 进程未停止、PID 文件非空以及日志文件不存在问题的具体分析和解决方案。 #### 1. 检查并清理 PID 文件 如果 PID 文件存在但对应的进程已经终止，这可能导致 Hadoop 认为服务仍在运行，从而阻止重新启动。需要手动删除这些 PID 文件。 ```bash find /tmp -name "*.pid" -exec rm -f {} \; ``` 上述命令会递归查找 `/tmp` 目录下的所有 `.pid` 文件并删除它们[^2]。 #### 2. 配置文件检查 确保 `hadoop-env.sh` 文件中定义了正确的用户变量以避免启动脚本因权限不足而失败。例如： ```bash export HDFS_NAMENODE_USER="hdfs" export HDFS_DATANODE_USER="hdfs" export HDFS_SECONDARYNAMENODE_USER="hdfs" export YARN_RESOURCEMANAGER_USER="yarn" export YARN_NODEMANAGER_USER="yarn" ``` 这些变量必须根据实际使用的用户名进行设置，通常建议使用专门创建的 Hadoop 用户而不是 root 用户[^1]。 #### 3. 日志路径验证 日志文件不存在可能是由于日志目录未正确配置或没有写入权限。在 `core-site.xml` 中确认日志目录是否正确指定： ```xml <property> <name>hadoop.log.dir</name> <value>/var/log/hadoop</value> </property> ``` 同时确保该目录存在并且 Hadoop 用户拥有读写权限： ```bash mkdir -p /var/log/hadoop chown -R hdfs:hadoop /var/log/hadoop chmod 750 /var/log/hadoop ``` #### 4. 格式化 NameNode 如果这是首次启动集群或者元数据丢失，需要重新格式化 NameNode： ```bash hdfs namenode -format ``` 注意：此操作将清除所有现有数据，请仅在必要时执行[^4]。 #### 5. 数据目录权限 DataNode 的数据存储路径必须具有正确的权限。检查 `hdfs-site.xml` 中的 `dfs.datanode.data.dir` 参数，并确保这些目录可由 HDFS 用户访问： ```xml <property> <name>dfs.datanode.data.dir</name> <value>/data/dfs/dn</value> </property> ``` 然后设置适当的权限： ```bash mkdir -p /data/dfs/dn chown -R hdfs:hadoop /data/dfs/dn chmod 750 /data/dfs/dn ``` #### 6. 启动服务 完成上述步骤后，尝试再次启动所有服务： ```bash start-all.sh ``` 若仍存在问题，查看具体的服务启动日志以获取更多调试信息。 ### 示例代码 以下是一个简单的 Python 脚本，用于检查 Hadoop 配置文件中是否存在关键参数： ```python def check_hadoop_config(config_path): with open(config_path, 'r') as file: content = file.read() if "dfs.datanode.data.dir" not in content: return "Error: dfs.datanode.data.dir is missing." if "hadoop.log.dir" not in content: return "Error: hadoop.log.dir is missing." return "All required parameters are present." result = check_hadoop_config("/etc/hadoop/conf/hdfs-site.xml") print(result) ```