Hive中利用insert从ods加载数据到dwd异常

最新推荐文章于 2023-06-26 20:46:47 发布

zhangyingchengqi

最新推荐文章于 2023-06-26 20:46:47 发布

阅读量575

点赞数 1

CC 4.0 BY-SA版权

本文链接：https://blog.csdn.net/zhangyingchengqi/article/details/120232859

hive 同时被 2 个专栏收录

7 篇文章

订阅专栏

数据仓库

2 篇文章

订阅专栏

本文探讨了Hive在移动外部表数据时遇到的IOException，原因在于源路径已存在的文件冲突。通过分析和提供解决方案，包括删除旧文件夹，解决Hadoop HDFS中rename操作的错误。

2021-09-12T08:23:20,736 ERROR [f53ee21b-bce6-4ddb-a0bd-9db513661e03 main] exec.Task: Failed with exception java.io.IOException: rename for src path: hdfs://yc/warehouse/gmall/dwd/dwd_start_log/dt=2021-09-10/.hive-staging_hive_2021-09-12_08-23-09_369_3424047669318712340-1/-ext-10000/000000_0 to dest path:hdfs://yc/warehouse/gmall/dwd/dwd_start_log/dt=2021-09-10/000000_0 returned false
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: rename for src path: hdfs://yc/warehouse/gmall/dwd/dwd_start_log/dt=2021-09-10/.hive-staging_hive_2021-09-12_08-23-09_369_3424047669318712340-1/-ext-10000/000000_0 to dest path:hdfs://yc/warehouse/gmall/dwd/dwd_start_log/dt=2021-09-10/000000_0 returned false
        at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:3204)
        at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3478)
        at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1650)
        at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1579)
        at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:503)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: rename for src path: hdfs://yc/warehouse/gmall/dwd/dwd_start_log/dt=2021-09-10/.hive-staging_hive_2021-09-12_08-23-09_369_3424047669318712340-1/-ext-10000/000000_0 to dest path:hdfs://yc/warehouse/gmall/dwd/dwd_start_log/dt=2021-09-10/000000_0 returned false
        at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:3184)
        at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:3174)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

异常分析如下:

1. 在前面的操作中曾经删除过dwd_start_log表，因为这是一张外部表，所以在删除时只删除了表的元数据，但是表对应的数据文件依然还在hdfs的指定位置上中
2. Hive把临时文件移动到正式目录的时候，会先对hdfs上的文件做一个判断。如果指定位置上不存在该文件夹，就在指定位置上创建此文件夹；如果该文件夹存在则直接将文件移过去，既把临时文件夹里的000000_0文件移动到正式目录下，可此时正式目录下已经有了一个叫000000_0的文件，所以就会报如上的错误.

解决方案:

hadoop fs -rm -r /warehouse/gmall/dwd/dwd_start_log/dt=2021-09-10/*