java.lang.IllegalArgumentException: Can not create a Path from a null string

[mhainfarecom]

Hi

I'm unable to insert data to HBase, because my job is failing with exception:

App > 16/07/06 12:16:58 task-result-getter-1 WARN TaskSetManager: Lost task 3.0 in stage 0.0 (TID 3, ip-172-31-19-56.eu-west-1.compute.internal): java.lang.IllegalArgumentException: Can not create a Path from a null string
App > at org.apache.hadoop.fs.Path.checkPathArg(Path.java:125)
App > at org.apache.hadoop.fs.Path.(Path.java:137)
App > at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1197)
App > at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1190)
App > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
App > at org.apache.spark.scheduler.Task.run(Task.scala:89)
App > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
App > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
App > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
App > at java.lang.Thread.run(Thread.java:745)

my hbase-site.xml is:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>zookeeper.recovery.retry</name>
    <value>3</value>
</property>
<property>
    <name>hbase.regionserver.info.port</name>
    <value>16030</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>remote_IP</value>
</property>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://remote_IP:9000/hbase</value>
</property>
<property>
    <name>hbase.fs.tmp.dir</name>
    <value>/user/${user.name}/hbase-staging</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/tmp/hadoop-${user.name}</value>
</property>
    <!-- Put any other property here, it will be used -->
</configuration>

my Hbase version:
HBase Version 1.0.0, revision=984db9a1cae088b996e997db9ce83f6d4bd565ad

Any suggesions?

[zhzhan]

Could you share your code in the writing path?

[jleach4]

I just upgraded our Spark implementation at Splice Machine and I hit the same issue. It looks like the output committer tries to write to the outputPath. I did not have
mapreduce.output.fileoutputformat.outputdir set which caused the issue.

    val committer = FileCommitProtocol.instantiate(
      className = classOf[HadoopMapReduceCommitProtocol].getName,
      jobId = stageId.toString,
      outputPath = conf.value.get("mapreduce.output.fileoutputformat.outputdir"),
      isAppend = false).asInstanceOf[HadoopMapReduceCommitProtocol]
    committer.setupJob(jobContext)

Hope this helps,
John

[mavencode01]

@zhzhan I just hit the same problem. @jleach4 were you able to resolve the issue?

[weiqingy]

@mavencode01 The issue is hitting https://issues.apache.org/jira/browse/SPARK-21549. For reference: https://issues.apache.org/jira/browse/SPARK-18191.

Workaround is to explicitly set the property mapreduce.output.fileoutputformat.outputdir to a dummy value (which is valid and writable by user - say /tmp).

[mavencode01]

@weiqingy thanks for the quick reply but I tried setting the property and the job still fails with same error.

--conf spark.hadoop.mapred.output.dir=/tmp --conf spark.hadoop.mapreduce.output.fileoutputformat.outputdir=/tmp

[weiqingy]

@mavencode01 You may want to set that property in mapred-site.xml, and restart hadoop cluster. Also, you can also use SHC branch-2.2 (https://github.com/hortonworks-spark/shc/tree/branch-2.2), which we make a fix for this issue.

[mavencode01]

@weiqingy sweet!!! Thanks for the quick reply. Appreciate