hadoop 报错：Error: Could not find or load main class

最新推荐文章于 2023-12-26 15:46:34 发布

原创最新推荐文章于 2023-12-26 15:46:34 发布 · 3.2w 阅读

7 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #error

Hadoop 专栏收录该内容

3 篇文章

订阅专栏

本文介绍了Hadoop MapReduce任务中常见的错误及其解决方法，包括如何设置正确的类路径和处理输出目录已存在的问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Error: Could not find or load main class

报这个错是因为当前运行的 .class 文件不在 hadoop-env.sh文件中
export HADOOP_CLASSPATH=${HADOOP_HOME}//你运行的*.class 文件所在路径

[john@localhost ch02]$ ls
input output pom.xml src target
[john@localhost ch02]$ export $HADOOP_CLASSPATH=target/classes
bash: export: `build/classes=target/classes': not a valid identifier
[john@localhost ch02]$ export HADOOP_CLASSPATH=target/classes
[john@localhost ch02]$ echo $HADOOP_CLASSPATH
target/classes
[john@localhost ch02]$ hadoop MaxTemperature input/sample.txt output
14/12/13 12:27:14 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/12/13 12:27:14 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/home/john/hadoop/book_res/code/hadoop-book-master/ch02/output already exists
   at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
   at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562)
   at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
   at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
   at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
   at MaxTemperature.main(MaxTemperature.java:31)

------

上面信息报错，output文件已经存在。hadoop输出文件路径需要，hadoop自己来建，我们创建就会报上面的错误。

修改方法：

[john@localhost ch02]$ rm -rf output/
[john@localhost ch02]$ ls
input pom.xml src target
[john@localhost ch02]$ hadoop MaxTemperature input/sample.txt output
14/12/13 12:28:40 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/12/13 12:28:40 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/12/13 12:28:41 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
14/12/13 12:28:41 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
14/12/13 12:28:41 INFO input.FileInputFormat: Total input paths to process : 1
14/12/13 12:28:41 INFO mapreduce.JobSubmitter: number of splits:1
14/12/13 12:28:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1165163551_0001
14/12/13 12:28:42 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/12/13 12:28:42 INFO mapreduce.Job: Running job: job_local1165163551_0001
14/12/13 12:28:42 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/12/13 12:28:42 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/12/13 12:28:42 INFO mapred.LocalJobRunner: Waiting for map tasks
14/12/13 12:28:42 INFO mapred.LocalJobRunner: Starting task: attempt_local1165163551_0001_m_000000_0
14/12/13 12:28:43 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/12/13 12:28:43 INFO mapred.MapTask: Processing split: file:/home/john/hadoop/book_res/code/hadoop-book-master/ch02/input/sample.txt:0+1485
14/12/13 12:28:43 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/12/13 12:28:43 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/12/13 12:28:43 INFO mapred.MapTask: soft limit at 83886080
14/12/13 12:28:43 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/12/13 12:28:43 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/12/13 12:28:43 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/12/13 12:28:43 INFO mapred.LocalJobRunner:
14/12/13 12:28:43 INFO mapred.MapTask: Starting flush of map output
14/12/13 12:28:43 INFO mapred.MapTask: Spilling map output
14/12/13 12:28:43 INFO mapred.MapTask: bufstart = 0; bufend = 99; bufvoid = 104857600
14/12/13 12:28:43 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
14/12/13 12:28:43 INFO mapred.MapTask: Finished spill 0
14/12/13 12:28:43 INFO mapred.Task: Task:attempt_local1165163551_0001_m_000000_0 is done. And is in the process of committing
14/12/13 12:28:43 INFO mapred.LocalJobRunner: map
14/12/13 12:28:43 INFO mapred.Task: Task 'attempt_local1165163551_0001_m_000000_0' done.
14/12/13 12:28:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local1165163551_0001_m_000000_0
14/12/13 12:28:43 INFO mapred.LocalJobRunner: map task executor complete.
14/12/13 12:28:43 INFO mapred.LocalJobRunner: Waiting for reduce tasks
14/12/13 12:28:43 INFO mapred.LocalJobRunner: Starting task: attempt_local1165163551_0001_r_000000_0
14/12/13 12:28:43 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/12/13 12:28:43 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@42cd2b59
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=333971456, maxSingleShuffleLimit=83492864, mergeThreshold=220421168, ioSortFactor=10, memToMemMergeOutputsThreshold=10
14/12/13 12:28:43 INFO reduce.EventFetcher: attempt_local1165163551_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
14/12/13 12:28:43 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1165163551_0001_m_000000_0 decomp: 123 len: 127 to MEMORY
14/12/13 12:28:43 INFO reduce.InMemoryMapOutput: Read 123 bytes from map-output for attempt_local1165163551_0001_m_000000_0
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 123, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->123
14/12/13 12:28:43 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
14/12/13 12:28:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
14/12/13 12:28:43 INFO mapred.Merger: Merging 1 sorted segments
14/12/13 12:28:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 116 bytes
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: Merged 1 segments, 123 bytes to disk to satisfy reduce memory limit
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: Merging 1 files, 127 bytes from disk
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
14/12/13 12:28:43 INFO mapred.Merger: Merging 1 sorted segments
14/12/13 12:28:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 116 bytes
14/12/13 12:28:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
14/12/13 12:28:43 INFO mapreduce.Job: Job job_local1165163551_0001 running in uber mode : false
14/12/13 12:28:43 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
14/12/13 12:28:43 INFO mapreduce.Job: map 100% reduce 0%
14/12/13 12:28:43 INFO mapred.Task: Task:attempt_local1165163551_0001_r_000000_0 is done. And is in the process of committing
14/12/13 12:28:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
14/12/13 12:28:43 INFO mapred.Task: Task attempt_local1165163551_0001_r_000000_0 is allowed to commit now
14/12/13 12:28:43 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1165163551_0001_r_000000_0' to file:/home/john/hadoop/book_res/code/hadoop-book-master/ch02/output/_temporary/0/task_local1165163551_0001_r_000000
14/12/13 12:28:43 INFO mapred.LocalJobRunner: reduce > reduce
14/12/13 12:28:43 INFO mapred.Task: Task 'attempt_local1165163551_0001_r_000000_0' done.
14/12/13 12:28:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local1165163551_0001_r_000000_0
14/12/13 12:28:43 INFO mapred.LocalJobRunner: reduce task executor complete.
14/12/13 12:28:44 INFO mapreduce.Job: map 100% reduce 100%
14/12/13 12:28:44 INFO mapreduce.Job: Job job_local1165163551_0001 completed successfully
14/12/13 12:28:44 INFO mapreduce.Job: Counters: 33
   File System Counters
       FILE: Number of bytes read=3652
       FILE: Number of bytes written=498027
       FILE: Number of read operations=0
       FILE: Number of large read operations=0
       FILE: Number of write operations=0
   Map-Reduce Framework
       Map input records=11
       Map output records=11
       Map output bytes=99
       Map output materialized bytes=127
       Input split bytes=142
       Combine input records=0
       Combine output records=0
       Reduce input groups=2
       Reduce shuffle bytes=127
       Reduce input records=11
       Reduce output records=2
       Spilled Records=22
       Shuffled Maps =1
       Failed Shuffles=0
       Merged Map outputs=1
       GC time elapsed (ms)=37
       CPU time spent (ms)=0
       Physical memory (bytes) snapshot=0
       Virtual memory (bytes) snapshot=0
       Total committed heap usage (bytes)=365953024
   Shuffle Errors
       BAD_ID=0
       CONNECTION=0
       IO_ERROR=0
       WRONG_LENGTH=0
       WRONG_MAP=0
       WRONG_REDUCE=0
   File Input Format Counters
       Bytes Read=1485
   File Output Format Counters
       Bytes Written=30
[john@localhost ch02]$