调试到Job job = Job.getInstance(conf)时报错：java.lang.IIIegalStateException异常，无法评估 org.apache.hadoop.mapre

于 2023-11-17 15:28:02 发布
阅读量326 收藏
点赞数
CC 4.0 BY-SA版权
分类专栏： 大数据 文章标签： java 开发语言
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Y_Aryan/article/details/134464185
文章讲述了在本地创建MapReduce项目时，若在Driver类设置断点可能导致IIIegalStateException异常。这种异常是非阻塞的，不影响程序执行。推荐在Mapper类或Reducer类设置断点进行调试，Driver类可酌情避免。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

在本地创建MapReduce项目，调试的时候断点如果加在Driver类中，就会发现走到创建Job实例的时候

Job job = Job.getInstance(conf);

调试出现报错：java.lang.IIIegalStateException异常，无法评估 org.apache.hadoop.mapreduce.Job.toString（）

如果遇到这个问题，可以忽略它，这是一个非阻塞异常，即使有这个异常，程序仍然可以正常执行到最后。

换句话说，即使运行这个项目没有任何问题，调试这个项目的时候也会报这个错。

因此建议断点加在Mapper类中，或者Reducer类中，非必要不需加在Driver类中。

java 版本文件导致 Job job = Job.getInstance(configuration); 报错
底层码农
03-27 1万+
Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) at org.apache.hadoop.security.SecurityUtil.getAuthen...
无法解析某个方法
qq_39091546的博客
03-31 2684
在java代码中遇到这个问题主要原因之一是采用了 this 调用了这个方法，但是this有时候并不一定是你认为的代表的类，可能代表了其他的类，一定要注意。 ...
一个简单的hadoop job测试
liufunan的专栏
09-06 959
1：编写map类package org.liufu.hadoop;import java.io.IOException; import java.net.InetAddress; import java.text.SimpleDateFormat; import java.util.Date;import org.apache.hadoop.io.IntWritable; import org.ap
【Hadoop】方法抛出‘java.lang.IllegalStateException‘异常. 无法评估 org.apache.hadoop.mapreduce.Job.toString()
sinat_38650186的博客
01-10 1612
虽然报错，但是非阻塞的错误，输出文件夹还是会创建并关闭job.
Hadoop源码篇之客户端Job
lixinkuan的博客
09-29 809
一、前述 剖析hadoop源码，首先从Client看起，因为Client在MapReduce的过程中承担了很多重要的角色。 二、MapReduce框架主类 代码如下： public static void main(String[] args) throws Exception { // System.setProperty("hadoop.home.dir", "E:/Hadoo...
java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.session.SessionState
半丶晨曦的博客
07-15 1255
java.lang.NoSuchMethodError:org.apache.hadoop.hive.ql.session.SessionState Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.session.SessionState.getPerfLogger()Lorg/
Exception in thread “main“ java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$PO
m0_55901666的博客
12-18 3528
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat;
mr执行报错：java.text.ParseException: Unparseable date: "1220" at java.text.DateFormat.parse(DateFormat.java:366) at com.sina.ud.common.CommonUtils.getReducersAndMemory(CommonUtils.java:671) at com.sina.ud.common.CommonUtils.getReducersAndMemory(CommonUtils.java:643) at com.sina.ud.job.game.GameKeywordsMapJob.run(GameKeywordsMapJob.java:89) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at com.sina.ud.job.game.GameKeywordsMapJob.main(GameKeywordsMapJob.java:42) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Error: org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120 at org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101) at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130) at org.apache.hadoop.mapred.Counters$Group.findCounter(Counters.java:369) at org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:314) at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:479) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:689) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:633) at org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.getCounter(TaskAttemptContextImpl.java:76) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getCounter(WrappedMapper.java:101) at com.sina.ud.map.keyword.GameKeywordsMapMapper.map(GameKeywordsMapMapper.java:124) at com.sina.ud.map.keyword.GameKeywordsMapMapper.map(GameKeywordsMapMapper.java:27) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at com.sina.hadoop.DelegatingMapper.run(DelegatingMapper.java:34) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:243) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1750) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:237)
07-11
其次，计数器错误：Error: org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120 这个错误表明在MapReduce作业中创建的计数器数量超过了Hadoop配置的最大值（默认120个...
mr执行又报错： Container id: container_e73_1742282717592_33201880_01_000226 Exit code: 67 Stack trace: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:128) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:108) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:540) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:474) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:97) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Shell output: main : command provided 1 main : run as user is mapred main : requested yarn user is sinadata Getting exit code file... Creating script paths... Writing pid file... Writing to tmp file /data13/hadoop/local/nmPrivate/application_1742282717592_33201880/container_e73_1742282717592_33201880_01_000226/container_e73_1742282717592_33201880_01_000226.pid.tmp Writing to cgroup task files... Creating local dirs... Launching container... Getting exit code file... Creating script paths... Container exited with a non-zero exit code 67. Last 4096 bytes of stderr : CounterGroup.findCounterImpl(AbstractCounterGroup.java:123) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130) at org.apache.hadoop.mapred.Counters$Group.findCounter(Counters.java:369) at org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:314) at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:479) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:689) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:633) at org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.getCounter(TaskAttemptContextImpl.java:76) at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.getCounter(WrappedReducer.java:94) at com.sina.ud.reduce.keyword.GameKeywordsMapReducer.reduce(GameKeywordsMapReducer.java:35) at com.sina.ud.reduce.keyword.GameKeywordsMapReducer.reduce(GameKeywordsMapReducer.java:14) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:243) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1750) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:237) Caused by: org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120 at org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101) ... 23 more org.apache.hadoop.mapreduce.counters.LimitExceededException: org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120 at org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:97) at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113) at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130) at org.apache.hadoop.mapred.Counters$Group.findCounter(Counters.java:369) at org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:314) at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:479) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:689) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:633) at org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.getCounter(TaskAttemptContextImpl.java:76) at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.getCounter(WrappedReducer.java:94) at com.sina.ud.reduce.keyword.GameKeywordsMapReducer.reduce(GameKeywordsMapReducer.java:35) at com.sina.ud.reduce.keyword.GameKeywordsMapReducer.reduce(GameKeywordsMapReducer.java:14) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:243) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1750) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:237) Caused by: org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120 at org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101) ... 23 more
最新发布
07-11
Caused by: org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120 这意味着在MapReduce作业中使用了过多的计数器（Counters），超过了Hadoop配置的最大值（默认为120个...
上述情况均正常的情况下，依然出现问题 [root@hadoop04 ~]# hadoop jar film001.jar CleanDriver /film/input /film/outputs/cleandata 25/03/14 18:21:36 INFO client.RMProxy: Connecting to ResourceManager at hadoop04/192.168.100.104:8032 Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://hadoop04:9000/film/input already exists at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at CleanDriver.main(CleanDriver.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:226) at org.apache.hadoop.util.RunJar.main(RunJar.java:141) [root@hadoop04 ~]#
03-15
Job job = Job.getInstance(conf, "Data Cleaner"); // 确保参数顺序正确 FileInputFormat.addInputPath(job, new Path(args[0])); // 输入路径 FileOutputFormat.setOutputPath(job, new Path(args[1])); // ...
elasticsearch5.5.1客户端所需jar包，解决必须用maven工程问题
08-19
elasticsearch5.5.1客户端所需jar包，可用于普通java工程使用elasticsearch客户端，解决必须用maven工程问题
几种任务调试的java实现方法与比较
wangclisjuandn的专栏
03-31 887
摘要:在游戏开中有一些任务调试需要程序实现处理，任务调度是指基于给定时间点，或给定时间间隔或者给定执行次数自动执行任务。本文由浅入深介绍三种任务调度的 Java 实现       在游戏开中有一些任务调试需要程序实现处理，任务调度是指基于给定时间点，或给定时间间隔或者给定执行次数自动执行任务。本文由浅入深介绍三种任务调度的 Java 实现：  • Timer  •
1.2.2.1Job的提交（源码解读）
寒暄的博客
01-29 1022
总目录:https://www.jianshu.com/p/e406a9bc93a9 Hadoop - 子目录：https://www.jianshu.com/p/9428e443b7fd Job的实例化 Configuration con = new Configuration(); //读取...
Apache Hadoop Mapreduce作业执行前篇之任务执行前准备（上）
猫君之上的博客
03-15 649
一.创建Job作业过程 1.获取Job作业对象 方式一： Configuration conf = new Configuration(); Job job=Job.getInstance(conf); public static Job getInstance(Configuration conf) throws IOException { JobConf jobConf =...
MapReduce笔记
yxinzju的专栏
05-20 500
今天在MapReduce程序时遇到个问题,错误是输出类型错误. 原因: Job job = Job.getInstance(conf, "TestPartitioner"); job.setJarByClass(TestRecordReader.class); job.setMapperClass(MyMapper.class); job.setReducerClass(MyRed
深入理解java.lang.IllegalArgumentException异常
热门推荐
博客
08-02 5万+
IllegalArgumentException是Java的一个标准异常类，继承自RuntimeException类。当方法接收到一个非法或不合理的参数时，就会抛出该异常。本篇博客深入探讨了java.lang.IllegalArgumentException异常的相关知识。我们了解了异常的概念和分类，以及异常处理机制。然后，重点介绍了IllegalArgumentException异常的定义、继承关系和触发条件。接着，我们讨论了该异常在方法参数校验、构造方法参数校验和API调用参数校验等常见场景中的应用。
Hadoop - Map/Reduce 通过理解org.apache.hadoop.mapreduce.Job类来学习hadoop的执行逻辑
Derekjiang的笔记簿
10-08 1万+
在我的上篇文章“Hadoop - Map/Reduce 通过WordCount例子的变化来了解新版hadoop接口的变化”中，已经跟大家提到，在Hadoop新的版本中，使用org.apache.hadoop.mapreduce.Job类替代了JobClient类。现在，我想通过对
第5节、Debug调试源码：获取job执行原理
weixin_45702895的博客
12-25 486
1.进入job类源码 （1）、在提交job的位置打断点，然后右键提交 （2）、双击选中waitForCompletion，强制进入，就到了Job类里面 2.在Job类中打断点 1. 在submit();处打断点 2. 进入submit(); 3. 在summit中打断点 以便于调试 3.1 ensureState(JobState.DEFINE); 判断Job的状态 3.2 setUseNewAPI(); 处理新旧API兼容性问题 3.3 强制进入connect(); 选择客户端：YARN或
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值