Spark on Kubernetes提交任务报错：Expected HTTP 101 response but was '403 Forbidden'

merrily01

于 2019-11-06 15:28:43 发布

阅读量7.7k

点赞数 2

分类专栏：随手记 Spark On Kubernetes 报错记录文章标签： spark kubernetes Expected HTTP 101 response but EKS安全补丁权限错误

本文链接：https://blog.csdn.net/merrily01/article/details/102935841

版权

报错记录同时被 3 个专栏收录

14 篇文章 ¥19.90 ¥99.00

订阅专栏

超级会员免费看

Spark On Kubernetes

9 篇文章 ¥19.90 ¥99.00

订阅专栏

超级会员免费看

随手记

17 篇文章

订阅专栏

在Spark 2.4.3和Kubernetes v1.16.2环境下，提交Spark作业遇到'Expected HTTP 101 response but was '403 Forbidden'错误。该问题是由于EKS安全补丁导致的权限错误。解决方案包括升级Spark到2.4.4以上版本或替换特定jar包。添加`--conf spark.kubernetes.container.image.pullPolicy=Always`以确保使用最新镜像，从而成功解决此问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

环境：

Spark版本： 2.4.3

Kubernetes版本：v1.16.2

问题：

提交spark-submit example.jar 以cluster方式到k8s集群，driver-pod报错如下：

19/11/06 07:06:54 INFO ExecutorPodsAllocator: Going to request 5 executors from Kubernetes.
19/11/06 07:06:54 WARN WatchConnectionManager: Exec Failure: HTTP 403, Status: 403 -
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.conc