springboot2.x,3.x打jar包在spark yarn 集群上部署运行

已于 2023-04-07 09:48:06 修改
阅读量2.8k 收藏 12
点赞数 2
CC 4.0 BY-SA版权
分类专栏： spark springboot 文章标签： spark jar spring boot
于 2022-02-22 10:51:38 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/poxiaomeng187/article/details/123050445

环境：

<java.version>1.8/17</java.version>
<scala.version>2.12.15</scala.version>
<scala.binary.version>2.12</scala.binary.version>
<spark.version>3.1.2/3.3.2</spark.version>
<janino.version>3.0.8/3.0.16</janino.version>
<hadoop.version>3.3.0</hadoop.version>
<scalikejdbc.version>2.5.2</scalikejdbc.version>
<fasterxml.jackson.version>2.10.5/2.14.2</fasterxml.jackson.version>

执行脚本

/home/software/spark-3.1.2/bin/spark-submit --class cnki.bdms.servicespark.BdcServiceSparkApplication \
	--conf spark.yarn.jars="hdfs://0.0.0.0:8020/spark/apps/jars/*,hdfs://0.0.0.0:8020/lib/*" \
	--driver-java-options "-Dorg.springframework.boot.logging.LoggingSystem=none -Dspring.profiles.active=test -Dspark.yarn.dist.files=/home/software/hadoop-3.3.0/etc/hadoop/yarn-site.xml" \
    --master yarn \
    --deploy-mode cluster \
    --driver-memory 4g \
    --executor-memory 2g \
    --executor-cores 1 \
    --queue default \
    /data/jar/bdcServiceSpark-1.0.0.jar

springboot程序

@SpringBootApplication(exclude= {DataSourceAutoConfiguration.class,QuartzAutoConfiguration.class, MongoAutoConfiguration.class, ElasticsearchDataAutoConfiguration.class, SecurityAutoConfiguration.class, RedisAutoConfiguration.class})
public class BdcServiceSparkApplication implements CommandLineRunner {
    public static void main(String[] args) {
        if(args==null||args.length==0){
            args=new String[]{"1495"};
//            System.out.println("缺少资源id,程序退出");
//            return;
        }
        SpringApplication.run(BdcServiceSparkApplication.class, args);
    }

    @Override
    public void run(String... args) throws Exception {
        System.out.println("==================程序开始====================");
        //应用程序的逻辑，这是是利用反射执行对应的服务类的方法
        WorkFlowDirectorServiceImpl flowDirectorService = SpringUtils.getBean(WorkFlowDirectorServiceImpl.class);
        int dataAnalyseId=Integer.parseInt(args[0]);
        flowDirectorService.direct(dataAnalyseId);
        SpringApplication.exit(SpringUtils.getApplicationContext());
        System.out.println("==================程序结束====================");
    }
}

打包

spark不支持使用spring-boot-maven-plugin打包的springboot项目结构，要改成

maven-shade-plugin 打包

<!--原来的构建方式-->
<!--    <build>-->
<!--        <plugins>-->
<!--            <plugin>-->
<!--                <groupId>org.springframework.boot</groupId>-->
<!--                <artifactId>spring-boot-maven-plugin</artifactId>-->
<!--                <configuration>-->
<!--                    <includeSystemScope>true</includeSystemScope>-->
<!--                </configuration>-->
<!--            </plugin>-->
<!--        </plugins>-->
<!--    </build>-->

<!--现在的构建方式-->
 <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <dependencies>
                    <dependency>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-maven-plugin</artifactId>
                        <version>2.4.11</version>
                    </dependency>
                </dependencies>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                    <transformers>
                        <transformer
                                implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                            <resource>META-INF/spring.handlers</resource>
                        </transformer>
                        <transformer
                                implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                            <resource>META-INF/spring.factories</resource>
                        </transformer>
                        <transformer
                                implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                            <resource>META-INF/spring.schemas</resource>
                        </transformer>
                        <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                        <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                            <mainClass>cnki.bdms.servicespark.BdcServiceSparkApplication</mainClass>
                        </transformer>
                    </transformers>
                </configuration>
            </plugin>
        </plugins>
    </build>

 

 这一块写成自己的启动类

版本冲突问题

1、 snakeyaml.jar冲突，保留springboot的版本，直接把spark.yarn.jars这个路径上的删除掉，保留最新版本

2、gson-2.4.jar包冲突，改成和springboot的版本一致的，比如我的是gson-2.7.jar，直接把spark.yarn.jars这个路径上的删除掉，换成gson-2.7.jar

3. guava-14.0.1.jar冲突 提示java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

升级到guava-20.0.jar，将guava-20.0.jar替换到spark.yarn.jars路径

4、jackson-databind版本太高，提示Scala moudle 2.10.0 require Jackson Databind version >=2.10.0 and <2.11.0 ,我这边环境里spark.yarn.jars里的是2.10.5，springboot里用的是2.11.4，采取方案是直接降级springboot里的jackson-databind

先排除所有的

<groupId>org.apache.hadoop</groupId> 下的jackson,如

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>${hadoop.version}</version>
    <exclusions>
        <exclusion>
            <groupId>com.fasterxml.jackson.module</groupId>
            <artifactId>*</artifactId>
        </exclusion>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<artifactId>一共5个，

hadoop-common hadoop-hdfs hadoop-mapreduce-client-core hadoop-yarn-common hadoop-client
排除所有<groupId>org.apache.spark</groupId>下的jackson
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_${scala.binary.version}</artifactId>
    <version>${spark.version}</version>
    <exclusions>
        <exclusion>
            <groupId>com.fasterxml.jackson.module</groupId>
            <artifactId>*</artifactId>
        </exclusion>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<artifactId>一共3个，spark-core,spark-sql,spark-yarn

具体的看自己项目的依赖，降级处理，然后添加依赖

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.fasterxml.jackson.dataformat</groupId>
                <artifactId>jackson-dataformat-yaml</artifactId>
                <version>2.10.5</version>
            </dependency>
            <dependency>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>jackson-databind</artifactId>
                <version>2.10.5</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

 也可以直接使用 dependencyManagement 控制版本这样就不用上面的挨个排除了

Maven 可以通过 dependencyManagement 元素对依赖进行管理，它具有以下 2 大特性：

  • 在该元素下声明的依赖不会实际引入到模块中，只有在 dependencies 元素下同样声明了该依赖，才会引入到模块中。
  • 该元素能够约束 dependencies 下依赖的使用，即 dependencies 声明的依赖若未指定版本，则使用 dependencyManagement 中指定的版本，否则将覆盖 dependencyManagement 中的版本。
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>jackson-core</artifactId>
                <version>${fasterxml.jackson.version}</version>
            </dependency>
            <dependency>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>jackson-databind</artifactId>
                <version>${fasterxml.jackson.version}</version>
            </dependency>
            <dependency>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>jackson-annotations</artifactId>
                <version>${fasterxml.jackson.version}</version>
            </dependency>
            <!-- Jackson module for scala object marshalling and unmarshalling -->
            <dependency>
                <groupId>com.fasterxml.jackson.module</groupId>
                <artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
                <version>${fasterxml.jackson.version}</version>
            </dependency>

            <dependency>
                <groupId>com.fasterxml.jackson.jaxrs</groupId>
                <artifactId>jackson-jaxrs-base</artifactId>
                <version>${fasterxml.jackson.version}</version>
            </dependency>
            <dependency>
                <groupId>com.fasterxml.jackson.jaxrs</groupId>
                <artifactId>jackson-jaxrs-json-provider</artifactId>
                <version>${fasterxml.jackson.version}</version>
            </dependency>
            <dependency>
                <groupId>com.fasterxml.jackson.module</groupId>
                <artifactId>jackson-module-jaxb-annotations</artifactId>
                <version>${fasterxml.jackson.version}</version>
            </dependency>
            <dependency>
                <groupId>com.fasterxml.jackson.module</groupId>
                <artifactId>jackson-module-paranamer</artifactId>
                <version>${fasterxml.jackson.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

2023-04-07

springboot3出来了，我们就是勇于尝试最新的，今天是基于springboot3.0.4版本

本次更新版本主要冲突的是日志，上面的打包也需要加点东西，主要是给<execution>加了个id,不然打不了包<id>springboot-spark</id> ，会报一个错Cannot find 'resource' in class org.apache.maven.plugins.shade.resource.ServicesResourceTransformer   或者       

 Cannot find 'resource' in class org.apache.maven.plugins.shade.resource.ManifestResourceTransformer

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <configuration>
                    <keepDependenciesWithProvidedScope>true</keepDependenciesWithProvidedScope>
                    <createDependencyReducedPom>true</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
                <executions>
                    <execution>
                        <id>springboot-spark</id>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/spring.handlers</resource>
                                </transformer>
                                <transformer
                                        implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                                    <resource>META-INF/spring.factories</resource>
                                </transformer>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/spring.schemas</resource>
                                </transformer>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>${start-class}</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

当你打完包后 使用spark-submit提交时会告诉你日志包冲突

Correct the classpath of your application so that it contains compatible versions of the classes org.springframework.boot.logging.log4j2.Log4J2LoggingSystem and org.apache.logging.log4j.util.PropertiesUtil

 

这个时候你肯定想到了方案，要么升级spark/jars下面的log4j日志包，要么降级springboot3的日志包，经过我实践证明，降级springboot3的日志版本不行，会提示你找不到一个什么日志类java.lang.NoClassDefFoundError，咱也不去深究了，本来降级就不可取

那只好升级spark 日志版本了，spark3.3.2版本log4j使用的是2.17，而spingboot使用的是2.19

这些删掉（自己做备份）

换成（可以使用springboot本身打包方式打成jar包，然后把里面的log4j包拿出来）

 这里还不行,还有个slf4j-api-1.7.32.jar要换成slf4j-api-2.0.6.jar

参考文章：

https://blog.csdn.net/hzs33/article/details/83183217

https://blog.csdn.net/laksdbaksjfgba/article/details/86023892

https://blog.csdn.net/suwei825/article/details/119236798

评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值