maven 解决依赖冲突

maven 解决依赖冲突 依赖冲突的表现 ClassNotFound

// class 名字会随着冲突的不同而不同
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tracing.SpanReceiverHost
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 21 more

NoSuchMethod

java.lang.NoSuchMethodError: org.elasticsearch.client.RestHighLevelClient.bulkAsync(Lorg/elasticsearch/action/bulk/BulkRequest;Lorg/elasticsearch/client/RequestOptions;Lorg/elasticsearch/action/ActionListener;)Lorg/elasticsearch/client/Cancellable;
	at org.apache.flink.streaming.connectors.elasticsearch7.Elasticsearch7ApiCallBridge.lambda$createBulkProcessorBuilder
mvn dependency:tree -Dverbose
(Elasticsearch7ApiCallBridge.java:82)
	at org.elasticsearch.action.bulk.Retry$RetryHandler.execute(Retry.java:202)
	at org.elasticsearch.action.bulk.Retry.withBackoff(Retry.java:62)

如何解决冲突 查找依赖冲突的来源

在解决冲突之前，我们需要知道冲突的 Jar 是从哪里引入的

如果在提交作业时，提示「可能有冲突」的 jar，则需要排查项目中哪里依赖了对应的 jar 包
如果是 NoSuchMethod 或者 ClassNotFound 等异常，则根据报错的 class 来查找对应的 jar 包（比如上文中 ClassNotFound 示例中需要查找 SpanReceiverHost 从哪个 jar 中引入，NoSuchMethod 异常示例中，则需要查找 org.elasticsearch.client.RestHighLevelClient 从哪里引入-- 从依赖栈我们知道是 elasticserach7 引入而来）

首先到项目的目录执行如下命令

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Flink-Demo 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ Flink-Demo ---
[INFO] org.example:Flink-Demo:jar:1.0-SNAPSHOT
[INFO] +- org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile
[INFO] |  +- org.apache.flink:flink-annotations:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  |  - (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-metrics-core:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  - (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-shaded-asm:jar:5.0.4-5.0:compile
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] |  +- com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[INFO] |  |  +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
[INFO] |  |  - org.objenesis:objenesis:jar:2.1:compile
[INFO] |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.15:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  - org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile
[INFO] +- org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] |  +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-java:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.flink:flink-shaded-asm:jar:5.0.4-5.0:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.commons:commons-lang3:jar:3.3.2:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.commons:commons-math3:jar:3.5:compile - omitted for duplicate)
[INFO] |  |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  |  - (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-queryable-state-client-java_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  +- (org.apache.flink:flink-shaded-netty:jar:4.1.24.Final-5.0:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile - omitted for duplicate)
[INFO] |  |  - (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-hadoop-fs:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  - (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  +- org.apache.flink:flink-shaded-netty:jar:4.1.24.Final-5.0:compile
[INFO] |  +- org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile
[INFO] |  +- (org.apache.flink:flink-shaded-asm:jar:5.0.4-5.0:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-shaded-jackson:jar:2.7.9-5.0:compile
[INFO] |  +- (org.apache.commons:commons-lang3:jar:3.3.2:compile - omitted for duplicate)
[INFO] |  +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] |  +- org.javassist:javassist:jar:3.19.0-GA:compile
[INFO] |  +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO] |  +- com.typesafe.akka:akka-actor_2.11:jar:2.4.20:compile
[INFO] |  |  +- com.typesafe:config:jar:1.3.0:compile
[INFO] |  |  - org.scala-lang.modules:scala-java8-compat_2.11:jar:0.7.0:compile
[INFO] |  +- com.typesafe.akka:akka-stream_2.11:jar:2.4.20:compile
[INFO] |  |  +- (com.typesafe.akka:akka-actor_2.11:jar:2.4.20:compile - omitted for duplicate)
[INFO] |  |  +- org.reactivestreams:reactive-streams:jar:1.0.0:compile
[INFO] |  |  - com.typesafe:ssl-config-core_2.11:jar:0.2.1:compile
[INFO] |  |     - org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] |  +- com.typesafe.akka:akka-protobuf_2.11:jar:2.4.20:compile
[INFO] |  +- com.typesafe.akka:akka-slf4j_2.11:jar:2.4.20:compile
[INFO] |  |  +- (com.typesafe.akka:akka-actor_2.11:jar:2.4.20:compile - omitted for duplicate)
[INFO] |  |  - (org.slf4j:slf4j-api:jar:1.7.16:compile - omitted for conflict with 1.7.15)
[INFO] |  +- org.clapper:grizzled-slf4j_2.11:jar:1.3.2:compile
[INFO] |  |  - (org.slf4j:slf4j-api:jar:1.7.9:compile - omitted for conflict with 1.7.15)
[INFO] |  +- com.github.scopt:scopt_2.11:jar:3.5.0:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.4:compile
[INFO] |  +- com.twitter:chill_2.11:jar:0.7.6:compile
[INFO] |  |  - com.twitter:chill-java:jar:0.7.6:compile
[INFO] |  +- com.github.oshi:oshi-core:jar:3.4.0:compile
[INFO] |  |  +- net.java.dev.jna:jna-platform:jar:4.2.2:compile
[INFO] |  |  |  - net.java.dev.jna:jna:jar:4.2.2:compile
[INFO] |  |  - (org.slf4j:slf4j-api:jar:1.7.22:compile - omitted for conflict with 1.7.15)
[INFO] |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  - (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] +- org.apache.flink:flink-streaming-java_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] |  +- (org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-clients_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  +- org.apache.flink:flink-optimizer_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  |  +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  |  +- (org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  |  +- (org.apache.flink:flink-java:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  |  +- (org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile - omitted for duplicate)
[INFO] |  |  |  +- (org.apache.flink:flink-shaded-jackson:jar:2.7.9-5.0:compile - omitted for duplicate)
[INFO] |  |  |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  |  |  - (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.flink:flink-java:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  +- (commons-cli:commons-cli:jar:1.3.1:compile - omitted for duplicate)
[INFO] |  |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  |  - (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- (org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile - omitted for duplicate)
[INFO] |  +- org.apache.commons:commons-math3:jar:3.5:compile
[INFO] |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  - (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.14:compile
[INFO] |  +- (org.slf4j:slf4j-api:jar:1.7.14:compile - omitted for conflict with 1.7.15)
[INFO] |  - (log4j:log4j:jar:1.2.17:compile - omitted for duplicate)
[INFO] - log4j:log4j:jar:1.2.17:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.437 s
[INFO] Finished at: 2021-11-25T11:22:25+08:00
[INFO] Final Memory: 13M/309M
[INFO] ------------------------------------------------------------------------

会得到类似如下的结果

org.apache.flink

在上述输出的日志中，打印出项目的依赖树以及版本冲突情况，其中

version managed from xxx, omitted for duplicated：这个依赖和另一个依赖版本冲突了，使用xxx为最终的版本，并忽略这个依赖
omitted for duplicated：这个依赖已经存在了，忽略这个依赖
omitted for conflict with xxx：这个依赖和另一个依赖版本冲突了，使用xxx为最终的版本，并忽略这个依赖

如何解决依赖冲突 将依赖的 scope 改成 provided

   ${flink.version}
   flink-core
   provided
   com.alibaba.occea           # 将 org.apache.flink:flink-core 对应的 jar 改成 provided 的

显示的 exclusive 掉相关依赖

 
    1.2.14
    TDB
    org.slf4j
    
        
            org.apache.maven.plugins
            slf4j-log4j12
        
    

对 class 进行 shade *** 作

 
   ...
   
       3.1.1
       maven-shade-plugin
       shade-plugin
       
          
             package
             shade
             
                org.yaml
             
          
       
       
 	     
 	        
 		       test