spark程序里使用了jackson去做scala对象的json序列化和反序列化。运行时出现java.lang.NoClassDefFoundError和java.lang.AbstractMethodError错误，网上搜了一下发现是jackson/guava和spark本身带的版本冲突。

1.在idea中，通过调整modules中依赖的先后次序，很快解决了问题：

2.发布到Spark集群上，由于某种原因一开始没有用maven来管理项目，所以还是想通过调整对第3方库的依赖次序来解决：

spark-submit --class noc.train.Train --master yarn-client --num-executors 10 --executor-cores 2 --driver-memory 10g --executor-memory 12g --conf spark.yarn.executor.memoryOverhead=2048 \             --conf spark.executor.extraClassPath=./guava-15.0.jar:./jackson-annotations-2.4.4.jar:./jackson-core-2.4.4.jar:./jackson-databind-2.4.4.jar:./jackson-module-scala_2.10-2.4.4.jar \             --conf spark.driver.extraClassPath=/home/ck/lib/guava-15.0.jar:/home/ck/lib/jackson-annotations-2.4.4.jar:/home/ck/lib/jackson-core-2.4.4.jar:/home/ck/lib/jackson-databind-2.4.4.jar:/home/ck/lib/jackson-module-scala_2.10-2.4.4.jar \             --jars /home/ck/lib/guava-15.0.jar,/home/ck/lib/jackson-annotations-2.4.4.jar,/home/ck/lib/jackson-core-2.4.4.jar,/home/ck/lib/jackson-databind-2.4.4.jar,/home/ck/lib/jackson-module-scala_2.10-2.4.4.jar \             /home/ck/noc_scala.jar

但错误依旧：

18/09/12 09:46:47 INFO scheduler.TaskSetManager: Lost task 198.0 in stage 7.0 (TID 897) on executor host-9-136: java.lang.NoClassDefFoundError (Could not initialize class noc.grid.Grid$) [duplicate 1]18/09/12 09:46:47 WARN scheduler.TaskSetManager: Lost task 58.0 in stage 7.0 (TID 890, host-9-136): java.lang.AbstractMethodError: noce.grid.Grid$$anon$1.com$fasterxml$jackson$module$scala$experimental$ScalaObjectMapper$_setter_$com$fasterxml$jackson$module$scala$experimental$ScalaObjectMapper$$typeCache_$eq(Lorg/spark-project/guava/cache/LoadingCache;)V    at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$class.$init$(ScalaObjectMapper.scala:50)

只好求助maven

        
     
      com.fasterxml.jackson.core
         
     
      jackson-core
         
     
      2.4.4
     
    
        
     
      com.fasterxml.jackson.core
         
     
      jackson-databind
         
     
      2.4.4
     
    
        
     
      com.fasterxml.jackson.core
         
     
      jackson-annotations
         
     
      2.4.4
     
    
        
     
      com.fasterxml.jackson.module
         
     
      jackson-module-scala_2.10
         
     
      2.4.4
     
    

然后增加maven-shade-plugin

        
     
      org.apache.maven.plugins
         
     
      maven-shade-plugin
         
     
      3.1.0
         
             
                  
       
        package
                   
                       
        
         shade
                    
                   
                       
                            
          
          
           com.fasterxml.jackson
           
          
           noc.com.fasterxml.jackson
           
                             
          
          
           com.google.guava
           
          
           noc.com.google.guava
           
                         
                    
               
          
     
    

Done！