Spark上下文广播变量抛出java.io.NotSerializableException，尽管其Serializable

我在Ubuntu上运行Spark 2.1 14.04 我想在spark中广播查找变量。的变量的类型的scala.collection.immutable.Map [字符串，为MyObject]Spark上下文广播变量抛出java.io.NotSerializableException，尽管其Serializable

MyObject的具有以下字段

类型字符串类型字符串的
'地址'
'的 '姓名' rangeSet”型com.google.common.collect的。{TreeRangeSet}

Exception in thread "main" java.io.NotSerializableException: com.google.common.collect.TreeRangeSet

Serialization stack: 
    - object not serializable (class: com.google.common.collect.TreeRangeSet, value: {[/101.32.168.0‥/101.32.181.255][/4626:7800:4048:0:0:0:0:0‥/4626:7800:4048:ffff:ffff:ffff:ffff:ffff]}) 
    - field (class: com.test.MyObject, name: rangeSet, type: class com.google.common.collect.TreeRangeSet) 
    - object (class com.test.MyObject, MyObject(Jack,Test,{[/101.32.168.0‥/101.32.181.255][/192.16.10.224‥/192.16.10.255][/4626:7800:4048:0:0:0:0:0‥/4626:7800:4048:ffff:ffff:ffff:ffff:ffff]})) 
    - writeObject data (class: scala.collection.immutable.HashMap$SerializationProxy) 
    - object (class scala.collection.immutable.HashMap$SerializationProxy, [email protected]) 
    - writeReplace data (class: scala.collection.immutable.HashMap$SerializationProxy)

MyObject.scala

import com.google.common.collect.{TreeRangeSet} 
@SerialVersionUID(123L) 
case class MyObject(name:String, address:String,rangeSet:TreeRangeSet[CustomInetAddress]) { 
}

CustomInetAddress.java

public class CustomInetAddress implements Comparable<CustomInetAddress>, Serializable { 

    private InetAddress inetAddress; 

    public CustomInetAddress(String ip) throws UnknownHostException { 
     this.inetAddress = InetAddress.getByName(ip); 
    } 

    public CustomInetAddress(InetAddress address) throws UnknownHostException { 
     this.inetAddress = address; 
    } 

    @Override 
    public int compareTo(final CustomInetAddress address){ 
     byte[] ba1 = this.inetAddress.getAddress(); 
     byte[] ba2 = address.inetAddress.getAddress(); 

     if(ba1.length < ba2.length) return -1; 
     if(ba1.length > ba2.length) return 1; 

     for(int i = 0; i < ba1.length; i++) { 
      int b1 = unsignedByteToInt(ba1[i]); 
      int b2 = unsignedByteToInt(ba2[i]); 
      if(b1 == b2) 
       continue; 
      if(b1 < b2) 
       return -1; 
      else 
       return 1; 
     } 
     return 0; 
    } 

    @Override 
    public String toString(){ 
     return this.inetAddress.toString(); 
    } 

    private int unsignedByteToInt(byte b) { 
     return (int) b & 0xFF; 
    } 
}

TreeRangeSet [CustomInetAddress]是实际的对象的类型。 CustomInetAddress有一个InetAddress类型的字段。所有这些都是可序列化的。我不知道为什么这是抛出异常。

来源

2017-08-01 Satheesh

消息明确：com.google.common.collect.TreeRangeSet未执行Serializable。

来源

2017-08-02 00:41:08 zsxwing

我使用的是22.0版本 https://google.github.io/guava/releases/22.0/api/docs/com/google/common/collect/TreeRangeSet.html – Satheesh

那么这可能是因为有多个番石榴罐在类路径和JVM中选错了一个。你可以看看依赖树吗？ – zsxwing

这是一个有效的点，但我的sbt驱逐所有旧版本，并添加了22.0 我发现问题是与TreeRangeSet类。尽管它实现了可序列化，但我无法腌制它。（在REPL中测试） – Satheesh

Spark上下文广播变量抛出java.io.NotSerializableException，尽管其Serializable

回答

相关问题