1. 首页
  2. 问答
  3. Hive给出SemanticException [Error 10014]：运行我的UDF时

Hive给出SemanticException [Error 10014]：运行我的UDF时

2017-04-14 1193 views
1

我有一个配置单元UDF，它执行GeoIP查找。Hive给出SemanticException [Error 10014]：运行我的UDF时

public static Text evaluate(Text inputFieldName, Text option, 
     Text databaseFileName) { 

    String inputField, fieldOption, dbFileName, result = null; 

    inputField = inputFieldName.toString(); 
    fieldOption = option.toString(); 
    dbFileName = databaseFileName.toString(); 
    ExtractData eed = new ExtractData(); 
    try { 
     result = eed.ExtractDB(inputField, fieldOption, 
       dbFileName); 
    } catch (IOException e) { 
     e.printStackTrace(); 
    } catch (GeoIp2Exception e) { 
     e.printStackTrace(); 
    } 
    return new Text(result); 

}

然后我做了一个罐子出来的这一点，跑在蜂巢以下CLI

add jar /location_of_jar/MyUDF.jar; 
add file /user/riyan/GeoIP2-Enterprise.mmdb; 
create temporary function samplefunction as 'com.package.name.App'; 
select samplefunction('172.73.14.54','country_name','/user/riyan/GeoIP2-Enterprise.mmdb') AS result;

我经过GeoIP2-Enterprise.mmdb数据库的位置到UDF。它在我的本地系统上正常工作。但是，当我做罐子了出来，并在命令行中运行它，它给了我一个错误说

FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments ''/user/riyan/GeoIP2-Enterprise.mmdb'': org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public static org.apache.hadoop.io.Text com.package.name.App.evaluate(org.apache.hadoop.io.Text,org.apache.hadoop.io.Text,org.apache.hadoop.io.Text) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:org.apache.hadoop.io.Text, country_name:org.apache.hadoop.io.Text, /user/riyan/GeoIP.mmdb:org.apache.hadoop.io.Text} of size 3

我也试图将参数从文本更改为字符串，并且给了我同样的异常。 有人能告诉我我做错了什么吗？ 感谢

编辑：添加以下部分

我跑了它在蜂巢调试模式，并根据您的answre得到这个

FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments ''./GeoIP2-Enterprise.mmdb'': org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String com.package.name.App.evaluate(java.lang.String,java.lang.String,java.lang.String) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:java.lang.String, countryCode:java.lang.String, ./GeoIP2-Enterprise.mmdb:java.lang.String} of size 3 
17/04/18 11:02:30 [main]: ERROR ql.Driver: FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments ''./GeoIP2-Enterprise.mmdb'': org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String com.bankofamerica.gisds.App.evaluate(java.lang.String,java.lang.String,java.lang.String) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:java.lang.String, countryCode:java.lang.String, ./GeoIP2-Enterprise.mmdb:java.lang.String} of size 3 
org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:7 Wrong arguments ''./GeoIP2-Enterprise.mmdb'': org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String com.package.name.App.evaluate(java.lang.String,java.lang.String,java.lang.String) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:java.lang.String, countryCode:java.lang.String, ./GeoIP2-Enterprise.mmdb:java.lang.String} of size 3 
     at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1184) 
     at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) 
     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) 
     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) 
     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132) 
     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) 
     at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:193) 
     at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:146) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10422) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10378) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3771) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3550) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8830) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8785) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9652) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9545) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10018) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10029) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9909) 
     at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223) 
     at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:488) 
     at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1274) 
     at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1391) 
     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1203) 
     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1193) 
     at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:220) 
     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172) 
     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:383) 
     at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:775) 
     at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:693) 
     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:628) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:606) 
     at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
     at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 
Caused by: org.apache.hadoop.hive.ql.exec.UDFArgumentException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String com.package.name.App.evaluate(java.lang.String,java.lang.String,java.lang.String) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:java.lang.String, countryCode:java.lang.String, ./GeoIP2-Enterprise.mmdb:java.lang.String} of size 3 
     at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:171) 
     at org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:233) 
     at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:959) 
     at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1176) 
     ... 36 more 
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String com.package.name.App.evaluate(java.lang.String,java.lang.String,java.lang.String) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:java.lang.String, countryCode:java.lang.String, ./GeoIP2-Enterprise.mmdb:java.lang.String} of size 3 
     at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:978) 
     at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:182) 
     at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:168) 
     ... 39 more 
Caused by: java.lang.reflect.InvocationTargetException 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:606) 
     at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:954) 
     ... 41 more 
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ObjectNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/Map;)V 
     at com.maxmind.db.Decoder.decodeMap(Decoder.java:285) 
     at com.maxmind.db.Decoder.decodeByType(Decoder.java:154) 
     at com.maxmind.db.Decoder.decode(Decoder.java:147) 
     at com.maxmind.db.Decoder.decodeMap(Decoder.java:281) 
     at com.maxmind.db.Decoder.decodeByType(Decoder.java:154) 
     at com.maxmind.db.Decoder.decode(Decoder.java:147) 
     at com.maxmind.db.Decoder.decode(Decoder.java:87) 
     at com.maxmind.db.Reader.<init>(Reader.java:132) 
     at com.maxmind.db.Reader.<init>(Reader.java:116) 
     at com.maxmind.geoip2.DatabaseReader.<init>(DatabaseReader.java:35) 
     at com.maxmind.geoip2.DatabaseReader.<init>(DatabaseReader.java:23) 
     at com.maxmind.geoip2.DatabaseReader$Builder.build(DatabaseReader.java:129) 
     at com.bankofamerica.gisds.ExtractEnterpriseData.ExtractEnterpriseDB(ExtractEnterpriseData.java:27) 
     at com.package.name.App.evaluate(App.java:73) 
     ... 46 more

来源

2017-04-14 Riyan Mohammed

+0

这个问题解决了吗？ –

回答

1

，它看起来像你缺少一些依赖你JAR文件。你如何编译包含UDF的项目？

可能缺少这一个在蜂房类路径

<dependency> 
    <groupId>com.fasterxml.jackson.core</groupId> 
    <artifactId>jackson-databind</artifactId> 
    <version>2.1.4</version> 
</dependency>

随着工作的周围，你可以尝试使用具有依赖性的罐子（不是一个好的做法这种情况下）编译它，但至少我们知道，如果这是您的问题

<build> 
    <plugins> 
    <plugin> 
     <artifactId>maven-assembly-plugin</artifactId> 
     <configuration> 
     <archive> 
      <manifest> 
      </manifest> 
     </archive> 
     <descriptorRefs> 
      <descriptorRef>jar-with-dependencies</descriptorRef> 
     </descriptorRefs> 
     </configuration> 
    </plugin> 
    </plugins> 
</build>

的另一种选择是加入这个依赖蜂房类路径，然后再试一次

https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind/2.1.4

来源

2017-04-14 19:21:18 hlagos

+0

没有工作....它仍然给我同样的错误...！ –

+0

它执行你的代码吗？取决于配置单元版本的错误可能不是很清楚，你可以尝试一些虚拟逻辑，如返回一个参数？ – hlagos

+0

我添加了一个新的虚拟评估函数，它只返回一个字符串。它的工作正常。 实际的评估函数在本地机器上运行良好...如果我在集群上运行，它给了我上述错误 –

0

失败的真正原因是由于下面的错误。

所致：java.lang.NoSuchMethodError：com.fasterxml.jackson.databind.node.ObjectNode（LCOM/fasterxml /杰克逊/数据绑定/节点/ JsonNodeFactory; Ljava/util的/地图;）V

看起来你可能在类路径中存在冲突的jar版本。

add -verbose：在启动它之前添加-verbose：hs2类的jvm选项，然后查看哪个jar被拾取 - 如果旧的或者udf不需要，则移除该jar。

来源

2017-12-10 08:46:04

相关问题

 相关问题