在Pig中运行UDF时出错

Question

我试图让一个UDF在猪中运行，但是我正面临着一些问题，当我尝试运行猪脚本时，它说错误无法实例化mathPow值为null的if任何机构都可以提供帮助，这会很棒。

由于

猪脚本如下：

REGISTER MathPower.jar 
A = load ‘input’ using PigStorage(‘,’); 
C = foreach A generate $0 as x, $1 as z; 
B = foreach A generate powUDF.mathUDF(x, z); 
dump B; 

输入文件包含： 2,3 4,5-

的Java是如下，NO exteranl librarbies有已被添加，我只是遵循一个教程。我使用的Java版本1.6和Eclipse：

package powUDF; 

import java.io.IOException; 
import org.apache.pig.EvalFunc; 
import org.apache.pig.data.Tuple; 
import org.apache.commons.logging.Log; 
import org.apache.*; 


public class mathUDF extends EvalFunc<Long> { 

/** 
* A simple UDF that takes a value and raises it to the power of a second 
* value. It can be used in a Pig Latin script as Pow(x, y), where x and y 
* are both expected to be ints. 
*/ 


public Long exec(Tuple input) throws IOException { 
try { 
    /* Rather than give you explicit arguments, UDFs are always handed 
     * a tuple. The UDF must know the arguments it expects and pull 
     * them out of the tuple. These next two lines get the first and 
     * second fields out of the input tuple that was handed in. Since 
     * Tuple.get returns Objects, we must cast them to Integers. If 
     * the case fails, an exception will be thrown. 
     */ 
    int base = (Integer)input.get(0); 
    int exponent = (Integer)input.get(1); 
    long result = 1; 

    /* Probably not the most efficient method...*/ 
    for (int i = 0; i < exponent; i++) { 
     long preresult = result; 
     result *= base; 
     if (preresult > result) { 
      // We overflowed. Give a warning, but do not throw an 
      // exception. 
      warn("Overflow!", PigWarning.TOO_LARGE_FOR_INT); 
      // Returning null will indicate to Pig that we failed but 
      // we want to continue execution. 
      return null; 
     } 
    } 
    return result; 
} catch (Exception e) { 
    // Throwing an exception will cause the task to fail. 
    throw new IOException("Something bad happened!", e); 
} 

}}

堆栈跟踪为

Pig Stack Trace 

---

错误1200：不能实例化 'powUDF2.mathUDF' 与参数'null'

Failed to parse: could not instantiate 'powUDF2.mathUDF' with arguments 'null' 
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193) 
    at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1571) 
    at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1544) 
    at org.apache.pig.PigServer.registerQuery(PigServer.java:516) 
    at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:991) 
    at  org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:412) 
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194) 
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170) 
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) 
    at org.apache.pig.Main.run(Main.java:538) 
    at org.apache.pig.Main.main(Main.java:157) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.apache.hadoop.util.RunJar.main(RunJar.java:160) 
Caused by: java.lang.RuntimeException: could not instantiate 'powUDF2.mathUDF' with arguments 'null' 
    at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:618) 
    at org.apache.pig.newplan.logical.expression.UserFuncExpression.getFieldSchema(UserFuncExpression.java:193) 
    at org.apache.pig.newplan.logical.optimizer.FieldSchemaResetter.execute(SchemaResetter.java:264) 
    at org.apache.pig.newplan.logical.expression.AllSameExpressionVisitor.visit(AllSameExpressionVisitor.java:143) 
    at org.apache.pig.newplan.logical.expression.UserFuncExpression.accept(UserFuncExpression.java:88) 
    at org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70) 
    at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) 
    at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visitAll(SchemaResetter.java:67) 
    at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:122) 
    at org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:246) 
    at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75) 
    at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114) 
    at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:76) 
    at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75) 
    at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) 
    at org.apache.pig.parser.LogicalPlanBuilder.expandAndResetVisitor(LogicalPlanBuilder.java:392) 
    at org.apache.pig.parser.LogicalPlanBuilder.buildForeachOp(LogicalPlanBuilder.java:924) 
    at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:14195) 
    at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1623) 
    at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799) 
    at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517) 
    at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392) 
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184) 
    ... 15 more 
Caused by: java.lang.Error: Unresolved compilation problems: 
    The type org.apache.commons.logging.Log cannot be resolved. It is indirectly referenced from required .class files 
    The import org.apache.commons.logging.Log cannot be resolved 
    The type org.apache.hadoop.io.WritableComparable cannot be resolved. It is indirectly referenced from required .class files 
    PigWarning cannot be resolved to a variable 

    at powUDF2.mathUDF.<init>(mathUDF.java:1) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513) 
    at java.lang.Class.newInstance0(Class.java:355) 
    at java.lang.Class.newInstance(Class.java:308) 
    at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:588) 
    ... 37 more 
================================================================================ 

Answer 1

当你建立你的jar时，你必须包含你的类所引用的所有必需的库。我使用Ant构建，以确保像这样的依赖性得到正确管理。尝试运行

jar -tf MathPower.jar 

并查看您是否在任何地方看到类org/apache/commons/logging/Log。您的UDF会导入它，但Pig无法找到它，如堆栈跟踪结束所示。同样，您似乎缺少与Hadoop交互所需的类：

The type org.apache.hadoop.io.WritableComparable cannot be resolved. 

请确保该类也包含在您构建的jar中。或者，您也可以使用包含要引用的类的jar，但我不确定这是否可行。