java - Cannot register pig UDF jar -


i'm having problems particular udf jar , have far been unable figure out begin. have way of testing jar command line , works. if register jar in pig script, pig fails create jar job. can register other jars without trouble , jar working until few days ago. here output when running pig script:

[michael@hadoop01 logitech-correlation]$ pig -f matchwithclassifier.pig -param date=20130301 -param siteid=0 2013-05-10 11:20:30,523 [main] info  org.apache.pig.main - apache pig version 0.10.0-cdh4.1.2 (rexported) compiled nov 01 2012, 18:38:58 2013-05-10 11:20:30,524 [main] info  org.apache.pig.main - logging error messages to: /home/michael/correlation/pig_1368210030521.log 2013-05-10 11:20:30,981 [main] info  org.apache.pig.backend.hadoop.executionengine.hexecutionengine - connecting hadoop file system at: hdfs://hadoop01/ 2013-05-10 11:20:31,346 [main] info  org.apache.pig.backend.hadoop.executionengine.hexecutionengine - connecting map-reduce job tracker at: hadoop01.dev.terapeak.com:8021 2013-05-10 11:20:32,143 [main] info  org.apache.pig.tools.pigstats.scriptstate - pig features used in script: filter 2013-05-10 11:20:32,390 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mrcompiler - file concatenation threshold: 100 optimistic? false 2013-05-10 11:20:32,422 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.multiqueryoptimizer - mr plan size before optimization: 1 2013-05-10 11:20:32,422 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.multiqueryoptimizer - mr plan size after optimization: 1 2013-05-10 11:20:32,508 [main] info  org.apache.pig.tools.pigstats.scriptstate - pig script settings added job 2013-05-10 11:20:32,518 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler - mapred.job.reduce.markreset.buffer.percent not set, set default 0.3 2013-05-10 11:20:32,522 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler - creating jar file job5623238576559565298.jar 2013-05-10 11:20:36,398 [main] error org.apache.pig.tools.grunt.grunt - error 2017: internal error creating job configuration. details @ logfile: /home/michael/correlation/pig_1368210030521.log

the stack trace below:

pig stack trace --------------- error 2017: internal error creating job configuration.  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcreationexception: error 2017: internal error creating job configuration.   @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler.getjob(jobcontrolcompiler.java:727)   @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler.compile(jobcontrolcompiler.java:259)   @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher.launchpig(mapreducelauncher.java:180)   @ org.apache.pig.pigserver.launchplan(pigserver.java:1275)   @ org.apache.pig.pigserver.executecompiledlogicalplan(pigserver.java:1260)   @ org.apache.pig.pigserver.execute(pigserver.java:1250)   @ org.apache.pig.pigserver.executebatch(pigserver.java:362)   @ org.apache.pig.tools.grunt.gruntparser.executebatch(gruntparser.java:132)   @ org.apache.pig.tools.grunt.gruntparser.parsestoponerror(gruntparser.java:193)   @ org.apache.pig.tools.grunt.gruntparser.parsestoponerror(gruntparser.java:165)   @ org.apache.pig.tools.grunt.grunt.exec(grunt.java:84)   @ org.apache.pig.main.run(main.java:430)   @ org.apache.pig.main.main(main.java:111)   @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)   @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57)   @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)   @ java.lang.reflect.method.invoke(method.java:616)   @ org.apache.hadoop.util.runjar.main(runjar.java:208) caused by: java.util.zip.zipexception: invalid distance far   @ java.util.zip.inflaterinputstream.read(inflaterinputstream.java:164)   @ java.util.zip.zipinputstream.read(zipinputstream.java:163)   @ java.util.jar.jarinputstream.read(jarinputstream.java:194)   @ java.io.filterinputstream.read(filterinputstream.java:107)   @ org.apache.pig.impl.util.jarmanager.addstream(jarmanager.java:242)   @ org.apache.pig.impl.util.jarmanager.mergejar(jarmanager.java:216)   @ org.apache.pig.impl.util.jarmanager.mergejar(jarmanager.java:206)   @ org.apache.pig.impl.util.jarmanager.createjar(jarmanager.java:126)   @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.jobcontrolcompiler.getjob(jobcontrolcompiler.java:411)   ... 17 more ================================================================================

based on think problem "java.util.zip.zipexception: invalid distance far back" exception. pig having issue reading jar?


Comments

Post a Comment

Popular posts from this blog

c# - Operator '==' incompatible with operand types 'Guid' and 'Guid' using DynamicExpression.ParseLambda<T, bool> -

i'm using dynamic linq library , there source code , basic docu , nuget version pm> install-package dynamiclinq i'm trying construct clause involves guids i have tried string "id == @0" . parameter array object[] value ( guid xxxx ) var whereclausesb = buildlogicalkeywhereclause2(entity, logicalkey); //build string var parms = buildparamarray(entity, logicalkey); // object[] var wherelambda = ofsi.bos.core.dynamicexpression.parselambda<t, bool>(whereclausesb.tostring(),parms); //parse an exception thrown in dynamicexpression.parselambda operator '==' incompatible operand types 'guid' , 'guid' i have tried guid , string.(fail) i tried , "id = @0" (fail). string == string works, int32==int32 not guid == guid not any ideas? try using equals method instead of == operator in string: "id.equals(@0)"
Read more