Class org.apache.hadoop.fs.s3a.S3A not found

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Class org.apache.hadoop.fs.s3a.S3A not found

cs user
Hello, 

Can someone tell me which version of Hadoop I should be able to find this class in?

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Class org.apache.hadoop.fs.s3a.S3A not found

Mingliang Liu-2
You may need the hadoop-aws-version.jar (and aws-java-sdk.jar) in your classpath.


Good luck,

L

On Dec 16, 2016, at 1:24 PM, cs user <[hidden email]> wrote:

Hello, 

Can someone tell me which version of Hadoop I should be able to find this class in?

Thanks!

Reply | Threaded
Open this post in threaded view
|

Re: Class org.apache.hadoop.fs.s3a.S3A not found

cs user
Hello, 

Thank you both for getting back to me so quickly,

Can you explain how I can get this to work with the latest version? I'm trying to submit a spark job with the following ( With the hadoop 3 release). 

Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages" "com.amazonaws:aws-java-sdk-pom:1.10.34,org.apache.hadoop:hadoop-aws:3.0.0-alpha1" "sparkr-shell"',

But then I get:

16/12/16 21:47:56 ERROR RBackendHandler: createSparkContext on org.apache.spark.api.r.RRDD failed
Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
  java.lang.IllegalAccessError: tried to access method org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(Lorg/apache/hadoop/metrics2/MetricsInfo;J)V from class org.apache.hadoop.fs.s3a.S3AInstrumentation


How can I set my environment to pull down the hadoop 2.8 release?

Thank you again for your help. 




On Fri, Dec 16, 2016 at 9:33 PM, Mingliang Liu <[hidden email]> wrote:
You may need the hadoop-aws-version.jar (and aws-java-sdk.jar) in your classpath.


Good luck,

L

On Dec 16, 2016, at 1:24 PM, cs user <[hidden email]> wrote:

Hello, 

Can someone tell me which version of Hadoop I should be able to find this class in?

Thanks!


Reply | Threaded
Open this post in threaded view
|

Re: Class org.apache.hadoop.fs.s3a.S3A not found

Ted Yu-3
MutableCounterLong is in hadoop-common jar.

Can you check the classpath to see if there is another hadoop-common jar (other than 3.0-alpha1) ?

Cheers

On Fri, Dec 16, 2016 at 1:49 PM, cs user <[hidden email]> wrote:
Hello, 

Thank you both for getting back to me so quickly,

Can you explain how I can get this to work with the latest version? I'm trying to submit a spark job with the following ( With the hadoop 3 release). 

Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages" "com.amazonaws:aws-java-sdk-pom:1.10.34,org.apache.hadoop:hadoop-aws:3.0.0-alpha1" "sparkr-shell"',

But then I get:

16/12/16 21:47:56 ERROR RBackendHandler: createSparkContext on org.apache.spark.api.r.RRDD failed
Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
  java.lang.IllegalAccessError: tried to access method org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(Lorg/apache/hadoop/metrics2/MetricsInfo;J)V from class org.apache.hadoop.fs.s3a.S3AInstrumentation


How can I set my environment to pull down the hadoop 2.8 release?

Thank you again for your help.