`hadoop-aws` reading S3 objects using S3 metadata content header / content-encoding

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

`hadoop-aws` reading S3 objects using S3 metadata content header / content-encoding

Colin Williams
Hello,

I'm reading via s3a objects with key "something.json" which are gzip
encoded with the content-encoding set as gzip. However spark on yarn
doesn't seem to interpret the content-encoding header and so not
reading via spark.read.json . Does anybody know how I can get spark to
read based on the content-encoding metadata via s3a?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]