[jira] [Commented] (HADOOP-5732) Add SFTP FileSystem

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (HADOOP-5732) Add SFTP FileSystem

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054270#comment-16054270 ]

Sergey Bahchissaraitsev commented on HADOOP-5732:
-------------------------------------------------

I have both configured on Hadoop 2.8.0:
{code:java}
fs.AbstractFileSystem.sftp.impl=org.apache.hadoop.fs.sftp.SFTPFileSystem
fs.sftp.impl=org.apache.hadoop.fs.sftp.SFTPFileSystem
{code}

I get the following error:
{code:java}
org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Seek not supported
        at org.apache.hadoop.fs.sftp.SFTPInputStream.seek(SFTPInputStream.java:66)
        at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:65)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:114)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:181)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
{code}

By looking at the code, it looks like the seek method is unsupported. Not sure what I am missing:
https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java#L66

> Add SFTP FileSystem
> -------------------
>
>                 Key: HADOOP-5732
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5732
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>         Environment: Any environment
>            Reporter: Íñigo Goiri
>            Assignee: ramtin
>            Priority: Minor
>             Fix For: 2.8.0, 3.0.0-alpha1
>
>         Attachments: HADOOP-5732.008.patch, HADOOP-5732.009.patch, HADOOP-5732.010.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> I have implemented a FileSystem that supports SFTP. It uses JSch (http://www.jcraft.com/jsch/) in order to manage SFTP.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]