java code - parsing string -outofindex error -File on HDFS

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

java code - parsing string -outofindex error -File on HDFS

dgoker

When i try to read a file from HDFS, i want to substring the record by java
code (javada line.substring(1,9)) but i am getting outofindex error. When i
do the same operation on same file by PIG code there is no problem.

I took following error, What shoul i do in Java.


java.lang.StringIndexOutOfBoundsException: String index out of range: 9
        at java.lang.String.substring(String.java:1934)
        at WordCount$Map.map(WordCount.java:19)
        at WordCount$Map.map(WordCount.java:12)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)

--
View this message in context: http://old.nabble.com/java-code---parsing-string--outofindex--error--File-on-HDFS-tp26322437p26322437.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

ETL using Hadoop ???

Palikala, Rajendra (CCL)
Hi All,

I am an experienced Informatica and Java Developer. I am very new to Hadoop. I have one basic question for the group. Please advise.

Can we do ETL (Example: load data from an oracle table to another oracle table) using Hadoop? If so, how can we implement that.

From my understanding, I see that Hive is for reporting purposes and not for ETL. Let me know if this is right.

Regards,
Rajendra

Reply | Threaded
Open this post in threaded view
|

Re: ETL using Hadoop ???

Dhruba Borthakur
Hi Rajendra,

We use Hive for a large data warehouse, details here:
http://wiki.apache.org/hadoop/Hive

thanks,
dhruba


On Thu, Nov 12, 2009 at 9:55 AM, Palikala, Rajendra (CCL) <
[hidden email]> wrote:

> Hi All,
>
> I am an experienced Informatica and Java Developer. I am very new to
> Hadoop. I have one basic question for the group. Please advise.
>
> Can we do ETL (Example: load data from an oracle table to another oracle
> table) using Hadoop? If so, how can we implement that.
>
> From my understanding, I see that Hive is for reporting purposes and not
> for ETL. Let me know if this is right.
>
> Regards,
> Rajendra
>
>


--
Connect to me at http://www.facebook.com/dhruba
Reply | Threaded
Open this post in threaded view
|

Re: ETL using Hadoop ???

Jeff Hammerbacher
Hey Rajendra,

For the "E" part of your task, you may want to check out Sqoop:
http://www.cloudera.com/hadoop-sqoop.

Regards,
Jeff

On Thu, Nov 12, 2009 at 10:02 AM, Dhruba Borthakur <[hidden email]> wrote:

> Hi Rajendra,
>
> We use Hive for a large data warehouse, details here:
> http://wiki.apache.org/hadoop/Hive
>
> thanks,
> dhruba
>
>
> On Thu, Nov 12, 2009 at 9:55 AM, Palikala, Rajendra (CCL) <
> [hidden email]> wrote:
>
> > Hi All,
> >
> > I am an experienced Informatica and Java Developer. I am very new to
> > Hadoop. I have one basic question for the group. Please advise.
> >
> > Can we do ETL (Example: load data from an oracle table to another oracle
> > table) using Hadoop? If so, how can we implement that.
> >
> > From my understanding, I see that Hive is for reporting purposes and not
> > for ETL. Let me know if this is right.
> >
> > Regards,
> > Rajendra
> >
> >
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
Reply | Threaded
Open this post in threaded view
|

RE: ETL using Hadoop ???

Palikala, Rajendra (CCL)
 
Jeff, This is great info. Thanks
-----Original Message-----
From: Jeff Hammerbacher [mailto:[hidden email]]
Sent: Thursday, November 12, 2009 4:16 PM
To: [hidden email]
Subject: Re: ETL using Hadoop ???

Hey Rajendra,

For the "E" part of your task, you may want to check out Sqoop:
http://www.cloudera.com/hadoop-sqoop.

Regards,
Jeff

On Thu, Nov 12, 2009 at 10:02 AM, Dhruba Borthakur <[hidden email]> wrote:

> Hi Rajendra,
>
> We use Hive for a large data warehouse, details here:
> http://wiki.apache.org/hadoop/Hive
>
> thanks,
> dhruba
>
>
> On Thu, Nov 12, 2009 at 9:55 AM, Palikala, Rajendra (CCL) <
> [hidden email]> wrote:
>
> > Hi All,
> >
> > I am an experienced Informatica and Java Developer. I am very new to
> > Hadoop. I have one basic question for the group. Please advise.
> >
> > Can we do ETL (Example: load data from an oracle table to another oracle
> > table) using Hadoop? If so, how can we implement that.
> >
> > From my understanding, I see that Hive is for reporting purposes and not
> > for ETL. Let me know if this is right.
> >
> > Regards,
> > Rajendra
> >
> >
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
Reply | Threaded
Open this post in threaded view
|

Re: ETL using Hadoop ???

Steve Morin
In reply to this post by Jeff Hammerbacher
Does sqoop also allow you to export data from hadoop to a db?  I
didn't see that in the documentation

-Steve

On Thu, Nov 12, 2009 at 1:16 PM, Jeff Hammerbacher <[hidden email]> wrote:

> Hey Rajendra,
>
> For the "E" part of your task, you may want to check out Sqoop:
> http://www.cloudera.com/hadoop-sqoop.
>
> Regards,
> Jeff
>
> On Thu, Nov 12, 2009 at 10:02 AM, Dhruba Borthakur <[hidden email]> wrote:
>
>> Hi Rajendra,
>>
>> We use Hive for a large data warehouse, details here:
>> http://wiki.apache.org/hadoop/Hive
>>
>> thanks,
>> dhruba
>>
>>
>> On Thu, Nov 12, 2009 at 9:55 AM, Palikala, Rajendra (CCL) <
>> [hidden email]> wrote:
>>
>> > Hi All,
>> >
>> > I am an experienced Informatica and Java Developer. I am very new to
>> > Hadoop. I have one basic question for the group. Please advise.
>> >
>> > Can we do ETL (Example: load data from an oracle table to another oracle
>> > table) using Hadoop? If so, how can we implement that.
>> >
>> > From my understanding, I see that Hive is for reporting purposes and not
>> > for ETL. Let me know if this is right.
>> >
>> > Regards,
>> > Rajendra
>> >
>> >
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: ETL using Hadoop ???

Jeff Hammerbacher
Hey Steve,

See https://issues.apache.org/jira/browse/MAPREDUCE-1168.

Regards,
Jeff

On Thu, Nov 12, 2009 at 1:35 PM, Steve Morin <[hidden email]> wrote:

> Does sqoop also allow you to export data from hadoop to a db?  I
> didn't see that in the documentation
>
> -Steve
>
> On Thu, Nov 12, 2009 at 1:16 PM, Jeff Hammerbacher <[hidden email]>
> wrote:
> > Hey Rajendra,
> >
> > For the "E" part of your task, you may want to check out Sqoop:
> > http://www.cloudera.com/hadoop-sqoop.
> >
> > Regards,
> > Jeff
> >
> > On Thu, Nov 12, 2009 at 10:02 AM, Dhruba Borthakur <[hidden email]>
> wrote:
> >
> >> Hi Rajendra,
> >>
> >> We use Hive for a large data warehouse, details here:
> >> http://wiki.apache.org/hadoop/Hive
> >>
> >> thanks,
> >> dhruba
> >>
> >>
> >> On Thu, Nov 12, 2009 at 9:55 AM, Palikala, Rajendra (CCL) <
> >> [hidden email]> wrote:
> >>
> >> > Hi All,
> >> >
> >> > I am an experienced Informatica and Java Developer. I am very new to
> >> > Hadoop. I have one basic question for the group. Please advise.
> >> >
> >> > Can we do ETL (Example: load data from an oracle table to another
> oracle
> >> > table) using Hadoop? If so, how can we implement that.
> >> >
> >> > From my understanding, I see that Hive is for reporting purposes and
> not
> >> > for ETL. Let me know if this is right.
> >> >
> >> > Regards,
> >> > Rajendra
> >> >
> >> >
> >>
> >>
> >> --
> >> Connect to me at http://www.facebook.com/dhruba
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: ETL using Hadoop ???

Steve Morin
Thanks
Jeff

On Thu, Nov 12, 2009 at 1:36 PM, Jeff Hammerbacher <[hidden email]> wrote:

> Hey Steve,
>
> See https://issues.apache.org/jira/browse/MAPREDUCE-1168.
>
> Regards,
> Jeff
>
> On Thu, Nov 12, 2009 at 1:35 PM, Steve Morin <[hidden email]> wrote:
>
>> Does sqoop also allow you to export data from hadoop to a db?  I
>> didn't see that in the documentation
>>
>> -Steve
>>
>> On Thu, Nov 12, 2009 at 1:16 PM, Jeff Hammerbacher <[hidden email]>
>> wrote:
>> > Hey Rajendra,
>> >
>> > For the "E" part of your task, you may want to check out Sqoop:
>> > http://www.cloudera.com/hadoop-sqoop.
>> >
>> > Regards,
>> > Jeff
>> >
>> > On Thu, Nov 12, 2009 at 10:02 AM, Dhruba Borthakur <[hidden email]>
>> wrote:
>> >
>> >> Hi Rajendra,
>> >>
>> >> We use Hive for a large data warehouse, details here:
>> >> http://wiki.apache.org/hadoop/Hive
>> >>
>> >> thanks,
>> >> dhruba
>> >>
>> >>
>> >> On Thu, Nov 12, 2009 at 9:55 AM, Palikala, Rajendra (CCL) <
>> >> [hidden email]> wrote:
>> >>
>> >> > Hi All,
>> >> >
>> >> > I am an experienced Informatica and Java Developer. I am very new to
>> >> > Hadoop. I have one basic question for the group. Please advise.
>> >> >
>> >> > Can we do ETL (Example: load data from an oracle table to another
>> oracle
>> >> > table) using Hadoop? If so, how can we implement that.
>> >> >
>> >> > From my understanding, I see that Hive is for reporting purposes and
>> not
>> >> > for ETL. Let me know if this is right.
>> >> >
>> >> > Regards,
>> >> > Rajendra
>> >> >
>> >> >
>> >>
>> >>
>> >> --
>> >> Connect to me at http://www.facebook.com/dhruba
>> >>
>> >
>>
>