Random weirdness — anyone explain, please?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Random weirdness — anyone explain, please?

BM-2
Hi!

I have quite odd Hadoop behavior. I wrote a client to my app that
simply is trying to talk to HDFS and do stuff. Version of Hadoop is
20.0. I still suspect CLASSPATH, but would be nice to know details.
So, here is a part of a traceback:
----------------------------------------------------
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
....
----------------------------------------------------

Now, could anybody please feel more light here what FileSystem is
trying to load in the configuration, once configuration was created
like: "... = new Configuration()"? I've looked at the source code, and
it does something this:
----------------------------------------------------
private static FileSystem createFileSystem(URI uri, Configuration conf
      ) throws IOException {
    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
    if (clazz == null) {
      throw new IOException("No FileSystem for scheme: " + uri.getScheme());
    }
    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
    fs.initialize(uri, conf);
    return fs;
}
----------------------------------------------------

...but I think I need either long hours to hack or just ask here what
the heck is trying to load and can not. :-) It sounds to me that
CLASSPATH was screwed up, but I've put all the required libraries in
Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
However, when I am trying to write there, my thread dies with the
traceback above.

If anybody at least suggest me where to dig more, I would greatly appreciate it.

--
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.
Reply | Threaded
Open this post in threaded view
|

Re: Random weirdness — anyone explain, please?

BM-2
One more thing. I am creating configuration like this (if it is wrong
— please tell me why and how is right to do it):
-------------------------------------------
        this.conf = new Configuration();
        this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
host, port));
        this.fs = FileSystem.get(this.conf);
-------------------------------------------

I don't see anything wrong with a code above, unless I just missing
something undocumented...
Anyone, please?

Thanks in advance.

--
Bo


On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
<[hidden email]> wrote:

> Hi!
>
> I have quite odd Hadoop behavior. I wrote a client to my app that
> simply is trying to talk to HDFS and do stuff. Version of Hadoop is
> 20.0. I still suspect CLASSPATH, but would be nice to know details.
> So, here is a part of a traceback:
> ----------------------------------------------------
>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>        at java.lang.Class.forName0(Native Method)
>        at java.lang.Class.forName(Class.java:247)
>        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
>        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
>        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
>        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
>        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
> ....
> ----------------------------------------------------
>
> Now, could anybody please feel more light here what FileSystem is
> trying to load in the configuration, once configuration was created
> like: "... = new Configuration()"? I've looked at the source code, and
> it does something this:
> ----------------------------------------------------
> private static FileSystem createFileSystem(URI uri, Configuration conf
>      ) throws IOException {
>    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
>    if (clazz == null) {
>      throw new IOException("No FileSystem for scheme: " + uri.getScheme());
>    }
>    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
>    fs.initialize(uri, conf);
>    return fs;
> }
> ----------------------------------------------------
>
> ...but I think I need either long hours to hack or just ask here what
> the heck is trying to load and can not. :-) It sounds to me that
> CLASSPATH was screwed up, but I've put all the required libraries in
> Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
> However, when I am trying to write there, my thread dies with the
> traceback above.
>
> If anybody at least suggest me where to dig more, I would greatly appreciate it.
>
> --
> Kind regards, BM
>
> Things, that are stupid at the beginning, rarely ends up wisely.
>



--
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.
Reply | Threaded
Open this post in threaded view
|

Re: Random weirdness — anyone explain, please?

Huy Phan
Hi Bogdan,
There's nothing wrong with your code, since you're talking about
CLASSPATH, I guess you didn't execute your script the right way.
How did you run your script anyway ? With or without hadoop command ?

Bogdan M. Maryniuk wrote:

> One more thing. I am creating configuration like this (if it is wrong
> — please tell me why and how is right to do it):
> -------------------------------------------
>         this.conf = new Configuration();
>         this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
> host, port));
>         this.fs = FileSystem.get(this.conf);
> -------------------------------------------
>
> I don't see anything wrong with a code above, unless I just missing
> something undocumented...
> Anyone, please?
>
> Thanks in advance.
>
> --
> Bo
>
>
> On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
> <[hidden email]> wrote:
>  
>> Hi!
>>
>> I have quite odd Hadoop behavior. I wrote a client to my app that
>> simply is trying to talk to HDFS and do stuff. Version of Hadoop is
>> 20.0. I still suspect CLASSPATH, but would be nice to know details.
>> So, here is a part of a traceback:
>> ----------------------------------------------------
>>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>        at java.lang.Class.forName0(Native Method)
>>        at java.lang.Class.forName(Class.java:247)
>>        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
>>        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
>>        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
>>        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
>>        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
>> ....
>> ----------------------------------------------------
>>
>> Now, could anybody please feel more light here what FileSystem is
>> trying to load in the configuration, once configuration was created
>> like: "... = new Configuration()"? I've looked at the source code, and
>> it does something this:
>> ----------------------------------------------------
>> private static FileSystem createFileSystem(URI uri, Configuration conf
>>      ) throws IOException {
>>    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
>>    if (clazz == null) {
>>      throw new IOException("No FileSystem for scheme: " + uri.getScheme());
>>    }
>>    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
>>    fs.initialize(uri, conf);
>>    return fs;
>> }
>> ----------------------------------------------------
>>
>> ...but I think I need either long hours to hack or just ask here what
>> the heck is trying to load and can not. :-) It sounds to me that
>> CLASSPATH was screwed up, but I've put all the required libraries in
>> Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
>> However, when I am trying to write there, my thread dies with the
>> traceback above.
>>
>> If anybody at least suggest me where to dig more, I would greatly appreciate it.
>>
>> --
>> Kind regards, BM
>>
>> Things, that are stupid at the beginning, rarely ends up wisely.
>>
>>    
>
>
>
>  

Reply | Threaded
Open this post in threaded view
|

Re: Random weirdness — anyone explain, please?

BM-2
On Thu, Oct 22, 2009 at 1:33 PM, Huy Phan <[hidden email]> wrote:
> Hi Bogdan,
> There's nothing wrong with your code, since you're talking about CLASSPATH,
> I guess you didn't execute your script the right way.
> How did you run your script anyway ? With or without hadoop command ?

Well, that's not a script really. I am trying to integrate the thing
with the GlassFish in order apps talk to Hadoop and put/get media from
there instead. So I wrote a client that is available over JNDI. All
Hadoop-related libs I put into common classpath, so they *are*
available. The only thing is probably that Catalina class loader could
mess up things, but not likely...

Anyway, I am looking to fix this. If somebody would fill more light
here what this method wants? What is interesting, that a test-case
works great, i.e. none of errors like this, file are written in the
HDFS and read perfectly. I suspect that Hadoop does some "fun" with
class loading, but I am not sure, since I did not invested time into
this yet (still hope someone will tell me where could be a problem).

--
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.
Reply | Threaded
Open this post in threaded view
|

RE: Random weirdness - anyone explain, please?

#YONG YONG CHENG#
Good Day,
 
Maybe you can try putting Hadoop jar files in the Extension classloader directory <domain name>/lib/ext.
 
Thanks.

________________________________

From: BM [mailto:[hidden email]]
Sent: Thu 10/22/2009 3:59 PM
To: [hidden email]
Subject: Re: Random weirdness - anyone explain, please?



On Thu, Oct 22, 2009 at 1:33 PM, Huy Phan <[hidden email]> wrote:
> Hi Bogdan,
> There's nothing wrong with your code, since you're talking about CLASSPATH,
> I guess you didn't execute your script the right way.
> How did you run your script anyway ? With or without hadoop command ?

Well, that's not a script really. I am trying to integrate the thing
with the GlassFish in order apps talk to Hadoop and put/get media from
there instead. So I wrote a client that is available over JNDI. All
Hadoop-related libs I put into common classpath, so they *are*
available. The only thing is probably that Catalina class loader could
mess up things, but not likely...

Anyway, I am looking to fix this. If somebody would fill more light
here what this method wants? What is interesting, that a test-case
works great, i.e. none of errors like this, file are written in the
HDFS and read perfectly. I suspect that Hadoop does some "fun" with
class loading, but I am not sure, since I did not invested time into
this yet (still hope someone will tell me where could be a problem).

--
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.


Reply | Threaded
Open this post in threaded view
|

Re: Random weirdness — anyone explain, please?

brien colwell
In reply to this post by BM-2
When I debug Hadoop locally in Eclipse I run jobs with the following
configurations. I access the HDFS with paths like below . It seems to me
this should be the same as what you have so I also suspect a classpath
issue.

        baseConf.set("fs.default.name", "file:///");

Path p = new Path("hdfs://<namenode>:<port>/path/to/stuff")
FileSystem fs = p.getFileSystem(baseConf) .





Bogdan M. Maryniuk wrote:

> One more thing. I am creating configuration like this (if it is wrong
> — please tell me why and how is right to do it):
> -------------------------------------------
>         this.conf = new Configuration();
>         this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
> host, port));
>         this.fs = FileSystem.get(this.conf);
> -------------------------------------------
>
> I don't see anything wrong with a code above, unless I just missing
> something undocumented...
> Anyone, please?
>
> Thanks in advance.
>
> --
> Bo
>
>
> On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
> <[hidden email]> wrote:
>  
>> Hi!
>>
>> I have quite odd Hadoop behavior. I wrote a client to my app that
>> simply is trying to talk to HDFS and do stuff. Version of Hadoop is
>> 20.0. I still suspect CLASSPATH, but would be nice to know details.
>> So, here is a part of a traceback:
>> ----------------------------------------------------
>>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>        at java.lang.Class.forName0(Native Method)
>>        at java.lang.Class.forName(Class.java:247)
>>        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
>>        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
>>        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
>>        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
>>        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
>> ....
>> ----------------------------------------------------
>>
>> Now, could anybody please feel more light here what FileSystem is
>> trying to load in the configuration, once configuration was created
>> like: "... = new Configuration()"? I've looked at the source code, and
>> it does something this:
>> ----------------------------------------------------
>> private static FileSystem createFileSystem(URI uri, Configuration conf
>>      ) throws IOException {
>>    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
>>    if (clazz == null) {
>>      throw new IOException("No FileSystem for scheme: " + uri.getScheme());
>>    }
>>    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
>>    fs.initialize(uri, conf);
>>    return fs;
>> }
>> ----------------------------------------------------
>>
>> ...but I think I need either long hours to hack or just ask here what
>> the heck is trying to load and can not. :-) It sounds to me that
>> CLASSPATH was screwed up, but I've put all the required libraries in
>> Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
>> However, when I am trying to write there, my thread dies with the
>> traceback above.
>>
>> If anybody at least suggest me where to dig more, I would greatly appreciate it.
>>
>> --
>> Kind regards, BM
>>
>> Things, that are stupid at the beginning, rarely ends up wisely.
>>
>>    
>
>
>
>  

Reply | Threaded
Open this post in threaded view
|

Re: Random weirdness - anyone explain, please?

BM-2
In reply to this post by #YONG YONG CHENG#
On Thu, Oct 22, 2009 at 10:43 PM, #YONG YONG CHENG#
<[hidden email]> wrote:
> Maybe you can try putting Hadoop jar files in the Extension classloader directory <domain name>/lib/ext.

Apparently, no. Now it works fine. Sounds like Webspace is just
Sun-style thing: bloated, big, slow, complicated and... wrong. On a
regular GlassFish it works fine.

Thanks! Case closed, was not Hadoop issue. :-)

--
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.
Reply | Threaded
Open this post in threaded view
|

Re: Random weirdness — anyone explain, please?

BM-2
In reply to this post by brien colwell
On Thu, Oct 22, 2009 at 11:56 PM, brien colwell <[hidden email]> wrote:
> When I debug Hadoop locally in Eclipse I run jobs with the following
> configurations. I access the HDFS with paths like below . It seems to me
> this should be the same as what you have so I also suspect a classpath
> issue.
>
>       baseConf.set("fs.default.name", "file:///");
>
> Path p = new Path("hdfs://<namenode>:<port>/path/to/stuff")
> FileSystem fs = p.getFileSystem(baseConf) .

Well, not my case: I am accessing a real cluster. That works fine, but
somewhat looks like Catalina class loader dislikes hadoop's forName
tricks and in some cases (depends on a Moon phase and solar
interference) does not works. All become OK after a regular restart.
Which is... weird and wrong. On a regular GlassFish it works fine and
stable, only Webspace freaks out.

I need to dig into it anyway, but sounds like not Hadoop issue anymore.

--
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.
Reply | Threaded
Open this post in threaded view
|

Re: Random weirdness — anyone explain, please?

Philip Zeyliger
In reply to this post by BM-2
On Wed, Oct 21, 2009 at 9:23 PM, Bogdan M. Maryniuk <
[hidden email]> wrote:

> Hi!
>
> I have quite odd Hadoop behavior. I wrote a client to my app that
> simply is trying to talk to HDFS and do stuff. Version of Hadoop is
> 20.0. I still suspect CLASSPATH, but would be nice to know details.
> So, here is a part of a traceback:
> ----------------------------------------------------
>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>        at java.lang.Class.forName0(Native Method)
>        at java.lang.Class.forName(Class.java:247)
>        at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
>        at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
>        at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
>        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
>        at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
> ....
> ----------------------------------------------------
>
> Now, could anybody please feel more light here what FileSystem is
> trying to load in the configuration, once configuration was created
> like: "... = new Configuration()"? I've looked at the source code, and
> it does something this:
> ----------------------------------------------------
> private static FileSystem createFileSystem(URI uri, Configuration conf
>      ) throws IOException {
>    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
>    if (clazz == null) {
>      throw new IOException("No FileSystem for scheme: " + uri.getScheme());
>    }
>    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
>    fs.initialize(uri, conf);
>    return fs;
> }
> ----------------------------------------------------
>

Here's what's going on here:

The FileSystem class has a static factory which takes the "scheme" of your
URL ("hdfs" or "s3" or "s3n" or "file"...) and maps that into an
implementation class by looking up a configuration variable that looks like
fs.hdfs.impl (or fs.s3.impl or fs.file.impl, ...).  It then loads that
implementation by reflection.  (Configuration.getClass is just a short hand
for read a configuration variable, and return the class that it represents.)

One thing you might try to debug is to dump the Configuration (using
configuration.writeXml(System.err) or similar), and making sure everything
there looks reasonable.  Typically HDFS is configured from from
core-default.xml, like so:

<property>
  <name>fs.hdfs.impl</name>
  <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
  <description>The FileSystem for hdfs: uris.</description>
</property>



-- Philip
Reply | Threaded
Open this post in threaded view
|

Syslog-ng and hadoop logs

Joe Hammerman
Hi all,

        We are working on integrating our hadoop logs which live on the datanodes into a central syslog-ng logging server. There seems to be a dearth of information available on the internet as how to effect this; I was hoping someone here could help out, either with a link or by providing examples from their configuration.

        Thanks in advance!

Regards,
        Joseph Hammerman
Reply | Threaded
Open this post in threaded view
|

Re: Syslog-ng and hadoop logs

Brian Bockelman
Hey Joe,

We added the following to log4j.properties:

log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.facility=local0
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.ConversionPattern=%p %c{2}: %m%n
log4j.appender.SYSLOG.SyslogHost=red
log4j.appender.SYSLOG.threshold=ERROR
log4j.appender.SYSLOG.Header=true
log4j.appender.SYSLOG.FacilityPrinting=true

IIRC, if you don't have have .Header=true, the generated packets are  
invalid and syslog chokes on them.  Fun!

Then, we add whatever we'd like to go to syslog; i.e.

log4j.logger.org.apache.hadoop.hdfs=INFO,SYSLOGM

or

log4j.rootLogger=DEBUG,SYSLOG,console,DRFA,EventCounter

Brian

On Oct 22, 2009, at 7:01 PM, Joe Hammerman wrote:

> Hi all,
>
> We are working on integrating our hadoop logs which live on the  
> datanodes into a central syslog-ng logging server. There seems to be  
> a dearth of information available on the internet as how to effect  
> this; I was hoping someone here could help out, either with a link  
> or by providing examples from their configuration.
>
> Thanks in advance!
>
> Regards,
> Joseph Hammerman


smime.p7s (3K) Download Attachment