Yarn not picking up the Resource manager configurations

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Yarn not picking up the Resource manager configurations

Dhanushka Parakrama
Hi All

I have setup the multi node hadoop cluster . but when is run the simple map reduce job it gives me the following error the in  the data nodes container log .


Im running the hadoop 2.7.3 version

In namenode which contain resource manager as well . i ran below job

hadoop jar hadoop-mapreduce-examples-2.7.3.jar wordcount /words  /output

Error
====

tryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-05-19 07:57:14,340 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-05-19 07:57:15,341 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)


Configurations
===========

vi .barshrc file

export JAVA_HOME=/home/ubuntu/jdk1.8.0_45
export HADOOP_HOME=/usr/local/hadoop/current
#export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export ZOOKEEPER_HOME=/usr/local/zookeeper/current

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin


vi .yarn-site.xml file


<configuration>

<!-- Site specific YARN configuration properties -->

<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>

<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>nn1.cluster.com</value>
</property>

 <property>
      <name>yarn.resourcemanager.scheduler.address</name>
      <value>nn1.cluster.com:8030</value>
 </property>


</configuration>



Can any expert help me with this issue , im stuck at this point

Thank You



 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Yarn not picking up the Resource manager configurations

Miklos Szegedi
Hi,

I usually get this error when the yarn-site.xml is not on the class path and it picks up a default 0.0.0.0. I would try:
1. bin/yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar pi 1 1000
2. explicitly adding $HADOOP_HOME/etc/hadoop to your client classpath
3. make sure $HADOOP_HOME/etc/hadoop contains your yarn-site.xml and it is not somewhere else
4. nn1.cluster.com: you actually need to attach to the resource manager host name not the name node, if they are running on different hosts.

I hope this helps.

Thanks,
Miklos


On Fri, May 19, 2017 at 1:39 AM, Dhanushka Parakrama <[hidden email]> wrote:
Hi All

I have setup the multi node hadoop cluster . but when is run the simple map reduce job it gives me the following error the in  the data nodes container log .


Im running the hadoop 2.7.3 version

In namenode which contain resource manager as well . i ran below job

hadoop jar hadoop-mapreduce-examples-2.7.3.jar wordcount /words  /output

Error
====

tryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-05-19 07:57:14,340 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-05-19 07:57:15,341 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)


Configurations
===========

vi .barshrc file

export JAVA_HOME=/home/ubuntu/jdk1.8.0_45
export HADOOP_HOME=/usr/local/hadoop/current
#export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export ZOOKEEPER_HOME=/usr/local/zookeeper/current

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin


vi .yarn-site.xml file


<configuration>

<!-- Site specific YARN configuration properties -->

<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>

<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>nn1.cluster.com</value>
</property>

 <property>
      <name>yarn.resourcemanager.scheduler.address</name>
      <value>nn1.cluster.com:8030</value>
 </property>


</configuration>



Can any expert help me with this issue , im stuck at this point

Thank You



 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Yarn not picking up the Resource manager configurations

Brahma Reddy Battula
In reply to this post by Dhanushka Parakrama

 

Looks configurations(mainly yarn-site.xml here which you pointed) are not loaded in classpath. Can you check the output of “hadoop classpath”..?

 

 

 

Regards

Brahma Reddy Battula

 

From: Dhanushka Parakrama [mailto:[hidden email]]
Sent: 19 May 2017 16:40
To: [hidden email]
Subject: Yarn not picking up the Resource manager configurations

 

Hi All

I have setup the multi node hadoop cluster . but when is run the simple map reduce job it gives me the following error the in  the data nodes container log .

Im running the hadoop 2.7.3 version

 

In namenode which contain resource manager as well . i ran below job

hadoop jar hadoop-mapreduce-examples-2.7.3.jar wordcount /words  /output

 

Error
====

tryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-05-19 07:57:14,340 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-05-19 07:57:15,341 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Configurations
===========

vi .barshrc file

export JAVA_HOME=/home/ubuntu/jdk1.8.0_45
export HADOOP_HOME=/usr/local/hadoop/current
#export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export ZOOKEEPER_HOME=/usr/local/zookeeper/current

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin

vi .yarn-site.xml file


<configuration>

<!-- Site specific YARN configuration properties -->

<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>

<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>nn1.cluster.com</value>
</property>

 <property>
      <name>yarn.resourcemanager.scheduler.address</name>
      <value>nn1.cluster.com:8030</value>
 </property>


</configuration>


Can any expert help me with this issue , im stuck at this point

Thank You

 


 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Yarn not picking up the Resource manager configurations

Gurmukh Singh
In reply to this post by Dhanushka Parakrama

it is  ".bashrc" not ".barshrc"

is port 8030 listening ?

netstat -tlpn | grep 8030

Can you give the output of:

$ hdfs getconf -confkey yarn.resourcemanager.scheduler.address

$ hdfs getconf -confkey fs.defaultFS

$ nslookup nn1.cluster1.com (if using DNS), else

 $ getent hosts nn1.cluster1.com

$ ifconfig from the host where resourcemanager is running

Refer my github for configs: https://github.com/netxillon/hadoop





On 19/5/17 6:39 pm, Dhanushka Parakrama wrote:
Hi All

I have setup the multi node hadoop cluster . but when is run the simple map reduce job it gives me the following error the in  the data nodes container log .


Im running the hadoop 2.7.3 version

In namenode which contain resource manager as well . i ran below job

hadoop jar hadoop-mapreduce-examples-2.7.3.jar wordcount /words  /output

Error
====

tryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-05-19 07:57:14,340 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-05-19 07:57:15,341 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)


Configurations
===========

vi .barshrc file

export JAVA_HOME=/home/ubuntu/jdk1.8.0_45
export HADOOP_HOME=/usr/local/hadoop/current
#export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export ZOOKEEPER_HOME=/usr/local/zookeeper/current

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin


vi .yarn-site.xml file


<configuration>

<!-- Site specific YARN configuration properties -->

<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>

<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>nn1.cluster.com</value>
</property>

 <property>
      <name>yarn.resourcemanager.scheduler.address</name>
      <value>nn1.cluster.com:8030</value>
 </property>


</configuration>



Can any expert help me with this issue , im stuck at this point

Thank You



 

Loading...