multi nic setup hadoop 2.7.7

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

multi nic setup hadoop 2.7.7

Micha
Hi,

I try to setup a hadoop cluser with 1 namenode and 5 datanodes with two
networks.
The cluster should use a 192.168 address for its own communication,
client traffic should happen over a 10 network.

Allthough reading a bunch of websides about this I cannot get this to to
work, there are allways connection errors showing up.

Maybe someone has done this and could help me with that...

The setup is:
each node has two nics.
The datanodes are 192.168.10.1 to 192.168.10.5,  the namenode is
192.168.10.6

The second nic in each node has network 10 address.

The clients connect via the 10 network to the cluster, also the dns
resolves to the 10 network

Listed in /etc/host are short names with the corresponding 192.168..
address aand the fqdn with the 10 address.
The cluster nodes should talk to each other via the 192 addresses, with
no firewall between them.

My config includes:

namenode:
dfs.namenode.rpc-bind-host: 0.0.0.0
dfs.namenode.servicerpc-bind-host: 0.0.0.

datanodes:
dfs.namenode.servicerpc-address: master01:9000
dfs.client.use.datanode.hostname: true

the slaves file contains the short names which are listed in the
/etc/hosts file.


With this config the cluster starts up without errors, hdfs dfs -ls /
works too, but "hdfs dfs -put ..." doesn't work:
DFSClient: Exception in createBlockOutputStream, Connection refused.
Abandoning BP....-10.x.x.x-7872394857:blk_23423433_1013
Excluding DatanodeInfoWithStorage(192.168.10.2 )

I thought the datanodes listen on 0.0.0 as shown in
https://hadoop.apache.org/docs/r2.7.7/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml



What I am missing?

I tried various combinations of config values, maybe overlooking the
obvious right one...


Thanks for helping,
 Michael



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]