question on data/task node specific configuration...

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

question on data/task node specific configuration...

Andy Sautins

    I've run into a situation where it would be helpful to set specific configuration variables local to a data/task node.  I've got a solution, but I'm curious if there is a best practice around this and if I'm doing it in a reasonable way.

    Basically what we've got is a number of machines that have 2 Cores/4GB of memory.  For those boxes we have some options configured higher than the default in mapred-site.xml ( specifically set to -Xmx1024m ).  We recently added a few additional boxes that have 2 Cores/2GB of memory and the causes those boxes to swap so we'd like to configure those boxes to set to -Xmx512m.  What I found is that if on the data/task node that if I change the value in mapred-site.xml it is overridden, but if I set the parameter using <final>true</final> it is used.  So effectively now I've got a different mapred-site.xml on each of the nodes.

   My question is, is this a reasonable way of going about this?  Is there a best practice for dealing with minor node-specific configuration differences?  What I'd really want is not only a mapred-site.xml, but a mapred-node.xml as well that has node specific overrides.  What I can't tell is if I make all the site level configuration changes in the name node mapred-site.xml and then the node specific mapred-site.xml files to node local changes if that does what I'm looking for.

   Any insight would be appreciated.