Reduce Task Allocation in Hadoop

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Reduce Task Allocation in Hadoop

Or Raz
Hey,
I am a researcher, and my focus is on Hadoop's performance mainly about the allocation of reduce Tasks. The idea is about interfering with the current allocation of tasks (Reduce tasks) in Hadoop (which is made by AM and RM of YARN), and it can be divided into the following questions (regarding Hadoop 2.6.5).

1. Where is the Class and function which I need to change (override probably) in AM for interfering with the current allocation of Reduce tasks? What is the API for that?
2. How can I do that? Do I need to write the whole Class for changing one function? Can I pass an argument to that function?
3. If the above idea is not feasible then can I make for two MapReduce Jobs the same allocation of Reduce tasks to their nodes in the cluster? 

Thanks,
Or