(no subject)

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

(no subject)

Shuubham Ojha

Hello All,

                       I am Shuubham Ojha a graduate researcher with the University Of Melbourne. We have developed a block placement strategy which optimises delay associated with reconstruction. As a result of this optimisation problem, we get a placement matrix for blocks which tells us which block has to be placed at which node. We have been able to implement this strategy in Hadoop 2 by tweaking the file placementmonitor.java and blockmover.java where placementmonitor.java monitors the placement process and calls blockmover.java when the placement is not according the strategy. However, I can't find any such file analogous to placementmonitor.java in Hadoop 3 although I think that the closest file which performs this function is balancer.java located in hadoop-hdfs-project. Can anybody please provide me more information on this front?


Warm Regards,

Shuubham Ojha

University Of Melbourne,

Victoria, Australia- 3010

Reply | Threaded
Open this post in threaded view
|

Re:

Wei-Chiu Chuang-3
[hidden email] 
Hi Shuubham,

Just like to clarify a bit. What's the purpose of this work? Is this for the general block placement policy in HDFS, or the balancer/mover/diskbalancer, or decommissioning/recommissioning? Block placement is determined by NameNode. Do you intend to shorten the time to decide where a block is placed? Do you want to reduce the time such that re-replication takes less time? 

I'm asking this because I don't think there's ever a placementmonitor or a blockmonitor class.

On Wed, Dec 19, 2018 at 10:36 PM Shuubham Ojha <[hidden email]> wrote:

Hello All,

                       I am Shuubham Ojha a graduate researcher with the University Of Melbourne. We have developed a block placement strategy which optimises delay associated with reconstruction. As a result of this optimisation problem, we get a placement matrix for blocks which tells us which block has to be placed at which node. We have been able to implement this strategy in Hadoop 2 by tweaking the file placementmonitor.java and blockmover.java where placementmonitor.java monitors the placement process and calls blockmover.java when the placement is not according the strategy. However, I can't find any such file analogous to placementmonitor.java in Hadoop 3 although I think that the closest file which performs this function is balancer.java located in hadoop-hdfs-project. Can anybody please provide me more information on this front?


Warm Regards,

Shuubham Ojha

University Of Melbourne,

Victoria, Australia- 3010

Reply | Threaded
Open this post in threaded view
|

Re:

Shuubham Ojha
Hi,
          Actually we just want to implement our block placement strategy in hadoop 3 and compare results. I think that we should see some improvements. Can you shed some more light on the difference between general block placement policy and balancer / mover thing.

Warm regards,
Shuubham Ojha   

On Thu, Dec 20, 2018 at 9:24 PM Wei-Chiu Chuang <[hidden email]> wrote:
[hidden email] 
Hi Shuubham,

Just like to clarify a bit. What's the purpose of this work? Is this for the general block placement policy in HDFS, or the balancer/mover/diskbalancer, or decommissioning/recommissioning? Block placement is determined by NameNode. Do you intend to shorten the time to decide where a block is placed? Do you want to reduce the time such that re-replication takes less time? 

I'm asking this because I don't think there's ever a placementmonitor or a blockmonitor class.

On Wed, Dec 19, 2018 at 10:36 PM Shuubham Ojha <[hidden email]> wrote:

Hello All,

                       I am Shuubham Ojha a graduate researcher with the University Of Melbourne. We have developed a block placement strategy which optimises delay associated with reconstruction. As a result of this optimisation problem, we get a placement matrix for blocks which tells us which block has to be placed at which node. We have been able to implement this strategy in Hadoop 2 by tweaking the file placementmonitor.java and blockmover.java where placementmonitor.java monitors the placement process and calls blockmover.java when the placement is not according the strategy. However, I can't find any such file analogous to placementmonitor.java in Hadoop 3 although I think that the closest file which performs this function is balancer.java located in hadoop-hdfs-project. Can anybody please provide me more information on this front?


Warm Regards,

Shuubham Ojha

University Of Melbourne,

Victoria, Australia- 3010