I am Shuubham Ojha a graduate researcher with
the University Of Melbourne. We have developed a block placement
strategy which optimises delay associated with reconstruction. As a
result of this optimisation problem, we get a placement matrix for
blocks which tells us which block has to be placed at which node. We
have been able to implement this strategy in Hadoop 2 by tweaking the
file placementmonitor.java and blockmover.java where placementmonitor.java monitors the placement process and calls blockmover.java when the placement is not according the strategy. However, I can't find any such file analogous to placementmonitor.java in Hadoop 3 although I think that the closest file which performs this function is balancer.java located in hadoop-hdfs-project. Can anybody please provide me more information on this front?
University Of Melbourne,
Victoria, Australia- 3010
Just like to clarify a bit. What's the purpose of this work? Is this for the general block placement policy in HDFS, or the balancer/mover/diskbalancer, or decommissioning/recommissioning? Block placement is determined by NameNode. Do you intend to shorten the time to decide where a block is placed? Do you want to reduce the time such that re-replication takes less time?
I'm asking this because I don't think there's ever a placementmonitor or a blockmonitor class.
On Wed, Dec 19, 2018 at 10:36 PM Shuubham Ojha <[hidden email]> wrote:
Actually we just want to implement our block placement strategy in hadoop 3 and compare results. I think that we should see some improvements. Can you shed some more light on the difference between general block placement policy and balancer / mover thing.
On Thu, Dec 20, 2018 at 9:24 PM Wei-Chiu Chuang <[hidden email]> wrote:
|Free forum by Nabble||Edit this page|