(no subject)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

(no subject)

Shuubham Ojha
Hello, I am trying to use Hadoop 3.1.1 on my cluster. I wish to experiment with the Hitchhiker Code which I believe was introduced in Hadoop 3 itself. I don't understand how do I activate the hitchhiker feature for the blocks of files I put on the datanode. I also don't know which erasure coding policy is being used by default on the uploaded blocks of files when I don't do anything. Any help regarding setting the erasure coding policy (and hitchhiker feature) would be appreciated.

It's a bit urgent.

Warm regards,
Shuubham Ojha
Reply | Threaded
Open this post in threaded view
|

Re:

Brahma Reddy Battula-3
By default, all built-in erasure coding policies are disabled, except the one defined in dfs.namenode.ec.system.default.policy which is enabled by default(RS-6-3-1024k) With this configuration, the default EC policy will be used when no policy name is passed as an argument in the ‘-setPolicy’ command.

You can enable set of policies through hdfs ec [-enablePolicy -policy <policyName>] command based on the size of the cluster and the desired fault-tolerance properties.

For instance, for a cluster with 9 racks, a policy like RS-10-4-1024k will not preserve rack-level fault-tolerance, and RS-6-3-1024k or RS-3-2-1024kmight be more appropriate.

Reference:


On Sun, Feb 17, 2019 at 8:53 PM Shuubham Ojha <[hidden email]> wrote:
Hello, I am trying to use Hadoop 3.1.1 on my cluster. I wish to experiment with the Hitchhiker Code which I believe was introduced in Hadoop 3 itself. I don't understand how do I activate the hitchhiker feature for the blocks of files I put on the datanode. I also don't know which erasure coding policy is being used by default on the uploaded blocks of files when I don't do anything. Any help regarding setting the erasure coding policy (and hitchhiker feature) would be appreciated.

It's a bit urgent.

Warm regards,
Shuubham Ojha


--



--Brahma Reddy Battula