How to skip fail map to done the job

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to skip fail map to done the job

梁景明
hi, I use hadoop0.20 and 8 nodes, there is a job that has 130 map to run
,and  completed 128 map,
but only 2 map fail ,and its fail in my case is accepted ,but the job fail
,the last 128 map also fail,
i can't get any result.
i found that SkipBadRecords maybe work for my case,but i used it ,and it
doesnt work,maybe something
wrong,here is my code , how to skip the bad fail map ,thanks for any help

 JobConf conf = new JobConf(ProductMR.class);
 conf.setJobName("ProductMR");
 conf.setOutputKeyClass(Text.class);
 conf.setOutputValueClass(Product.class);
 conf.setMapperClass(Map.class);
 conf.setReducerClass(Reduce.class);
 conf.setMapOutputCompressorClass(DefaultCodec.class);
 conf.setInputFormat(SequenceFileInputFormat.class);
 conf.setOutputFormat(SequenceFileOutputFormat.class);
 String objpath = "abc1";
 SequenceFileInputFormat.addInputPath(conf, new Path(objpath));
 SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);
 SkipBadRecords.setAttemptsToStartSkipping(conf, 0);
 SkipBadRecords.setSkipOutputPath(conf, new Path("data/product/skip/"));
 String output = "abc";
 SequenceFileOutputFormat.setOutputPath(conf, new Path(output));
 JobClient.runJob(conf);
Reply | Threaded
Open this post in threaded view
|

Re: How to skip fail map to done the job

Amogh Vasekar
For skipping failed tasks try : mapred.max.map.failures.percent

Amogh
On 10/21/09 8:58 AM, "梁景明" <[hidden email]> wrote:

hi, I use hadoop0.20 and 8 nodes, there is a job that has 130 map to run
,and  completed 128 map,
but only 2 map fail ,and its fail in my case is accepted ,but the job fail
,the last 128 map also fail,
i can't get any result.
i found that SkipBadRecords maybe work for my case,but i used it ,and it
doesnt work,maybe something
wrong,here is my code , how to skip the bad fail map ,thanks for any help

 JobConf conf = new JobConf(ProductMR.class);
 conf.setJobName("ProductMR");
 conf.setOutputKeyClass(Text.class);
 conf.setOutputValueClass(Product.class);
 conf.setMapperClass(Map.class);
 conf.setReducerClass(Reduce.class);
 conf.setMapOutputCompressorClass(DefaultCodec.class);
 conf.setInputFormat(SequenceFileInputFormat.class);
 conf.setOutputFormat(SequenceFileOutputFormat.class);
 String objpath = "abc1";
 SequenceFileInputFormat.addInputPath(conf, new Path(objpath));
 SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);
 SkipBadRecords.setAttemptsToStartSkipping(conf, 0);
 SkipBadRecords.setSkipOutputPath(conf, new Path("data/product/skip/"));
 String output = "abc";
 SequenceFileOutputFormat.setOutputPath(conf, new Path(output));
 JobClient.runJob(conf);

Reply | Threaded
Open this post in threaded view
|

Re: How to skip fail map to done the job

梁景明
thank you ,it works

2009/10/21 Amogh Vasekar <[hidden email]>

> For skipping failed tasks try : mapred.max.map.failures.percent
>
> Amogh
> On 10/21/09 8:58 AM, "梁景明" <[hidden email]> wrote:
>
> hi, I use hadoop0.20 and 8 nodes, there is a job that has 130 map to run
> ,and  completed 128 map,
> but only 2 map fail ,and its fail in my case is accepted ,but the job fail
> ,the last 128 map also fail,
> i can't get any result.
> i found that SkipBadRecords maybe work for my case,but i used it ,and it
> doesnt work,maybe something
> wrong,here is my code , how to skip the bad fail map ,thanks for any help
>
>  JobConf conf = new JobConf(ProductMR.class);
>  conf.setJobName("ProductMR");
>  conf.setOutputKeyClass(Text.class);
>  conf.setOutputValueClass(Product.class);
>  conf.setMapperClass(Map.class);
>  conf.setReducerClass(Reduce.class);
>  conf.setMapOutputCompressorClass(DefaultCodec.class);
>  conf.setInputFormat(SequenceFileInputFormat.class);
>  conf.setOutputFormat(SequenceFileOutputFormat.class);
>  String objpath = "abc1";
>  SequenceFileInputFormat.addInputPath(conf, new Path(objpath));
>  SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);
>  SkipBadRecords.setAttemptsToStartSkipping(conf, 0);
>  SkipBadRecords.setSkipOutputPath(conf, new Path("data/product/skip/"));
>  String output = "abc";
>  SequenceFileOutputFormat.setOutputPath(conf, new Path(output));
>  JobClient.runJob(conf);
>
>