mapreduce application which only has 17000+ map tasks runs very slow on yarn after 16000+ map completed

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

mapreduce application which only has 17000+ map tasks runs very slow on yarn after 16000+ map completed

esri_lxc
Hi everyone!

        I'm running a simple sql(select xx,xx... from viewXXX where xxxxx) using hive0.13.1 on hadoop2.6.0(the framework is MRv2, not tez). After submitting it, I find that it's a MR job which has only 17000+ map tasks and no reduce tasks.

        The job runs very quickly in the early 15 minutes(all 400+ containers on my cluster(20+ nodes) are allocated to run tasks during this period), but become very slow after that(no other jobs running on my cluster).

        I run it a couple of times and find that the number of containers allocated to the job decreases(not strictly but roughly) as the time go on, and after about 15 minutes the number of containers allocated to the job becomes 1(which is the ApplicationMaster's container)! Then the AM is always waitting for RM to give it a container to run map task. RM is not busy(no much GC) and has a lot of containers available(I find that in the RM log), but it assign AM 1 container per MINUTE. So the job finally takes 7 hours to finish.:(

       Parts of my AM,RM log is in the attachment.
       Any help will be appreciated!


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

=?GBK?B?Uk0gbG9nLnR4dA==?= (11M) Download Attachment
=?GBK?B?QU0gbG9nLnR4dA==?= (197K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: mapreduce application which only has 17000+ map tasks runs very slow on yarn after 16000+ map completed

Alexey Eremihin

We were experiencing a similar issue with fair scheduler dynamic allocation.

In our case there were most of resources allocated to application reducers and mappers did not have enough resources to start.

That was cleanly seen on MR Application master page.

The solution for it was to specify mapreduce.job.reduce.slowstart.completedmaps to 1. Yes it might a bit delay short queries but for large queries that is essential to allocate enough resources for mappers



On 13.09.2018 03:51, [hidden email] wrote:
Hi everyone!

        I'm running a simple sql(select xx,xx... from viewXXX where xxxxx) using hive0.13.1 on hadoop2.6.0(the framework is MRv2, not tez). After submitting it, I find that it's a MR job which has only 17000+ map tasks and no reduce tasks.

        The job runs very quickly in the early 15 minutes(all 400+ containers on my cluster(20+ nodes) are allocated to run tasks during this period), but become very slow after that(no other jobs running on my cluster).

        I run it a couple of times and find that the number of containers allocated to the job decreases(not strictly but roughly) as the time go on, and after about 15 minutes the number of containers allocated to the job becomes 1(which is the ApplicationMaster's container)! Then the AM is always waitting for RM to give it a container to run map task. RM is not busy(no much GC) and has a lot of containers available(I find that in the RM log), but it assign AM 1 container per MINUTE. So the job finally takes 7 hours to finish.:(

       Parts of my AM,RM log is in the attachment.
       Any help will be appreciated!


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]