The AvailableVCores of the scheduler queue is a negative number

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

The AvailableVCores of the scheduler queue is a negative number

Huang Meilong
Hi,
I'm getting metrics of scheduler queue with jmx:

I found some negative data points of AvailableVCores, is this a bug of YARN?


timestamp: 1550565127000, yarn.QueueMetrics.root.AvailableVCores=-31
timestamp: 1550565156000, yarn.QueueMetrics.root.AvailableVCores=-31
timestamp: 1550565186000, yarn.QueueMetrics.root.AvailableVCores=-32
timestamp: 1550565220000, yarn.QueueMetrics.root.AvailableVCores=14
timestamp: 1550565250000, yarn.QueueMetrics.root.AvailableVCores=14
Reply | Threaded
Open this post in threaded view
|

答复: The AvailableVCores of the scheduler queue is a negative number

Huang Meilong
hadoop version: 2.7.2
scheduler type: capacity scheduler

发件人: Huang Meilong
发送时间: 2019年2月19日 17:22
收件人: [hidden email]
主题: The AvailableVCores of the scheduler queue is a negative number
 
Hi,
I'm getting metrics of scheduler queue with jmx:

I found some negative data points of AvailableVCores, is this a bug of YARN?


timestamp: 1550565127000, yarn.QueueMetrics.root.AvailableVCores=-31
timestamp: 1550565156000, yarn.QueueMetrics.root.AvailableVCores=-31
timestamp: 1550565186000, yarn.QueueMetrics.root.AvailableVCores=-32
timestamp: 1550565220000, yarn.QueueMetrics.root.AvailableVCores=14
timestamp: 1550565250000, yarn.QueueMetrics.root.AvailableVCores=14
Reply | Threaded
Open this post in threaded view
|

Re: The AvailableVCores of the scheduler queue is a negative number

Prabhu Josephraj
In reply to this post by Huang Meilong
This is expected for DefaultResourceCalculator (Memory based scheduling) where it allocates requested n memory and 1 core (logical) per container.
Say a node has 100GB and 5 cores, 15 containers requested each with 10 GB, 10 containers will be allocated and available node resource will be 0GB and -5 cores. It stops scheduling once memory is exhausted without considering the cores.

In case of DominantResourceCalculator, it considers both memory and cpu, it stops scheduling once any of memory and cpu is exhausted. In above example, it stops scheduling after 5 containers allocated (50GB and 5 cores) with remaining 50GB and 0 cores. 

On Tue, Feb 19, 2019 at 9:59 PM Huang Meilong <[hidden email]> wrote:
Hi,
I'm getting metrics of scheduler queue with jmx:

I found some negative data points of AvailableVCores, is this a bug of YARN?


timestamp: 1550565127000, yarn.QueueMetrics.root.AvailableVCores=-31
timestamp: 1550565156000, yarn.QueueMetrics.root.AvailableVCores=-31
timestamp: 1550565186000, yarn.QueueMetrics.root.AvailableVCores=-32
timestamp: 1550565220000, yarn.QueueMetrics.root.AvailableVCores=14
timestamp: 1550565250000, yarn.QueueMetrics.root.AvailableVCores=14
Reply | Threaded
Open this post in threaded view
|

答复: The AvailableVCores of the scheduler queue is a negative number

Huang Meilong
Thank you Prabhu!

发件人: Prabhu Josephraj <[hidden email]>
发送时间: 2019年2月20日 0:49
收件人: Huang Meilong
抄送: [hidden email]
主题: Re: The AvailableVCores of the scheduler queue is a negative number
 
This is expected for DefaultResourceCalculator (Memory based scheduling) where it allocates requested n memory and 1 core (logical) per container.
Say a node has 100GB and 5 cores, 15 containers requested each with 10 GB, 10 containers will be allocated and available node resource will be 0GB and -5 cores. It stops scheduling once memory is exhausted without considering the cores.

In case of DominantResourceCalculator, it considers both memory and cpu, it stops scheduling once any of memory and cpu is exhausted. In above example, it stops scheduling after 5 containers allocated (50GB and 5 cores) with remaining 50GB and 0 cores. 

On Tue, Feb 19, 2019 at 9:59 PM Huang Meilong <[hidden email]> wrote:
Hi,
I'm getting metrics of scheduler queue with jmx:

I found some negative data points of AvailableVCores, is this a bug of YARN?


timestamp: 1550565127000, yarn.QueueMetrics.root.AvailableVCores=-31
timestamp: 1550565156000, yarn.QueueMetrics.root.AvailableVCores=-31
timestamp: 1550565186000, yarn.QueueMetrics.root.AvailableVCores=-32
timestamp: 1550565220000, yarn.QueueMetrics.root.AvailableVCores=14
timestamp: 1550565250000, yarn.QueueMetrics.root.AvailableVCores=14