Retrieve number of physical cores from node managers

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Retrieve number of physical cores from node managers

Shashank Prabhakara
Hi All,

Is there an existing API which will retrieve the number of physical cores available in the worker nodes of the cluster(not sure about the term physical cores, but essentially actual number of cpu cores as opposed to vcores) ?

I was not able to find any in RM or NM APIs for 2.8, so if it really is not available, how would you suggest I proceed - my best guess would be a distributed shell-like application with an AM that would run one container per node on all nodes and essentially execute "nproc --all".

I'm exploring my options right now so anything related to physical cores will be helpful, whether it's total cores at cluster level or at a worker level and whether it is the number of physical cores available to yarn or the total irrespective of yarn config.

Thanks in advance.

Regards,
Shashank
Reply | Threaded
Open this post in threaded view
|

Re: Retrieve number of physical cores from node managers

Sandeep Khurana
Hey Shashank 

How are you doing?  I an not sure about existing API but I had written a yarn custom application myself to get info from nodes directly by running Linux command.  
Would be nice to know any API way of doing it . Thx 

On Mon, Aug 13, 2018, 15:47 Shashank Prabhakara <[hidden email]> wrote:
Hi All,

Is there an existing API which will retrieve the number of physical cores available in the worker nodes of the cluster(not sure about the term physical cores, but essentially actual number of cpu cores as opposed to vcores) ?

I was not able to find any in RM or NM APIs for 2.8, so if it really is not available, how would you suggest I proceed - my best guess would be a distributed shell-like application with an AM that would run one container per node on all nodes and essentially execute "nproc --all".

I'm exploring my options right now so anything related to physical cores will be helpful, whether it's total cores at cluster level or at a worker level and whether it is the number of physical cores available to yarn or the total irrespective of yarn config.

Thanks in advance.

Regards,
Shashank