YARN RM keep trying to set new AMRMToken for applications during the master key rolling period

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

YARN RM keep trying to set new AMRMToken for applications during the master key rolling period

Paul Lam
Hi,

AFAIK, YARN RM rolls master keys used to generate AMRMToken periodically (default to 1 day). Before the new master key takes effect, there's a window that the AMs can update its AMRMToken, during which RM would generate a new AMRMToken and send it back to the AM when it detects an AMRMToken using the old master key in the AM reports.

I'm using hadoop-2.6.0-cdh5.6.0 and I find that in the RM logs, during the whole window RM keeping trying to create and set new tokens for the applications until the rolling period is ended, maybe 100+ times for a single application. It seems that the new AMRMToken is ignored by the AM. But it's strange that the applications could run for from days to weeks before it fails with "Invalid AMRMToken" error.

I tried to and didn't find logs to confirm that the AMRMToken is updated on the AM or NM side, so I'm suspecting it the token update fails somehow. Could anyone help me with this? Thanks a lot!

Best,
Paul Lam


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]