Memory error during hdfs dfs -format

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory error during hdfs dfs -format

Dmitry Goldenberg-2
Hi,

We've got a task in Ansible which returns a MemoryError during HDFS installation on a box with 64 GB memory total, 30 GB free at the moment.

It appears that during the execution of the hdfs dfs -format command, a Python process is spawned which gobbles up ~32ish GB of memory and then the Ansible deploy fails.

Any ideas as to how we could curtail / manage memory consumption better? 

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Memory error during hdfs dfs -format

Arpit Agarwal-2
Hi Dmitry, HDFS commands in Apache Hadoop do not launch any python processes. You'll have to find out more about what the process/command is.


On Tue, Feb 19, 2019, 11:23 Dmitry Goldenberg <[hidden email] wrote:
Hi,

We've got a task in Ansible which returns a MemoryError during HDFS installation on a box with 64 GB memory total, 30 GB free at the moment.

It appears that during the execution of the hdfs dfs -format command, a Python process is spawned which gobbles up ~32ish GB of memory and then the Ansible deploy fails.

Any ideas as to how we could curtail / manage memory consumption better? 

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Memory error during hdfs dfs -format

Anu Engineer
In reply to this post by Dmitry Goldenberg-2

I don’t know of any python process in the Hadoop path that gobbles up that much memory.

Would it be possible that you have some kind of memory (–mx) flags in Namenode Options (Probably in HADOOP_OPTS),

such that the min and max is set to the same value, say 32GB – so that namenode when it boots up is reserving that much memory?

 

Thanks

Anu

 

 

From: Dmitry Goldenberg <[hidden email]>
Date: Tuesday, February 19, 2019 at 11:25 AM
To: "[hidden email]" <[hidden email]>
Subject: Memory error during hdfs dfs -format

 

Hi,

 

We've got a task in Ansible which returns a MemoryError during HDFS installation on a box with 64 GB memory total, 30 GB free at the moment.

 

It appears that during the execution of the hdfs dfs -format command, a Python process is spawned which gobbles up ~32ish GB of memory and then the Ansible deploy fails.

 

Any ideas as to how we could curtail / manage memory consumption better? 

 

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: Memory error during hdfs dfs -format

Dmitry Goldenberg-2
Hi Anu,

I think this is most likely an Ansible issue.

Thanks for the info...

- Dmitry

On Tue, Feb 19, 2019 at 2:40 PM Anu Engineer <[hidden email]> wrote:

I don’t know of any python process in the Hadoop path that gobbles up that much memory.

Would it be possible that you have some kind of memory (–mx) flags in Namenode Options (Probably in HADOOP_OPTS),

such that the min and max is set to the same value, say 32GB – so that namenode when it boots up is reserving that much memory?

 

Thanks

Anu

 

 

From: Dmitry Goldenberg <[hidden email]>
Date: Tuesday, February 19, 2019 at 11:25 AM
To: "[hidden email]" <[hidden email]>
Subject: Memory error during hdfs dfs -format

 

Hi,

 

We've got a task in Ansible which returns a MemoryError during HDFS installation on a box with 64 GB memory total, 30 GB free at the moment.

 

It appears that during the execution of the hdfs dfs -format command, a Python process is spawned which gobbles up ~32ish GB of memory and then the Ansible deploy fails.

 

Any ideas as to how we could curtail / manage memory consumption better? 

 

Thanks