What is the best way to analyze io latency in hdfs?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

What is the best way to analyze io latency in hdfs?

Daegyu Han
Hi all,

I'm currently studying HDFS, and I want to analyze HDFS io latency.

I know that C / C ++ programs can use perf and ftrace under Linux to analyze user level and kernel level latency measurements and overhead.

I would like to analyze the read io latency in HDFS to user level (HDFS) and system level (kernel I / O stack). 

Which way is the best?

Thank you.


Reply | Threaded
Open this post in threaded view
|

Re: What is the best way to analyze io latency in hdfs?

Julien Laurenceau
Hi,
On Linux you can monitor système call of any process using:

strace -p PIDofHDFSdatanode

It can be very verbose but the information will be there.

Did you try metrics available in ambari or cloudera manager ?

Regards

Le mar. 20 août 2019 à 02:47, Daegyu Han <[hidden email]> a écrit :
Hi all,

I'm currently studying HDFS, and I want to analyze HDFS io latency.

I know that C / C ++ programs can use perf and ftrace under Linux to analyze user level and kernel level latency measurements and overhead.

I would like to analyze the read io latency in HDFS to user level (HDFS) and system level (kernel I / O stack). 

Which way is the best?

Thank you.