question about nnbench

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

question about nnbench

Xiangfei Ni

Hi Expert,

  I am using nnbench to test the namenode performance,I am using this command:

  hadoop jar /opt/hadoop-2.7.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.4.jar nnbench -operation create_write -maps 10 -reduces 5 -numberOfFiles 100 -replicationFactorPerFile 2 -blockSize 128 -bytesToWrite 20  -readFileAfterOpen true

  and I got below output:

18/05/17 16:59:29 INFO hdfs.NNBench: -------------- NNBench -------------- :

18/05/17 16:59:29 INFO hdfs.NNBench:                                Version: NameNode Benchmark 0.4

18/05/17 16:59:29 INFO hdfs.NNBench:                            Date & time: 2018-05-17 16:59:29,870

18/05/17 16:59:29 INFO hdfs.NNBench:

18/05/17 16:59:29 INFO hdfs.NNBench:                         Test Operation: create_write

18/05/17 16:59:29 INFO hdfs.NNBench:                             Start time: 2018-05-17 16:59:22,127

18/05/17 16:59:29 INFO hdfs.NNBench:                            Maps to run: 10

18/05/17 16:59:29 INFO hdfs.NNBench:                         Reduces to run: 5

18/05/17 16:59:29 INFO hdfs.NNBench:                     Block Size (bytes): 128

18/05/17 16:59:29 INFO hdfs.NNBench:                         Bytes to write: 20

18/05/17 16:59:29 INFO hdfs.NNBench:                     Bytes per checksum: 1

18/05/17 16:59:29 INFO hdfs.NNBench:                        Number of files: 100

18/05/17 16:59:29 INFO hdfs.NNBench:                     Replication factor: 2

18/05/17 16:59:29 INFO hdfs.NNBench:             Successful file operations: 0

18/05/17 16:59:29 INFO hdfs.NNBench:

18/05/17 16:59:29 INFO hdfs.NNBench:         # maps that missed the barrier: 0

18/05/17 16:59:29 INFO hdfs.NNBench:                           # exceptions: 10000

18/05/17 16:59:29 INFO hdfs.NNBench:

18/05/17 16:59:29 INFO hdfs.NNBench:                TPS: Create/Write/Close: 0

18/05/17 16:59:29 INFO hdfs.NNBench: Avg exec time (ms): Create/Write/Close: Infinity

18/05/17 16:59:29 INFO hdfs.NNBench:             Avg Lat (ms): Create/Write: NaN

18/05/17 16:59:29 INFO hdfs.NNBench:                    Avg Lat (ms): Close: NaN

18/05/17 16:59:29 INFO hdfs.NNBench:

18/05/17 16:59:29 INFO hdfs.NNBench:                  RAW DATA: AL Total #1: 0

18/05/17 16:59:29 INFO hdfs.NNBench:                  RAW DATA: AL Total #2: 0

18/05/17 16:59:29 INFO hdfs.NNBench:               RAW DATA: TPS Total (ms): 14924

18/05/17 16:59:29 INFO hdfs.NNBench:        RAW DATA: Longest Map Time (ms): 1838.0

18/05/17 16:59:29 INFO hdfs.NNBench:                    RAW DATA: Late maps: 0

18/05/17 16:59:29 INFO hdfs.NNBench:              RAW DATA: # of exceptions: 10000

 

My question is that how to understand the output,how to measure the performance by the output.is there any official instruction about this?

 

Yet another performance test :

[apache@namenode01 sbin]$ hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs hdfs://namenode01:8020 -op create -threads 100 -files 100000 -filesPerDir 4 -close

18/05/18 11:55:48 INFO namenode.NNThroughputBenchmark: Starting benchmark: create

18/05/18 11:55:48 INFO namenode.NNThroughputBenchmark: Generate 100000 intputs for create

18/05/18 11:55:49 FATAL namenode.NNThroughputBenchmark: Log level = ERROR

18/05/18 11:55:49 INFO namenode.NNThroughputBenchmark: Starting 100000 create(s).

18/05/18 11:56:03 INFO namenode.NNThroughputBenchmark:

18/05/18 11:56:03 INFO namenode.NNThroughputBenchmark: --- create inputs ---

18/05/18 11:56:03 INFO namenode.NNThroughputBenchmark: nrFiles = 100000

18/05/18 11:56:03 INFO namenode.NNThroughputBenchmark: nrThreads = 100

18/05/18 11:56:03 INFO namenode.NNThroughputBenchmark: nrFilesPerDir = 4

18/05/18 11:56:03 INFO namenode.NNThroughputBenchmark: --- create stats  ---

18/05/18 11:56:03 INFO namenode.NNThroughputBenchmark: # operations: 100000  ###number of files has been created

18/05/18 11:56:03 INFO namenode.NNThroughputBenchmark: Elapsed Time: 14041   ###how much time has been used to create 100000 files

18/05/18 11:56:03 INFO namenode.NNThroughputBenchmark:  Ops per sec: 7121.999857560003  ### how many files has been created per second

18/05/18 11:56:03 INFO namenode.NNThroughputBenchmark: Average Time: 6   ######## what dose this number mean?how to calculate this?###########

 

Which test tool do you prefer?

 

Any help would be highly appreciated!

 

 

 

Best Regards,

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516