Hive query failed with 'Cannot obtain block length' exception

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Hive query failed with 'Cannot obtain block length' exception

Igor Kuzmenko
Some of my hive queries failing into this exception:

Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: java.io.IOException: Cannot obtain block length for LocatedBlock{BP-2028945280-10.35.63.1-1481200988158:blk_1088997479_15461734; getBlockSize()=6980; corrupt=false; offset=0; locs=
» DatanodeInfoWithStorage
» 10.35.63.20:50010,DS-885ec97e-e948-457f-9ea9-ebc923b487d5,DISK
, DatanodeInfoWithStorage
» 10.35.63.43:50010,DS-bae7e35e-fe36-4aa8-9aad-caffbe89736c,DISK
, DatanodeInfoWithStorage
» 10.35.63.42:50010,DS-e38d612d-1b93-4aed-b490-3a133656ecc5,DISK
}


I've found problem block on 10.35.63.20 data node: 

find /data/ -name blk_1088997479*
 /data/7/hadoop/hdfs/data/current/BP-2028945280-10.35.63.1-1481200988158/current/rbw/blk_1088997479
 /data/7/hadoop/hdfs/data/current/BP-2028945280-10.35.63.1-1481200988158/current/rbw/blk_1088997479_15461734.meta


fsck shows system is healthy:

 Total size:    48329290532734 B (Total open files size: 386901241354 B)
 Total dirs:    290047
 Total files:   417880
 Total symlinks:                0 (Files currently being written: 3765)
 Total blocks (validated):      583003 (avg. block size 82897155 B) (Total open file blocks (not validated): 5003)
 Minimally replicated blocks:   583003 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.001897
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          30
 Number of racks:               3
FSCK ended at Fri May 26 13:03:30 MSK 2017 in 6845 milliseconds


The filesystem under path '/' is HEALTHY


And I can't find a file associated with this block.
Execution of 'hadoop fsck / -files -blocks | grep blk_1088997479' returns nothing.

Any advice what should I do next to localize the problem?