当前位置：首页 > news >正文

【flinkx】【hdfs】【ing】Cannot obtain block length for LocatedBlock

news 2025/6/27 7:57:54

一. 任务描述

使用flinkx去跑HDFS到HIVE的任务时，出现如下报错：

CannotObtainBlockLengthException

com.dtstack.flinkx.throwable.FlinkxRuntimeException: can't get file size from hdfs, file = hdfs://xxx/.data/540240453caeb6fe4b3f118410a05315_235_0at com.dtstack.flinkx.connector.hdfs.sink.BaseHdfsOutputFormat.getCurrentFileSize(BaseHdfsOutputFormat.java:169) ~[flinkx-connector-hdfs-1.12-SNAPSHOT.jar:?]at com.dtstack.flinkx.sink.format.BaseFileOutputFormat.checkCurrentFileSize(BaseFileOutputFormat.java:137) ~[flinkx-core-1.12-SNAPSHOT.jar:?]at com.dtstack.flinkx.sink.format.BaseFileOutputFormat.writeSingleRecordInternal(BaseFileOutputFormat.java:128) ~[flinkx-core-1.12-SNAPSHOT.jar:?]at com.dtstack.flinkx.sink.format.BaseRichOutputFormat.writeSingleRecord(BaseRichOutputFormat.java:427) ~[flinkx-core-1.12-SNAPSHOT.jar:?]at com.dtstack.flinkx.sink.format.BaseRichOutputFormat.writeRecord(BaseRichOutputFormat.java:244) ~[flinkx-core-1.12-SNAPSHOT.jar:?]at com.dtstack.flinkx.sink.format.BaseRichOutputFormat.writeRecord(BaseRichOutputFormat.java:84) ~[flinkx-core-1.12-SNAPSHOT.jar:?]at com.dtstack.flinkx.sink.DtOutputFormatSinkFunction.invoke(DtOutputFormatSinkFunction.java:117) ~[flinkx-core-1.12-SNAPSHOT.jar:?]at org.apache.flink.table.runtime.operators.sink.SinkOperator.processElement(SinkOperator.java:72) ~[flink-table-blink_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at StreamExecCalc$12.processElement(Unknown Source) ~[?:?]at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at com.dtstack.flinkx.source.DtInputFormatSourceFunction.run(DtInputFormatSourceFunction.java:135) ~[flinkx-core-1.12-SNAPSHOT.jar:?]at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66) ~[flink-dist_2.12-1.12.7.jar:1.12.7]at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:267) ~[flink-dist_2.12-1.12.7.jar:1.12.7]
Caused by: org.apache.hadoop.hdfs.CannotObtainBlockLengthException: Cannot obtain block length for LocatedBlock{BP-908532681-10.203.2.20-1593930010000:blk_2513177302_1108300914318; getBlockSize()=0; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.203.70.78:9866,DS-9aa641e8-4ce8-436b-9eaa-fd3eb676185e,DISK], DatanodeInfoWithStorage[10.203.22.205:9866,DS-cb07688e-a349-46eb-8929-71d8fd6156dd,DISK], DatanodeInfoWithStorage[10.203.34.146:9866,DS-efa9e16d-405b-4a05-8e52-0e35d681c3d2,DISK]]}at org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:363) ~[hadoop-hdfs-client-3.2.1U20.jar:?]at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270) ~[hadoop-hdfs-client-3.2.1U20.jar:?]at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:201) ~[hadoop-hdfs-client-3.2.1U20.jar:?]at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:185) ~[hadoop-hdfs-client-3.2.1U20.jar:?]at org.apache.hadoop.hdfs.DFSClient.openInternal(DFSClient.java:1048) ~[hadoop-hdfs-client-3.2.1U20.jar:?]at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1011) ~[hadoop-hdfs-client-3.2.1U20.jar:?]at org.apache.hadoop.hdfs.DistributedFileSystem$5.doCall(DistributedFileSystem.java:359) ~[hadoop-hdfs-client-3.2.1U20.jar:?]at org.apache.hadoop.hdfs.DistributedFileSystem$5.doCall(DistributedFileSystem.java:355) ~[hadoop-hdfs-client-3.2.1U20.jar:?]at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.2.1U20.jar:?]at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:367) ~[hadoop-hdfs-client-3.2.1U20.jar:?]at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:164) ~[hadoop-common-3.2.1U20.jar:?]at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:285) ~[hadoop-common-3.2.1U20.jar:?]at org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:500) ~[hadoop-common-3.2.1U20.jar:?]at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:912) ~[hadoop-common-3.2.1U20.jar:?]at com.dtstack.flinkx.connector.hdfs.sink.BaseHdfsOutputFormat.getCurrentFileSize(BaseHdfsOutputFormat.java:166) ~[flinkx-connector-hdfs-1.12-SNAPSHOT.jar:?]... 24 more

看报错可以得到当时任务的一些消息

文件没有了还去获取文件
文件还在写的时候去获取文件

暂时怀疑flinkx的hdfs写操作会偶发的出现，或者当达到什么机制时后，会出现。

二、google解决方案

Usually when you see “Cannot obtain block length for LocatedBlock”, this means the file is still in being-written state, i.e., it has not been closed yet, and the reader cannot successfully identify its current length by communicating with corresponding DataNodes. There are multiple possibilities here, .e.g., there may be temporary network connection issue between the reader and the DataNodes, or the original writing failed some while ago and the under-construction replicas are somehow missing.

In general you run fsck command to get more information about the file. You can also trigger lease recovery for further debugging. Run command:

hdfs debug recoverLease -path <path-of-the-file> -retries <retrytimes>

This command will ask the NameNode to try to recover the lease for the file, and based on the NameNode log you may track to detailed DataNodes to understand the states of the replicas. The command may successfully close the file if there are still healthy replicas. Otherwise we can get more internal details about the file/block state.

当看到这个消息时：Cannot obtain block length for LocatedBlock,说明文件仍然是写的状态，即写还没有关闭，client无法成功的和datanodes通讯获取到当前文件的长度。

有以下几种可能的原因会出现这种问题：