Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
Reduce the output stream buffer size of a DFSClient remote read from 8KB to 512 bytes.
Description
this issue is similar to HDFS-14535.
public static BlockReader newBlockReader(String file, ExtendedBlock block, Token<BlockTokenIdentifier> blockToken, long startOffset, long len, boolean verifyChecksum, String clientName, Peer peer, DatanodeID datanodeID, PeerCache peerCache, CachingStrategy cachingStrategy, int networkDistance) throws IOException { // in and out will be closed when sock is closed (by the caller) final DataOutputStream out = new DataOutputStream(new BufferedOutputStream( peer.getOutputStream())); new Sender(out).readBlock(block, blockToken, clientName, startOffset, len, verifyChecksum, cachingStrategy); } public BufferedOutputStream(OutputStream out) { this(out, 8192); }
Sender#readBlock parameter( block,blockToken, clientName, startOffset, len, verifyChecksum, cachingStrategy) could not use such a big buffer.
So i think it should reduce BufferedOutputStream buffer.
Attachments
Attachments
Issue Links
- is related to
-
HDFS-14844 Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable
- Resolved
- relates to
-
HDFS-14535 The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
- Resolved