[HDFS-5182] BlockReaderLocal must allow zero-copy reads only when the DN believes it's valid - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 3.0.0-alpha1
Fix Version/s: 2.6.0
Component/s: hdfs-client
Labels:
None

Target Version/s:

Description

BlockReaderLocal must allow zero-copy reads only when the DN believes it's valid. This implies adding a new field to the response to REQUEST_SHORT_CIRCUIT_FDS. We also need some kind of heartbeat from the client to the DN, so that the DN can inform the client when the mapped region is no longer locked into memory.

Attachments

Issue Links

is blocked by

HDFS-5634 allow BlockReaderLocal to switch between checksumming and not

Closed

relates to

HDFS-5521 skip checksums when reading a cached block via non-local reads

Resolved

HDFS-5394 fix race conditions in DN caching and uncaching

Closed

Sub-Tasks

1.	add ShortCircuitSharedMemorySegment	Closed	Colin McCabe
2.	Unify mmap cache and short-circuit file descriptor cache	Closed	Colin McCabe
3.	Make BlockReaderFactory#BlockReaderPeer a static class	Closed	Colin McCabe
4.	Minor cleanups to ShortCircuitReplica, FsDatasetCache, and DomainSocketWatcher	Closed	Colin McCabe
5.	add DomainSocket#shutdown method	Closed	Colin McCabe
6.	The DFSClient and DataNode should use shared memory segments to communicate short-circuit information	Closed	Colin McCabe
7.	fix DFSClient issue without libhadoop.so and some other ShortCircuitShm cleanups	Closed	Colin McCabe
8.	add dfs.client.mmap.enabled	Closed	Colin McCabe
9.	Allow dfs.datanode.shared.file.descriptor.path to contain multiple entries and fall back when needed	Closed	Colin McCabe
10.	Fix a case where zero-copy or no-checksum reads were not allowed even when the block was cached	Closed	Colin McCabe
11.	The DataNode should use its shared memory segment to mark short-circuit replicas that have been unlinked as stale	Closed	Colin McCabe
12.	Forcibly timeout misbehaving DFSClients that try to do no-checksum reads that extend too long	Closed	Colin McCabe

Activity

People

Assignee:: Colin McCabe

Reporter:: Colin McCabe

Votes:: 0 Vote for this issue

Watchers:: 12 Start watching this issue

Dates

Created:: 11/Sep/13 00:03

Updated:: 12/May/16 18:16

Resolved:: 08/Sep/14 20:03