Details
Description
In HDFS-15406, we have a small change to greatly reduce the runtime and locking time of the datanode DirectoryScanner. They may be room for further improvement.
From the scan step, we have captured a snapshot of what is on disk. After calling `dataset.getFinalizedBlocks(bpid);` we have taken a snapshot of in memory. The two snapshots are never 100% in sync as things are always changing as the disk is scanned.
We are only comparing finalized blocks, so they should not really change:
- If a block is deleted after our snapshot, our snapshot will not see it and that is OK.
- A finalized block could be appended. If that happens both the genstamp and length will change, but that should be handled by reconcile when it calls `FSDatasetImpl.checkAndUpdate()`, and there is nothing stopping blocks being appended after they have been scanned from disk, but before they have been compared with memory.
My suspicion is that we can do all the comparison work outside of the lock and checkAndUpdate() re-checks any differences later under the lock on a block by block basis.
Attachments
Attachments
Issue Links
- fixes
-
HDFS-16013 DirectoryScan operation holds dataset lock for long time
- Resolved
- is related to
-
HDFS-15160 ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock
- Resolved
-
HDFS-15406 Improve the speed of Datanode Block Scan
- Resolved