Description
When a large amount of data is deleted, we may see slow progress to clear it out due to the large backlog of pending deletes, even if all deletion services are running for their full intervals and collecting their maximum number of entries. In order to speed up deletion further, more threads will need to be configured. However, the deletion algorithms in OM and SCM currently only support single threaded implementations. The algorithms for these jobs need to be updates to partition the pending delete key space to support any number of deletion threads working over it.
This Jira is relevant for the following components and these may be done in separate subtasks:
- OM open key cleanup
- OM key delete
- OM directory delete
- SCM block delete
Datanode block deletion already supports a configurable thread pool that divides work by container.
Attachments
Issue Links
- is required by
-
HDDS-11511 All deletion services should have consistent metrics
- Open
-
HDDS-11513 All deletion configurations should be configurable without restart
- Open