Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-11506 Improvements for large scale deletion
  3. HDDS-11510

All deletion services should support multiple threads

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • OM, SCM
    • None

    Description

      When a large amount of data is deleted, we may see slow progress to clear it out due to the large backlog of pending deletes, even if all deletion services are running for their full intervals and collecting their maximum number of entries. In order to speed up deletion further, more threads will need to be configured. However, the deletion algorithms in OM and SCM currently only support single threaded implementations. The algorithms for these jobs need to be updates to partition the pending delete key space to support any number of deletion threads working over it.

      This Jira is relevant for the following components and these may be done in separate subtasks:

      • OM open key cleanup
      • OM key delete
      • OM directory delete
      • SCM block delete
        Datanode block deletion already supports a configurable thread pool that divides work by container.
         

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              erose Ethan Rose
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: