Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12352

Delay in checkpointing Trash can leave trash for 2 intervals before deleting

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • 2.7.1
    • 2.8.0, 3.0.0-alpha1
    • trash
    • None
    • Reviewed
    • Fixes an Trash related issue wherein a delay in the periodic checkpointing of one user's directory causes the subsequent user directory checkpoints to carry a newer timestamp, thereby delaying their eventual deletion.

    Description

      When the fs.trash.checkpoint.interval and the fs.trash.interval are set non-zero and the same, it is possible for trash to be left for two intervals.

      The TrashPolicyDefault will use a floor and ceiling function to ensure that the Trash will be checkpointed every "interval" of minutes.

      Each user's trash is checkpointed individually. The time resolution of the checkpoint timestamp is to the second.

      If the seconds switch while one user is checkpointing, then the next user's timestamp will be later.

      This will cause the next user's checkpoint to not be deleted at the next interval.

      I have recreated this in a lab cluster

      I also have a suggestion for a patch that I can upload later tonight after testing it further.

      Attachments

        1. HDFS-8118.patch
          2 kB
          Casey Brotherton
        2. HDFS-8118.001.patch
          2 kB
          Casey Brotherton

        Activity

          People

            caseyjbrotherton Casey Brotherton
            caseyjbrotherton Casey Brotherton
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: