Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15281

Distcp to add no-rename copy option

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.3.0, 3.2.1, 3.1.3
    • tools/distcp
    • None

    Description

      Currently Distcp uploads a file by two strategies

      1. append parts
      2. copy to temp then rename

      option 2 executes the following sequence in promoteTmpToTarget

          if ((fs.exists(target) && !fs.delete(target, false))
        || (!fs.exists(target.getParent()) && !fs.mkdirs(target.getParent()))
        || !fs.rename(tmpTarget, target)) {
      throw new IOException("Failed to promote tmp-file:" + tmpTarget
                              + " to: " + target);
    }

      For any object store, that's a lot of HTTP requests; for S3A you are looking at 12+ requests and an O(data) copy call.

      This is not a good upload strategy for any store which manifests its output atomically at the end of the write().

      Proposed: add a switch to write directly to the dest path, which can be supplied as either a conf option (distcp.direct.write = true) or a CLI option (-direct).

      Attachments

        1. HADOOP-15281-branch-2-002.patch
          26 kB
          Andrew Olson
        2. HADOOP-15281-branch-2-001.patch
          26 kB
          Steve Loughran
        3. HADOOP-15281-004.patch
          22 kB
          Andrew Olson
        4. HADOOP-15281-003.patch
          22 kB
          Andrew Olson
        5. HADOOP-15281-002.patch
          22 kB
          Andrew Olson
        6. HADOOP-15281-001.patch
          21 kB
          Andrew Olson

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16096 HADOOP-15281/distcp -Xdirect needs to use commons-logging on 3.1

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is depended upon by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-10007 distcp / mv is not working on ftp

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15788 Improve Distcp for long-haul/cloud deployments

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15577 Update distcp to use zero-rename s3 committers

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-16047 Avoid expensive rename when DistCp is writing to S3

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13622 `-atomic` should not be supported while using `distcp` command in object file system

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          is required by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-10007 distcp / mv is not working on ftp

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15209 DistCp to eliminate needless deletion of files under already-deleted directories

          • Major - Major loss of function.
          • Resolved
          supercedes

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16260 Allow Distcp to create a new tempTarget file per File

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-12046 Avoid creating "._COPYING_" temporary file when copying file to Swift file system

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #469

          Activity

            People

              noslowerdna Andrew Olson
              stevel@apache.org Steve Loughran
              Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: