Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15788

Improve Distcp for long-haul/cloud deployments

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.0
    • None
    • tools/distcp
    • None

    Description

      There are a number of outstanding distcp options related to: extensibility, failure reporting/cleanup, long-haul options, cloud performance.

      Hadoop 3.1 added some speedups; follow this up with others.

      Attachments

        Issue Links

          depends upon

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8233 Turn CRC checking off for 0 byte size and differing blocksizes

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13002 distcp behaves differently through code compared to toolrunner invocation from command-line

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14631 Distcp should add a default atomicWorkPath properties when using atomic

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-15790 distcp app should fail if m/r job fails

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-12900 distcp -delete should show a counter of files deleted

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14567 DistCP NullPointerException when -atomic is set but -tmp is not

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HDFS-9455 In distcp, Invalid Argument Error thrown in case of filesystem operation failure

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-15300 distcp -update to WASB and ADL copies up all the files, always

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14544 DistCp documentation for command line options is misaligned.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-15789 DistCp does not clean staging folder if class extends DistCp

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. KNOX-482 Support DistCp via Knox

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-11234 distcp performance is suboptimal for high bandwidth/high latency setups

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-11043 Use Path Instead of String in DistCp Tests

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14086 Improve DistCp Speed for small files

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6840 Distcp to support cutoff time

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15281 Distcp to add no-rename copy option

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-3889 distcp overwrites files even when there are missing checksums

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              Unassigned Unassigned
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: