Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8079

Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • None
    • native
    • Windows, Azure, Native Platform, Cloud, PaaS, IaaS

    Description

      This JIRA is intended to capture discussion around proposed work to enhance Apache Hadoop to run well on Windows. Apache Hadoop has worked on Microsoft Windows since its inception, but Windows support has never been a priority. Currently Windows works as a development and testing platform for Hadoop, but Hadoop is not natively integrated, full-featured or performance and scalability tuned for Windows Server or Windows Azure. We would like to change this and engage in a dialog with the broader community on the architectural design points for making Windows (enterprise and cloud) an excellent runtime and deployment environment for Hadoop.

      The Isotope team at Microsoft (names below) has developed an Apache Hadoop 1.0 patch set that addresses these performance, integration and feature gaps, allowing Apache Hadoop to be used with Azure and Windows Server without recourse to virtualization technologies such as Cygwin. We have significant interest in the deployment of Hadoop across many multi-tenant, PaaS and IaaS environments - which bring their own unique requirements.

      Microsoft has recently completed a CCLA with Apache and would like to contribute these enhancements back to the Apache Hadoop community.

      In the interest of improving Apache Hadoop so that it runs more smoothly on all platforms, including Windows, we propose first contributing this work to the Apache community by attaching it to this JIRA. From there we would like to work with the community to refine the patch set until it is ready to be merged into the Apache trunk.

      Your feedback solicited,

      Alexander Stojanovic
      Min Wei
      David Lao
      Lengning Liu
      David Zhang
      Asad Khan

      Attachments

        1. windows-cmd-scripts.patch
          27 kB
          David Lao
        2. security.patch
          6 kB
          David Lao
        3. microsoft-windowsazure-api-0.1.2.jar
          659 kB
          Min Wei
        4. mapred-tasks.patch
          25 kB
          David Lao
        5. hadoopcmdscripts.zip
          10 kB
          David Lao
        6. hadoop-8079-branch-1-win.patch
          90 kB
          David Lao
        7. hadoop-8079.patch
          26 kB
          Min Wei
        8. hadoop-8079.AzureBlobStore.patch
          0.6 kB
          Min Wei
        9. general-utils-windows.patch
          10 kB
          David Lao
        10. azurenative.zip
          11 kB
          Min Wei

        Issue Links

          incorporates

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8374 Improve support for hard link manipulation on Windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8409 Fix TestCommandLineJobSubmission and TestGenericOptionsParser to work for windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8411 TestStorageDirecotyFailure, TestTaskLogsTruncater, TestWebHdfsUrl and TestSecurityUtil fail on Windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8412 TestModTime, TestDelegationToken and TestAuthenticationToken fail intermittently on Windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8414 Address problems related to localhost resolving to 127.0.0.1 on Windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8421 Verify and fix build of c++ targets in Hadoop on Windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8424 Web UI broken on Windows because classpath not setup correctly

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8486 Resource leak - Close the open resource handles (File handles) before throwing the exception from the SequenceFile constructor

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8534 Some tests leave a config file open causing failure on windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-3424 TestDatanodeBlockScanner and TestReplication fail intermittently on Windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4201 Getting PID not working on Windows. Termination of Task/TaskJVM's not working

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4263 Use taskkill /T to terminate tasks on Windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4321 DefaultTaskController fails to launch tasks on Windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4368 TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4369 Fix streaming job failures with WindowsResourceCalculatorPlugin

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8454 Fix the ‘chmod =[perm]’ bug in winutils

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8544 Move an assertion location in 'winutils chmod'

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8440 HarFileSystem.decodeHarURI fails for URIs whose host contains numbers

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-8101 Access Control support for Non-secure deployment of Hadoop on Windows

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-3898 Hadoop for Windows - Interfacing with Windows to manage MR tasks

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4203 Create equivalent of ProcfsBasedProcessTree for Windows

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4260 Use JobObject to spawn tasks on Windows

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. HADOOP-8645 Stabilize branch-1-win

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8139 Path does not allow metachars to be escaped

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4322 Fix command-line length abort issues on Windows

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-6814 [WINDOWS] HBase on Windows

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-8900 BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match decompressed size

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3564 Design enhancements to the pluggable blockplacementpolicy

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3566 Custom Replication Policy for Azure

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-240 Rename ProcessTree.isSetsidAvailable

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              stojanovic Alexander Stojanovic
              Votes:
              0 Vote for this issue
              Watchers:
              66 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 2,016h
                  2,016h
                  Remaining:
                  Remaining Estimate - 2,016h
                  2,016h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified