Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17193

Compile Hadoop on Windows natively

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.3
    • None
    • build, native
    • None

    • Windows 10

    Description

      Hadoop compiles flawlessly on Linux whereas, one has to hack around to compile it on Windows. The reason is obvious, there's a very large proportion of C/C++ codebase that's written in a platform specific manner that's amenable to Linux.

      Thus, compiling Hadoop on Windows involves using the MinGW or the Cygwin toolchains. This third party dependency might be seen as a risk for those who want to deploy Hadoop on Windows nodes in production. The primary reason for this is that the MinGW/Cygwin runtimes on Windows don't produce core dumps. Whereas the Visual C++ runtime is able to do so, which greatly enables debugging.

      Another reason to use the Visual C++ compiler for compiling Hadoop on Windows is that Visual C++ is native to Windows and one will have access to a large collection of tools that can be used for instrumentation, telemetry and fine-tuning the settings of Hadoop to run better on Windows.

      Modern C++ has a plethora of APIs that one can use to write safe, cross-platform code, which can be leveraged to refactor the existing Linux specific code, thereby enabling compilation on Windows using the native Visual C++ compiler.

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18133 Add Dockerfile for Windows 10

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is a parent of

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16300 Use libcrypto in Windows for libhdfspp

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16665 Fix duplicate sources for hdfspp_test_shim_static

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17196 Fix C/C++ standard warnings

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18357 Retarget solution file to VS2019

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18428 Parameterize platform toolset version

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16178 Make recursive rmdir in libhdfs++ cross platform

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16205 Make hdfs_allowSnapshot tool cross platform

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16564 Use uint32_t for hdfs_find

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16654 Link OpenSSL lib for CMake deps check

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16667 Use malloc for buffer allocation in uriparser2

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16680 Skip libhdfspp Valgrind tests on Windows

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16681 Do not pass GCC flags for MSVC in libhdfspp

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15385 Upgrade boost library to 1.72

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15740 Make basename cross-platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15843 [libhdfs++] Make write cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15909 Make fnmatch cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15955 Make explicit_bzero cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15962 Make strcasecmp cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15971 Make mkstemp cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15976 Make mkdtemp cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16251 Make hdfs_cat tool cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16260 Make hdfs_deleteSnapshot tool cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16267 Make hdfs_df tool cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16278 Make HDFS snapshot tools cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16285 Make HDFS ownership tools cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16407 Make hdfs_du tool cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16419 Make HDFS data transfer tools cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16445 Make HDFS count, mkdir, rm cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16462 Make HDFS get tool cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16463 Make dirent cross platform compatible

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16464 Create only libhdfspp static libraries for Windows

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16465 Remove redundant strings.h inclusions

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16466 Implement Linux permission flags on Windows

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16467 Ensure Protobuf generated headers are included first

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16468 Define ssize_t for Windows

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16469 Locate protoc-gen-hrpc across platforms

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16470 Make HDFS find tool cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16471 Make HDFS ls tool cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16472 Make HDFS setrep tool cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16473 Make HDFS stat tool cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16474 Make HDFS tail tool cross platform

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16666 Pass CMake args for Windows in pom.xml

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-11078 Set env vars in a cross platform compatible way

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15903 Refactor X-Platform library

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18729 Fix mvnsite on Windows 10

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17399 lz4 sources missing for native Visual Studio project

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-13223 winutils.exe is a bug nexus and should be killed with an axe.

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15712 Upgrade googletest to 1.10.0

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              gaurava Gautham Banasandra
              gaurava Gautham Banasandra
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: