Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13341

Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 3.0.0-alpha2
    • scripts
    • None
    • Incompatible change
    • Hide
      <!-- markdown -->
      Users:
      * Ability to set per-command+sub-command options from the command line.
      * Makes daemon environment variable options consistent across the project. (See deprecation list below)
      * HADOOP\_CLIENT\_OPTS is now honored for every non-daemon sub-command. Prior to this change, many sub-commands did not use it.

      Developers:
      * No longer need to do custom handling for options in the case section of the shell scripts.
      * Consolidates all \_OPTS handling into hadoop-functions.sh to enable future projects.
      * All daemons running with secure mode features now get \_SECURE\_EXTRA\_OPTS support.

      \_OPTS Changes:

      | Old | New |
      |:---- |:---- |
      | HADOOP\_BALANCER\_OPTS | HDFS\_BALANCER\_OPTS |
      | HADOOP\_DATANODE\_OPTS | HDFS\_DATANODE\_OPTS |
      | HADOOP\_DN\_SECURE_EXTRA_OPTS | HDFS\_DATANODE\_SECURE\_EXTRA\_OPTS |
      | HADOOP\_JOB\_HISTORYSERVER\_OPTS | MAPRED\_HISTORYSERVER\_OPTS |
      | HADOOP\_JOURNALNODE\_OPTS | HDFS\_JOURNALNODE\_OPTS |
      | HADOOP\_MOVER\_OPTS | HDFS\_MOVER\_OPTS |
      | HADOOP\_NAMENODE\_OPTS | HDFS\_NAMENODE\_OPTS |
      | HADOOP\_NFS3\_OPTS | HDFS\_NFS3\_OPTS |
      | HADOOP\_NFS3\_SECURE\_EXTRA\_OPTS | HDFS\_NFS3\_SECURE\_EXTRA\_OPTS |
      | HADOOP\_PORTMAP\_OPTS | HDFS\_PORTMAP\_OPTS |
      | HADOOP\_SECONDARYNAMENODE\_OPTS | HDFS\_SECONDARYNAMENODE\_OPTS |
      | HADOOP\_ZKFC\_OPTS | HDFS\_ZKFC\_OPTS |
      Show
      <!-- markdown --> Users: * Ability to set per-command+sub-command options from the command line. * Makes daemon environment variable options consistent across the project. (See deprecation list below) * HADOOP\_CLIENT\_OPTS is now honored for every non-daemon sub-command. Prior to this change, many sub-commands did not use it. Developers: * No longer need to do custom handling for options in the case section of the shell scripts. * Consolidates all \_OPTS handling into hadoop-functions.sh to enable future projects. * All daemons running with secure mode features now get \_SECURE\_EXTRA\_OPTS support. \_OPTS Changes: | Old | New | |:---- |:---- | | HADOOP\_BALANCER\_OPTS | HDFS\_BALANCER\_OPTS | | HADOOP\_DATANODE\_OPTS | HDFS\_DATANODE\_OPTS | | HADOOP\_DN\_SECURE_EXTRA_OPTS | HDFS\_DATANODE\_SECURE\_EXTRA\_OPTS | | HADOOP\_JOB\_HISTORYSERVER\_OPTS | MAPRED\_HISTORYSERVER\_OPTS | | HADOOP\_JOURNALNODE\_OPTS | HDFS\_JOURNALNODE\_OPTS | | HADOOP\_MOVER\_OPTS | HDFS\_MOVER\_OPTS | | HADOOP\_NAMENODE\_OPTS | HDFS\_NAMENODE\_OPTS | | HADOOP\_NFS3\_OPTS | HDFS\_NFS3\_OPTS | | HADOOP\_NFS3\_SECURE\_EXTRA\_OPTS | HDFS\_NFS3\_SECURE\_EXTRA\_OPTS | | HADOOP\_PORTMAP\_OPTS | HDFS\_PORTMAP\_OPTS | | HADOOP\_SECONDARYNAMENODE\_OPTS | HDFS\_SECONDARYNAMENODE\_OPTS | | HADOOP\_ZKFC\_OPTS | HDFS\_ZKFC\_OPTS |

    Description

      Big features like YARN-2928 demonstrate that even senior level Hadoop developers forget that daemons need a custom _OPTS env var. We can replace all of the custom vars with generic handling just like we do for the username check.

      For example, with generic handling in place:

      Old Var New Var
      HADOOP_NAMENODE_OPTS HDFS_NAMENODE_OPTS
      YARN_RESOURCEMANAGER_OPTS YARN_RESOURCEMANAGER_OPTS
      n/a YARN_TIMELINEREADER_OPTS
      n/a HADOOP_DISTCP_OPTS
      n/a MAPRED_DISTCP_OPTS
      HADOOP_DN_SECURE_EXTRA_OPTS HDFS_DATANODE_SECURE_EXTRA_OPTS
      HADOOP_NFS3_SECURE_EXTRA_OPTS HDFS_NFS3_SECURE_EXTRA_OPTS
      HADOOP_JOB_HISTORYSERVER_OPTS MAPRED_HISTORYSERVER_OPTS

      This makes it:

      a) consistent across the entire project
      b) consistent for every subcommand
      c) eliminates almost all of the custom appending in the case statements

      It's worth pointing out that subcommands like distcp that sometimes need a higher than normal client-side heapsize or custom options are a huge win. Combined with .hadooprc and/or dynamic subcommands, it means users can easily do customizations based upon their needs without a lot of weirdo shell aliasing or one line shell scripts off to the side.

      Attachments

        1. HADOOP-13341.00.patch
          54 kB
          Allen Wittenauer

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6904 HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is required by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13365 Convert _OPTS to arrays to enable spaces in file paths

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13595 Rework hadoop_usage to be broken up by clients/daemons/etc.

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          links to

          Web Link GitHub Pull Request #126

          Activity

            People

              aw Allen Wittenauer
              aw Allen Wittenauer
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: