Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11252

RPC client does not time out by default

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.5.0
    • 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • ipc
    • None
    • Incompatible change, Reviewed
    • Hide
      This fix includes public method interface change.
      A follow-up JIRA issue for this incompatibility for branch-2.7 is HADOOP-13579.
      Show
      This fix includes public method interface change. A follow-up JIRA issue for this incompatibility for branch-2.7 is HADOOP-13579 .

    Description

      The RPC client has a default timeout set to 0 when no timeout is passed in. This means that the network connection created will not timeout when used to write data. The issue has shown in YARN-2578 and HDFS-4858. Timeouts for writes then fall back to the tcp level retry (configured via tcp_retries2) and timeouts between the 15-30 minutes. Which is too long for a default behaviour.

      Using 0 as the default value for timeout is incorrect. We should use a sane value for the timeout and the "ipc.ping.interval" configuration value is a logical choice for it. The default behaviour should be changed from 0 to the value read for the ping interval from the Configuration.

      Fixing it in common makes more sense than finding and changing all other points in the code that do not pass in a timeout.

      Offending code lines:
      https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java#L488
      and
      https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java#L350

      Attachments

        1. HADOOP-11252.patch
          5 kB
          wilfreds#1
        2. HADOOP-11252.004.patch
          9 kB
          Masatake Iwasaki
        3. HADOOP-11252.003.patch
          9 kB
          Masatake Iwasaki
        4. HADOOP-11252.002.patch
          10 kB
          Masatake Iwasaki

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14958 CLONE - Fix source-level compatibility after HADOOP-11252

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13579 Fix source-level compatibility after HADOOP-11252

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          duplicates

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-9654 IPC timeout doesn't seem to be kicking in

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Task - A task that needs to be done. HADOOP-11574 Uber-JIRA: improve Hadoop network resilience & diagnostics

          • Major - Major loss of function.
          • Open
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. YARN-2578 NM does not failover timely if RM node network connection fails

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. YARN-5119 Timeout HA issue: Enable IPC ping for all calls by default

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. YARN-2714 Localizer thread might stuck if NM is OOM

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12672 RPC timeout should not override IPC ping interval

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              iwasakims Masatake Iwasaki
              wilfreds Wilfred Spiegelenburg
              Votes:
              2 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: