Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-6748

Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.94.6, 0.95.0
    • 0.94.6, 0.95.0
    • master
    • None
    • Reviewed

    Description

      You can ealily understand the problem from the below logs:

      [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3
[2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2
[2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1
[2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0
[2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
[2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
[2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807
[2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806
[2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805
[2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804
[2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775803
...................
[2012-09-01 11:41:03,307] [ERROR] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.zookeeper.ClientCnxn 623] Caught unexpected throwable
java.lang.StackOverflowError

      Attachments

        1. hbase-6748.patch
          7 kB
          Jeffrey Zhong
        2. hbase-6748_1.patch
          8 kB
          Jeffrey Zhong

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-7914 Port the fix of HBASE-6748 into 0.94 branch

          • Major - Major loss of function.
          • Closed

          Activity

            People

              jeffreyz Jeffrey Zhong
              jeason Jieshan Bean
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: