Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-1735

Insert data in hbase through phoenix JDBC when restart hbase service

    XMLWordPrintableJSON

Details

    Description

      use four clients put data in hbase through phoenix-JDBC when restart hbase service。I discover hbase break down.
      use:
      hbase hbck

      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table CARD_INDEX_SALT
      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table SYSTEM.CATALOG
      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table TZJT_CAMERA_SALT
      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table T
      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table TT
      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:12:00,026 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table TT3
      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:12:00,026 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table hbase:namespace
      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table TABLENAME
      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table SYSTEM.SEQUENCE
      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table SYSTEM.STATS
      ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
      2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
      ERROR: Found inconsistency in table LOCATION_INDEX_SALT
      2015-03-14 10:12:01,248 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181 sessionTimeout=30000 watcher=hbase Fsck, quorum=hadoop1:2181,hadoop2:2181,hadoop3:2181, baseZNode=/hbase-unsecure
      2015-03-14 10:12:01,248 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hadoop1:2181,hadoop2:2181,hadoop3:2181
      2015-03-14 10:12:01,249 INFO [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Opening socket connection to server hadoop1/10.165.97.7:2181. Will not attempt to authenticate using SASL (unknown error)
      2015-03-14 10:12:01,249 INFO [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Socket connection established to hadoop1/10.165.97.7:2181, initiating session
      2015-03-14 10:12:01,252 INFO [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Session establishment complete on server hadoop1/10.165.97.7:2181, sessionid = 0x14c1337679e00f9, negotiated timeout = 30000
      Summary:
      IDX_CC is okay.
      Number of regions: 0
      Deployed on:
      CARD_INDEX is okay.
      Number of regions: 0
      Deployed on:
      US_POPULATION is okay.
      Number of regions: 1
      Deployed on: hadoop4,60020,1426296975241
      EXAMPLE2 is okay.
      Number of regions: 1
      Deployed on: hadoop2,60020,1426296977835
      Table SKIPDEMO is inconsistent.
      Number of regions: 9
      Deployed on: hadoop3,60020,1426296976617 hadoop4,60020,1426296975241
      STUDENT is okay.
      Number of regions: 0
      Deployed on:
      LOCATION_INDEX is okay.
      Number of regions: 0
      Deployed on:
      TZJT_CAMERA is okay.
      Number of regions: 0
      Deployed on:
      hbase:meta is okay.
      Number of regions: 1
      Deployed on: hadoop2,60020,1426296977835
      CARD_INDEX_SALT is okay.
      Number of regions: 0
      Deployed on:
      SYSTEM.CATALOG is okay.
      Number of regions: 0
      Deployed on:
      TZJT_CAMERA_SALT is okay.
      Number of regions: 0
      Deployed on:
      T is okay.
      Number of regions: 0
      Deployed on:
      TT is okay.
      Number of regions: 0
      Deployed on:
      TT3 is okay.
      Number of regions: 0
      Deployed on:
      hbase:namespace is okay.
      Number of regions: 0
      Deployed on:
      TABLENAME is okay.
      Number of regions: 0
      Deployed on:
      SYSTEM.SEQUENCE is okay.
      Number of regions: 0
      Deployed on:
      SYSTEM.STATS is okay.
      Number of regions: 0
      Deployed on:
      LOCATION_INDEX_SALT is okay.
      Number of regions: 0
      Deployed on:
      462 inconsistencies detected.
      Status: INCONSISTENT
      2015-03-14 10:12:01,255 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService

      so,use 'hbase hbck -fixMeta -fixAssignments hbase:namespace ','hbase hbck -repair hbase:namespace' show:

      2015-03-14 10:18:14,988 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned:

      {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

      2015-03-14 10:18:15,994 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned:

      {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

      2015-03-14 10:18:17,000 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned:

      {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

      2015-03-14 10:18:18,005 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned:

      {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

      2015-03-14 10:18:19,011 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned:

      {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

      2015-03-14 10:18:24,041 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
      2015-03-14 10:18:24,041 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24c15ce67b80047
      2015-03-14 10:18:24,045 INFO [main] zookeeper.ZooKeeper: Session: 0x24c15ce67b80047 closed
      2015-03-14 10:18:24,045 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
      2015-03-14 10:18:24,045 DEBUG [main] ipc.RpcClient: Stopping rpc client
      Exception in thread "main" java.io.IOException: Region

      {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

      failed to move out of transition within timeout 120000ms
      at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:141)
      at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1798)
      at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1969)
      at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1579)
      at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:473)
      at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:492)
      at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4213)
      at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4022)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
      at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4010)

      ------------------------------
      the hbase still not normal.
      I don't know why ?
      Is bug ?
      My operational problems?
      Can someone help me
      Thanks!

      Attachments

        Activity

          People

            Unassigned Unassigned
            limu wubiandelvbo
            Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 48h
                48h
                Remaining:
                Remaining Estimate - 48h
                48h
                Logged:
                Time Spent - Not Specified
                Not Specified