Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
apurtel cluster
Description
content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 is problem region. below is snippet from master log. Looks to me like things are proceeding in order.
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Server XX.XX.XX.51:60020 is overloaded. Server load: 74 avg: 48.0, slop: 0.1 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Choosing to reassign 26 regions. mostLoadedRegions has 10 regions in it. 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,07730cecbb5b13364cbe17af5778f983,1230448573272 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,29be0a07dece4e9bb318c4d253807287,1230356680860 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,23893c82d792ff5c9d3ff99123bc1773,1230390013767 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,e76475012b59500d25000aa7658bc279,1230498762282 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region result_domain,cn.com.sg.www,1230498785950 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678 2008-12-29 12:19:32,074 INFO org.apache.hadoop.hbase.master.RegionManager: Skipped 0 region(s) that are in transition states 2008-12-29 12:19:32,076 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from XX.XX.XX.51:60020 2008-12-29 12:19:32,076 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,42b19221f3a60945595b18017d6bacfa,1230357228381 open on XX.XX.XX.51:60020 2008-12-29 12:19:32,076 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,42b19221f3a60945595b18017d6bacfa,1230357228381 in region .META.,,1 with startcode 1230571104777 and server XX.XX.XX.51:60020 2008-12-29 12:19:32,077 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from XX.XX.XX.51:60020 2008-12-29 12:19:32,077 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,3c619bc7a2c824fb3c4f5d747b0f2f12,1230354673241 open on XX.XX.XX.51:60020 2008-12-29 12:19:32,077 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,3c619bc7a2c824fb3c4f5d747b0f2f12,1230354673241 in region .META.,,1 with startcode 1230571104777 and server XX.XX.XX.51:60020 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,36946541ed9a62f419cf7238d32a6a38,1230448587552, false 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,58221f5f3a79b7dd5e23c339c507a7c6,1230487059978, false 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,94b170cae87992c33c3913d74bb257e4,1230498360162, false 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,bb34d860d76c1eabbe2eb7d2ce8148d9,1230416266955, false 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,8ac5fbdd63b313ce252838a189c73438,1230530000480, false 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,f9f5b2d035349cf4292ed7dd7e9823f0,1230505507313, false 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,4b30e2a8b7949ed62e097824969f8bca,1230448501152, false 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,0c4b7c2825f674510a511233e9e0c539,1230530122456, false 2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,43f83c42e08bb0f4b1f6d1125f39f018,1230498357831, false 2008-12-29 12:19:32,299 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,43f83c42e08bb0f4b1f6d1125f39f018,1230498357831 to server XX.XX.XX.34:60020 2008-12-29 12:19:32,300 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,bb34d860d76c1eabbe2eb7d2ce8148d9,1230416266955 to server XX.XX.XX.34:60020 2008-12-29 12:19:32,301 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,94b170cae87992c33c3913d74bb257e4,1230498360162 to server XX.XX.XX.34:60020 2008-12-29 12:19:32,301 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,8ac5fbdd63b313ce252838a189c73438,1230530000480 to server XX.XX.XX.34:60020 2008-12-29 12:19:32,302 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,36946541ed9a62f419cf7238d32a6a38,1230448587552 to server XX.XX.XX.34:60020 2008-12-29 12:19:32,302 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,58221f5f3a79b7dd5e23c339c507a7c6,1230487059978 to server XX.XX.XX.34:60020 2008-12-29 12:19:32,303 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,f9f5b2d035349cf4292ed7dd7e9823f0,1230505507313 to server XX.XX.XX.34:60020 2008-12-29 12:19:32,303 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,4b30e2a8b7949ed62e097824969f8bca,1230448501152 to server XX.XX.XX.34:60020 2008-12-29 12:19:32,304 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,0c4b7c2825f674510a511233e9e0c539,1230530122456 to server XX.XX.XX.34:60020 2008-12-29 12:19:35,080 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426 from XX.XX.XX.51:60020 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,07730cecbb5b13364cbe17af5778f983,1230448573272 from XX.XX.XX.51:60020 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426, false 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882 from XX.XX.XX.51:60020 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,07730cecbb5b13364cbe17af5778f983,1230448573272, false 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,29be0a07dece4e9bb318c4d253807287,1230356680860 from XX.XX.XX.51:60020 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882, false 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,23893c82d792ff5c9d3ff99123bc1773,1230390013767 from XX.XX.XX.51:60020 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,29be0a07dece4e9bb318c4d253807287,1230356680860, false 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,e76475012b59500d25000aa7658bc279,1230498762282 from XX.XX.XX.51:60020 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,23893c82d792ff5c9d3ff99123bc1773,1230390013767, false 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: result_domain,cn.com.sg.www,1230498785950 from XX.XX.XX.51:60020 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,e76475012b59500d25000aa7658bc279,1230498762282, false 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914 from XX.XX.XX.51:60020 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of result_domain,cn.com.sg.www,1230498785950, false 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 from XX.XX.XX.51:60020 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914, false 2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678 from XX.XX.XX.51:60020 2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322, false 2008-12-29 12:19:35,082 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: ProcessRegionClose of content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678, false 2008-12-29 12:19:35,083 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000, call regionServerReport(address: XX.XX.XX.51:60020, startcode: 1230571104777, load: (requests=0, regions=64, usedHeap=99, maxHeap=1777), [Lorg. apache.hadoop.hbase.HMsg;@3e2dce4e, [Lorg.apache.hadoop.hbase.HRegionInfo;@314382c6) from XX.XX.XX.51:55371: error: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as clo sing. Region: content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as closing. Region: content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1108) at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:816) at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:377) at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:327) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:240) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:570) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)