Server hanging on startup

While starting, the server hangs indefinitely. Everything works fine when I first start the server with no index created yet but if I fill the index then stop and start the server, it hangs. Could it be a lock that is never released?

Here is what I get in a full thread dump:

2013-02-06 16:28:52
Full thread dump OpenJDK 64-Bit Server VM (23.2-b09 mixed mode):

"searcherExecutor-4-thread-1" prio=10 tid=0x00007fbdfc16a800 nid=0x42c6 in Object.wait() [0x00007fbe0ab10000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0x00000000c34c1c48> (a java.lang.Object)
  at java.lang.Object.wait(Object.java:503)
  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1492)
• locked <0x00000000c34c1c48> (a java.lang.Object)
  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1312)
  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1247)
  at org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:94)
  at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:213)
  at org.apache.solr.spelling.SpellCheckCollator.collate(SpellCheckCollator.java:112)
  at org.apache.solr.handler.component.SpellCheckComponent.addCollationsToResponse(SpellCheckComponent.java:203)
  at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:180)
  at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
  at org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:64)
  at org.apache.solr.core.SolrCore$5.call(SolrCore.java:1594)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  at java.lang.Thread.run(Thread.java:722)

"coreLoadExecutor-3-thread-1" prio=10 tid=0x00007fbe04194000 nid=0x42c5 in Object.wait() [0x00007fbe0ac11000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0x00000000c34c1c48> (a java.lang.Object)
  at java.lang.Object.wait(Object.java:503)
  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1492)
• locked <0x00000000c34c1c48> (a java.lang.Object)
  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1312)
  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1247)
  at org.apache.solr.handler.ReplicationHandler.getIndexVersion(ReplicationHandler.java:495)
  at org.apache.solr.handler.ReplicationHandler.getStatistics(ReplicationHandler.java:518)
  at org.apache.solr.core.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo(JmxMonitoredMap.java:232)
  at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
  at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
  at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:512)
  at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
  at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
  at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:636)
  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:809)
  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:607)
  at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:1003)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1033)
  at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629)
  at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:624)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  at java.lang.Thread.run(Thread.java:722)

"Thread-3" daemon prio=10 tid=0x00007fbe40742800 nid=0x42c2 waiting on condition [0x00007fbe0ad42000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000ef8e6a08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  at java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:662)
  at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:340)
  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:98)
  at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:273)
  at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:254)
  at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:372)
  at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:98)
  at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4461)
• locked <0x00000000c2e916f8> (a java.util.HashMap)
  at org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5133)
  at org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5128)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  at java.lang.Thread.run(Thread.java:722)

"GC Daemon" daemon prio=10 tid=0x00007fbe405af800 nid=0x42bf in Object.wait() [0x00007fbe383c6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0x00000000c2e46c30> (a sun.misc.GC$LatencyLock)
  at sun.misc.GC$Daemon.run(GC.java:117)
• locked <0x00000000c2e46c30> (a sun.misc.GC$LatencyLock)

"Service Thread" daemon prio=10 tid=0x00007fbe4010c000 nid=0x42bb runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007fbe4010a000 nid=0x42ba waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007fbe40107000 nid=0x42b9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007fbe40104800 nid=0x42b8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007fbe400ad800 nid=0x42b7 in Object.wait() [0x00007fbe444d0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0x00000000c2e46ea8> (a java.lang.ref.ReferenceQueue$Lock)
  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
• locked <0x00000000c2e46ea8> (a java.lang.ref.ReferenceQueue$Lock)
  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
  at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

"Reference Handler" daemon prio=10 tid=0x00007fbe400ab800 nid=0x42b6 in Object.wait() [0x00007fbe445d1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0x00000000c2e46c10> (a java.lang.ref.Reference$Lock)
  at java.lang.Object.wait(Object.java:503)
  at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
• locked <0x00000000c2e46c10> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00007fbe40009800 nid=0x42b1 waiting on condition [0x00007fbe49487000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000c34bdc68> (a java.util.concurrent.FutureTask$Sync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
  at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
  at java.util.concurrent.FutureTask.get(FutureTask.java:111)
  at org.apache.tomcat.util.threads.DedicatedThreadExecutor.execute(DedicatedThreadExecutor.java:56)
  at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5127)
• locked <0x00000000c2e90d70> (a org.apache.catalina.core.StandardContext)
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
• locked <0x00000000c2e90d70> (a org.apache.catalina.core.StandardContext)
  at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1035)
• locked <0x00000000c2e90a08> (a org.apache.catalina.core.StandardHost)
  at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:774)
• locked <0x00000000c2e90a08> (a org.apache.catalina.core.StandardHost)
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
• locked <0x00000000c2e90a08> (a org.apache.catalina.core.StandardHost)
  at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1035)
• locked <0x00000000c2e90850> (a org.apache.catalina.core.StandardEngine)
  at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:291)
• locked <0x00000000c2e90850> (a org.apache.catalina.core.StandardEngine)
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
• locked <0x00000000c2e90850> (a org.apache.catalina.core.StandardEngine)
  at org.apache.catalina.core.StandardService.startInternal(StandardService.java:443)
• locked <0x00000000c2e90850> (a org.apache.catalina.core.StandardEngine)
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
• locked <0x00000000c2e8f640> (a org.apache.catalina.core.StandardService)
  at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:724)
• locked <0x00000000c2e8f628> (a [Lorg.apache.catalina.Service
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
• locked <0x00000000c2e471c0> (a org.apache.catalina.core.StandardServer)
  at org.apache.catalina.startup.Catalina.start(Catalina.java:620)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:601)
  at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:304)
  at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:432)

"VM Thread" prio=10 tid=0x00007fbe400a3000 nid=0x42b5 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fbe40017000 nid=0x42b3 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fbe40019000 nid=0x42b4 runnable

"VM Periodic Task Thread" prio=10 tid=0x00007fbe40116800 nid=0x42bc waiting on condition

JNI global references: 189

Heap
PSYoungGen total 65280K, used 48438K [0x00000000eba00000, 0x00000000f3980000, 0x0000000100000000)
eden space 62720K, 73% used [0x00000000eba00000,0x00000000ee6d19b8,0x00000000ef740000)
from space 2560K, 99% used [0x00000000ef740000,0x00000000ef9bc010,0x00000000ef9c0000)
to space 2560K, 0% used [0x00000000f3700000,0x00000000f3700000,0x00000000f3980000)
ParOldGen total 41728K, used 9077K [0x00000000c2e00000, 0x00000000c56c0000, 0x00000000eba00000)
object space 41728K, 21% used [0x00000000c2e00000,0x00000000c36dd478,0x00000000c56c0000)
PSPermGen total 25280K, used 25218K [0x00000000b8800000, 0x00000000ba0b0000, 0x00000000c2e00000)
object space 25280K, 99% used [0x00000000b8800000,0x00000000ba0a0b50,0x00000000ba0b0000)