Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-4660

[Java Broker] A deadlock can sporadically occur on broker shutdown when stopping a Jetty server which is in the process of sending a response over HTTPS

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.21
    • None
    • Broker-J
    • None
    • RHEL 5.3, JDK 1.7.0_14

    Description

      When running a system test suite org.apache.qpid.systest.rest.BasicAuthRestTest with JDK 1.7 a deadlock might occur on shutting down the broker whilst it still tries to send some data over HTTPS:

      Found one Java-level deadlock:
      =============================
      "qtp538763660-62":
        waiting to lock monitor 0x000000000c91c440 (object 0x00000000f9404e28, a sun.security.ssl.SSLSocketImpl),
        which is held by "Broker-Configuration-Thread"
      "Broker-Configuration-Thread":
        waiting for ownable synchronizer 0x00000000f95c6278, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
        which is held by "qtp538763660-62"
      
      Java stack information for the threads listed above:
      ===================================================
      "qtp538763660-62":
              at sun.security.ssl.SSLSocketImpl.getConnectionState(SSLSocketImpl.java:649)
              - waiting to lock <0x00000000f9404e28> (a sun.security.ssl.SSLSocketImpl)
              at sun.security.ssl.SSLSocketImpl.isClosed(SSLSocketImpl.java:1446)
              at java.net.Socket.getTcpNoDelay(Socket.java:953)
              at sun.security.ssl.BaseSSLSocketImpl.getTcpNoDelay(BaseSSLSocketImpl.java:345)
              at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:819)
              at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
              at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
              - locked <0x00000000f95c60b8> (a sun.security.ssl.AppOutputStream)
              at org.eclipse.jetty.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:354)
              at org.eclipse.jetty.io.bio.StreamEndPoint.flush(StreamEndPoint.java:159)
              at org.eclipse.jetty.io.bio.StreamEndPoint.flush(StreamEndPoint.java:189)
              at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:876)
              at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:438)
              at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:94)
              at org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1016)
              at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:173)
              at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:101)
              at org.eclipse.jetty.util.ByteArrayISO8859Writer.writeTo(ByteArrayISO8859Writer.java:102)
              at org.eclipse.jetty.server.Response.sendError(Response.java:374)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:562)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
              at org.eclipse.jetty.server.Server.handle(Server.java:348)
              at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
              at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
              at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884)
              at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938)
              at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630)
              at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
              at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
              at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
              at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:665)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
              at java.lang.Thread.run(Thread.java:722)
      "Broker-Configuration-Thread":
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000000f95c6278> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
              at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
              at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
              at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:799)
              at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:672)
              at sun.security.ssl.SSLSocketImpl.sendAlert(SSLSocketImpl.java:2007)
              at sun.security.ssl.SSLSocketImpl.warning(SSLSocketImpl.java:1834)
              at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1602)
              - locked <0x00000000f9404e28> (a sun.security.ssl.SSLSocketImpl)
              at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1538)
              at org.eclipse.jetty.io.bio.SocketEndPoint.close(SocketEndPoint.java:156)
              at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.close(SocketConnector.java:233)
              at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.close(SslSocketConnector.java:613)
              at org.eclipse.jetty.server.bio.SocketConnector.doStop(SocketConnector.java:167)
              at org.eclipse.jetty.server.ssl.SslSocketConnector.doStop(SslSocketConnector.java:371)
              at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:84)
              - locked <0x00000000f94069a0> (a java.lang.Object)
              at org.eclipse.jetty.server.Server.doStop(Server.java:318)
              at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:84)
              - locked <0x00000000f9406dc0> (a java.lang.Object)
              at org.apache.qpid.server.management.plugin.HttpManagement.stop(HttpManagement.java:187)
              at org.apache.qpid.server.management.plugin.HttpManagement.setState(HttpManagement.java:156)
              at org.apache.qpid.server.model.adapter.AbstractAdapter.setDesiredState(AbstractAdapter.java:88)
              at org.apache.qpid.server.model.adapter.BrokerAdapter.changeState(BrokerAdapter.java:776)
              - locked <0x00000000f958c040> (a java.util.HashMap)
              at org.apache.qpid.server.model.adapter.BrokerAdapter.setState(BrokerAdapter.java:749)
              at org.apache.qpid.server.model.adapter.AbstractAdapter.setDesiredState(AbstractAdapter.java:88)
              at org.apache.qpid.server.configuration.updater.ChangeStateTask.call(ChangeStateTask.java:59)
              at org.apache.qpid.server.configuration.updater.ChangeStateTask.call(ChangeStateTask.java:28)
              at org.apache.qpid.server.configuration.updater.TaskExecutor.executeTask(TaskExecutor.java:213)
              at org.apache.qpid.server.configuration.updater.TaskExecutor.access$200(TaskExecutor.java:46)
              at org.apache.qpid.server.configuration.updater.TaskExecutor$CallableWrapper$1.run(TaskExecutor.java:252)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.qpid.server.configuration.updater.TaskExecutor$CallableWrapper.call(TaskExecutor.java:247)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
              at java.util.concurrent.FutureTask.run(FutureTask.java:166)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:722)
      
      Found 1 deadlock.
      
      

      It seems that the issue does not exist on JDK 1.6. I've been running tests for 2 days with JDK 1.6.0_33 but was not able to reproduce it.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              orudyy Alex Rudyy
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: