Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-5711

Zeppelin0.10: deadlock on zeppelin-server

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.10.1
    • None
    • zeppelin-server

    Description

      there is deadlock in zeppelin server, which is a fatal flaw.

      the jstack of zeppelin-server process is :

      Found one Java-level deadlock:
      =============================
      "qtp1593180232-305212":
        waiting to lock monitor 0x00007f541003fce8 (object 0x0000000681692280, a java.util.HashMap),
        which is held by "qtp1593180232-302665"
      "qtp1593180232-302665":
        waiting to lock monitor 0x00007f54100b23d8 (object 0x0000000755589698, a org.apache.zeppelin.socket.NotebookSocket),
        which is held by "qtp1593180232-302348"
      "qtp1593180232-302348":
        waiting to lock monitor 0x00007f541003fce8 (object 0x0000000681692280, a java.util.HashMap),
        which is held by "qtp1593180232-302665"

      Java stack information for the threads listed above:
      ===================================================
      "qtp1593180232-305212":
          at org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:170)
          - waiting to lock <0x0000000681692280> (a java.util.HashMap)
          at org.apache.zeppelin.socket.NotebookServer.onClose(NotebookServer.java:476)
          at org.apache.zeppelin.socket.NotebookSocket.onWebSocketClose(NotebookSocket.java:47)
          at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
          at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:393)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:316)
          at org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection$CallbackBridge.writeSuccess(AbstractWebSocketConnection.java:86)
          at org.eclipse.jetty.websocket.common.io.FrameFlusher.notifyCallbackSuccess(FrameFlusher.java:359)
          at org.eclipse.jetty.websocket.common.io.FrameFlusher.succeedEntries(FrameFlusher.java:288)
          at org.eclipse.jetty.websocket.common.io.FrameFlusher.succeeded(FrameFlusher.java:280)
          at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:293)
          at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
          at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
          at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
          at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
          at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:581)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:510)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
          at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
          at java.lang.Thread.run(Thread.java:748)

      "qtp1593180232-302665":
          at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70)
          - waiting to lock <0x0000000755589698> (a org.apache.zeppelin.socket.NotebookSocket)
          at org.apache.zeppelin.socket.ConnectionManager.broadcast(ConnectionManager.java:242)
          at org.apache.zeppelin.socket.ConnectionManager.checkCollaborativeStatus(ConnectionManager.java:209)
          at org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromNote(ConnectionManager.java:185)
          - locked <0x0000000681692280> (a java.util.HashMap)
          at org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:173)
          - locked <0x0000000681692280> (a java.util.HashMap)
          at org.apache.zeppelin.socket.NotebookServer.onClose(NotebookServer.java:476)
          at org.apache.zeppelin.socket.NotebookSocket.onWebSocketClose(NotebookSocket.java:47)
          at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
          at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:393)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.remoteClose(AbstractWebSocketConnection.java:274)
          at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:109)
          at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
          at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:147)
          at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:112)
          at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:71)
          at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
          at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
          at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
          at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
          at java.lang.Thread.run(Thread.java:748)
      "qtp1593180232-302348":

      at org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:170)
          - waiting to lock <0x0000000681692280> (a java.util.HashMap)
          at org.apache.zeppelin.socket.NotebookServer.onClose(NotebookServer.java:476)
          at org.apache.zeppelin.socket.NotebookSocket.onWebSocketClose(NotebookSocket.java:47)
          at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
          at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:393)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:225)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection$Flusher.onCompleteFailure(AbstractWebSocketConnection.java:100)
          at org.eclipse.jetty.util.IteratingCallback.failed(IteratingCallback.java:402)
          at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
          at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
          at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
          at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
          at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
          at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:581)
          at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextOutgoingFrame(AbstractExtension.java:155)
          at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextOutgoingFrame(PerMessageDeflateExtension.java:123)
          at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.access$1100(CompressExtension.java:44)
          at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.compress(CompressExtension.java:591)
          at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.deflate(CompressExtension.java:488)
          at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.process(CompressExtension.java:468)
          at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
          at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
          at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.outgoingFrame(CompressExtension.java:244)
          at org.eclipse.jetty.websocket.common.extensions.ExtensionStack$Flusher.process(ExtensionStack.java:400)
          at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
          at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
          at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:277)
          at org.eclipse.jetty.websocket.common.WebSocketSession.outgoingFrame(WebSocketSession.java:359)
          at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.uncheckedSendFrame(WebSocketRemoteEndpoint.java:306)
          at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendAsyncFrame(WebSocketRemoteEndpoint.java:240)
          at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendStringByFuture(WebSocketRemoteEndpoint.java:403)
          at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70)
          - locked <0x0000000755589698> (a org.apache.zeppelin.socket.NotebookSocket)
          at org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:823)
          at org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:819)
          at org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:141)
          at org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:121)
          at org.apache.zeppelin.socket.NotebookServer.getNote(NotebookServer.java:818)
          at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:311)
          at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58)
          at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:231)
          at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
          at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
          at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:179)
          at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:150)
          at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
          at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:147)
          at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:112)
          at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:168)
          at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:92)
          at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
          at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
          at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
          at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
          at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
          at java.lang.Thread.run(Thread.java:748)

      Found 1 deadlock.

      Attachments

        Activity

          People

            Unassigned Unassigned
            zhaoming.chen zhaoming.chen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: