Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.3.0, 1.4.0
-
None
Description
I am using sshd-core for a server machine (S) that accepts incoming connections and port forwarding requests.
There are client machines (C) that run servers that should be accessible by a tunnel to the server.
On the client machines (C) also an implementation using sshd-core is running that establish the connection to the server (S) and initiate the port forwarding.
Other clients are using the tunnelled connection to communicate with the servers that are running on the client machines (C).
Sometimes I realized that no data is transferred anymore (through the tunnels).
All the worker reside in the waitFor function and no one wakes them up.
"sshd-SshServer[67de991c]-nio2-thread-3" - Thread t@125 java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) - waiting on <132c6b60> (a java.lang.Object) at org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244) at org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984) at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276) at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256) at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at sun.nio.ch.Invoker$2.run(Invoker.java:218) at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - locked <6d02d7ed> (a java.util.concurrent.ThreadPoolExecutor$Worker) "sshd-SshServer[67de991c]-nio2-thread-2" - Thread t@124 java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) - waiting on <7e9f4eff> (a java.lang.Object) at org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244) at org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984) at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276) at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256) at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at sun.nio.ch.Invoker$2.run(Invoker.java:218) at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - locked <35fbf3e8> (a java.util.concurrent.ThreadPoolExecutor$Worker) "sshd-SshServer[67de991c]-nio2-thread-1" - Thread t@122 java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) - waiting on <49ce93a9> (a java.lang.Object) at org.apache.sshd.client.channel.AbstractClientChannel.waitFor(AbstractClientChannel.java:244) at org.apache.sshd.common.forward.DefaultTcpipForwarder$StaticIoHandler.messageReceived(DefaultTcpipForwarder.java:984) at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:276) at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:256) at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:253) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553) at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:304) at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:249) at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:243) at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:239) at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:235) at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:231) at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:227) at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:178) at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:156) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$45/1071326492.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at sun.nio.ch.Invoker$2.run(Invoker.java:218) at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - locked <7d1c59e6> (a java.util.concurrent.ThreadPoolExecutor$Worker) "sshd-SshServer[67de991c]-timer-thread-1" - Thread t@105 java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) - parking to wait for <655c080c> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None
To eliminate some other code that could trigger that error I created a "minimal" example – a simple test application – that could be used to demonstrate the hang (for me it is reproducible using that code).
Please have a look at https://github.com/maggu2810/sshd-deadlock/tree/first-report where you could also find a readme with a short description about the code.
