Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Later
-
None
-
None
Description
Description of problem:
---------------------------------
If the JMS client is connected to a cluster and if the broker the client is connected to, is killed while the client is in the midst of producing messages, the client appears to be hung and eventually the failover times out.
The problem only occurs if the producer (or consumer) is using a an addressing string. If you run the same example using the BURL syntax then there is no
issue.
(*)The resume method in o.a.q.transport.Connection takes a lock when in enters the resume method and in turn it tries to call the "resumed" methods in the AMQSession_0_10 class.
(*)That method will essentially recreate the producers and consumers which will cause the address string to be re-evaluated. In doing so a synchronous ExchangeQuery is made.
The IoReceiverThread tries to dispatch it, but unable to do as it cannot grab the 'lock' as it is already held by the failover thread. The failover thread is not releasing the lock as it is waiting for the response
from the exchange query method.
Hence the client appears to be stuck until the wait for the response times out and the failover operation fails
Below is a partial thread dump that illustrates the above issue.
Following are two snippets from a thread dump that illustrates the issue.
"IoReceiver - /10.16.44.241:5673" daemon prio=10 tid=0x000000000cdf0800
nid=0x1afe waiting for monitor entry [0x0000000041c1a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.qpid.transport.Connection.getSession(Connection.java:440)
- waiting to lock <0x00002aab6075e0e8> (a java.lang.Object)
at org.apache.qpid.transport.Connection.dispatch(Connection.java:380)
at
org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:64)
at
org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:40)
at
org.apache.qpid.transport.MethodDelegate.sessionAttached(MethodDelegate.java:68)
at org.apache.qpid.transport.SessionAttached.dispatch(SessionAttached.java:91)
at
org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:49)
at
org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:40)
at org.apache.qpid.transport.Method.delegate(Method.java:163)
at org.apache.qpid.transport.Connection.received(Connection.java:348)
at org.apache.qpid.transport.Connection.received(Connection.java:55)
at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:98)
at org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:184)
at org.apache.qpid.transport.network.Assembler.frame(Assembler.java:132)
at org.apache.qpid.transport.network.Frame.delegate(Frame.java:133)
at org.apache.qpid.transport.network.Assembler.received(Assembler.java:103)
at org.apache.qpid.transport.network.Assembler.received(Assembler.java:48)
at org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:187)
at
org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:103)
at
org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:42)
at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:128)
at java.lang.Thread.run(Thread.java:636)
IoReceiver - localhost/127.0.0.1:5672" daemon prio=10 tid=0x00002aabbc1a8000
nid=0x1aee in Object.wait() [0x0000000040bee000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab6155be60> (a
org.apache.qpid.transport.Session$ResultFuture)
at org.apache.qpid.transport.util.Waiter.await(Waiter.java:54)
at org.apache.qpid.transport.Session$ResultFuture.get(Session.java:837) - locked <0x00002aab6155be60> (a
org.apache.qpid.transport.Session$ResultFuture)
at org.apache.qpid.transport.Session$ResultFuture.get(Session.java:859)
at
org.apache.qpid.client.AMQSession_0_10.resolveAddressType(AMQSession_0_10.java:1219)
at
org.apache.qpid.client.AMQSession_0_10.handleAddressBasedDestination(AMQSession_0_10.java:1160)
at
org.apache.qpid.client.BasicMessageProducer_0_10.declareDestination(BasicMessageProducer_0_10.java:81)
at
org.apache.qpid.client.BasicMessageProducer.resubscribe(BasicMessageProducer.java:174)
at
org.apache.qpid.client.AMQSession.resubscribeProducers(AMQSession.java:2782)
at org.apache.qpid.client.AMQSession.resubscribe(AMQSession.java:2038)
at org.apache.qpid.client.AMQSession_0_10.resumed(AMQSession_0_10.java:871)
at org.apache.qpid.transport.Session.resume(Session.java:265) - locked <0x00002aab61761350> (a [Lorg.apache.qpid.transport.Method
at org.apache.qpid.transport.Connection.resume(Connection.java:452) - locked <0x00002aab6075e0e8> (a java.lang.Object)
at
org.apache.qpid.client.AMQConnectionDelegate_0_10.closed(AMQConnectionDelegate_0_10.java:243)
at org.apache.qpid.transport.Connection.closed(Connection.java:530)
at org.apache.qpid.transport.network.Assembler.closed(Assembler.java:113)
at
org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:202)
at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:150)
at java.lang.Thread.run(Thread.java:636)