qpid_messaging (amqp1.0) randomly hangs when closing the connection.

We are using qpid_messaging to communicate with an ActiveMQ broker using the amqp1.0 protocol.

We have noticed during both large scale tests and in production that our program sometimes hangs when it is closing the connection to the broker. This is a very rare occurrence, but happens often enough to be annoying.

While the design of the amqp1.0 implementation is generally nice, I have noticed a lack of life cycle management for the helper objects, especially the ones interacting with background threads. The background thread is shut down when closing the last remaining connection, but not enough care is being taken to make sure it is in a safe state to do so.

If an event is being processed when the connection is closed, then a deadlock might occur, as seen in this stack-trace:

#0  0x00007fd104f26945 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1  0x00007fd10111a420 in qpid::sys::Condition::wait (this=<optimized out>, mutex=...) at /qpid-cpp-1.38.0/src/qpid/sys/posix/Condition.h:59
#2  0x00007fd10112489b in wait (this=0xbaff20) at /qpid-cpp-1.38.0/src/qpid/sys/Monitor.h:41
#3  qpid::sys::TimerTask::cancel (this=0xbafef0) at /qpid-cpp-1.38.0/src/qpid/sys/Timer.cpp:94
#4  0x00007fd105bb367f in qpid::messaging::amqp::ConnectionContext::close (this=0xb7b350) at /qpid-cpp-1.38.0/src/qpid/messaging/amqp/ConnectionContext.cpp:246