Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-5139

HA transactions block a thread, can deadlock the broker.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.24
    • 0.25
    • C++ Clustering
    • None

    Description

      When the client sends a "prepare" command for a transaction, the thread handling that command is blocked until all backups have responded with their prepare status. This can easily deadlock the broker if there are more concurrent transactions than worker threads.

      To reproduce you need a way to delay completion of a transaction. The following regression test illustrates, it will be added to TransactionTests in ha_tests.py as part of the fix:

          def test_tx_block_threads(self):
              """Verify that TXs blocked in commit don't block broker threads."""
              cluster = HaCluster(self, 2, args=["--worker-threads=2"])
              sessions = [cluster[0].connect().session(transactional=True) for i in xrange(2)]
              for s in sessions: s.sender("foo;{create:always}").send("foo")
              self.assertEqual(2, len(cluster[1].agent().tx_queues()))
              os.kill(cluster[1].pid, signal.SIGSTOP) # Freeze backup so tx can't complete.
              threads = [ Thread(target=s.commit) for s in sessions]
              for t in threads: t.start()
              cluster[0].ready(timeout=1) # Should not block
              os.kill(cluster[1].pid, signal.SIGCONT) # Allow tx to complete.
              for t in threads: t.join()
              c.close()
      

      Attachments

        Issue Links

          Activity

            People

              aconway Alan Conway
              aconway Alan Conway
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: