Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-2871

QPID management is stuck in a waiting state for broker sync during rapid federation/defederation using the Java management API

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Later
    • 0.6
    • JIRA Cleanup
    • Java Common
    • None
    • RHEL 5.5, MRG-M 1,1, QPID Java Management API 0.6, 100GB hard drive, 4GB RAM, virtualized using VMWare

    Description

      We used the QPID Java Management API to create a Java library to replicate the qpid-route CLI tool. When we rapidly federated, un-federated, and re-federated brokers, the broker's management component would become unresponsive to any other commands. Looking for locked or waiting threads, we noticed that the following class was consistently in a waiting state.

      org.apache.qpid.console.Broker in the qpid-management-console.jar

      Looking at the latest source we noticed that in the Broker.java waitForSync method the timeout check that throws a ConsoleException (line 498) is outside of the while loop (line 487-496) that is waiting for the broker to sync. We downloaded the source, put the timeout check inside the while loop, and compiled a new version of the qpid-management-console.jar. When we ran our original scenario, the broker's management component never became unresponsive. We did get some ConsoleExceptions thrown as we expected when we rapdily sent federation commands. The line numbers above were taken from revision 998162 (http://svn.apache.org/viewvc/qpid/trunk/qpid/java/management/console/src/main/java/org/apache/qpid/console/Broker.java?view=markup).

      Updated code for the waitForSync method:
      public void waitForSync(int timeout)
      {
      synchronized (lockObject)
      {
      long start = System.currentTimeMillis();
      while (syncInFlight)
      {
      try

      { lockObject.wait(SYNC_TIME); }

      catch (Exception e)

      { throw new ConsoleException(e); }

      long duration = System.currentTimeMillis() - start;
      if (duration > timeout)

      { throw new ConsoleException("Timeout waiting for Broker to Sync"); }

      }
      }
      }

      Attachments

        Activity

          People

            Unassigned Unassigned
            jwongamentra Jason Wong
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified