Uploaded image for project: 'Qpid Dispatch'
  1. Qpid Dispatch
  2. DISPATCH-2353

More than configured consumers count

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.16.0
    • None
    • Router Node, Routing Engine
    • None
    • PROD

    Description

      Hi There,
       
      We have few issues with the qpid-dispatch router where we are seeing multiple consumers are getting created randomly by our qpid dispatch routers. We are unable to trace nor able to replicate this issue in our lower env's. Below is the architecture diagram. Please provide any suggestions or feedback to help us understand this issue better. Thanks 
       
      Flow
      Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch Routers acting up as consumers in front of brokers (1.16 version) ---> 4 Broker pairs of activemq(2.18 version) masters and slaves (independent pairs
       
      Problem statement # 1 
      We are seeing more than the configured number of consumers count in our activemq hawtio console causing messages to sit in delivering count which are Un browsable since those messages are currently being delivered to its consumers. Having only two qpid dispatch routers(acting as consumers) infront of our activemq brokers, the count should always remain 2 but at times it is going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s to release the stuck/bad consumer so that messages can be processed/consumed. Restarting of the client(microservice) and broker does NOT help in this case. Only component that helps is QPID. 
       
      Problem statement # 2
      At times we see messages going to delivering count even when there are only two configured consumers(2qpids) showing in activemq hawtio console. We don't know the cause to why the messages get stuck in delivering count. To resolve this, we tried restarting the consumer service but that did not help. Next up we tried restarting the brokers, that did not help and noticed that all the stuck delivering messages, the broker slowly replayed back and Eventually the stuck messages came back to delivering count post broker restart. To resolve this issue, we are having to bounce the qpid’s to release the messages and that fixes the issue. 
       
      Version of qpid-dispatch router
      qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
      qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm

       

      As always any suggestions would be greatly appreciated. Thanks

      Attachments

        Activity

          People

            Unassigned Unassigned
            ekta-awasthi Ekta
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: