Uploaded image for project: 'ActiveMQ Classic'
  1. ActiveMQ Classic
  2. AMQ-7366

ActiveMQ broker always hangs up at intervals in the case of high concurrent connections

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Blocker
    • Resolution: Unresolved
    • 5.15.0
    • 5.15.0
    • Broker, Connector, MQTT
    • activeMQ version:5.15.0

      linux OS version:Red Hat Enterprise Linux Server release 6.9 (Santiago)

      linux server configuration: Memery:32G、CPU:16、disk:200G

       

    • Important

    Description

      My broker JVM configuration is "- xms16g - xmx16g - xss256k - XX: metaspacesize = 128M".

      The transport connector in activemq.xml is configured as follows:

      <transportConnector name="sslmqtt" uri="mqtt+ssl://0.0.0.0:1883?maximumConnections=200000&wireFormat.maxFrameSize=104857600&jms.watchTopicAdvisories=false&transport.defaultKeepAlive=30000"/>

      At present, it can support 15000 client connections. But every two days or less, ActiveMQ broker will hang up. When I check the logs of ActiveMQ broker, I find the following errors:

      1.

      org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://180.162.0.129:50014org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://180.162.0.129:50014 at org.apache.activemq.transport.mqtt.MQTTInactivityMonitor$1$1.run(MQTTInactivityMonitor.java:82)[activemq-mqtt-5.15.0.jar:5.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]2019-12-17 13:48:28,123 | DEBUG | No CONNECT frame received in time for tcp:///101.80.38.31:4751@1883! Throwing InactivityIOException. | org.apache.activemq.transport.mqtt.MQTTInactivityMonitor | InactivityMonitor ReadCheck2019-12-17 13:48:28,124 | DEBUG | Transport Connection to: tcp://101.88.157.200:6892 failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://101.88.157.200:6892 | org.apache.activemq.broker.TransportConnection.Transport | MQTTInactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@ddca0bb[State = -1, empty queue]org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://101.88.157.200:6892 at org.apache.activemq.transport.mqtt.MQTTInactivityMonitor$1$1.run(MQTTInactivityMonitor.java:82)[activemq-mqtt-5.15.0.jar:5.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]2019-12-17 13:48:28,136 | DEBUG | No CONNECT frame received in time for tcp:///114.83.88.242:36665@1883! Throwing InactivityIOException. | org.apache.activemq.transport.mqtt.MQTTInactivityMonitor | InactivityMonitor ReadCheck

      2.

      javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

      javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)[:1.8.0_151] at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)[:1.8.0_151] at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938)[:1.8.0_151] at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)[:1.8.0_151] at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)[activemq-client-5.15.0.jar:5.15.0] at java.io.DataInputStream.readByte(DataInputStream.java:265)[:1.8.0_151] at org.apache.activemq.transport.mqtt.MQTTWireFormat.unmarshal(MQTTWireFormat.java:86)[activemq-mqtt-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.15.0.jar:5.15.0] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]

      3.

       java.lang.OutOfMemoryError:unable to create new native thread

       

      At the same time, the number of threads in ActiveMQ service keeps skyrocketing, reaching 60000, resulting in memory overflow。I found that when the channel was inactive for too  long is kicked out by the broker, the old thread is not destroyed. At this time, new requests come in, and the thread will still be created continuously, resulting in memory overflow.

      My requirement is that ActiveMQ broker can support 30000 client connections.How can I solve this problem? Can someone help me

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            levin wong zhiwei wang
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: