Details
-
Bug
-
Status: Open
-
Blocker
-
Resolution: Unresolved
-
5.15.0
-
activeMQ version:5.15.0
linux OS version:Red Hat Enterprise Linux Server release 6.9 (Santiago)
linux server configuration: Memery:32G、CPU:16、disk:200G
-
Important
Description
My broker JVM configuration is "- xms16g - xmx16g - xss256k - XX: metaspacesize = 128M".
The transport connector in activemq.xml is configured as follows:
<transportConnector name="sslmqtt" uri="mqtt+ssl://0.0.0.0:1883?maximumConnections=200000&wireFormat.maxFrameSize=104857600&jms.watchTopicAdvisories=false&transport.defaultKeepAlive=30000"/>
At present, it can support 15000 client connections. But every two days or less, ActiveMQ broker will hang up. When I check the logs of ActiveMQ broker, I find the following errors:
1.
org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://180.162.0.129:50014org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://180.162.0.129:50014 at org.apache.activemq.transport.mqtt.MQTTInactivityMonitor$1$1.run(MQTTInactivityMonitor.java:82)[activemq-mqtt-5.15.0.jar:5.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]2019-12-17 13:48:28,123 | DEBUG | No CONNECT frame received in time for tcp:///101.80.38.31:4751@1883! Throwing InactivityIOException. | org.apache.activemq.transport.mqtt.MQTTInactivityMonitor | InactivityMonitor ReadCheck2019-12-17 13:48:28,124 | DEBUG | Transport Connection to: tcp://101.88.157.200:6892 failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://101.88.157.200:6892 | org.apache.activemq.broker.TransportConnection.Transport | MQTTInactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@ddca0bb[State = -1, empty queue]org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://101.88.157.200:6892 at org.apache.activemq.transport.mqtt.MQTTInactivityMonitor$1$1.run(MQTTInactivityMonitor.java:82)[activemq-mqtt-5.15.0.jar:5.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]2019-12-17 13:48:28,136 | DEBUG | No CONNECT frame received in time for tcp:///114.83.88.242:36665@1883! Throwing InactivityIOException. | org.apache.activemq.transport.mqtt.MQTTInactivityMonitor | InactivityMonitor ReadCheck
2.
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)[:1.8.0_151] at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)[:1.8.0_151] at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938)[:1.8.0_151] at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)[:1.8.0_151] at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)[activemq-client-5.15.0.jar:5.15.0] at java.io.DataInputStream.readByte(DataInputStream.java:265)[:1.8.0_151] at org.apache.activemq.transport.mqtt.MQTTWireFormat.unmarshal(MQTTWireFormat.java:86)[activemq-mqtt-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.15.0.jar:5.15.0] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]
3.
java.lang.OutOfMemoryError:unable to create new native thread
At the same time, the number of threads in ActiveMQ service keeps skyrocketing, reaching 60000, resulting in memory overflow。I found that when the channel was inactive for too long is kicked out by the broker, the old thread is not destroyed. At this time, new requests come in, and the thread will still be created continuously, resulting in memory overflow.
My requirement is that ActiveMQ broker can support 30000 client connections.How can I solve this problem? Can someone help me