Description
When running a XML-RPC server under heavy load, it eventually deadlocks inside the thread pool that manages the 'worker' threads which handle the individual XML-RPC requests - the classes involved are org.apache.xmlrpc.util.ThreadPool and org.apache.xmlrpc.util.ThreadPool$Poolable. jstack on the hung process shows:
----------
Found one Java-level deadlock:
=============================
"XML-RPC-13":
waiting to lock monitor 0x08d10bec (object 0xbb6959c0, a org.apache.xmlrpc.util.ThreadPool),
which is held by "XML-RPC Weblistener"
"XML-RPC Weblistener":
waiting to lock monitor 0x08d1186c (object 0xbd2ed340, a org.apache.xmlrpc.util.ThreadPool$Poolable$1),
which is held by "XML-RPC-2"
"XML-RPC-2":
waiting to lock monitor 0x08d112f4 (object 0xbd2ed570, a org.apache.xmlrpc.util.ThreadPool$Poolable),
which is held by "XML-RPC Weblistener"
Java stack information for the threads listed above:
===================================================
"XML-RPC-13":
at org.apache.xmlrpc.util.ThreadPool.repool(Unknown Source)
- waiting to lock <0xbb6959c0> (a org.apache.xmlrpc.util.ThreadPool)
at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(Unknown Source)
"XML-RPC Weblistener":
at org.apache.xmlrpc.util.ThreadPool$Poolable.start(Unknown Source) - waiting to lock <0xbd2ed340> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)
- locked <0xbd2ed570> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
at org.apache.xmlrpc.util.ThreadPool.startTask(Unknown Source) - locked <0xbb6959c0> (a org.apache.xmlrpc.util.ThreadPool)
at org.apache.xmlrpc.webserver.WebServer.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
"XML-RPC-2":
at org.apache.xmlrpc.util.ThreadPool$Poolable.isShuttingDown(Unknown Source) - waiting to lock <0xbd2ed570> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
at org.apache.xmlrpc.util.ThreadPool$Poolable.access$000(Unknown Source)
at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(Unknown Source) - locked <0xbd2ed340> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)
Found 1 deadlock.
----------
And another slight variant on the above, from a debug build of revision 769436.
----------
Found one Java-level deadlock:
=============================
"XML-RPC-6":
waiting to lock monitor 0x0870d8ec (object 0xbac020f8, a org.apache.xmlrpc.util.ThreadPool),
which is held by "XML-RPC Weblistener"
"XML-RPC Weblistener":
waiting to lock monitor 0x0814de4c (object 0xbad73820, a org.apache.xmlrpc.util.ThreadPool$Poolable$1),
which is held by "XML-RPC-5"
"XML-RPC-5":
waiting to lock monitor 0x0814eacc (object 0xbad73b48, a org.apache.xmlrpc.util.ThreadPool$Poolable),
which is held by "XML-RPC Weblistener"
Java stack information for the threads listed above:
===================================================
"XML-RPC-6":
at org.apache.xmlrpc.util.ThreadPool.repool(ThreadPool.java:136)
- waiting to lock <0xbac020f8> (a org.apache.xmlrpc.util.ThreadPool)
at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(ThreadPool.java:70)
"XML-RPC Weblistener":
at org.apache.xmlrpc.util.ThreadPool$Poolable.start(ThreadPool.java:106) - waiting to lock <0xbad73820> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)
- locked <0xbad73b48> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
at org.apache.xmlrpc.util.ThreadPool.startTask(ThreadPool.java:168) - locked <0xbac020f8> (a org.apache.xmlrpc.util.ThreadPool)
at org.apache.xmlrpc.webserver.WebServer.run(WebServer.java:338)
at java.lang.Thread.run(Thread.java:619)
"XML-RPC-5":
at org.apache.xmlrpc.util.ThreadPool$Poolable.getTask(ThreadPool.java:99) - waiting to lock <0xbad73b48> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
at org.apache.xmlrpc.util.ThreadPool$Poolable.access$100(ThreadPool.java:47)
at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(ThreadPool.java:59) - locked <0xbad73820> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)
Found 1 deadlock.
----------
Attachments
Attachments
Issue Links
- relates to
-
XMLRPC-167 XmlRpcClient is supposed to be thread-safe but it isn't
- Open