Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Using latest proton and dispatch master code as of 3 hours ago.
Testing router TCP adapter on a machine with 32 cores / 64 threads.
I gave the router 64 worker threads, then used 'hey' load generator to send it HTTP requests to a TCP listener which router forwarded to Nginx on same machine.
Multiple tests with increasing number of parallel senders: 10, 20, 30,...Each sender throttled to 10 messages per second.
It survived many tests, but crashed around test with 200 senders.
I believe this is easily repeatable – I will go check that now.
Here is the thread that crashed:
#0 0x00007f33186a0684 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00007f33186e2848 in lock (m=<optimized out>)
at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#2 process (tsk=<optimized out>)
at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#3 next_event_batch (p=0x10ed970, can_block=true)
at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40)
at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
And here are all the threads:
(gdb) thread apply all bt
Thread 65 (Thread 0x7f3244ff9640 (LWP 36500)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f326811a578) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 64 (Thread 0x7f327ffff640 (LWP 36481)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proacto-Type <RET> for more, q to quit, c to continue without paging-
r/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 63 (Thread 0x7f322f7fe640 (LWP 36502)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32c8063af8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
-Type <RET> for more, q to quit, c to continue without paging-c
Thread 62 (Thread 0x7f322d7fa640 (LWP 36506)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 61 (Thread 0x7f322dffb640 (LWP 36505)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 60 (Thread 0x7f32d17fa640 (LWP 36471)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32880d36a8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 59 (Thread 0x7f322effd640 (LWP 36503)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f325c074f68) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 58 (Thread 0x7f327e7fc640 (LWP 36484)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32f0087df8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 57 (Thread 0x7f3276ffd640 (LWP 36489)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f328c0e1e48) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 56 (Thread 0x7f32757fa640 (LWP 36492)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 55 (Thread 0x7f32477fe640 (LWP 36495)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 54 (Thread 0x7f32d2ffd640 (LWP 36468)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 53 (Thread 0x7f3245ffb640 (LWP 36498)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32a80994e8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 52 (Thread 0x7f322e7fc640 (LWP 36504)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 51 (Thread 0x7f32467fc640 (LWP 36497)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f329001ce38) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 50 (Thread 0x7f32da7fc640 (LWP 36463)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 49 (Thread 0x7f3247fff640 (LWP 36494)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 48 (Thread 0x7f3274ff9640 (LWP 36493)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 47 (Thread 0x7f3275ffb640 (LWP 36491)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 46 (Thread 0x7f32767fc640 (LWP 36490)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 45 (Thread 0x7f32777fe640 (LWP 36488)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32e4080b78) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 44 (Thread 0x7f327cff9640 (LWP 36487)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 43 (Thread 0x7f32a17fa640 (LWP 36478)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 42 (Thread 0x7f327effd640 (LWP 36483)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 41 (Thread 0x7f327f7fe640 (LWP 36482)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 40 (Thread 0x7f32d37fe640 (LWP 36460)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 39 (Thread 0x7f32daffd640 (LWP 36462)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 38 (Thread 0x7f3277fff640 (LWP 36480)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 37 (Thread 0x7f32a0ff9640 (LWP 36479)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f328407b3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 36 (Thread 0x7f32d0ff9640 (LWP 36472)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f32a1ffb640 (LWP 36477)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f31fc1293f8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7f32a2ffd640 (LWP 36475)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32b404a638) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 33 (Thread 0x7f32a37fe640 (LWP 36474)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7f32a27fc640 (LWP 36476)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7f32a3fff640 (LWP 36473)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f32d27fc640 (LWP 36469)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f329c0828d8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f33017fa640 (LWP 36451)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f32d1ffb640 (LWP 36470)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f32d8ff9640 (LWP 36466)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f32d3fff640 (LWP 36467)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f3238096088) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f32fbfff640 (LWP 36453)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f3246ffd640 (LWP 36496)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f32fb7fe640 (LWP 36454)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f322ffff640 (LWP 36501)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32180ef358) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f32d97fa640 (LWP 36465)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f33027fc640 (LWP 36449)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f33037fe640 (LWP 36446)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f32db7fe640 (LWP 36461)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f32d9ffb640 (LWP 36464)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2d79 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 suspend (ts=<optimized out>, p=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:397
#4 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2459
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f3309369640 (LWP 36443)):
#0 0x00007f33186a0687 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00007f331878f219 in sys_mutex_lock (mutex=<optimized out>) at /home/mick/latest/qpid-dispatch/src/posix/threading.c:57
#2 0x00007f33187a0ec2 in qdr_delivery_continue_peers_CT (core=core@entry=0x118e8b0, in_dlv=in_dlv@entry=0x7f325805ea48, more=more@entry=true) at /home/mick/latest/qpid-dispatch/src/router_core/delivery.c:1062
#3 0x00007f33187a1bb7 in qdr_delivery_continue_CT (action=<optimized out>, action=<optimized out>, discard=false, core=0x118e8b0) at /home/mick/latest/qpid-dispatch/src/router_core/delivery.c:1126
#4 qdr_delivery_continue_CT (core=0x118e8b0, action=<optimized out>, discard=<optimized out>) at /home/mick/latest/qpid-dispatch/src/router_core/delivery.c:1103
#5 0x00007f33187ac2ed in router_core_thread (arg=0x118e8b0) at /home/mick/latest/qpid-dispatch/src/router_core/router_core_thread.c:239
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f32dbfff640 (LWP 36459)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dd6d0 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 poller_done (p=p@entry=0x10ed970, ts=ts@entry=0x7f32b0000b60) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2649
#4 0x00007f33186e327a in next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2452
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f32f8ff9640 (LWP 36458)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f32fa7fc640 (LWP 36455)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f32f97fa640 (LWP 36457)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f32f9ffb640 (LWP 36456)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32180eed48) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f3301ffb640 (LWP 36450)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f3300ff9640 (LWP 36452)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f3204155058) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f3302ffd640 (LWP 36448)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32600fd028) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f327dffb640 (LWP 36485)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2d79 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 suspend (ts=<optimized out>, p=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:397
#4 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2459
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f32faffd640 (LWP 36447)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2509 in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f327d7fa640 (LWP 36486)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32dc076678) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f3308b68640 (LWP 36444)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32180dff68) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f3303fff640 (LWP 36445)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186e2b7e in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#4 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#6 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f33179222c0 (LWP 36442)):
#0 0x00007f33186a7ea0 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33186a08f5 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007f33186dfc5f in lock (m=0x10edc90) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#3 pni_raw_connection_done (rc=0x10ed3b8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423
#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696
#5 pn_proactor_done (p=0x10ed970, batch=batch@entry=0x7f32a403fec8) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676
#6 0x00007f33187c1a11 in thread_run (arg=arg@entry=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1140
#7 0x00007f33187c20b0 in qd_server_run (qd=<optimized out>) at /home/mick/latest/qpid-dispatch/src/server.c:1484
#8 0x00000000004026a8 in main_process (config_path=0x7ffede736db3 "./1_router.conf", python_pkgdir=<optimized out>, test_hooks=<optimized out>, fd=2) at /home/mick/latest/qpid-dispatch/router/src/main.c:113
#9 0x000000000040240b in main (argc=3, argv=0x7ffede735188) at /home/mick/latest/qpid-dispatch/router/src/main.c:367
Thread 1 (Thread 0x7f32457fa640 (LWP 36499)):
#0 0x00007f33186a0684 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00007f33186e2848 in lock (m=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326
#2 process (tsk=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248
#3 next_event_batch (p=0x10ed970, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423
#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at /home/mick/latest/qpid-dispatch/src/server.c:1107
#5 0x00007f331869e3f9 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6
(gdb)
{color:#de350b}}
Attachments
Issue Links
- Testing discovered
-
-
- Closed
-