Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Private Beta
-
None
-
None
Description
I deleted a twitter table at the same time that I was inserting into it heavily. It appears that nearly immediately thereafter, one of the tservers SEGVed:
#0 0x0000003fca2093a0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00007f61005859df in lock (this=0x8089228191029b41) at /var/lib/jenkins/cdep/kudu/src/kudu/util/mutex.h:29
#2 lock_guard (this=0x8089228191029b41) at /var/lib/jenkins/cdep/kudu/src/kudu/util/locks.h:210
#3 kudu::ArenaBase<true>::memory_footprint (this=0x8089228191029b41) at /var/lib/jenkins/cdep/kudu/src/kudu/util/memory/arena.cc:155
#4 0x00007f610431469b in memory_footprint (this=<value optimized out>) at /var/lib/jenkins/cdep/kudu/src/kudu/tablet/memrowset.h:237
#5 kudu::tablet::Tablet::MemRowSetSize (this=<value optimized out>) at /var/lib/jenkins/cdep/kudu/src/kudu/tablet/tablet.cc:1260
#6 0x00007f6100598000 in operator() (this=<value optimized out>, writer=0x7f601dd2c1f0) at /usr/include/boost/function/function_template.hpp:1013
#7 value (this=<value optimized out>, writer=0x7f601dd2c1f0) at /var/lib/jenkins/cdep/kudu/src/kudu/util/metrics.h:514
#8 kudu::FunctionGauge<unsigned long>::WriteValue (this=<value optimized out>, writer=0x7f601dd2c1f0) at /var/lib/jenkins/cdep/kudu/src/kudu/util/metrics.h:517
#9 0x00007f61005935eb in kudu::Gauge::WriteAsJson (this=0x2408b00, name="kudu.tabletserver.tablet.tablet-fc196677da194b9091d9c59afe3df9e0.memrowset_size", writer=0x7f601dd2c1f0, granularity=<value optimized out>)
at /var/lib/jenkins/cdep/kudu/src/kudu/util/metrics.cc:318
#10 0x00007f6100594def in kudu::MetricRegistry::WriteAsJson (this=0x7f601dd2c0f0, writer=0x7f601dd2c1f0, requested_metrics=std::vector of length -17506349242431, capacity -17472094237159 =
,
requested_detail_metrics=<value optimized out>) at /var/lib/jenkins/cdep/kudu/src/kudu/util/metrics.cc:273
#11 0x00007f610481dc2d in kudu::WriteMetricsAsJson (metrics=0x22f5cc0, req=<value optimized out>, output=<value optimized out>) at /var/lib/jenkins/cdep/kudu/src/kudu/server/default-path-handlers.cc:192
#12 0x00007f610482bc75 in operator() (this=0x2306000, handler=<value optimized out>, connection=0x6872e000, request_info=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013
#13 kudu::Webserver::RunPathHandler (this=0x2306000, handler=<value optimized out>, connection=0x6872e000, request_info=<value optimized out>) at /var/lib/jenkins/cdep/kudu/src/kudu/server/webserver.cc:290
#14 0x00007f610482c311 in kudu::Webserver::BeginRequestCallback (this=0x2306000, connection=0x6872e000, request_info=0x6872e000) at /var/lib/jenkins/cdep/kudu/src/kudu/server/webserver.cc:268
#15 0x00007f610483cdd8 in handle_request () from /var/lib/jenkins/cdep/kudu/build/release/libserver_process.so
#16 0x00007f610483e6e8 in process_new_connection () from /var/lib/jenkins/cdep/kudu/build/release/libserver_process.so
#17 0x00007f610483ed48 in worker_thread () from /var/lib/jenkins/cdep/kudu/build/release/libserver_process.so
#18 0x0000003fca2079d1 in start_thread () from /lib64/libpthread.so.0
#19 0x0000003fc9ee88fd in clone () from /lib64/libc.so.6
I'm guessing this has something to do with not unregistering metrics