Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
ghx-label-12
Description
There are multiple issues with shutting down the ThriftServer:
- The docs for ThriftServer say "TODO: shutdown is buggy (which only harms tests)"
- TSAN reports a thread leak when using the ThriftServer in statestore-test
- Attempts to shutdown the ThriftServer in the Statestore don't seem to work properly
- The TAcceptQueueServer actually uses a volatile boolean called stop_ to coordinate the shutdown (see TAcceptQueueServer::stop()) which is not-thread safe (TSAN complains about this as well)
According to the docs this only affects test, specifically statestore-test. We should consider whether it is worth fixing the shutdown logic in the ThriftServer.
WARNING: ThreadSanitizer: thread leak (pid=67752) Thread T39 (tid=67794, finished) created by main thread at: #0 pthread_create /mnt/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 (statestore-test+0x18903bb) #1 boost::thread::start_thread_noexcept() <null> (statestore-test+0x2f30b59) #2 boost::thread::thread<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugInfo*, impala: :Promise<long, (impala::PromiseMode)0>*>(void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugIn fo*, impala::Promise<long, (impala::PromiseMode)0>*) /home/systest/Impala/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:419:13 (statestore-test+0x1ebf8eb) #3 impala::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /home/systest/Impala/be/src/util/thread.cc:317:13 (statestore-test+0x1ebcbd5) #4 impala::Status impala::Thread::Create<void (impala::ThriftServer::ThriftServerEventProcessor::*)(), impala::ThriftServer::ThriftServerEventProcessor*>(std::string const&, std::string const&, void (impala::ThriftServer::ThriftServerEventProcessor::* const&)(), imp ala::ThriftServer::ThriftServerEventProcessor* const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /home/systest/Impala/be/src/util/thread.h:81:12 (statestore-test+0x2707497) #5 impala::ThriftServer::ThriftServerEventProcessor::StartAndWaitForServer() /home/systest/Impala/be/src/rpc/thrift-server.cc:116:43 (statestore-test+0x27046f2) #6 impala::ThriftServer::Start() /home/systest/Impala/be/src/rpc/thrift-server.cc:447:60 (statestore-test+0x27063cc) #7 impala::Statestore::Init(int) /home/systest/Impala/be/src/statestore/statestore.cc:478:59 (statestore-test+0x1d7a431) #8 impala::StatestoreTest_SmokeTest_Test::TestBody() /home/systest/Impala/be/src/statestore/statestore-test.cc:55:132 (statestore-test+0x18fb381) #9 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (statestore-test+0x41051e2) #10 __libc_start_main /build/glibc-LK5gWL/glibc-2.23/csu/../csu/libc-start.c:291 (libc.so.6+0x2082f)