Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 3.2.0
-
None
-
ghx-label-6
Description
Test output:
45/99 Test #45: disk-io-mgr-test .................***Exception: Other 43.29 sec Turning perftools heap leak checking off [==========] Running 25 tests from 1 test case. [----------] Global test environment set-up. [----------] 25 tests from DiskIoMgrTest [ RUN ] DiskIoMgrTest.SingleWriter 19/01/16 15:57:09 INFO util.JvmPauseMonitor: Starting JVM pause monitor [ OK ] DiskIoMgrTest.SingleWriter (3407 ms) [ RUN ] DiskIoMgrTest.InvalidWrite [ OK ] DiskIoMgrTest.InvalidWrite (281 ms) [ RUN ] DiskIoMgrTest.WriteErrors [ OK ] DiskIoMgrTest.WriteErrors (235 ms) [ RUN ] DiskIoMgrTest.SingleWriterCancel [ OK ] DiskIoMgrTest.SingleWriterCancel (1165 ms) [ RUN ] DiskIoMgrTest.SingleReader [ OK ] DiskIoMgrTest.SingleReader (5835 ms) [ RUN ] DiskIoMgrTest.SingleReaderSubRanges [ OK ] DiskIoMgrTest.SingleReaderSubRanges (16404 ms) [ RUN ] DiskIoMgrTest.AddScanRangeTest [ OK ] DiskIoMgrTest.AddScanRangeTest (1210 ms) [ RUN ] DiskIoMgrTest.SyncReadTest *** Check failure stack trace: *** @ 0x4825dcc @ 0x4827671 @ 0x48257a6 @ 0x4828d6d @ 0x1af39ec @ 0x1ae90a4 @ 0x1ac30ea @ 0x1accad3 @ 0x1acc660 @ 0x1acbf3e @ 0x1acb62d @ 0x1b03671 @ 0x1f79988 @ 0x1f82b60 @ 0x1f82a84 @ 0x1f82a47 @ 0x3751579 @ 0x3ea4807850 @ 0x3ea44e894c Wrote minidump to /data/jenkins/workspace/<...>/repos/Impala/logs/be_tests/minidumps/disk-io-mgr-test/5bbf76f7-e5d6-4ac9-bdae9d9b-065c32ec.dmp
Error:
Operating system: Linux 0.0.0 Linux 2.6.32-358.14.1.el6.centos.plus.x86_64 #1 SMP Tue Jul 16 21:33:24 UTC 2013 x86_64 CPU: amd64 family 6 model 45 stepping 7 8 CPUs GPU: UNKNOWN Crash reason: SIGABRT Crash address: 0x45200002fa1 Process uptime: not available Thread 205 (crashed) 0 libc-2.12.so + 0x328e5 rax = 0x0000000000000000 rdx = 0x0000000000000006 rcx = 0xffffffffffffffff rbx = 0x0000000006adf9c0 rsi = 0x0000000000000563 rdi = 0x0000000000002fa1 rbp = 0x00007f8009b8ffe0 rsp = 0x00007f8009b8fc78 r8 = 0x00007f8009b8fd00 r9 = 0x0000000000000563 r10 = 0x0000000000000008 r11 = 0x0000000000000202 r12 = 0x0000000006adfa40 r13 = 0x000000000000001f r14 = 0x0000000006ae7384 r15 = 0x0000000006adf9c0 rip = 0x0000003ea44328e5 Found by: given as instruction pointer in context 1 libc-2.12.so + 0x340c5 rbp = 0x00007f8009b8ffe0 rsp = 0x00007f8009b8fc80 rip = 0x0000003ea44340c5 Found by: stack scanning 2 disk-io-mgr-test!boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> > >::operator()() [bind_template.hpp : 20 + 0x21] rbp = 0x00007f8009b8ffe0 rsp = 0x00007f8009b8fc88 rip = 0x0000000001acbf3e Found by: stack scanning 3 disk-io-mgr-test!google::LogMessage::Flush() + 0x157 rbx = 0x0000000000000007 rbp = 0x0000000006adf980 rsp = 0x00007f8009b8fff0 rip = 0x00000000048257a7 Found by: call frame info 4 disk-io-mgr-test!google::LogMessageFatal::~LogMessageFatal() + 0xe rbx = 0x00007f8009b90110 rbp = 0x00007f8009b903f0 rsp = 0x00007f8009b90070 r12 = 0x0000000000000001 r13 = 0x0000000006aee8b8 r14 = 0x000000000c213538 r15 = 0x0000000000000007 rip = 0x0000000004828d6e Found by: call frame info 5 disk-io-mgr-test!impala::io::LocalFileReader::ReadFromPos(long, unsigned char*, long, long*, bool*) [local-file-reader.cc : 67 + 0x10] rbx = 0x0000000000000001 rbp = 0x00007f8009b903f0 rsp = 0x00007f8009b90090 r12 = 0x0000000000000001 r13 = 0x0000000006aee8b8 r14 = 0x000000000c213538 r15 = 0x0000000000000007 rip = 0x0000000001af39ed Found by: call frame info 6 disk-io-mgr-test!impala::io::ScanRange::DoRead(int) [scan-range.cc : 219 + 0x5b] rbx = 0x000000000c4f71e0 rbp = 0x00007f8009b90620 rsp = 0x00007f8009b90400 r12 = 0x0000000001af36e4 r13 = 0x000000000000000d r14 = 0x000000000c213538 r15 = 0x0000000000000007 rip = 0x0000000001ae90a5 Found by: call frame info 7 disk-io-mgr-test!impala::io::DiskQueue::DiskThreadLoop(impala::io::DiskIoMgr*) [disk-io-mgr.cc : 425 + 0x17] rbx = 0x000000000c0e0f00 rbp = 0x00007f8009b906c0 rsp = 0x00007f8009b90630 r12 = 0x00007fff99de21c0 r13 = 0x00007fff99de1a90 r14 = 0x000000000c4f5dc8 r15 = 0x0000000000000007 rip = 0x0000000001ac30eb Found by: call frame info 8 disk-io-mgr-test!boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>::operator()(impala::io::DiskQueue*, impala::io::DiskIoMgr*) const [mem_fn_template.hpp : 165 + 0x1e] rbx = 0x000000000c0e0f00 rbp = 0x00007f8009b906f0 rsp = 0x00007f8009b906d0 r12 = 0x00007fff99de21c0 r13 = 0x00007fff99de1a90 r14 = 0x000000000c4f5dc8 r15 = 0x0000000000000007 rip = 0x0000000001accad4 Found by: call frame info 9 disk-io-mgr-test!void boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> >::operator()<boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>&, boost::_bi::list0&, int) [bind.hpp : 313 + 0x4f] rbx = 0x000000000c0e0f00 rbp = 0x00007f8009b90730 rsp = 0x00007f8009b90700 r12 = 0x00007fff99de21c0 r13 = 0x00007fff99de1a90 r14 = 0x000000000c4f5dc8 r15 = 0x0000000000000007 rip = 0x0000000001acc661 Found by: call frame info 10 disk-io-mgr-test!boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> > >::operator()() [bind_template.hpp : 20 + 0x22] rbx = 0x0000000000000563 rbp = 0x00007f8009b90780 rsp = 0x00007f8009b90740 r12 = 0x0000000000000000 r13 = 0x00007fff99de1a90 r14 = 0x000000000c4f5dc8 r15 = 0x0000000000000007 rip = 0x0000000001acbf3f Found by: call frame info 11 disk-io-mgr-test!boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> > >, void>::invoke(boost::detail::function::function_buffer&) [function_template.hpp : 153 + 0xc] rbx = 0x0000000000000563 rbp = 0x00007f8009b907b0 rsp = 0x00007f8009b90790 r12 = 0x0000000000000000 r13 = 0x00007fff99de1a90 r14 = 0x000000000c4f5dc8 r15 = 0x0000000000000007 rip = 0x0000000001acb62e Found by: call frame info 12 disk-io-mgr-test!boost::function0<void>::operator()() const [function_template.hpp : 767 + 0x11] rbx = 0x0000000000000563 rbp = 0x00007f8009b907f0 rsp = 0x00007f8009b907c0 r12 = 0x0000000000000000 r13 = 0x00007fff99de1a90 r14 = 0x000000000c4f5dc8 r15 = 0x0000000000000007 rip = 0x0000000001b03672 Found by: call frame info 13 disk-io-mgr-test!impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) [thread.cc : 359 + 0xf] rbx = 0x0000000000000563 rbp = 0x00007f8009b90b50 rsp = 0x00007f8009b90800 r12 = 0x0000000000000000 r13 = 0x00007fff99de1a90 r14 = 0x000000000c4f5dc8 r15 = 0x0000000000000007 rip = 0x0000000001f79989 Found by: call frame info 14 disk-io-mgr-test!void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) [bind.hpp : 525 + 0x15] rbx = 0x0000000001f79622 rbp = 0x00007f8009b90bc0 rsp = 0x00007f8009b90b60 r12 = 0x0000000000000000 r13 = 0x00007fff99de1a90 r14 = 0x000000000c4f5dc8
It can be reproduced by looping the test for a few minutes, e.g.
$ ninja -j $IMPALA_BUILD_THREADS disk-io-mgr-test && while ./be/build/latest/runtime/io/disk-io-mgr-test --gtest_filter=*SyncReadTest*; do date; don
Attachments
Issue Links
- relates to
-
IMPALA-4249 Clarify lifetime of DiskIoMgr::ScanRange objects
- Open