Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3186

Stress test crash: wait until impalad is quiesced before starting next test: [libstdc++.so.6+0x5cad0] __dynamic_cast+0x50

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Later
    • Impala 2.6.0
    • None
    • Infrastructure

    Description

      There was a crash on the EC2 stress test cluster. In the hs_err_pid file we find:

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      #  SIGSEGV (0xb) at pc=0x00007f61f241dad0, pid=18718, tid=140039557764864
      #
      # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # C  [libstdc++.so.6+0x5cad0]  __dynamic_cast+0x50
      #
      # Core dump written. Default location: /var/log/impalad/core or core.18718
      #
      # If you would like to submit a bug report, please visit:
      #   http://bugreport.sun.com/bugreport/crash.jsp
      

      and in the impalad.ERROR log:

      Log file created at: 2016/03/15 00:04:31
      Running on machine: impala-stress-cdh5-trunk2-2.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      E0315 00:04:31.310118 18718 logging.cc:120] stderr will be logged to this file.
      E0315 07:50:01.608139  6880 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:01.747846  6884 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:02.445477  8224 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:10.240705  8572 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:10.397904  7070 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:12.076678  7198 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:19.973160  8696 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:20.319783  8233 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:29.652158  7819 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:30.150673  7084 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:35.144541  9215 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:35.868957  8489 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:39.297054  7494 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-3.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      Picked up JAVA_TOOL_OPTIONS:
      hdfsOpenFile(hdfs://impala-stress-cdh5-trunk2-1.vpc.cloudera.com:8020/user/hive/warehouse/tpch_3_nested_parquet.db/customer/000001_0): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error:
      FSDataInputStream#close error:
      terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
        what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument
      ~                                                                                          
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              tarmstrong Tim Armstrong
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: