Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Later
-
Impala 2.6.0
-
None
Description
There was a crash on the EC2 stress test cluster. In the hs_err_pid file we find:
# # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f61f241dad0, pid=18718, tid=140039557764864 # # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libstdc++.so.6+0x5cad0] __dynamic_cast+0x50 # # Core dump written. Default location: /var/log/impalad/core or core.18718 # # If you would like to submit a bug report, please visit: # http://bugreport.sun.com/bugreport/crash.jsp
and in the impalad.ERROR log:
Log file created at: 2016/03/15 00:04:31 Running on machine: impala-stress-cdh5-trunk2-2.vpc.cloudera.com Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg E0315 00:04:31.310118 18718 logging.cc:120] stderr will be logged to this file. E0315 07:50:01.608139 6880 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:01.747846 6884 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:02.445477 8224 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:10.240705 8572 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:10.397904 7070 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:12.076678 7198 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:19.973160 8696 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:20.319783 8233 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:29.652158 7819 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:30.150673 7084 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:35.144541 9215 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:35.868957 8489 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out) E0315 07:50:39.297054 7494 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-3.vpc.cloudera.com:22000 (connect() failed: Connection timed out) Picked up JAVA_TOOL_OPTIONS: hdfsOpenFile(hdfs://impala-stress-cdh5-trunk2-1.vpc.cloudera.com:8020/user/hive/warehouse/tpch_3_nested_parquet.db/customer/000001_0): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: FSDataInputStream#close error: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >' what(): boost: mutex lock failed in pthread_mutex_lock: Invalid argument ~
Attachments
Issue Links
- is blocked by
-
IMPALA-3299 Stress test failures: Couldn't open transport for impala-stress-cdh5-trunk2-5.vpc.cloudera.com:22000
- Resolved