Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Duplicate
-
Impala 2.5.0
-
None
-
Ubuntu Trusty, CDH 5.7.0
Description
Impala becomes unresponsive, causing a full outage for our reporting system. Impalad logs show various errors as follows:
Time Log Level Source Log Message May 16, 3:20:59.953 PM ERROR logging.cc:120 stderr will be logged to this file. May 16, 3:27:38.652 PM ERROR data-stream-sender.cc:261 channel send status: Sender timed out waiting for receiver fragment instance: a045d3abcd1bed4f:a116563837c8ebda May 16, 3:27:39.738 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:44.784 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:46.055 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:46.371 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:46.906 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:51.003 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:52.479 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:54.227 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:54.298 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:54.305 PM ERROR data-stream-sender.cc:261 channel send status: Sender timed out waiting for receiver fragment instance: 7446d34d9eefb9f:436f61bcd3c78eb3 May 16, 3:27:55.022 PM ERROR data-stream-sender.cc:261 channel send status: Sender timed out waiting for receiver fragment instance: b44445bcb782b5a4:5cef13b26af20aa1 May 16, 3:27:55.646 PM ERROR data-stream-sender.cc:261 channel send status: Sender timed out waiting for receiver fragment instance: 3249c6494703ecc7:3b9186fbed00dc May 16, 3:27:56.042 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:56.277 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:56.749 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:57.126 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:27:59.030 PM ERROR data-stream-sender.cc:261 channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out) May 16, 3:28:03.573 PM ERROR data-stream-sender.cc:261 channel send status: Sender timed out waiting for receiver fragment instance: c64b22033608d609:10e070d34aa299dd
Attachments
Attachments
Issue Links
- duplicates
-
IMPALA-3764 Fuzz-test HDFS scanners
- Resolved
- is duplicated by
-
IMPALA-7057 crash when DiskIoMgr::GetFreeBuffer(long*)+0xa9
- Resolved
- relates to
-
IMPALA-2877 Unexpected exits with impala::DiskIoMgr::GetFreeBuffer in dump
- Resolved