[IMPALA-3097] Query with empty result set never completes in stress test - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Not A Bug
Affects Version/s: Impala 2.5.0
Fix Version/s: None
Component/s: Distributed Exec
Labels:
- hang
- stress

Target Version:

Impala 2.5.0

Description

The query is stuck in the "RUNNING" state indefinitely. I saw this happen with the same query on two different stress runs. Cancelling the query manually works fine. I've attached the profile from the stuck query. Here is the summary from the debug webpage:

Operator              #Hosts   Avg Time   Max Time  #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                         
--------------------------------------------------------------------------------------------------------------------------------
17:MERGING-EXCHANGE        0    0.000ns    0.000ns      0       1.00K          0        -1.00 B  UNPARTITIONED                  
10:TOP-N                   1  173.000ms  173.000ms      0       1.00K    4.00 KB       93.41 KB                                 
09:HASH JOIN               1   46s754ms   46s754ms      0     411.00K    8.00 KB      952.77 KB  INNER JOIN, BROADCAST          
|--16:EXCHANGE             1    0.000ns    0.000ns      0      36.96K          0              0  BROADCAST                      
|  15:AGGREGATE            9   33s355ms         1m      0      36.96K    2.25 MB       10.00 MB  FINALIZE                       
|  14:EXCHANGE             9    0.000ns    0.000ns      0      36.96K          0              0  HASH(ss_ticket_number,ss_cu... 
|  07:AGGREGATE            9   31s986ms   51s763ms      0      36.96K    1.28 MB       10.00 MB  STREAMING                      
|  06:HASH JOIN            9    4s539ms   25s807ms      0      36.96K    2.03 MB        3.49 KB  INNER JOIN, BROADCAST          
|  |--13:EXCHANGE          9    0.000ns    0.000ns      0          82          0              0  BROADCAST                      
|  |  02:SCAN HDFS         1   35s427ms   35s427ms      0          82   70.00 KB       32.00 MB  tpcds_8_decimal_parquet.store  
|  05:HASH JOIN            9    1s906ms    4s437ms      0      36.96K    2.03 MB       62.78 KB  INNER JOIN, BROADCAST          
|  |--12:EXCHANGE          9    0.000ns    0.000ns     90       7.30K          0              0  BROADCAST                      
|  |  01:SCAN HDFS         1    3s726ms    3s726ms     90       7.30K    1.16 MB       16.00 MB  tpcds_8_decimal_parquet.dat... 
|  04:HASH JOIN            9    1s603ms    3s571ms      0      36.96K    2.04 MB       16.89 KB  INNER JOIN, BROADCAST          
|  |--11:EXCHANGE          9    0.000ns    0.000ns    520         398          0              0  BROADCAST                      
|  |  03:SCAN HDFS         1   53s171ms   53s171ms    520         398  379.20 KB       64.00 MB  tpcds_8_decimal_parquet.hou... 
|  00:SCAN HDFS            9      3m55s      5m54s      0     668.55K  288.62 KB       64.00 MB  tpcds_8_decimal_parquet.sto... 
08:SCAN HDFS               1    0.000ns    0.000ns      0     411.00K          0      160.00 MB  tpcds_8_decimal_parquet.cus...

The same query completes very quickly. Here's a summary from a successful run:

Fetched 0 row(s) in 2.53s
[impala-stress-cdh5-big-2.vpc.cloudera.com:21000] > summary;
+---------------------+--------+----------+----------+-------+------------+-----------+---------------+------------------------------------------------+
| Operator            | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem  | Est. Peak Mem | Detail                                         |
+---------------------+--------+----------+----------+-------+------------+-----------+---------------+------------------------------------------------+
| 17:MERGING-EXCHANGE | 1      | 0ns      | 0ns      | 0     | 1.00K      | 0 B       | -1 B          | UNPARTITIONED                                  |
| 10:TOP-N            | 1      | 0ns      | 0ns      | 0     | 1.00K      | 20.00 KB  | 93.41 KB      |                                                |
| 09:HASH JOIN        | 1      | 274.00ms | 274.00ms | 0     | 411.00K    | 5.08 MB   | 952.77 KB     | INNER JOIN, BROADCAST                          |
| |--16:EXCHANGE      | 1      | 0ns      | 0ns      | 0     | 36.96K     | 0 B       | 0 B           | BROADCAST                                      |
| |  15:AGGREGATE     | 9      | 263.89ms | 407.00ms | 0     | 36.96K     | 2.26 MB   | 10.00 MB      | FINALIZE                                       |
| |  14:EXCHANGE      | 9      | 0ns      | 0ns      | 0     | 36.96K     | 0 B       | 0 B           | HASH(ss_ticket_number,ss_customer_sk)          |
| |  07:AGGREGATE     | 9      | 246.00ms | 450.00ms | 0     | 36.96K     | 1.28 MB   | 10.00 MB      | STREAMING                                      |
| |  06:HASH JOIN     | 9      | 12.56ms  | 22.00ms  | 0     | 36.96K     | 2.03 MB   | 3.49 KB       | INNER JOIN, BROADCAST                          |
| |  |--13:EXCHANGE   | 9      | 0ns      | 0ns      | 0     | 82         | 0 B       | 0 B           | BROADCAST                                      |
| |  |  02:SCAN HDFS  | 1      | 10.00ms  | 10.00ms  | 0     | 82         | 70.00 KB  | 32.00 MB      | tpcds_8_decimal_parquet.store                  |
| |  05:HASH JOIN     | 9      | 18.22ms  | 29.00ms  | 0     | 36.96K     | 2.03 MB   | 62.78 KB      | INNER JOIN, BROADCAST                          |
| |  |--12:EXCHANGE   | 9      | 0ns      | 0ns      | 810   | 7.30K      | 0 B       | 0 B           | BROADCAST                                      |
| |  |  01:SCAN HDFS  | 1      | 145.00ms | 145.00ms | 90    | 7.30K      | 1.16 MB   | 16.00 MB      | tpcds_8_decimal_parquet.date_dim               |
| |  04:HASH JOIN     | 9      | 15.00ms  | 19.00ms  | 0     | 36.96K     | 2.04 MB   | 16.89 KB      | INNER JOIN, BROADCAST                          |
| |  |--11:EXCHANGE   | 9      | 0ns      | 0ns      | 4.68K | 398        | 0 B       | 0 B           | BROADCAST                                      |
| |  |  03:SCAN HDFS  | 1      | 52.00ms  | 52.00ms  | 520   | 398        | 379.20 KB | 64.00 MB      | tpcds_8_decimal_parquet.household_demographics |
| |  00:SCAN HDFS     | 9      | 233.89ms | 287.99ms | 0     | 668.55K    | 288.62 KB | 64.00 MB      | tpcds_8_decimal_parquet.store_sales            |
| 08:SCAN HDFS        | 1      | 808.00ms | 808.00ms | 0     | 411.00K    | 9.46 MB   | 160.00 MB     | tpcds_8_decimal_parquet.customer               |
+---------------------+--------+----------+----------+-------+------------+-----------+---------------+------------------------------------------------+
[impala-stress-cdh5-big-2.vpc.cloudera.com:21000] >

Logs from all the nodes with the query id:

=======
big-2
======
I0226 13:51:11.894414  7616 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c592 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=13
I0226 13:51:11.898619 19187 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c589 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=4
I0226 13:51:13.085256 25641 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c592
I0226 13:51:14.113623 25694 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c589
I0226 13:51:15.076217 25641 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c592
I0226 13:51:15.285073 25694 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c589
I0226 13:52:13.727200  6420 data-stream-mgr.cc:113] Datastream sender timed-out waiting for recvr for fragment instance: f84898b3478b4999:14c030501327c592 (time-out was: 1m). If query was cancelled, this is not an error.
I0226 13:52:16.576380 25694 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c589
I0226 13:52:22.067354 25641 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c592

=======
big-3
======
I0226 13:50:19.714898 21862 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c587 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=2
I0226 13:50:20.055209  2988 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c587
I0226 13:50:20.213335  2988 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c587
I0226 13:50:20.577216 17671 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c590 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=11
I0226 13:50:20.823226  3016 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c590
I0226 13:50:21.109196  3016 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c590
I0226 13:50:48.250717  2988 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c587
I0226 13:50:52.610519  3016 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c590
I0226 13:57:44.063164  3016 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c590, node=11
I0226 13:57:44.063314  3016 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c590 node_id=11
I0226 13:57:44.063418  3016 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c590, node=12
I0226 13:57:44.063465  3016 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c590 node_id=12
I0226 13:57:44.063527  3016 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c590, node=13
I0226 13:57:44.063572  3016 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c590 node_id=13

=======
big-4
======
I0226 13:50:20.243297 32712 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c58d coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=8
I0226 13:50:20.540153 19826 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c58d
I0226 13:50:20.722810 19826 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c58d
I0226 13:50:21.197211  1881 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c596 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=17
I0226 13:50:21.327072 29743 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c598 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=19
I0226 13:50:21.571300  4248 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c59a coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=21
I0226 13:50:21.574506 19884 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c596
I0226 13:50:21.835398 19890 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c598
I0226 13:50:22.003244 19915 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c59a
I0226 13:50:22.123579 19890 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c598
I0226 13:50:22.125180 19915 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c59a
I0226 13:50:22.127692 19884 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c596
I0226 13:50:24.867311 19890 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c598
I0226 13:50:25.352212 19915 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c59a
I0226 13:50:49.338429 19826 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c58d
I0226 13:50:53.700314 19884 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c596
I0226 13:58:09.674340 19884 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c596, node=11
I0226 13:58:09.674530 19884 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c596 node_id=11
I0226 13:58:09.674633 19884 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c596, node=12
I0226 13:58:09.674681 19884 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c596 node_id=12
I0226 13:58:09.674743 19884 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c596, node=13
I0226 13:58:09.674788 19884 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c596 node_id=13

=======
big-5
======
I0226 13:50:19.755312 23342 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c588 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=3
I0226 13:50:21.410038 32260 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c591 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=12
I0226 13:50:21.174175  8771 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c588
I0226 13:50:21.659816  8771 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c588
I0226 13:50:21.829635  8793 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c591
I0226 13:50:22.305151  8793 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c591
I0226 13:50:45.914347  8771 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c588
I0226 13:50:55.147889  8793 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c591
I0226 13:55:21.892947  8793 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c591, node=11
I0226 13:55:21.893091  8793 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c591 node_id=11
I0226 13:55:21.893182  8793 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c591, node=12
I0226 13:55:21.893230  8793 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c591 node_id=12
I0226 13:55:21.893295  8793 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c591, node=13
I0226 13:55:21.893339  8793 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c591 node_id=13

=======
big-6
======
I0226 13:50:20.428678 27872 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c58e coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=9
I0226 13:50:20.626041 13316 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c58e
I0226 13:50:20.892022 13316 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c58e
I0226 13:50:21.372905 27436 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c597 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=18
I0226 13:50:21.495224 13370 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c597
I0226 13:50:21.559825 27865 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c599 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=20
I0226 13:50:21.587496 13386 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c599
I0226 13:50:21.654352 13370 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c597
I0226 13:50:21.658922 13386 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c599
I0226 13:50:23.751642 13386 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c599
I0226 13:50:36.250512 13316 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c58e
I0226 13:50:39.544792 13370 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c597
I0226 13:52:16.314110 13370 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c597, node=11
I0226 13:52:16.314259 13370 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c597 node_id=11
I0226 13:52:16.314355 13370 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c597, node=12
I0226 13:52:16.314415 13370 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c597 node_id=12
I0226 13:52:16.314476 13370 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c597, node=13
I0226 13:52:16.314518 13370 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c597 node_id=13

=======
big-7
======
I0226 13:50:19.593989 31016 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c585 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=0
I0226 13:50:19.832082  3130 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c585
I0226 13:50:19.939898 21110 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c58a coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=5
I0226 13:50:20.116478  3130 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c585
I0226 13:50:20.139652  3148 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c58a
I0226 13:50:20.434000  3148 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c58a
I0226 13:50:21.255945 31054 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c593 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=14
I0226 13:50:22.147454  3181 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c593
I0226 13:50:22.550940  3181 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c593
I0226 13:51:07.543025  3130 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c585
I0226 13:51:08.868098  3148 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c58a
I0226 13:51:15.812692  3181 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c593
I0226 13:58:13.523514  3181 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c593, node=11
I0226 13:58:13.523697  3181 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c593 node_id=11
I0226 13:58:13.523807  3181 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c593, node=12
I0226 13:58:13.523861  3181 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c593 node_id=12
I0226 13:58:13.523936  3181 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c593, node=13
I0226 13:58:13.524094  3181 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c593 node_id=13

=======
big-8
======
I0226 13:50:20.116541   579 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c58c coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=7
I0226 13:50:20.647146 19560 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c58c
I0226 13:50:20.926245 19560 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c58c
I0226 13:50:21.428442  3244 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c595 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=16
I0226 13:50:22.028971 19618 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c595
I0226 13:50:22.342458 19618 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c595
I0226 13:50:56.443527 19560 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c58c
I0226 13:51:01.306488 19618 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c595
I0226 13:58:01.699436 19618 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c595, node=11
I0226 13:58:01.699628 19618 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c595 node_id=11
I0226 13:58:01.699779 19618 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c595, node=12
I0226 13:58:01.699858 19618 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c595 node_id=12
I0226 13:58:01.699964 19618 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c595, node=13
I0226 13:58:01.700029 19618 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c595 node_id=13

=======
big-9
======
I0226 13:50:09.087669 19711 coordinator.cc:382] Exec() query_id=f84898b3478b4999:14c030501327c583
I0226 13:50:09.136206 19711 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c584
I0226 13:50:09.158352 19711 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c584
I0226 13:50:09.159289 19711 coordinator.cc:532] starting 22 fragment instances for query f84898b3478b4999:14c030501327c583
I0226 13:50:20.048096 21630 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c58b coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=6
I0226 13:50:20.351111  6438 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c58b
I0226 13:50:20.559540  6438 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c58b
I0226 13:50:21.192937   860 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c594 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=15
I0226 13:50:21.523375  6476 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c594
I0226 13:50:21.798583  6476 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c594
I0226 13:50:48.894707  6438 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c58b
I0226 13:50:55.705461  6476 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c594
I0226 13:51:14.253968  7501 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c584
I0226 13:52:10.874662  2588 coordinator.cc:1558] Fragment instance 21(f84898b3478b4999:14c030501327c59a) on host impala-stress-cdh5-big-4.vpc.cloudera.com:22000 completed, 21 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:52:10.878027  2588 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000
I0226 13:52:15.715076 23798 coordinator.cc:1558] Fragment instance 20(f84898b3478b4999:14c030501327c599) on host impala-stress-cdh5-big-6.vpc.cloudera.com:22000 completed, 20 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:52:15.732395 23798 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000
I0226 13:52:15.732601  2121 coordinator.cc:1558] Fragment instance 19(f84898b3478b4999:14c030501327c598) on host impala-stress-cdh5-big-4.vpc.cloudera.com:22000 completed, 19 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:52:15.732801  2121 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000
I0226 13:52:43.861732   635 coordinator.cc:1558] Fragment instance 18(f84898b3478b4999:14c030501327c597) on host impala-stress-cdh5-big-6.vpc.cloudera.com:22000 completed, 18 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:52:43.867935   635 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000
I0226 13:55:37.713892 20285 coordinator.cc:1558] Fragment instance 12(f84898b3478b4999:14c030501327c591) on host impala-stress-cdh5-big-5.vpc.cloudera.com:22000 completed, 17 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:55:37.810827 20285 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000
I0226 13:56:35.505439  6476 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c594, node=11
I0226 13:56:35.505605  6476 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c594 node_id=11
I0226 13:56:35.505717  6476 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c594, node=12
I0226 13:56:35.505795  6476 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c594 node_id=12
I0226 13:56:35.505869  6476 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c594, node=13
I0226 13:56:35.506235  6476 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c594 node_id=13
I0226 13:57:05.517987 23190 coordinator.cc:1558] Fragment instance 15(f84898b3478b4999:14c030501327c594) on host impala-stress-cdh5-big-9.vpc.cloudera.com:22000 completed, 16 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:57:05.518095 23190 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000
I0226 13:57:57.388262 23815 coordinator.cc:1558] Fragment instance 11(f84898b3478b4999:14c030501327c590) on host impala-stress-cdh5-big-3.vpc.cloudera.com:22000 completed, 15 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:57:57.388391 23815 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000
I0226 13:58:14.745719  2493 coordinator.cc:1558] Fragment instance 14(f84898b3478b4999:14c030501327c593) on host impala-stress-cdh5-big-7.vpc.cloudera.com:22000 completed, 14 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:58:14.748761  2493 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000
I0226 13:58:14.750422 20703 coordinator.cc:1558] Fragment instance 10(f84898b3478b4999:14c030501327c58f) on host impala-stress-cdh5-big-10.vpc.cloudera.com:22000 completed, 13 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:58:14.772533 20703 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000
I0226 13:58:18.349151 24353 coordinator.cc:1558] Fragment instance 16(f84898b3478b4999:14c030501327c595) on host impala-stress-cdh5-big-8.vpc.cloudera.com:22000 completed, 12 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:58:18.388654 24353 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000
I0226 13:58:18.389120  1503 coordinator.cc:1558] Fragment instance 17(f84898b3478b4999:14c030501327c596) on host impala-stress-cdh5-big-4.vpc.cloudera.com:22000 completed, 11 remaining: query_id=f84898b3478b4999:14c030501327c583
I0226 13:58:18.392668  1503 coordinator.cc:1570] query_id=f84898b3478b4999:14c030501327c583: first in-progress backend: impala-stress-cdh5-big-7.vpc.cloudera.com:22000

=======
big-10
======
I0226 13:50:19.704885 28272 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c586 coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=1
I0226 13:50:19.753043  6166 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c586
I0226 13:50:19.862658  6166 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c586
I0226 13:50:20.480628 23307 fragment-mgr.cc:36] ExecPlanFragment() instance_id=f84898b3478b4999:14c030501327c58f coord=impala-stress-cdh5-big-9.vpc.cloudera.com:22000 fragment instance#=10
I0226 13:50:20.909868  6199 plan-fragment-executor.cc:92] Prepare(): query_id=f84898b3478b4999:14c030501327c583 instance_id=f84898b3478b4999:14c030501327c58f
I0226 13:50:21.521169  6199 plan-fragment-executor.cc:202] descriptor table for fragment=f84898b3478b4999:14c030501327c58f
I0226 13:50:54.550410  6166 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c586
I0226 13:51:01.114915  6199 plan-fragment-executor.cc:311] Open(): instance_id=f84898b3478b4999:14c030501327c58f
I0226 13:57:55.947114  6199 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c58f, node=11
I0226 13:57:55.947247  6199 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c58f node_id=11
I0226 13:57:55.947350  6199 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c58f, node=12
I0226 13:57:55.947419  6199 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c58f node_id=12
I0226 13:57:55.947507  6199 data-stream-mgr.cc:207] DeregisterRecvr(): fragment_instance_id=f84898b3478b4999:14c030501327c58f, node=13
I0226 13:57:55.947556  6199 data-stream-recvr.cc:233] cancelled stream: fragment_instance_id_=f84898b3478b4999:14c030501327c58f node_id=13

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

profile.txt
27/Feb/16 01:19
144 kB
Tim Armstrong

Activity

People

Assignee:: Henry Robinson

Reporter:: Tim Armstrong

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 27/Feb/16 01:45

Updated:: 29/Feb/16 18:36

Resolved:: 29/Feb/16 18:36