Details
-
Test
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
-
ghx-label-5
Description
Google Cloud will add a line in /etc/hosts for resolving the hostname of the instance. E.g.
[systest@quanlong-compile impala]$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 x.x.x.x quanlong-compile.gce.cloudera.com quanlong-compile # Added by Google x.x.x.x metadata.google.internal # Added by Google 127.0.0.1 quanlong-compile quanlong-compile.gce.cloudera.com
(The IPs are manually masked)
This causes connection refused failure in TestGracefulShutdown::test_shutdown_idle
tests/custom_cluster/test_restart_services.py:295: in test_shutdown_idle query_options={}) tests/common/impala_test_suite.py:814: in wrapper return function(*args, **kwargs) tests/common/impala_test_suite.py:822: in execute_query_expect_success result = cls.__execute_query(impalad_client, query, query_options, user) tests/common/impala_test_suite.py:923: in __execute_query return impalad_client.execute(query, user=user) tests/common/impala_connection.py:205: in execute return self.__beeswax_client.execute(sql_stmt, user=user) tests/beeswax/impala_beeswax.py:187: in execute handle = self.__execute_query(query_string.strip(), user=user) tests/beeswax/impala_beeswax.py:363: in __execute_query handle = self.execute_query_async(query_string, user=user) tests/beeswax/impala_beeswax.py:357: in execute_query_async handle = self.__do_rpc(lambda: self.imp_service.query(query,)) tests/beeswax/impala_beeswax.py:520: in __do_rpc raise ImpalaBeeswaxException(self.__build_error_message(b), b) E ImpalaBeeswaxException: ImpalaBeeswaxException: E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'> E MESSAGE: Rpc to 127.0.0.1:27001 failed with error 'RemoteShutdown() RPC failed: Network error: Client connection negotiation failed: client connection to 127.0.0.1:27001: connect: Connection refused (error 111)'
The failed command is ":shutdown('localhost:27001')".
Checked with the impalad, it's listening on "quanlong-compile.gce.cloudera.com:27001":
[systest@quanlong-compile impala]$ ps aux | grep 28771 systest 28771 11.0 3.5 22579424 2165200 pts/1 Sl 23:28 0:15 /home/systest/impala/be/build/latest/service/impalad -disconnected_session_timeout 21600 -kudu_client_rpc_timeout_ms 0 -kudu_master_hosts localhost -mem_limit=12884 901888 -logbufsecs=5 -v=1 -max_log_files=10 -log_filename=impalad_node1 -log_dir=/tmp/ -beeswax_port=21001 -hs2_port=21051 -hs2_http_port=28001 -krpc_port=27001 -state_store_subscriber_port=23001 -webserver_port=25001 --shutdown_ grace_period_s=1 --hostname=quanlong-compile.gce.cloudera.com --default_query_options= systest 30727 0.0 0.0 112828 1032 pts/1 S+ 23:31 0:00 grep --color=auto 28771 [systest@quanlong-compile impala]$ lsof -p 28771 | grep 27001 impalad 28771 systest 341u IPv4 577338 0t0 TCP quanlong-compile.gce.cloudera.com:27001 (LISTEN)
The shutdown command works with hostname:
[localhost.localdomain:21050] default> :shutdown('localhost:27001'); Query: :shutdown('localhost:27001') Query submitted at: 2021-03-07 23:39:25 (Coordinator: http://quanlong-compile.gce.cloudera.com:25000) ERROR: Rpc to 127.0.0.1:27001 failed with error 'RemoteShutdown() RPC failed: Network error: Client connection negotiation failed: client connection to 127.0.0.1:27001: connect: Connection refused (error 111)' [localhost.localdomain:21050] default> :shutdown('quanlong-compile.gce.cloudera.com:27001'); Query: :shutdown('quanlong-compile.gce.cloudera.com:27001') Query submitted at: 2021-03-07 23:41:35 (Coordinator: http://quanlong-compile.gce.cloudera.com:25000) Query progress can be monitored at: http://quanlong-compile.gce.cloudera.com:25000/query_plan?query_id=81475574dca0db03:6e2f767d00000000 +-------------------------------------------------------------------------------------------------------------------------------------------+ | summary | +-------------------------------------------------------------------------------------------------------------------------------------------+ | shutdown grace period left: 1s000ms, deadline left: 1h, queries registered on coordinator: 0, queries executing: 0, fragment instances: 0 | +-------------------------------------------------------------------------------------------------------------------------------------------+ Fetched 1 row(s) in 0.01s
Removing the line "x.x.x.x quanlong-compile.gce.cloudera.com quanlong-compile # Added by Google" can pass the test. Maybe we should add the logics in bin/bootstrap_system.sh when running in GCE instance.
Attachments
Issue Links
- relates to
-
IMPALA-7712 Impala read from and write to GCS
- Resolved