Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10562

TestGracefulShutdown::test_shutdown_idle failes in GCE instance

    XMLWordPrintableJSON

Details

    • Test
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None
    • ghx-label-5

    Description

      Google Cloud will add a line in /etc/hosts for resolving the hostname of the instance. E.g.

      [systest@quanlong-compile impala]$ cat /etc/hosts                                                                                                                                                                                    
      127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
      ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
      x.x.x.x quanlong-compile.gce.cloudera.com quanlong-compile  # Added by Google
      x.x.x.x metadata.google.internal  # Added by Google
      
      127.0.0.1 quanlong-compile quanlong-compile.gce.cloudera.com
      

      (The IPs are manually masked)

      This causes connection refused failure in TestGracefulShutdown::test_shutdown_idle

      tests/custom_cluster/test_restart_services.py:295: in test_shutdown_idle
          query_options={})
      tests/common/impala_test_suite.py:814: in wrapper
          return function(*args, **kwargs)
      tests/common/impala_test_suite.py:822: in execute_query_expect_success
          result = cls.__execute_query(impalad_client, query, query_options, user)
      tests/common/impala_test_suite.py:923: in __execute_query
          return impalad_client.execute(query, user=user)
      tests/common/impala_connection.py:205: in execute
          return self.__beeswax_client.execute(sql_stmt, user=user)
      tests/beeswax/impala_beeswax.py:187: in execute
          handle = self.__execute_query(query_string.strip(), user=user)
      tests/beeswax/impala_beeswax.py:363: in __execute_query
          handle = self.execute_query_async(query_string, user=user)
      tests/beeswax/impala_beeswax.py:357: in execute_query_async
          handle = self.__do_rpc(lambda: self.imp_service.query(query,))
      tests/beeswax/impala_beeswax.py:520: in __do_rpc
          raise ImpalaBeeswaxException(self.__build_error_message(b), b)
      E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      E    INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
      E    MESSAGE: Rpc to 127.0.0.1:27001 failed with error 'RemoteShutdown() RPC failed: Network error: Client connection negotiation failed: client connection to 127.0.0.1:27001: connect: Connection refused (error 111)'
      

      The failed command is ":shutdown('localhost:27001')".

      Checked with the impalad, it's listening on "quanlong-compile.gce.cloudera.com:27001":

      [systest@quanlong-compile impala]$ ps aux | grep 28771
      systest  28771 11.0  3.5 22579424 2165200 pts/1 Sl  23:28   0:15 /home/systest/impala/be/build/latest/service/impalad -disconnected_session_timeout 21600 -kudu_client_rpc_timeout_ms 0 -kudu_master_hosts localhost -mem_limit=12884
      901888 -logbufsecs=5 -v=1 -max_log_files=10 -log_filename=impalad_node1 -log_dir=/tmp/ -beeswax_port=21001 -hs2_port=21051 -hs2_http_port=28001 -krpc_port=27001 -state_store_subscriber_port=23001 -webserver_port=25001 --shutdown_
      grace_period_s=1 --hostname=quanlong-compile.gce.cloudera.com --default_query_options=
      systest  30727  0.0  0.0 112828  1032 pts/1    S+   23:31   0:00 grep --color=auto 28771
      [systest@quanlong-compile impala]$ lsof -p 28771 | grep 27001
      impalad 28771 systest  341u     IPv4             577338       0t0       TCP quanlong-compile.gce.cloudera.com:27001 (LISTEN)
      

      The shutdown command works with hostname:

      [localhost.localdomain:21050] default> :shutdown('localhost:27001');
      Query: :shutdown('localhost:27001')
      Query submitted at: 2021-03-07 23:39:25 (Coordinator: http://quanlong-compile.gce.cloudera.com:25000)
      ERROR: Rpc to 127.0.0.1:27001 failed with error 'RemoteShutdown() RPC failed: Network error: Client connection negotiation failed: client connection to 127.0.0.1:27001: connect: Connection refused (error 111)'
      
      [localhost.localdomain:21050] default> :shutdown('quanlong-compile.gce.cloudera.com:27001');                                                                                                                                         
      Query: :shutdown('quanlong-compile.gce.cloudera.com:27001')
      Query submitted at: 2021-03-07 23:41:35 (Coordinator: http://quanlong-compile.gce.cloudera.com:25000)
      Query progress can be monitored at: http://quanlong-compile.gce.cloudera.com:25000/query_plan?query_id=81475574dca0db03:6e2f767d00000000
      +-------------------------------------------------------------------------------------------------------------------------------------------+
      | summary                                                                                                                                   |
      +-------------------------------------------------------------------------------------------------------------------------------------------+
      | shutdown grace period left: 1s000ms, deadline left: 1h, queries registered on coordinator: 0, queries executing: 0, fragment instances: 0 |
      +-------------------------------------------------------------------------------------------------------------------------------------------+
      Fetched 1 row(s) in 0.01s
      

      Removing the line "x.x.x.x quanlong-compile.gce.cloudera.com quanlong-compile # Added by Google" can pass the test. Maybe we should add the logics in bin/bootstrap_system.sh when running in GCE instance.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              stigahuang Quanlong Huang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: