Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 3.1.0
-
ghx-label-3
Description
All of the tests from query_test.test_kudu.TestCreateExternalTable fail with an error like:
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:704: in err_if_rpc_not_ok
raise HiveServer2Error(resp.status.errorMessage)
E HiveServer2Error: ImpalaRuntimeException: Error making 'createDatabase' RPC to Hive Metastore:
E CAUSED BY: AlreadyExistsException: Database testcreateexternaltable_23808_vu8cqo already exists
It looks like the failures all happen at once in a single process. The first test to fail is test_kudu.TestCreateExternalTable.test_col_types. It takes 52 seconds where all the other tests take no time. It also has an extra error on stderr:
-- connecting to: localhost:21000
MainThread: Failed to open transport (tries_left=3)
Traceback (most recent call last):
File "/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py", line 940, in _execute
return func(request)
File "/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", line 265, in ExecuteStatement
return self.recv_ExecuteStatement()
File "/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", line 276, in recv_ExecuteStatement
(fname, mtype, rseqid) = self._iprot.readMessageBegin()
File "/data/jenkins/workspace/impala-asf-master-core-s3/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
sz = self.readI32()
File "/data/jenkins/workspace/impala-asf-master-core-s3/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 206, in readI32
buff = self.trans.readAll(4)
File "/data/jenkins/workspace/impala-asf-master-core-s3/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py", line 58, in readAll
chunk = self.read(sz - have)
File "/data/jenkins/workspace/impala-asf-master-core-s3/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py", line 159, in read
self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
File "/data/jenkins/workspace/impala-asf-master-core-s3/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py", line 105, in read
buff = self.handle.recv(sz)
timeout: timed out
MainThread: Error closing Impala cursor: Invalid session id: f54064f9a4604f23:fb686144269fc8b1
The other failures don't have this.
This happened only once, so it is definitely intermittent. This has some similarity to IMPALA-6933, but this looks like a repeated failure in a single process, not a concurrency issue.