Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10316

load_nested.py failed due to out of memory during Jenkins GVO

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • None
    • Infrastructure
    • ghx-label-12

    Description

      The following job failed due to out of memory:

      https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/12588 (please click on "Don't keep this build forever" once this issue is resolved)

      Relevant log lines:

      02:33:42 Loading nested orc data (logging to /home/ubuntu/Impala/logs/data_loading/load-nested.log)... 
      02:35:39     FAILED (Took: 1 min 57 sec)
      02:35:39     '/home/ubuntu/Impala/testdata/bin/load_nested.py -t tpch_nested_orc_def -f orc/def' failed. Tail of log:
      02:35:39 2020-11-11 02:35:06,225 INFO:load_nested[348]:Executing: 
      02:35:39 
      02:35:39       CREATE EXTERNAL TABLE supplier
      02:35:39       STORED AS orc
      02:35:39       TBLPROPERTIES('orc.compress' = 'ZLIB','external.table.purge'='TRUE')
      02:35:39       AS SELECT * FROM tmp_supplier
      02:35:39 Traceback (most recent call last):
      02:35:39   File "/home/ubuntu/Impala/testdata/bin/load_nested.py", line 415, in <module>
      02:35:39     load()
      02:35:39   File "/home/ubuntu/Impala/testdata/bin/load_nested.py", line 349, in load
      02:35:39     hive.execute(stmt)
      02:35:39   File "/home/ubuntu/Impala/tests/comparison/db_connection.py", line 206, in execute
      02:35:39     return self._cursor.execute(sql, *args, **kwargs)
      02:35:39   File "/home/ubuntu/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py", line 331, in execute
      02:35:39     self._wait_to_finish()  # make execute synchronous
      02:35:39   File "/home/ubuntu/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py", line 413, in _wait_to_finish
      02:35:39     raise OperationalError(resp.errorMessage)
      02:35:39 impala.error.OperationalError: Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1605060173780_0039_2_00, diagnostics=[Task failed, taskId=task_1605060173780_0039_2_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Container container_1605060173780_0039_01_000002 finished with diagnostics set to [Container failed, exitCode=-104. [2020-11-11 02:35:11.768]Container [pid=16810,containerID=container_1605060173780_0039_01_000002] is running 7729152B beyond the 'PHYSICAL' memory limit. Current usage: 1.0 GB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing container.

      Attachments

        Issue Links

          Activity

            People

              MikaelSmith Michael Smith
              boroknagyz Zoltán Borók-Nagy
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: